[ { "path": ".github/ISSUE_TEMPLATE/bug-report.yml", "content": "---\nname: \"🐛 Bug Report\"\ndescription: Report a bug\ntitle: \"(short issue description)\"\nlabels: [bug, needs-triage]\nassignees: []\nbody:\n - type: textarea\n id: description\n attributes:\n label: Describe the bug\n description: What is the problem? Provide a clear description of your issue and the steps you took that produced it.\n validations:\n required: true\n - type: textarea\n id: modelname\n attributes:\n label: Model Name\n description: Provide Model Name\n validations:\n required: true\n - type: textarea\n id: workloadtype\n attributes:\n label: Describe the workload type \n description: Note the type of workload (such as Inference or Training) and any specific details about the workload configuration.\n validations:\n required: true\n - type: textarea\n id: instancetype\n attributes:\n label: Instance Type\n description: |\n Provide the AWS EC2 instance type you used to run the workload (such as `inf2.xlarge`, `trn1.32xlarge`, `trn2.48xlarge` etc.)\n validations:\n required: true\n - type: textarea\n id: release\n attributes:\n label: Release version \n description: |\n Provide the Neuron SDK release version (such as `2.25.0`) you are using, and all relevant Neuron component versions. \n ```\n apt list --installed | grep -i -e neuron\n pip list | grep -i -e neuron -e torch -e transformers -e jax\n ```\n - type: textarea\n id: reproduction\n attributes:\n label: Reproduction Steps\n description: |\n Provide the type of the model and links to any tutorials you may have used, as additional context.\n Provide a self-contained, concise snippet of code that can be used to reproduce the issue.\n For more complex issues provide a repo with the smallest sample that reproduces the bug.\n \n Avoid including business logic or unrelated code as it makes diagnosis more difficult.\n The code sample should be an SSCCE. See http://sscce.org/ for details. In short, please provide a code sample that we can copy/paste, run and reproduce.\n validations:\n required: true\n - type: checkboxes\n id: regression\n attributes:\n label: Regression Issue\n description: Is this as regression (did it work in a previous version and not now)?\n If this is a regression, provide the Neuron SDK release version where this configuration worked for you.\n options:\n - label: Select this option if this issue appears to be a regression.\n required: false\n - type: textarea\n id: solution\n attributes:\n label: Possible Solution\n description: |\n Suggest a fix or reason for the bug, if you know:\n validations:\n required: false\n - type: textarea\n id: context\n attributes:\n label: Logs/Context/Additional Information\n description: |\n Anything else that might be relevant for troubleshooting this bug. Providing context helps us come up with a solution that is most useful in the real world. When applicable, please provide HLOs and compiler commands.\n validations:\n required: false\n\n" }, { "path": ".github/ISSUE_TEMPLATE/config.yml", "content": "blank_issues_enabled: false" }, { "path": ".github/ISSUE_TEMPLATE/documentation.yml", "content": "---\nname: \"📕 Documentation Issue\"\ndescription: Report an issue in the documentation and Developer Guide\ntitle: \"(short issue description)\"\nlabels: [documentation, needs-triage]\nassignees: []\nbody:\n - type: textarea\n id: description\n attributes:\n label: Describe the issue\n description: A clear and concise description of the issue.\n validations:\n required: true\n\n - type: textarea\n id: links\n attributes:\n label: Links\n description: |\n Include links to affected documentation page(s).\n validations:\n required: true\n" }, { "path": ".github/ISSUE_TEMPLATE/feature-request.yml", "content": "---\nname: 🚀 Feature Request\ndescription: Suggest an idea for this project\ntitle: \"(short issue description)\"\nlabels: [feature-request, needs-triage]\nassignees: []\nbody:\n - type: textarea\n id: description\n attributes:\n label: Describe the feature\n description: A clear and concise description of the feature you are proposing.\n validations:\n required: true\n - type: textarea\n id: use-case\n attributes:\n label: Use Case\n description: |\n Why do you need this feature?\n validations:\n required: true\n - type: textarea\n id: solution\n attributes:\n label: Proposed Solution\n description: |\n Provide detailed suggestions or requirements for this proposed feature. If you have them, include any reference implementation details (or even links to prototypes).\n validations:\n required: false\n - type: textarea\n id: other\n attributes:\n label: Other Information\n description: |\n Any additional details or information you can provide, including links to related content or similar issues.\n validations:\n required: false\n - type: checkboxes\n id: ack\n attributes:\n label: Acknowledgements\n options:\n - label: I may be able to implement this feature request\n required: false\n" }, { "path": ".github/pull_request_template.md", "content": "**IMPORTANT!** _If this is a documentation PR for a specific release, this PR must go the corresponding release branch_ (`release-X.XX.X`). _If it is an \"out-of-band\" doc update, the PR must go to the_ `master` _branch_.\n\n\n## Required PR information\n\nTo expedite approvals and merges for releases, provide the following information (select the `...` button to the right at the top of your PR message to edit it):\n\n> **AWS email alias**: {_your-name_}@amazon.com\n\n>**Description**: {_What this documentation change is and why you made it. If you have a corresponding Jira ticket or content plan, link it here. The more details you provide around any decisions you made when preparing the docs, the less annoying comments you'll get preparing to release it._}\n\n> **Date this must be published by**: {_If empty, we will assume the release date for the branch you're merging into._}\n\n> **Link to ReadTheDocs staging for this branch's doc changes**: https://awsdocs-neuron-staging.readthedocs-hosted.com/en/{YOUR_BRANCH_NAME_HERE}/\n\n> **Set the `docs-review-needed` label on the PR for tracking.**\n\n## Before you request approvals\n\n> Run a spelling and grammar check over your prose and make the changes it suggests. VSCode has a number of extensions (cSpell, LTeX) that you can use. You can also provide the rendered HTML for (or a cut-and-paste of) your pages to an AI and have it correct your spelling, grammar, and formatting issues. If you need an advanced prompt, contact @erickson-doug.\n\n## Approvers\n\nWe require 3-4 approvers to merge for non-trivial content changes (where a \"trivial\" change is a typo/grammar fix or a minor update to the format syntax):\n\n1. A senior+ engineer who will review your documentation for technical accuracy and clarity in communicating the technical concepts in your work\n2. A product manager for your Neuron component area who will review it for customer relevance and product/component/feature messaging\n3. The lead tech writer (@erickson-doug) who will review your work for overall doc design and quality, and perform the merge when all approvals are met\n4. (For PRs with code/notebook submissions) A QA/test engineer who can run your code and confirm the results.\n\nMake sure you get a commitment from these reviewers in advance! It's hard to get good quality doc reviews in order in the 11th hour of a release.\n\n**Note**: For trivial changes, you only need @erickson-doug's approval. He will merge your content once he's confirmed the fixes on staging.\n\n## Doc review checklist\n\n### Engineering reviewer checklist\n\n- [ ] I've confirmed that the contributions in this PR meet the current [AWS Neuron writing guidelines](https://quip-amazon.com/m97CAO0kQFEU/Writing-for-AWS-Neuron).\n- [ ] I've confirmed that the documentation submitted is technically correct to the best of my knowledge.\n- [ ] I've confirmed that the documentation submitted has no spelling or grammar errors or use of internal jargon/terminology.\n- [ ] I've verified the changes render correctly on RTD (link above).\n- [ ] (If code is included) I've run tests to verify the contents of the change.\n\n---\n\n## For PRs that include code or notebook examples\n\n**MANDATORY: PR must include test run output**\n\nProvide this information for the QA reviewer in order to expedite their review.\n\n**Test run output:**\nSpecify the release version, instance size and type, OS type and test output.\n\n**For Training tutorials:**\n\n{Convergence graph for training tutorials}\n\n{Performance metrics `average_throughput`, `latency_p50`, `latency_p99` and MFU% if available}\n\nMake sure this PR contains correct classification terms (Alpha, Beta, and Stable).\n\nIf possible, provide your results or a link to them for the reviewer to check your work.\n\n## Code example/notebook content PR checklist\n\n- [ ] (If applicable) I've automated a test to safeguard my changes from regression.\n- [ ] (If applicable) I've posted test collateral to prove my change was effective and not harmful.\n- [ ] (If applicable) I've added someone from QA to the list of reviewers. Do this if you didn't make an automated test or feel it's appropriate for another reason.\n- [ ] (If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the pre-approved Amazon license list. See https://inside.amazon.com/en/services/legal/us/OpenSource/Pages/BlessedOpenSourceLicenses.aspx.\n\nBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice." }, { "path": ".github/stale_issue_mark_close_workflow.yml", "content": "name: Close inactive issues\non:\n schedule:\n - cron: \"30 1 * * *\"\n\njobs:\n close-issues:\n runs-on: ubuntu-latest\n permissions:\n issues: write\n pull-requests: write\n steps:\n - uses: actions/stale@v5\n with:\n days-before-issue-stale: 30\n days-before-issue-close: 14\n stale-issue-label: \"stale\"\n stale-issue-message: \"This issue is stale because it has been open for 30 days with no activity.\"\n close-issue-message: \"This issue was closed because it has been inactive for 14 days since being marked as stale.\"\n days-before-pr-stale: -1\n days-before-pr-close: -1\n repo-token: ${{ secrets.GITHUB_TOKEN }}\n" }, { "path": ".github/workflows/acknowledge-new-issue.yml", "content": "name: Acknowledge New Issue\n\non:\n issues:\n types: [opened]\n\npermissions:\n issues: write\n\njobs:\n acknowledge:\n runs-on: ubuntu-latest\n steps:\n - name: Comment on issue\n uses: actions/github-script@v7\n with:\n script: |\n const creator = context.payload.issue.user.login;\n await github.rest.issues.createComment({\n owner: context.repo.owner,\n repo: context.repo.repo,\n issue_number: context.payload.issue.number,\n body: `Hi @${creator}, Thank you for filing the issue! We will take a look and get back to you.`\n });\n" }, { "path": ".github/workflows/auto-label-issues.yml", "content": "# Auto-label issues based on content keywords\nname: auto-label-issues\n\non:\n issues:\n types: [opened]\n\njobs:\n auto-label-issues:\n runs-on: ubuntu-latest\n permissions:\n issues: write\n steps:\n - name: Analyze issue content\n id: analyze_content\n uses: actions/github-script@v7\n env:\n GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n ISSUE_TITLE: ${{ github.event.issue.title }}\n ISSUE_BODY: ${{ github.event.issue.body }}\n with:\n script: |\n const title = process.env.ISSUE_TITLE || '';\n const body = process.env.ISSUE_BODY || '';\n const content = `${title} ${body}`;\n \n const labels = [];\n \n // =============================================================================\n // LABEL CONFIGURATION - Easy to update dictionary\n // Add keywords, typos, or synonyms to the arrays below\n // =============================================================================\n \n const labelConfig = {\n // ----- Issue Type Labels (mutually exclusive) -----\n bug: {\n keywords: [\n // Standard terms\n 'bug', 'error', 'crash', 'fail', 'failed', 'failure', 'failing',\n 'broken', 'exception', 'traceback', 'segfault', 'segmentation fault',\n // Synonyms\n 'issue', 'problem', 'defect', 'fault', 'glitch', 'malfunction',\n 'wrong', 'incorrect', 'unexpected',\n 'hang', 'hanging', 'hung', 'freeze', 'frozen',\n 'timeout', 'timed out',\n 'oom', 'out of memory', 'memory error',\n 'nan', 'diverge', 'diverged',\n // Common typos\n 'bugg', 'bgu', 'eror', 'errror', 'crahs', 'fial', 'brokn', 'broke'\n ],\n patterns: [/not\\s*work/i, /doesn'?t\\s*work/i, /won'?t\\s*work/i, /can'?t\\s*work/i]\n },\n \n documentation: {\n keywords: [\n // Standard terms\n 'doc', 'docs', 'documentation', 'readme',\n 'guide', 'tutorial', 'howto', 'how-to', 'how to',\n 'typo', 'typos', 'spelling', 'grammar',\n 'example', 'examples', 'sample', 'samples',\n 'instruction', 'instructions',\n 'clarify', 'clarification', 'unclear', 'confusing',\n 'outdated', 'out of date', 'stale',\n 'missing documentation', 'missing docs',\n 'broken link', 'dead link', '404',\n // Common typos\n 'documention', 'documenation', 'documentaion', 'tutoral', 'toturial'\n ],\n patterns: [/issue\\s*on\\s*page/i, /page\\s*.*\\.html/i]\n },\n \n 'feature-request': {\n keywords: [\n // Standard terms\n 'feature', 'feature request', 'feature-request',\n 'enhancement', 'improvement',\n 'implement', 'implementation',\n 'new feature', 'add feature',\n 'support for', 'add support',\n 'would be nice', 'would be great', 'would be helpful',\n 'suggestion', 'suggest', 'proposal', 'propose',\n 'wishlist', 'wish list',\n // Common typos\n 'feture', 'featrue', 'enchancement', 'improvment'\n ],\n patterns: [/add\\s+support\\s+for/i, /please\\s+add/i, /would\\s+be\\s+(nice|great|helpful)/i]\n },\n \n // ----- Hardware Labels (independent - multiple can be applied) -----\n Trn1: {\n keywords: [\n 'trn1', 'trn-1', 'trn 1', 'trn1n',\n 'trn1.2xlarge', 'trn1.32xlarge', 'trn1n.32xlarge',\n 'trainium', 'trainium1', 'trainium 1', 'trainium-1',\n // Common typos\n 'tranium', 'trainuim', 'trn-1n'\n ],\n patterns: [/trn1n?(?:\\.[0-9]*xlarge)?/i, /trainium\\s*1?(?!\\s*2)/i]\n },\n \n Trn2: {\n keywords: [\n 'trn2', 'trn-2', 'trn 2',\n 'trn2.48xlarge',\n 'trainium2', 'trainium 2', 'trainium-2',\n // Common typos\n 'tranium2', 'trainuim2'\n ],\n patterns: [/trn2(?:\\.[0-9]*xlarge)?/i, /trainium\\s*2/i]\n },\n \n Inf1: {\n keywords: [\n 'inf1', 'inf-1', 'inf 1',\n 'inf1.xlarge', 'inf1.2xlarge', 'inf1.6xlarge', 'inf1.24xlarge',\n 'inferentia', 'inferentia1', 'inferentia 1', 'inferentia-1',\n // Common typos\n 'infertia', 'inferntia', 'infernita'\n ],\n patterns: [/inf1(?:\\.[0-9]*xlarge)?/i, /inferentia\\s*1?(?!\\s*2)/i]\n },\n \n Inf2: {\n keywords: [\n 'inf2', 'inf-2', 'inf 2',\n 'inf2.xlarge', 'inf2.8xlarge', 'inf2.24xlarge', 'inf2.48xlarge',\n 'inferentia2', 'inferentia 2', 'inferentia-2',\n // Common typos\n 'infertia2', 'inferntia2', 'infernita2'\n ],\n patterns: [/inf2(?:\\.[0-9]*xlarge)?/i, /inferentia\\s*2/i]\n },\n \n // ----- Use Case Labels (independent - both can be applied) -----\n Inference: {\n keywords: [\n // Standard terms\n 'inference', 'inferencing',\n 'predict', 'prediction', 'predictions', 'predicting',\n 'serving', 'serve', 'server',\n 'batch inference', 'real-time', 'realtime',\n 'endpoint', 'endpoints',\n // Common typos\n 'infernce', 'inferance', 'prediciton', 'deploymnet'\n ],\n patterns: [/infer(?:ence|ring)?/i, /predict(?:ion|ing)?/i, /deploy(?:ment|ing)?/i]\n },\n \n Training: {\n keywords: [\n // Standard terms\n 'training', 'train', 'trained',\n 'fine-tune', 'finetune', 'fine tune', 'finetuning', 'fine-tuning',\n 'pretrain', 'pre-train', 'pretraining', 'pre-training',\n 'learning', 'learn',\n 'gradient', 'gradients',\n 'backward', 'backprop', 'backpropagation',\n 'loss', 'convergence', 'converge',\n 'epoch', 'epochs',\n 'checkpoint', 'checkpointing',\n // Common typos\n 'trainig', 'traning', 'trainin', 'fintune', 'finetunning'\n ],\n patterns: [/train(?:ing|ed)?/i, /fine[\\s-]?tun(?:e|ing)/i, /pre[\\s-]?train(?:ing)?/i]\n }\n };\n \n // =============================================================================\n // MATCHING LOGIC\n // =============================================================================\n \n function matchesLabel(config) {\n const contentLower = content.toLowerCase();\n // Check keywords (case-insensitive substring match)\n for (const keyword of config.keywords) {\n if (contentLower.includes(keyword.toLowerCase())) {\n return true;\n }\n }\n // Check regex patterns\n for (const pattern of config.patterns) {\n if (pattern.test(content)) {\n return true;\n }\n }\n return false;\n }\n \n // Issue Type Labels - MUTUALLY EXCLUSIVE (priority: bug > documentation > feature-request)\n if (matchesLabel(labelConfig.bug)) {\n labels.push('bug');\n } else if (matchesLabel(labelConfig.documentation)) {\n labels.push('documentation');\n } else if (matchesLabel(labelConfig['feature-request'])) {\n labels.push('feature-request');\n }\n \n // Hardware/Instance Type Labels - INDEPENDENT (multiple can be applied)\n if (matchesLabel(labelConfig.Trn1)) {\n labels.push('Trn1');\n }\n if (matchesLabel(labelConfig.Trn2)) {\n labels.push('Trn2');\n }\n if (matchesLabel(labelConfig.Inf1)) {\n labels.push('Inf1');\n }\n if (matchesLabel(labelConfig.Inf2)) {\n labels.push('Inf2');\n }\n \n // Use Case Labels - INDEPENDENT (both can be applied)\n if (matchesLabel(labelConfig.Inference)) {\n labels.push('Inference');\n }\n if (matchesLabel(labelConfig.Training)) {\n labels.push('Training');\n }\n \n core.setOutput('labels', labels.join(','));\n core.setOutput('has_labels', labels.length > 0);\n\n - name: Apply labels to issue\n if: steps.analyze_content.outputs.has_labels == 'true'\n env:\n GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n run: |\n IFS=',' read -ra LABELS <<< \"${{ steps.analyze_content.outputs.labels }}\"\n for label in \"${LABELS[@]}\"; do\n gh issue edit ${{ github.event.issue.number }} --add-label \"$label\" -R ${{ github.repository }}\n done\n" }, { "path": ".gitignore", "content": "_build/\n__pycache__/\n.venv/\n.DS_Store\nsrc/examples/pytorch/libtorch_demo.tar.gz\nsrc/neuronperf.tar.gz\n*-checkpoint.ipynb\n.idea/\n.vscode/\nnki/*/generated/\nuncommitted/" }, { "path": ".readthedocs.yml", "content": "# .readthedocs.yml\n# Read the Docs configuration file\n# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details\n\n# Required\nversion: 2\n\n# Set the version of Python and other tools you might need\nbuild:\n os: \"ubuntu-22.04\"\n tools:\n python: \"3.10\"\n# jobs:\n# pre_build:\n# - python -m sphinx -b linkcheck . _build/linkcheck\n\n# Build documentation in the docs/ directory with Sphinx\nsphinx:\n configuration: conf.py\n\n#conda\n#conda:\n# file: readthedocs-environment.yml\n\n# Build documentation with MkDocs\n#mkdocs:\n# configuration: mkdocs.yml\n\n# Optionally build your docs in additional formats such as PDF\n#formats:\n# - pdf\n\n# Optionally set the version of Python and requirements required to build your docs\npython:\n install:\n - requirements: requirements.txt" }, { "path": "CODEOWNERS", "content": "# This file creates codeowners for the documentation. It will allow setting code reviewers for all Pull requests to merge to the master branch \n# Each line is a file pattern followed by one or more owners.\n\n# Reference guide - https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-code-owners#example-[…]ners-file\n# Example - These owners will be the default owners for everything in\n# the repo. Unless a later match takes precedence,\n# @global-owner1 and @global-owner2 will be requested for\n# review when someone opens a pull request.\n# * @global-owner1 @global-owner2\n\n* @aws-maens @micwade-aws @musunita @aws-sadaf @rgrandhiamzn @eshalakhotia @jluntamazon @jeffhataws @aws-rhsoln @hannanjgaws @PrashantSaraf @aws-donkrets @aws-singhada @gsnaws @awsjoshir @sidjoshiaws @pinak-p @vikas-paliwal-aws @aarondou @mrinalks @erickson-doug @lnixaws @micwade-aws \n\nsrc/examples/mxnet/ @aws-rhsoln @aws-sadaf @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\nneuron-guide/neuron-frameworks/mxnet-neuron/ @aws-rhsoln @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\nneuron-guide/neuron-frameworks/mxnet-neuron/tutorials/ @musunita @aws-rhsoln @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\n\nsrc/examples/tensorflow/ @awshaichen @aws-sadaf @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\nneuron-guide/neuron-frameworks/tensorflow-neuron/ @awshaichen @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\nneuron-guide/neuron-frameworks/tensorflow-neuron/tutorials/ @musunita @awshaichen @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\n\n\nsrc/examples/pytorch/ @jluntamazon @aws-sadaf @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\nneuron-guide/neuron-frameworks/pytorch-neuron/ @jluntamazon @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\nneuron-guide/neuron-frameworks/pytorch-neuron/tutorials/ @musunita @jluntamazon @aws-maens @vikas-paliwal-aws @rgrandhiamzn @eshalakhotia\n\nlibraries/nxd-inference/ @huntingcarlisle @lccasagrande @lipovsek-aws @erickson-doug @eshalakhotia @pinak-p @hannanjgaws @akhil-aws @ahimsh-aws @rgrandhiamzn @yahavb @FThompsonAWS @gsnaws @sidjoshiaws @jluntamazon @musunita" }, { "path": "CONTRIBUTING.md", "content": "# Contributing Guidelines\n\nThank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional\ndocumentation, we greatly value feedback and contributions from our community.\n\nPlease read through this document before submitting any issues or pull requests to ensure we have all the necessary\ninformation to effectively respond to your bug report or contribution.\n\n## Reporting Bugs/Feature Requests\n\nWe welcome you to use the GitHub issue tracker to report bugs or suggest features.\n\nWhen filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already\nreported the issue. Please try to include as much information as you can. Details like these are incredibly useful:\n\n* A reproducible test case or series of steps\n* The version of our code being used\n* Any modifications you've made relevant to the bug\n* Anything unusual about your environment or deployment\n\n## Contributing Workflow (via Pull Requests)\n\nContributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:\n\n1. You are working against the latest source on the *master* branch.\n2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.\n3. You open an issue to discuss any significant work - we would hate for your time to be wasted.\n\n**Important**: Currently, local doc builds require a Python 3.9 environment. If you are on MacOS, you can install it from the terminal with `brew install python@3.9`. Add it to your working path with `brew link python@3.9` and confirm it works by running `python3.9 --version`.\n\n### Docker Build\n\nIf you don't have Python 3.9/3.10 or a compatible gcc toolchain, use the Docker workflow:\n\n```bash\n./build.sh build # Build Docker image (first time only)\n./build.sh html # Build HTML docs to _build/html/\n./build.sh shell # Interactive shell for debugging\n./build.sh clean # Remove _build/ directory\n```\n\n### Manual Build\n\nTo send us a pull request, please:\n\n1. Clone the repository locally:\n\n ```bash\n git clone git@github.com:YOUR-USERNAME/private-aws-neuron-sdk-staging.git\n ```\n\n2. Install the build dependencies. This requires a Python 3.9 installation and venv:\n\n ```bash\n cd .. # The root folder where you have your cloned Git repos; don't run this in the repo folder but one level up or you'll have venv files in your repo folder\n python3.9 -m venv venv && . venv/bin/activate\n pip install -U pip\n cd private-aws-neuron-sdk-staging\n pip install -r requirements.txt\n ```\n\n3. Build the documentation into HTML. This command will allow you to view the\n rendered documentation by opening the generated `_build/html/index.html`. On first run, this will take about 15 mins. Subsequent html generations are incremental and will take less time.\n\n Run:\n\n ```bash\n sphinx-build -b html . _build/html\n ```\n\n Or leverage the make file and run:\n\n ```bash\n make html\n ```\n\n If this doesn't work, try this command:\n\n ```bash\n sphinx-build -C -b html . _build/html\n ```\n\n For speedier builds in multiprocessor environments, run:\n\n ```bash\n sphinx-build -b html . _build/html -j auto\n ```\n\n **NOTE**: If you get an error for the spelling extension, like `Extension error: Could not import extension sphinxcontrib.spelling (exception: The 'enchant' C library was not found and maybe needs to be installed. See https://pyenchant.github.io/pyenchant/install.html`, run `brew install enchant`.\n\n4. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.\n5. Rebuild the documentation with `sphinx-build -b html . _build/html`. Always ensure that the docs build without errors and that your changes look correct before pushing your changes to remote.\n * If you encounter errors that are unclear, run the build in verbose mode with `sphinx-build -vv -b html . _build/html`.\n6. Commit your changes to your branch with a clear, scoped commit messages. Bad: \"fixed stuff\". Good: \"Updated ref IDs in all containers topics\".\n7. Push your changes to remote (`git push origin`) and create a PR from your branch into `master` or the standing release branch (example: `release-2.27.0`). Answer any default questions in the pull request interface.\n * See: [pull request guide](https://help.github.com/articles/creating-a-pull-request/)).\n8. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.\n\nUpdated process documentation can be found here: [Runbook: Authoring a topic for the Neuron documentation](https://quip-amazon.com/e9B9AM7Npb17/Runbook-Authoring-a-topic-for-the-Neuron-documentation).\n\n## Updating the sitemap\n\nIf you add or remove a topic, you must recreate the sitemap. To do so:\n\n1. From a shell, `cd` to the root of this repo (`private-aws-neuron-sdk-staging`) on your local machine.\n2. Run the following command: `python3 ./_utilities/create_sitemap.py`. This will generate the sitemap as `sitemap.xml` in the root folder of the repo.\n3. Rename the `sitemap.xml` file to `sitemap1.xml`.\n4. Move the `sitemap1.xml` file to the `/static` folder, copying over the previous version.\n5. Delete the generated `sitemap.xml` file from the root (**not** from `/static`) if you did a copy instead of a move.\n6. Push a PR with the updated sitemap to remote and request DougEric review/approve it.\n\n## Finding contributions to work on\n\nLooking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.\n * Or, if you're so inclined, get on DougEric's Christmas card list by fixing broken links, formatting errors, removing stale topics, and fixing spelling/grammar errors.\n\n## Code of Conduct\n\nThis project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).\nFor more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact\nopensource-codeofconduct@amazon.com with any additional questions or comments.\n\n## Security issue notifications\n\nIf you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.\n\n## Licensing\n\nSee the [LICENSE-DOCUMENTATION](./LICENSE-DOCUMENTATION), [LICENSE-SAMPLECODE](./LICENSE-SAMPLECODE) and [LICENSE-SUMMARY-DOCS-SAMPLES](./LICENSE-SUMMARY-DOCS-SAMPLES) files for our project's licensing. We will ask you to confirm the licensing of your contribution.\n\nWe may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger chan\n" }, { "path": "Dockerfile", "content": "FROM python:3.10-slim\n\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n make enchant-2 git pandoc \\\n && rm -rf /var/lib/apt/lists/* \\\n && pandoc --version\n\nCOPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv\n\nWORKDIR /docs\n\nCOPY requirements.txt .\nRUN uv pip install --system -r requirements.txt --extra-index-url=https://pypi.org/simple\n\nENTRYPOINT [\"/bin/bash\"]\n" }, { "path": "LICENSE-DOCUMENTATION", "content": "*** Documentation:\n\nCreative Commons Attribution-ShareAlike 4.0 International Public License\n\nBy exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-ShareAlike 4.0 International Public License (\"Public License\"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.\n\nSection 1 – Definitions.\n\t\n a.\tAdapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.\n\t\n b.\tAdapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License.\n\t\n c.\tBY-SA Compatible License means a license listed at creativecommons.org/compatiblelicenses, approved by Creative Commons as essentially the equivalent of this Public License.\n\t\n d.\tCopyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights.\n\t\n e.\tEffective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.\n\t\n f.\tExceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.\n\t\n g.\tLicense Elements means the license attributes listed in the name of a Creative Commons Public License. The License Elements of this Public License are Attribution and ShareAlike.\n\t\n h.\tLicensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License.\n\t\n i.\tLicensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.\n\t\n j.\tLicensor means the individual(s) or entity(ies) granting rights under this Public License.\n\t\n k.\tShare means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.\n\t\n l.\tSui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.\n\t\n m.\tYou means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning.\n\nSection 2 – Scope.\n\t\n a.\tLicense grant.\n\t\n 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to:\n\n A. reproduce and Share the Licensed Material, in whole or in part; and\t\n\n B. produce, reproduce, and Share Adapted Material.\n\t\n 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.\n\t\n 3. Term. The term of this Public License is specified in Section 6(a).\n\t\n 4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material.\n\t\n 5. Downstream recipients.\n\n A. Offer from the Licensor – Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.\n\t\n B. Additional offer from the Licensor – Adapted Material. Every recipient of Adapted Material from You automatically receives an offer from the Licensor to exercise the Licensed Rights in the Adapted Material under the conditions of the Adapter’s License You apply.\n\t\n C. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.\n\t\n 6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i).\n\t\n b.\tOther rights.\n\t\n 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.\n\t\n 2. Patent and trademark rights are not licensed under this Public License.\n\t\n 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties.\n\nSection 3 – License Conditions.\n\nYour exercise of the Licensed Rights is expressly made subject to the following conditions.\n\t\n a.\tAttribution.\n\t\n 1. If You Share the Licensed Material (including in modified form), You must:\n\n A. retain the following if it is supplied by the Licensor with the Licensed Material:\n\n i.\tidentification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);\n\n ii.\ta copyright notice;\n\n iii. a notice that refers to this Public License;\n\n iv.\ta notice that refers to the disclaimer of warranties;\n\n v.\ta URI or hyperlink to the Licensed Material to the extent reasonably practicable;\n\n B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and\n\n C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.\n\t\n 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.\n\t\n 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable.\n\t\n b.\tShareAlike.In addition to the conditions in Section 3(a), if You Share Adapted Material You produce, the following conditions also apply.\n\t\n 1. The Adapter’s License You apply must be a Creative Commons license with the same License Elements, this version or later, or a BY-SA Compatible License.\n\t\n 2. You must include the text of, or the URI or hyperlink to, the Adapter's License You apply. You may satisfy this condition in any reasonable manner based on the medium, means, and context in which You Share Adapted Material.\n\t\n 3. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, Adapted Material that restrict exercise of the rights granted under the Adapter's License You apply.\n\nSection 4 – Sui Generis Database Rights.\n\nWhere the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:\n\t\n a.\tfor the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database;\n\t\n b.\tif You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material, including for purposes of Section 3(b); and\n\t\n c.\tYou must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database.\nFor the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights.\n\nSection 5 – Disclaimer of Warranties and Limitation of Liability.\n\t\n a.\tUnless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.\n\t\n b.\tTo the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.\n\t\n c.\tThe disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.\n\nSection 6 – Term and Termination.\n\t\n a.\tThis Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.\n\t\n b.\tWhere Your right to use the Licensed Material has terminated under Section 6(a), it reinstates:\n\t\n 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or\n\t\n 2. upon express reinstatement by the Licensor.\n\t\n c.\tFor the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License.\n\t\n d.\tFor the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.\n\t\n e.\tSections 1, 5, 6, 7, and 8 survive termination of this Public License.\n\nSection 7 – Other Terms and Conditions.\n\t\n a.\tThe Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.\n\t\n b.\tAny arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.\n\nSection 8 – Interpretation.\n\t\n a.\tFor the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.\n\t\n b.\tTo the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.\n\t\n c.\tNo term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.\n\t\n d.\tNothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.\n" }, { "path": "LICENSE-SAMPLECODE", "content": "Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this\nsoftware and associated documentation files (the \"Software\"), to deal in the Software\nwithout restriction, including without limitation the rights to use, copy, modify,\nmerge, publish, distribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,\nINCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A\nPARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT\nHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\nSOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n" }, { "path": "LICENSE-SUMMARY-DOCS-SAMPLES", "content": "*** Documentation and Sample Code:\n\nCopyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n\nThe documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International License. See the LICENSE file.\n\nThe sample code within this documentation is made available under the MIT-0 license. See the LICENSE-SAMPLECODE file.\n" }, { "path": "Makefile", "content": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the environment for the first two.\nSPHINXOPTS ?=\nSPHINXBUILD ?= sphinx-build\nSOURCEDIR = $(CURDIR)\nBUILDDIR = _build\n\n# Put it first so that \"make\" without argument is like \"make help\".\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n.PHONY: help Makefile clean\n\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option. $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\nclean:\n\t-rm -rf $(BUILDDIR)/*\n\n" }, { "path": "README.md", "content": "![neuron](./images/Site-Merch_Neuron-ML-SDK_Editorial.png)\n\n# AWS Neuron\n\n## Neuron SDK Overview\n\nAWS Neuron is a software development kit (SDK) enabling high-performance deep learning acceleration using AWS Inferentia and Trainium, AWS's custom designed machine learning accelerators. With Neuron, you can develop, profile, and deploy high-performance machine learning workloads on top of accelerated EC2 instances, e.g. Inf1 and Trn1.\n\nNeuron includes a compiler, runtime driver, as well as debug and profiling utilities with a TensorBoard plugin for visualization, and is pre-integrated into popular machine learning frameworks like Pytorch, TensorFlow and MXNet, to provide a seamless machine learning acceleration workflow.\n\n## Neuron SDK’s documentation\n\nFor full documentations including user guide, Howtos and Tutorials see [Neuron SDK’s documentation](https://awsdocs-neuron.readthedocs-hosted.com/)\n\n## Support\nIf none of the github and online resources have an answer to your question, checkout the AWS Neuron [support forum](https://forums.aws.amazon.com/forum.jspa?forumID=355).\n" }, { "path": "_backup-setup/neuron-setup/multiframework/multi-framework-ubuntu22-neuron-dlami.rst", "content": ".. _setup-ubuntu22-multi-framework-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nGet Started with Neuron on Ubuntu 22 with Neuron Multi-Framework DLAMI\n======================================================================\n\nYou can quickly get started on Ubuntu 22 using the Neuron Deep Learning AMI (DLAMI). Then, start using one of the multiple frameworks or libraries that Neuron SDK supports by\nactivating the corresponding virtual environment. Each virtual environment comes pre-installed with Neuron libraries needed for you to get started. The Neuron DLAMI supports all Neuron instances (Inf1/Inf2/Trn1/Trn1n/Trn2/Trn3)\nand is updated with each Neuron SDK release. To start using the latest version of the Neuron DLAMI, use the following steps:\n\nStep 1: Launch the instance using Neuron DLAMI\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOnce you open the `EC2 Console `_, select your desired AWS region and choose \"Launch Instance\". Under AMI selection select the \"Quick Start\"\nand \"Ubuntu\", choose the \"Deep Learning AMI Neuron (Ubuntu 22.04)\"(see screenshot below). Once you have selected the AMI, select the desired Neuron Instance(Inf1/Inf2/Trn1/Trn1n/Trn2/Trn3) , \nconfigure disk size and other criteria, launch the instance\n\n.. image:: /images/neuron-multi-framework-dlami-quick-start.png\n :scale: 20%\n :align: center\n\n\n.. note::\n If you are looking to use the Neuron DLAMI in your cloud automation flows , Neuron also supports :ref:`SSM parameters ` to easily retrieve the latest DLAMI id.\n\n\n\nStep 2: Activate the desired virtual environment \n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ \n\nYou can activate one of the virtual environments depending on the library or framework you are interested in:\n\n1. Get the desired virtual environment name for the framework/library by referring to :ref:`the Neuron DLAMI overview `.\n2. Activate the virtual environment by using:\n\n ::\n\n source /opt//bin/activate\n\n\nAfter you have activated the desired virtual environment , you can try out one of the tutorials listed in the corresponding framework or library training and inference section.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "_backup-setup/neuron-setup/multiframework/multi-framework-ubuntu24-neuron-dlami.rst", "content": ".. _setup-ubuntu24-multi-framework-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nGet Started with Neuron on Ubuntu 24 with Neuron Multi-Framework DLAMI\n======================================================================\n\nYou can quickly get started on Ubuntu 24 using the Neuron Deep Learning AMI (DLAMI). Then, start using one of the multiple frameworks or libraries that Neuron SDK supports by\nactivating the corresponding virtual environment. Each virtual environment comes pre-installed with Neuron libraries needed for you to get started. The Neuron DLAMI supports all Neuron instances (Inf2/Trn1/Trn1n/Trn2/Trn3)\nand is updated with each Neuron SDK release. To start using the latest version of the Neuron DLAMI, use the following steps:\n\nStep 1: Launch the instance using Neuron DLAMI\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOnce you open the `EC2 Console `_, select your desired AWS region and choose \"Launch Instance\". Under AMI selection select the \"Quick Start\"\nand \"Ubuntu\", choose the \"Deep Learning AMI Neuron (Ubuntu 24.04)\"(see screenshot below). Once you have selected the AMI, select the desired Neuron Instance(Inf2/Trn1/Trn1n/Trn2/Trn3), \nconfigure disk size and other criteria, launch the instance\n\n.. image:: /images/neuron-multi-framework-dlami-U24-quick-start.png\n :scale: 20%\n :align: center\n\n\n.. note::\n If you are looking to use the Neuron DLAMI in your cloud automation flows , Neuron also supports :ref:`SSM parameters ` to easily retrieve the latest DLAMI id.\n\n\n\nStep 2: Activate the desired virtual environment \n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ \n\nYou can activate one of the virtual environments depending on the library or framework you are interested in:\n\n1. Get the desired virtual environment name for the framework/library by referring to :ref:`the Neuron DLAMI overview `.\n2. Activate the virtual environment by using:\n\n ::\n\n source /opt//bin/activate\n\n\nAfter you have activated the desired virtual environment , you can try out one of the tutorials listed in the corresponding framework or library training and inference section.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/amazon-linux/torch-neuron-al2-base-dlami.rst", "content": ".. _setup-torch-neuron-al2-base-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Amazon Linux 2 with DLAMI Base\n=======================================================================\n\n.. note::\n As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the :ref:`AL2 Migration guide `\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron` for Inference.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instance sizes and pricing see: `Inf1 web page `_\n * Check for the latest version of the `DLAMI Base AMI `_ and copy the AMI name that starts with \"Deep Learning Base Neuron AMI (Amazon Linux 2) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see a matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-torch-neuron-al2.txt\n\n.. include :: /archive/torch-neuron/setup/pytorch-update-al2.rst\n\n.. include :: /archive/torch-neuron/setup/pytorch-install-prev-al2.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/amazon-linux/torch-neuron-al2-pytorch-dlami.rst", "content": ".. _setup-torch-neuron-al2-pytorch-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Amazon Linux 2 with Pytorch DLAMI\n=========================================================================\n\n.. note::\n As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the :ref:`AL2 Migration guide `\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Check for the latest version of the `DLAMI Neuron Pytorch 1.13 AMI `_ and copy the AMI name that starts with \"Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see an exact matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Update Neuron Drivers\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=driver_runtime_tools --framework=pytorch --framework-version=1.13.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1\n\n.. dropdown:: Get Started With Pytorch DLAMI\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 98\n :end-line: 99\n\n.. card:: Visit PyTorch Neuron(``torch-neuron``) for Inference section\n :link: inference-torch-neuron\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. card:: Visit PyTorch Neuron section for more\n :class-body: sphinx-design-class-body-small\n :link: neuron-pytorch\n :link-type: ref\n\n.. include:: /archive/torch-neuron/setup/pytorch-update-al2-dlami.rst\n\n.. include:: /archive/torch-neuron/setup/pytorch-install-prev-al2.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/amazon-linux/torch-neuron-al2.rst", "content": ".. _setup-torch-neuron-al2:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Amazon Linux 2\n=========================================================\n\n.. note::\n As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the :ref:`AL2 Migration guide `\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron` for Inference.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Amazon Linux 2 AMI(HVM) - Kernel 5.10\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-torch-neuron-al2.txt\n\n.. include :: /archive/torch-neuron/setup/pytorch-update-al2.rst\n\n.. include :: /archive/torch-neuron/setup/pytorch-install-prev-al2.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/amazon-linux/torch-neuron-al2023.rst", "content": ".. _setup-torch-neuron-al2023:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Amazon Linux 2023\n===========================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron` for Inference.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Amazon Linux 2023 AMI\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-torch-neuron-al2023.txt\n\n.. include:: /archive/torch-neuron/setup/pytorch-update-al2023.rst\n\n.. include:: /archive/torch-neuron/setup/pytorch-install-prev-al2023.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/ubuntu/torch-neuron-ubuntu20-base-dlami.rst", "content": ".. _setup-torch-neuron-u20-base-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Ubuntu 20 with DLAMI Base\n==================================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron` for Inference.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instance sizes and pricing see: `Inf1 web page `_\n * Check for the latest version of the `DLAMI Base AMI `_ and copy the AMI name that starts with \"Deep Learning Base Neuron AMI (Ubuntu 20.04) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see a matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-torch-neuron-u20.txt\n\n.. include:: /archive/torch-neuron/setup/pytorch-update-u20.rst\n\n.. include:: /archive/torch-neuron/setup/pytorch-install-prev-u20.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/ubuntu/torch-neuron-ubuntu20-pytorch-dlami.rst", "content": ".. _setup-torch-neuron-u20-pytorch-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Ubuntu 20 with Pytorch DLAMI\n=====================================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Check for the latest version of the `DLAMI Neuron Pytorch 1.13 AMI `_ and copy the AMI name that starts with \"Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see an exact matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n\n.. dropdown:: Update Neuron Drivers\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=driver_runtime_tools --framework=pytorch --framework-version=1.13.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1\n\n\n.. dropdown:: Get Started With Pytorch DLAMI\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 101\n :end-line: 102\n\n.. card:: PyTorch Neuron(``torch-neuron``) for Inference\n :link: inference-torch-neuron\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. card:: Visit PyTorch Neuron section for more\n :class-body: sphinx-design-class-body-small\n :link: neuron-pytorch\n :link-type: ref\n\n.. include:: /archive/torch-neuron/setup/pytorch-update-u20-dlami.rst\n\n.. include:: /archive/torch-neuron/setup/pytorch-install-prev-u20.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/ubuntu/torch-neuron-ubuntu20.rst", "content": ".. _setup-torch-neuron-u20:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Ubuntu 20\n====================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron` for Inference.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Ubuntu Server 20 AMI\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-torch-neuron-u20.txt\n\n.. include:: /archive/torch-neuron/setup/pytorch-update-u20.rst\n\n.. include:: /archive/torch-neuron/setup/pytorch-install-prev-u20.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuron/ubuntu/torch-neuron-ubuntu22.rst", "content": ".. _setup-torch-neuron-u22:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuron\") Setup on Ubuntu 22\n=====================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`setup-torch-neuron` for Inference.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Ubuntu Server 22 AMI\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n.. include:: /includes/setup/tab-inference-torch-neuron-u22.txt\n\n.. include:: /archive/torch-neuron/setup/pytorch-update-u22.rst\n\n.. include:: /archive/torch-neuron/setup/pytorch-install-prev-u22.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/amazon-linux/torch-neuronx-al2-base-dlami.rst", "content": ".. _setup-torch-neuronx-al2-base-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Amazon Linux 2 with DLAMI Base\n=========================================================================\n\n.. note::\n As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the :ref:`AL2 Migration guide `\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instance sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Check for the latest version of the `DLAMI Base AMI `_ and copy the AMI name that starts with \"Deep Learning Base Neuron AMI (Amazon Linux 2) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see a matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 2\n :end-line: 3\n\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-al2.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-al2.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/amazon-linux/torch-neuronx-al2-pytorch-dlami.rst", "content": ".. _setup-torch-neuronx-al2-dlami-pytorch:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Amazon Linux 2 with DLAMI Pytorch\n===========================================================================\n\n.. note::\n As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the :ref:`AL2 Migration guide `\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Check for the latest version of the `DLAMI Neuron Pytorch 1.13 AMI `_ and copy the AMI name that starts with \"Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see an exact matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Update Neuron Drivers\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=driver_runtime_tools --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1\n\n.. dropdown:: Get Started With Pytorch DLAMI\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 50\n :end-line: 51\n\n\n.. card:: Visit PyTorch Neuron(``torch-neuronx``) for Inference section\n :link: inference-torch-neuronx\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\n.. card:: Visit PyTorch Neuron(``torch-neuronx``) for Training section\n :link: training-torch-neuronx\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-al2-dlami.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/amazon-linux/torch-neuronx-al2.rst", "content": ".. _setup-torch-neuronx-al2:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Amazon Linux 2\n=========================================================\n\n.. note::\n As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the :ref:`AL2 Migration guide `\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Select Amazon Linux 2 AMI(HVM) - Kernel 5.10\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 2\n :end-line: 3\n\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-al2.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-al2.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/amazon-linux/torch-neuronx-al2023.rst", "content": ".. _setup-torch-neuronx-al2023:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Amazon Linux 2023\n============================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Select Amazon Linux 2023 AMI\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 239\n :end-line: 240\n\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-al2023.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-al2023.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2023.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu20-base-dlami.rst", "content": ".. _setup-torch-neuronx-ubuntu20-base-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Ubuntu 20 with DLAMI Base\n====================================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instance sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Check for the latest version of the `DLAMI Base AMI `_ and copy the AMI name that starts with \"Deep Learning Base Neuron AMI (Ubuntu 20.04) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see a matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 5\n :end-line: 6\n\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-u20.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-u20.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u20.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu20-pytorch-dlami.rst", "content": ".. _setup-torch-neuronx-ubuntu20-dlami-pytorch:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Ubuntu 20 with DLAMI Pytorch\n======================================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Check for the latest version of the `DLAMI Neuron Pytorch 1.13 AMI `_ and copy the AMI name that starts with \"Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see an exact matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Update Neuron Drivers\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=driver_runtime_tools --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1\n\n.. dropdown:: Get Started With Pytorch DLAMI\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 53\n :end-line: 54\n\n.. card:: Visit PyTorch Neuron(``torch-neuronx``) for Inference section\n :link: inference-torch-neuronx\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\n.. card:: Visit PyTorch Neuron(``torch-neuronx``) for Training section\n :link: training-torch-neuronx\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n \n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-u20-dlami.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u20.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu20.rst", "content": ".. _setup-torch-neuronx-ubuntu20:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :width: 100%\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Ubuntu 20\n===================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n.. include:: /setup/install-templates/trn1-ga-warning.txt\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Select Ubuntu Server 20 AMI\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 5\n :end-line: 6\n\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-u20.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-u20.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u20.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu22.rst", "content": ".. _setup-torch-neuronx-ubuntu22:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Ubuntu 22\n=====================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n.. include:: /setup/install-templates/trn1-ga-warning.txt\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Select Ubuntu Server 22 AMI\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 242\n :end-line: 243\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-u22.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-u22.rst\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u22.rst" }, { "path": "_backup-setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu24.rst", "content": ".. _setup-torch-neuronx-ubuntu24:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nPyTorch Neuron (\"torch-neuronx\") Setup on Ubuntu 24\n=====================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of PyTorch Neuron (``torch-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`pytorch-neuronx-main` for both Inference and Training.\n\n.. include:: /setup/install-templates/trn1-ga-warning.txt\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Trn1 web page `_, `Inf2 web page `_\n * Select Ubuntu Server 24 AMI\n * When launching a Trn1, please adjust your primary EBS volume size to a minimum of 512GB.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 299\n :end-line: 300\n\n.. include:: /includes/setup/tab-inference-torch-neuronx-u24.txt\n\n.. include:: /frameworks/torch/torch-neuronx/setup/pytorch-update-u24.rst\n\n" }, { "path": "_content-types/conceptual-deep-dive.rst", "content": ".. meta::\n :description: {short description here}\n :date_updated: {planned date of publication here}\n\n.. _{RST page ref string here}:\n\n================================================================================\nDeep dive: {concept/practice/technique name; use sentence-case, not title case!}\n================================================================================\n\n.. {SEO-friendly intro paragraph, no more than 3 sentences total.} \nThis topic explores {subjects} in depth and discusses the technical details of it from the perspective of an AWS Neuron expert. Some experience in {related subjects here} is required to understand it in full.\n\nWhat you should know before reading\n-----------------------------------\n\n.. {If there is anything the reader should know before diving into this material, note it here and provide any supporting links. This also helps LLMs training on this content have greater technical context for this subject.}\n\nBefore you start, you must be familiar with the following:\n\n- **Concept 1:** {Brief description. Link to a related topic if necessary.}\n- **Concept 2:** {Brief description. Link to a related topic if necessary.}\n\nOverview\n---------\n\n.. {Your first section, which should cover the subject from the title at a high level. If appropriate, note when this concept is applicable in Neuron components and developer workflows. Starting off with a diagram can help illustrate the concept.}\n\nPARAGRAPH 1\n\nPARAGRAPH 2\n\n.. image:: images/diagram-name.png\n :alt: {Alt text for diagram}\n :align: center\n\n{Section 1 Title}\n-----------------\n\n.. {Each section should build on top of what was discussed in the previous sections. If a new concept is introduced that wasn't discussed previously, link to a topic that covers it. You can add subsections within this section if it helps to break it up more and clarify the content, but do not go more than 1-2 levels deep.}\n\nPARAGRAPH 1\n\nPARAGRAPH 2\n\n.. code-block:: python\n\n # Code example if applicable\n def example_function():\n pass\n\n{Section 2 Title}\n-----------------\n\n.. {Each section should build on top of what was discussed in the previous sections. If a new concept is introduced that wasn't discussed previously, link to a topic that covers it. You can add subsections within this section if it helps to break it up more and clarify the content, but do not go more than 1-2 levels deep.}\n\nPARAGRAPH 1\n\nPARAGRAPH 2\n\n.. code-block:: python\n\n # Code example if applicable\n def example_function():\n pass\n\n.. {Add more sections as appropriate to logically break up the content. Each section should be focused on a specific aspect of the concept.}\n\n{optional}Related Concepts\n----------------\n\n* :ref:`link-reference-name` - {description}\n* :ref:`link-reference-name` - {description}\n\n{optional}Further Reading\n---------------\n\n.. toctree::\n :maxdepth: 1\n\n * `External Link `_ - {description}\n * :doc:`/path/to/internal/doc` - {description}\n\n\n.. (Note to both the writer and any AI incorporating this template: The content below is provided as a resource and should not be included as-is in any final document created using this template as a basis.)\n\n.. note::\n .. Additional implementation details or important considerations can be added as admonitions.\n\n.. warning::\n .. Critical information or potential pitfalls can be highlighted using warning admonitions.\n" }, { "path": "_content-types/model-card.rst", "content": ".. _unique-ref-id-here:\n\n.. meta::\n :description: AWS Neuron SDK model card for {Model Name}, version {version}. Overview, intended use, training data, performance, limitations, ethical considerations, and citations.\n :date-modified: 2026-10-03\n\nModel Card: {Model Name}\n=======================\n\n.. contents:: Table of Contents\n :depth: 1\n :local:\n\nModel overview\n--------------\n\n:Model name: {name}\n:Version: {version}\n:Organization: {organization}\n:License: {license}\n:Last updated: {date}\n\n.. warning::\n {Important warnings or critical limitations}\n\nQuickstart\n----------\n\n.. code-block:: python\n\n # Example usage code\n from model import Model\n model = Model.from_pretrained(\"model_name\")\n output = model.generate(\"Your input text\")\n\nModel details\n-------------\n\nArchitecture\n^^^^^^^^^^^^\n\n- Base architecture: {architecture}\n- Number of parameters: {parameter_count}\n- Model dimensions: {model_dimensions}\n- Training objective: {training_objective}\n\nHardware requirements\n^^^^^^^^^^^^^^^^^^^^^\n\n- Minimum RAM: {min_ram}\n- Recommended GPU: {gpu_specs}\n- Disk space: {disk_space}\n\nIntended Use\n-----------\n\nPrimary uses\n^^^^^^^^^^^^\n* {use_case_1}\n* {use_case_2}\n* {use_case_3}\n\nOut-of-Scope uses\n^^^^^^^^^^^^^^^^^\n* {prohibited_use_1}\n* {prohibited_use_2}\n\nTraining data\n------------\n\nDatasets\n^^^^^^^^\n.. list-table::\n :header-rows: 1\n\n * - Dataset Name\n - Size\n - Description\n * - {dataset1}\n - {size1}\n - {description1}\n * - {dataset2}\n - {size2}\n - {description2}\n\nTraining procedure\n^^^^^^^^^^^^^^^^^^\n* Training hardware: {hardware_details}\n* Training time: {duration}\n* Training cost: {cost_estimate}\n* Carbon footprint: {carbon_impact}\n\nPerformance and limitations\n---------------------------\n\nBenchmarks\n^^^^^^^^^\n.. list-table::\n :header-rows: 1\n\n * - Benchmark\n - Score\n - Details\n * - {benchmark1}\n - {score1}\n - {details1}\n * - {benchmark2}\n - {score2}\n - {details2}\n\nKnown limitations\n^^^^^^^^^^^^^^^^^\n* {limitation_1}\n* {limitation_2}\n\nBias and fairness\n^^^^^^^^^^^^^^^^^\n* {bias_consideration_1}\n* {bias_consideration_2}\n\nEthical considerations\n----------------------\n\nPotential risks\n^^^^^^^^^^^^^^^\n* {risk_1}\n* {risk_2}\n\nMitigation strategies\n^^^^^^^^^^^^^^^^^^^^^\n* {strategy_1}\n* {strategy_2}\n\nModel details and notes\n----------------------\n\n{Provide detailed information about the model, its training, evaluation, and any other relevant aspects. Create the sections as needed.}\n\n{Section 1 title}\n^^^^^^^^^^^^^^^^^\n{Details for section 1.}\n\n{Section 2 title}\n^^^^^^^^^^^^^^^^^\n{Details for section 2.}\n\n{. . .}\n\nCitations\n---------\n\n.. code-block:: bibtex\n\n @article{model_paper,\n title={},\n author={},\n journal={},\n year={}\n }\n\nVersion history\n---------------\n\n.. list-table::\n :header-rows: 1\n\n * - Version\n - Date\n - Changes\n * - {version1}\n - {date1}\n - {changes1}\n * - {version2}\n - {date2}\n - {changes2}\n\nContact\n-------\n\n:Documentation Issues: {link_to_issues}\n:Support: {support_contact}\n:Website: {website_url}\n" }, { "path": "_content-types/procedural-how-to.rst", "content": ".. meta::\n :description: {short description here}\n :date_updated: {planned date of publication here}\n\n.. _{RST page ref string here}:\n\n========================================================================\nHow to {verb phrase with specific features or models that will be used}\n========================================================================\n\nTask overview\n-------------\n\n.. {SEO-friendly intro paragraph, no more than 3 sentences total.} \nThis topic discusses how to {description of task or process here} using the AWS Neuron SDK. {Short description of what the task will accomplish.}\n\nPrerequisites\n-------------\n\n- **Prerequisite 1:** Description. Link to a related topic if necessary.\n- **Prerequisite 2:** Description. Link to a related topic if necessary.\n\nInstructions\n------------\n\n**1:** {First step; start with verb/action}\n\n.. {Describe what the user will do in this step, starting with a verb. If applicable, include any commands or code examples that illustrate the step.}\n\n.. code-block:: bash\n\n # Command or code example\n command --flag value\n\n.. {Additional detail if needed.}\n\n.. note::\n .. {Optional; important information or caveats about this step}\n\n**2:** {Second step; start with verb/action}\n\n.. .. {Describe what the user will do in this step, starting with a verb. If applicable, include any commands or code examples that illustrate the step.}\n\n.. code-block:: python\n\n # Code example if applicable\n def example():\n pass\n\n.. {Additional detail if needed.}\n\n.. note::\n .. {Optional; important information or caveats about this step}\n\n.. **{More discrete steps as needed, following the same pattern as above.}**\n\n**N:** {Last step; start with verb/action}\n\n.. {Final step instructions}\n\nConfirm your work\n-----------------\n\nTo confirm you have successfully completed this task, {how to verify the task was done correctly}:\n\n.. {Provide them with a way to know they’ve done everything correctly. This could be a screenshot, command-line output, a tool to launch, or specific settings to check.}\n\n.. code-block:: bash\n\n # Verification command if applicable\n verify-command --check\n\nCommon issues\n-------------\n\nUh oh! Did you encounter an error or other issue while working through this task? Here are some commonly encountered issues and how to address them.\n\n.. rubric:: {Problem 1}\n\n- **Possible solution**: {detailed solution}\n\n.. rubric:: {Problem 2}\n\n- **Possible solution**: {detailed solution}\n\nRelated information\n-------------------\n\n.. toctree::\n :maxdepth: 1\n\n * `External Link `_ - {description}\n * :doc:`/path/to/internal/doc` - {description}\n" }, { "path": "_content-types/procedural-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"raw\",\n \"metadata\": {\n \"raw_mimetype\": \"text/restructuredtext\",\n \"vscode\": {\n \"languageId\": \"raw\"\n }\n },\n \"source\": [\n \".. meta::\\n\",\n \" :description: {SEO-friendly short description of the tutorial. Include 'Neuron' and any keywords such as the language mode and framework.}\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Tutorial: {title starting with verb}\\n\",\n \"\\n\",\n \"This tutorial guides you through using the AWS Neuron SDK to {description of what the reader will accomplish in this tutorial, using a specific component or framework}.\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Overview\\n\",\n \"\\n\",\n \"{Briefly summarize the purpose and outcome of this end-to-end tutorial}.\\n\",\n \"{State what users will learn or achieve by completing the tutorial}.\"\n ]\n },\n {\n \"cell_type\": \"raw\",\n \"metadata\": {\n \"raw_mimetype\": \"text/restructuredtext\",\n \"vscode\": {\n \"languageId\": \"raw\"\n }\n },\n \"source\": [\n \".. contents:: Table of contents\\n\",\n \" :local:\\n\",\n \" :depth: 2\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Before you start\\n\",\n \"\\n\",\n \"To successfully complete this tutorial, you must have completed the following steps in advance:\\n\\n\",\n \"- Downloaded and installed the [AWS Neuron SDK](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/index.html) for {component}.\\n\",\n \"- {prerequisite 2 description here. If the user must read a topic in advance or perform any complex preparations, provide a link to a topic or download}\\n\",\n \"- {prerequisite 3 description here}\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Setup\\n\",\n \"\\n\",\n \"{Describe any initial local setup required before starting the tutorial.}\\n\",\n \"{Include any code-specific installation, configuration, or environment setup steps.}\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Example setup command (Remove these comments and add the CLI commands, env variable declarations, or other operations for the user to prepare their environment.)\\n\",\n \"# pip install package_name\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Tutorial steps\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 1: {Title, starting with an infinitive verb like 'Load...', 'Create...', etc.}\\n\",\n \"\\n\",\n \"{Describe the first main step. Provide code, commands, or configuration as needed.}\\n\",\n \"\\n\",\n \"{Optional} {Add any important notes, caveats, or warnings for this step.}\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print(\\\"Code goes here!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 2: {Title, starting with an infinitive verb like 'Load...', 'Create...', etc.}\\n\",\n \"\\n\",\n \"{Describe the second main step.}\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print(\\\"Code goes here!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step 3: {Title, starting with an infinitive verb like 'Load...', 'Create...', etc.}\\n\",\n \"\\n\",\n \"Describe the third main step.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print(\\\"Code goes here!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Step N: {Title, starting with an infinitive verb like 'Load...', 'Create...', etc.}\\n\",\n \"\\n\",\n \"Describe the last main step.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print(\\\"Code goes here!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\nCode completed. Now, let's run it...\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run the code\\n\",\n \"\\n\",\n \"To run this code, {action to take to run the code}:\\n\",\n \"Include commands, expected outputs, or checks to perform.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Example verification command\\n\",\n \"# python foo.py\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"If your code works, you will see output like this:\\n\\n\",\n \"```\\n\",\n \"Loading glorp inhume logic...Done!\\n\",\n \"Configuring extubation channel instances...Done!\\n\\n\",\n \"1111 | 2222 | 3333\\n\",\n \"4444 | 5555 | 6666\\n\\n\",\n \"Average glorps inhumed and extubated: 420\\n\",\n \"Time to max glorp: 8 seconds\\n\",\n \"```\\n\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\nCongratulations! You now know how to {goal of tutorial}. If your code did not run or did not produce similar results, see the [Common issues](#Common issues) section below.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Common issues\\n\",\n \"\\n\",\n \"Here are some common errors and mistakes you can make when developing code using the approach in this tutorial, and how you may be able to address them:\\n\\n\",\n \"- {describe error, symptoms, and possible solution}\\n\",\n \"- {describe error, symptoms, and possible solution}\"\n ]\n },\n\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## (Optional) Next steps\\n\",\n \"\\n\",\n \"{Suggest what users might want to do next after completing the tutorial.\\n\",\n \"Link to related topics or advanced guides.}\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Related topics\\n\",\n \"\\n\",\n \"- [Related topic 1](link_here)\\n\",\n \"- [Related topic 2](link_here)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.5\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "_content-types/reference-kernel-api.rst", "content": ".. meta::\n :description: API reference for the {kernel-name} kernel included in the NKI Library .\n :date-modified: MM/DD/YYYY\n\n.. currentmodule:: {kernel namespace}.{kernel module path}\n\nRMSNorm-Quant Kernel API Reference\n==================================\n\nThis topic provides the API reference for the ``{kernel name}`` kernel. The kernel performs optional RMS normalization followed by quantization to ``fp8``.\n\nThe kernel supports:\n\n* {feature 1}\n* {feature 2}\n* {feature 3}\n* ... {more features as needed}\n\nBackground\n-----------\n\nThe ``{kernel}`` kernel ... {description of kernel functionality based in sources}\n\nFor detailed information about the mathematical operations and implementation details, refer to the :doc:`{kernel name} Kernel Design Specification `.\n\nAPI Reference\n--------------\n\n{kernel argument class name}\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. py:class:: {kernel argument class name}\n\n {kernel name} Kernel arguments.\n\n .. py:attribute:: {attribute-1}\n :type: {attribute-1-type}\n\n {description from docstring}\n\n .. py:attribute:: {attribute-1}\n :type: {attribute-1-type}\n\n {description from docstring}\n\n {more attributes as needed}\n\n .. py:method:: {method syntax} -> {return type}\n\n {description from docstring}\n\n .. py:method:: {method syntax} -> {return type}\n\n {description from docstring}\n\n **Raises**:\n\n * **{exception-1}** – {when exception is raised}\n * **{exception-1}** – {when exception is raised}\n\n{kernel API function name in code}\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. py:function:: rmsnorm_quant_kernel(hidden: nt.tensor, ln_w: nt.tensor, kargs: RmsNormQuantKernelArgs)\n\n {definition of method used to instantiate or invoke kernel here, from source docstrings}\n \n {params and types with descriptions from source docstrings}\n\nImplementation Details\n-----------------------\n\nThe kernel implementation includes several key optimizations:\n\n1. **{optimization-or-feature}**: {description}\n2. **{optimization-or-feature}**: {description}\n3. **{optimization-or-feature}**: {description}\n\nExample\n--------\n\nThe following is a simple example of how to use the ``{kernel}`` kernel:\n\n.. code-block:: python\n\n # Code here -- need usage example in pedagogical style.\n\nSee Also\n--------\n\n* :doc:`{kernel} `\n\n" }, { "path": "_content-types/release-notes-templates/compiler.rst", "content": ".. _neuron-2-XX-0-compiler:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK compiler component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: Neuron Compiler release notes\n====================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/containers.rst", "content": ".. _neuron-2-XX-0-dlc:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK Deep Learning Containers (DLC) component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.0: Neuron Deep Learning Containers release notes\n====================================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/dlami.rst", "content": ".. _neuron-2-XX-0-dlami:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK Deep Learning AWS Machine Images (DLAMIs) component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: Neuron Deep Learning AWS Machine Images release notes\n============================================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.X release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/index.rst", "content": ".. _neuron-2-XX-0-whatsnew:\n.. _latest-neuron-release:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X release notes\n===================================\n\n**Date of release**: Month Day, 2026\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n\n PyTorch support \n JAX support \n NxD Inference \n NxD Training \n NxD Core \n Neuron Compiler \n NKI \n Neuron Runtime \n Developer tools \n Deep Learning AMIs \n Deep Learning Containers \n Release artifacts <../releasecontent>\n\nWhat's new?\n-----------\n\nAWS and Annapurna Labs are excited to bring you release version 2.XX.X of the Neuron SDK! In this release you'll find improvements to...\n\n* . . .\n* . . .\n* . . .\n\n.. contents:: In this release\n :local:\n :depth: 1\n\nRelease highlights\n------------------\n\nVersion 2.XX.X brings some exciting new features! HYPE TEXT HERE\n\nHIGHLIGHT 1\n^^^^^^^^^^^\nHYPE TEXT HERE \n\n* TALKING POINT 1\n* TALKING POINT 2\n* . . .\n\nUSE CASE DESCRIPTION HERE\n\nFor more details, see :doc:`DOC LINK `\n\nHIGHLIGHT 2\n^^^^^^^^^^^\nHYPE TEXT HERE \n\n* TALKING POINT 1\n* TALKING POINT 2\n* . . .\n\nUSE CASE DESCRIPTION HERE\n\nFor more details, see :doc:`DOC LINK `\n\nHIGHLIGHT 3\n^^^^^^^^^^^\nHYPE TEXT HERE \n\n* TALKING POINT 1\n* TALKING POINT 2\n* . . .\n\nUSE CASE DESCRIPTION HERE\n\nFor more details, see :doc:`DOC LINK `\n\nOther important changes\n^^^^^^^^^^^^^^^^^^^^^^^\n\nThis release also includes the following improvements\n\n* . . . LINK TO COMPONENT RELEASE NOTE PAGE\n* . . . LINK TO COMPONENT RELEASE NOTE PAGE\n* . . . LINK TO COMPONENT RELEASE NOTE PAGE\n* . . . LINK TO COMPONENT RELEASE NOTE PAGE\n\nComponent release notes\n-----------------------\n\nSelect a card below to review detailed release notes for each component of the Neuron SDK version 2.XX.X. These component release notes contain details on specific new and improved features, as well as breaking changes, bug fixes, and known issues for that component area of the Neuron SDK.\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-pytorch\n :link-type: ref\n\n **PyTorch support** 2.XX.0 release notes\n ^^^\n Neuron features and solutions that support the PyTorch ML framework.\n +++\n Supports: ``Inf2``, ``Trn1`` / ``Trn1n``, ``Trn2``\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-jax\n :link-type: ref\n\n **JAX support** 2.XX.0 release notes\n ^^^\n Neuron features and solutions that support the JAX ML framework.\n +++\n Supports: ``Inf2``, ``Trn1`` / ``Trn1n``, ``Trn2``\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-nxd-training\n :link-type: ref\n\n **NxD Training** 2.XX.0 release notes\n ^^^\n Neuron features and tools for LLM and agent ML model training.\n +++\n Supports: ``Trn1`` / ``Trn1n``, ``Trn2``\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-nxd-inference\n :link-type: ref\n\n **NxD Inference** 2.XX.0 release notes\n ^^^\n Neuron features and tools for LLM and agent ML model inference.\n +++\n Supports: ``Inf2``, ``Trn1`` / ``Trn1n``, ``Trn2``\n \n .. grid-item-card::\n :link: neuron-2-XX-0-nxd-core\n :link-type: ref\n\n **NxD Core** 2.XX.0 release notes\n ^^^\n Common features and tools for Neuron-based training and inference.\n +++\n Supports: ``Trn1`` / ``Trn1n``, ``Trn2``\n \n .. grid-item-card:: \n :link: neuron-2-XX-0-compiler\n :link-type: ref\n\n **Neuron Compiler** 2.XX.0 release notes\n ^^^\n The Neuron compiler for AWS Trainium and Inferentia, and its libraries and tools.\n +++\n Supports: ``Inf2``, ``Trn1`` / ``Trn1n``, ``Trn2``\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-nki\n :link-type: ref\n\n **Neuron Kernel Interface (NKI)** 2.XX.0 release notes\n ^^^\n Neuron's Python-based programming interface for developing and optimizing Neuron kernels.\n +++\n Supports: ``Inf2``, ``Trn1``, ``Trn1n``\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-runtime\n :link-type: ref\n\n **Neuron Runtime** 2.XX.0 release notes\n ^^^\n The Neuron kernel driver and C++ libraries for AWS Inferentia and Trainium instances.\n +++\n Supports: ``Inf1``, ``Inf2``, ``Trn1`` / ``Trn1n``\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-tools\n :link-type: ref\n\n **Neuron Developer Tools** 2.XX.0 release notes\n ^^^\n Tools that support end-to-end development for AWS Neuron.\n +++\n Supports: ``Inf1``, ``Inf2``, ``Trn1`` / ``Trn1n``\n\n\n .. grid-item-card:: \n :link: neuron-2-XX-0-dlami\n :link-type: ref\n\n **Neuron Deep Learning AWS Machine Images (DLAMIs)** 2.XX.0 release notes\n ^^^\n AWS-specific machine images for building and deploying Neuron-based ML solutions.\n +++\n Supports: ``Inf1``, ``Inf2``, ``Trn1`` / ``Trn1n``\n \n .. grid-item-card:: \n :link: neuron-2-XX-0-dlc\n :link-type: ref\n\n **Neuron Deep Learning Containers (DLCs)** 2.XX.0 release notes\n ^^^\n AWS-specific container definitions for building and deploying Neuron-based ML solutions.\n +++\n Supports: ``Inf1``, ``Inf2``, ``Trn1`` / ``Trn1n``\n\n .. grid-item-card::\n :link: latest-neuron-release-artifacts\n :link-type: ref\n \n **Neuron 2.XX.0 release artifacts**\n ^^^\n The libraries and packages updated in this release.\n\nSupport announcements\n---------------------\n\nThis section signals the official end-of-support or end of support for specific features, tools, and APIs.\n\nEnd-of-support announcements\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n*An \"end-of-support (EoS)\" announcement is a notification that a feature, tool, or API will not be supported in the future. Plan accordingly!*\n\n* END-OF-SUPPORT ANNOUNCEMENT 1 (link to announcement here)\n* . . .\n\nEnding support in 2.XX.X\n^^^^^^^^^^^^^^^^^^^^^^^^\n\n\"End of support\" means that AWS Neuron no longer supports the feature, tool, or API indicated in the note as of this release.\n\n* ENDING SUPPORT ANNOUNCEMENT 1 (link to announcement here)\n* . . .\n\nPrevious releases\n-----------------\n\n* :doc:`Neuron 2.27.0 `\n* :doc:`Neuron 2.26.0 `\n* :doc:`Neuron 2.25.0 `\n* :doc:`Earlier releases `\n\n* :ref:`prev-rn`\n* :ref:`pre-release-content`\n* :ref:`prev-n1-rn`\n" }, { "path": "_content-types/release-notes-templates/nki.rst", "content": ".. _neuron-2-XX-0-nki:\n\n.. meta::\n :description: The official release notes for the AWS Neuron Kernel Interface (NKI) component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.25.0: Neuron Kernel Interace (NKI) release notes\n=================================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.25.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.25.0:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/nx-jax.rst", "content": ".. _neuron-2-XX-0-jax:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK JAX support component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: JAX support release notes\n================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.25.0 release notes home `\n\nReleased versions\n-----------------\n* ``0.6.1.1.0.*``\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.25.0:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . .\n" }, { "path": "_content-types/release-notes-templates/nx-pytorch.rst", "content": ".. _neuron-2-XX-0-pytorch:\n\n.. meta::\n :description: The official release notes for AWS Neuron SDK PyTorch support, version X.XX.0. Release date: XX/XX/XXXX.\n\nAWS Neuron SDK X.XX.0: PyTorch support release notes\n====================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nReleased versions\n-----------------\n\n* ... \n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE WHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\nHere's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . .\n" }, { "path": "_content-types/release-notes-templates/nxd-core.rst", "content": ".. _neuron-2-XX-0-nxd-core:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK NxD Core component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: NxD Core release notes\n=============================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/nxd-inference.rst", "content": ".. _neuron-2-XX-0-nxd-inference:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK Transformers for Inference component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: NxD Inference release notes\n==================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n* \nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/nxd-training.rst", "content": ".. _neuron-2-XX-0-nxd-training:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK NxD Training component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.25.0: NxD Training release notes\n=================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.25.0:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/runtime.rst", "content": ".. _neuron-2-XX-0-runtime:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK Runtime component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: Neuron Runtime release notes\n===================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_content-types/release-notes-templates/tools.rst", "content": ".. _neuron-2-XX-0-tools:\n\n.. meta::\n :description: The official release notes for the AWS Neuron SDK Developer Tools component, version X.XX.0. Release date: XX/XX/2026.\n\nAWS Neuron SDK 2.XX.X: Developer Tools release notes\n====================================================\n\n**Date of release**: Month Day, 2026\n\n.. contents:: In this release\n :local:\n :depth: 1\n\n* Go back to the :ref:`AWS Neuron 2.XX.0 release notes home `\n\nImprovements\n------------\n\n*Improvements are significant new or improved features and solutions introduced this release of the AWS Neuron SDK. Read on to learn about them!*\n\nFeature 1\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 2\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nFeature 3\n^^^^^^^^^\n\nUSER-FACING DESCRIPTION OF IMPROVEMENT (WHAT WILL IT DO FOR DEV CUSTOMERS), WHY WE MADE THE IMPROVEMENT, LINK TO SUPPORTING DOC PAGE\n\nBehavioral changes\n------------------\n\n*Behavioral changes are small, user-facing changes that you may notice after upgrading to this version.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HOW THE USER MAY EXPERIENCE IT, IF APPLICABLE.\n* . . .\n\nBreaking changes\n----------------\n\n*Sometimes we have to break something now to make the experience better in the longer term. Breaking changes are changes that may require you to update your own code, tools, and configurations.*\n\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* CHANGE DESCRIPTION SENTENCE. NOTE HWHEN THE USER MAY ENCOUNTER IT. PROVIDE A WORKAROUND, IF POSSIBLE.\n* . . .\n\nBug fixes\n---------\n\n Here's what we fixed in 2.XX.X:\n\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* SHORT SENTENCE DESCRIBING BUG FIX.\n* . . .\n\nKnown issues\n------------\n\n*Something doesn't work. Check here to find out if we already knew about it. We hope to fix these soon!*\n\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* SENTENCE DESCRIBING ISSUE AND WHEN THE USER WILL ENCOUNTER IT.\n* . . ." }, { "path": "_ext/archive.py", "content": "# This file creates a downloadable archive from each directory listed in src_dirs.\n# You can modify or add additional archive_handler functions here to create additional archives.\n\nimport os, tarfile\n\ndef archive_handler(app):\n old_cwd = os.getcwd()\n src_dirs = ['src/examples/pytorch', 'src']\n target_dirs = ['libtorch_demo', 'neuronperf']\n archive_names = [name + '.tar.gz' for name in target_dirs]\n\n for src_dir, target_dir, archive_name in zip(src_dirs, target_dirs, archive_names):\n os.chdir(src_dir)\n\n try:\n os.remove(archive_name)\n except OSError:\n pass\n\n with tarfile.open(archive_name, 'w:gz') as tar:\n tar.add(target_dir)\n\n os.chdir(old_cwd)\n\ndef setup(app):\n app.connect('builder-inited', archive_handler)\n\n return {\n 'version': '1.0',\n 'parallel_read_safe': True,\n 'parallel_write_safe': True,\n }\n" }, { "path": "_ext/df_tables.py", "content": "import os\nfrom docutils.parsers.rst import Directive, directives\nfrom docutils.parsers.rst.directives.tables import CSVTable\n\nclass DFTable(CSVTable):\n CSVTable.option_spec['df-arg'] = directives.unchanged\n df = None\n\n def __init__(self, name, arguments, options, content, lineno,\n content_offset, block_text, state, state_machine):\n\n super().__init__(name, arguments, options, content, lineno,\n content_offset, block_text, state, state_machine)\n\n def get_csv_data(self):\n return self.df.to_csv(index=False).splitlines(), None\n\n def run(self):\n source_file_name = self.state_machine.document.attributes[\"source\"]\n dirname = os.path.abspath(os.path.dirname(source_file_name))\n os.chdir(dirname)\n\n code = \"\\n\".join(map(str, self.content))\n ns = {}\n\n try:\n exec(\"\\n\".join( [\"import numpy as np\", \"import pandas as pd\", ]), ns)\n\n variable_name = \"df\"\n if self.options.get(\"df-var\"):\n variable_name = self.options.get(\"df-var\")\n\n exec(code, ns)\n self.df = ns[variable_name]\n \n except Exception as e:\n raise self.error(str(e))\n\n return super().run()\n\n \ndef setup(app):\n setup.app = app\n setup.config = app.config\n setup.confdir = app.confdir\n app.add_directive(\"df-table\", DFTable)\n\n metadata = {\n \"parallel_read_safe\": True,\n \"parallel_write_safe\": True,\n \"version\": 0.1,\n }\n return metadata" }, { "path": "_ext/local_documenter.py", "content": "import os\nimport sys\n\nfrom sphinx.ext.autodoc import ModuleDocumenter, FunctionDocumenter\n\n\nclass LocalModuleDocumenter(ModuleDocumenter):\n \"\"\"\n Provides identical functionality to \"automodule\", but allows the module\n function names to be overridden with the \"module-name\" option.\n\n This also allows local python files to be documented as if they were\n imported from an actual package by temporarily adding the directory of the\n RST file to the python path.\n \"\"\"\n option_spec = dict(ModuleDocumenter.option_spec)\n option_spec['module-name'] = lambda x = None: x\n\n def import_object(self, *args):\n \"\"\"Find modules local to the RST document directory\"\"\"\n local = os.path.join(self.env.app.srcdir, os.path.dirname(self.env.docname))\n sys.path.append(local)\n result = super().import_object(*args)\n sys.path.remove(local)\n return result\n\n def get_module_members(self):\n \"\"\"Add module name override to local files\"\"\"\n members = super().get_module_members()\n name = self.options.module_name\n if name is not None:\n for member in members.values():\n if callable(member.object):\n setattr(member.object, 'module_name_override', name)\n return members\n\n\nclass LocalFunctionDocumenter(FunctionDocumenter):\n def format_name(self) -> str:\n \"\"\"Apply module name override to local functions\"\"\"\n # Use overridden module path if it is provided\n if hasattr(self.object, 'module_name_override'):\n self.objpath = self.object.module_name_override.split('.') + [self.objpath[-1]]\n return super().format_name()\n\n\ndef setup(app):\n app.add_autodocumenter(LocalFunctionDocumenter)\n app.add_autodocumenter(LocalModuleDocumenter)\n" }, { "path": "_ext/neuron_tag.py", "content": "import os\n\nfrom docutils import nodes\nfrom docutils.statemachine import ViewList\n\nfrom sphinx.util.docutils import SphinxDirective\nfrom sphinx.util.nodes import nested_parse_with_titles\n\n\n# =============================================================================\n# Legacy add/clear lists (used only for files NOT handled by explicit overrides)\n# =============================================================================\n\n# These lists use substring matching via in_list(). They apply ONLY when no\n# explicit_override was set. As more paths get explicit overrides, entries\n# here become dead code. Kept for backward compatibility with paths not yet\n# explicitly overridden.\n\nadd_inf1_tag = [\n 'about-neuron/arch',\n 'archive/mxnet-neuron',\n 'about-neuron/announcements/index',\n 'archive/tensorflow/tensorflow-neuron/',\n]\n\nadd_trn1_tag = [\n 'frameworks/neuron-customops/',\n 'neuron-customops/',\n 'frameworks/torch/inference-torch-neuronx',\n 'libraries/nemo-megatron/',\n 'libraries/nxd-training/',\n]\n\nadd_trn2_tag = [\n 'libraries/nxd-training/',\n 'about-neuron/models/',\n]\n\nadd_trn3_tag = [\n 'about-neuron/arch/neuron-hardware/neuron-core-v4',\n 'about-neuron/arch/neuron-hardware/trn3-arch',\n]\n\nadd_neuronx_tag = [\n 'frameworks/torch/torch-neuronx/',\n 'archive/tensorflow/tensorflow-neuronx/',\n 'frameworks/torch/inference-torch-neuronx/',\n 'libraries/neuronx-distributed/',\n 'libraries/nxd-training',\n 'setup/tensorflow-neuronx',\n]\n\nclear_inf1_tag = [\n 'about-neuron/arch/neuron-features/neuron-caching',\n 'about-neuron/arch/neuron-features/eager-debug-mode',\n 'about-neuron/arch/neuron-features/collective-communication-operations',\n 'about-neuron/arch/neuron-features/dynamic-shapes',\n 'about-neuron/arch/neuron-features/control-flow',\n 'about-neuron/arch/neuron-features/custom-c++-operators',\n 'about-neuron/arch/neuron-features/collective-communication',\n 'about-neuron/arch/neuron-features/rounding-modes',\n 'about-neuron/arch/neuron-hardware/trn1-arch',\n 'about-neuron/arch/neuron-hardware/inf2-arch',\n 'about-neuron/arch/neuron-hardware/inferentia2',\n 'about-neuron/arch/neuron-hardware/trainium',\n 'about-neuron/arch/neuron-hardware/neuron-core-v2',\n 'about-neuron/arch/neuron-hardware/trn2-arch',\n 'about-neuron/arch/neuron-hardware/trn3-arch',\n 'about-neuron/arch/neuron-hardware/neuron-core-v3',\n 'about-neuron/arch/neuron-hardware/neuron-core-v4',\n 'about-neuron/benchmarks/trn1-performance',\n 'about-neuron/benchmarks/trn1/',\n 'about-neuron/benchmarks/inf2/inf2-performance',\n 'about-neuron/faq/training/',\n 'about-neuron/models/inference-inf2-trn1-samples',\n 'about-neuron/models/training-trn1-samples',\n 'about-neuron/models/training-inference-trn2-samples',\n 'about-neuron/appnotes/neuronx-cc/neuronx-cc-training-mixed-precision',\n 'about-neuron/appnotes/transformers-neuronx/generative-llm-inference-with-neuron',\n 'about-neuron/appnotes/torch-neuronx/torch-neuronx-dataparallel-app-note',\n 'about-neuron/calculator/neuron-calculator',\n 'about-neuron/announcements/neuron2.x/dlami-pytorch-introduce',\n 'about-neuron/announcements/neuron2.x/sm-training-trn1-introduce',\n 'about-neuron/announcements/neuron2.x/sm-training-dlc-2.9.1',\n 'devflows/training',\n 'devflows/inference/byoc-hosting-devflow-inf2',\n 'compiler/neuronx-cc/',\n 'about-neuron/appnotes/perf/neuronx-cc/',\n 'frameworks/torch/torch-neuronx/',\n 'frameworks/torch/training',\n 'frameworks/torch/inference-torch-neuronx',\n 'archive/tensorflow/tensorflow-neuronx/',\n 'archive/tensorflow/tensorflow-neuronx-inference',\n 'frameworks/torch/torch-neuronx/transformers-neuronx/readme',\n 'release-notes/neuron-cc/index',\n 'release-notes/runtime/aws-neuronx-collectives/',\n 'release-notes/torch/torch-neuronx/',\n 'release-notes/torch/transformers-neuronx/index',\n 'release-notes/tensorflow/tensorflow-neuronx/',\n 'release-notes/compiler/neuronx-cc/',\n 'tools/tutorials/tutorial-tensorboard-scalars-mnist',\n 'tools/tutorials/tutorial-neuron-monitor-mnist',\n 'tools/tensorboard/getting-started-tensorboard-neuronx-plugin',\n 'tools/neuron-sys-tools/nccom-test',\n 'setup/torch-neuronx',\n 'setup/tensorflow-neuronx',\n 'setup/neuron-setup/tensorflow/neuronx/',\n 'setup/neuron-setup/pytorch/neuronx/',\n 'nki/',\n 'frameworks/jax/',\n 'libraries/nxd-training/',\n '/release-notes/components/nki',\n '/release-notes/components/nki-lib',\n '/release-notes/components/compiler'\n]\n\nclear_inf2_tag = [\n 'frameworks/torch/torch-neuronx/training',\n 'frameworks/torch/training',\n 'archive/torch-neuron/inference-torch-neuron',\n 'archive/tensorflow/tensorflow-neuron-inference',\n 'frameworks/jax/',\n 'about-neuron/arch/neuron-hardware/trn1-arch',\n 'about-neuron/arch/neuron-hardware/trainium',\n 'about-neuron/arch/neuron-hardware/trn2-arch',\n 'about-neuron/arch/neuron-hardware/trn3-arch',\n 'about-neuron/arch/neuron-hardware/neuron-core-v3',\n 'about-neuron/arch/neuron-hardware/neuron-core-v4',\n 'about-neuron/arch/neuron-features/logical-neuroncore-config',\n 'about-neuron/benchmarks/trn1/trn1-inference-performance',\n 'about-neuron/benchmarks/trn1/trn1-training-performance',\n 'about-neuron/models/training-trn1-samples',\n 'about-neuron/models/training-inference-trn2-samples',\n 'about-neuron/announcements/neuron2.x/announce-neuron-trn2',\n 'neuronx-distributed/nxd-training',\n 'libraries/nxd-training/',\n 'tools/neuron-sys-tools/nccom-test',\n 'release-notes/runtime/aws-neuronx-collectives/',\n]\n\nclear_trn1_tag = [\n 'about-neuron/arch/neuron-hardware/inf2-arch',\n 'about-neuron/arch/neuron-hardware/inferentia2',\n 'about-neuron/arch/neuron-hardware/trn2-arch',\n 'about-neuron/arch/neuron-hardware/trn3-arch',\n 'about-neuron/arch/neuron-hardware/trainium2',\n 'about-neuron/arch/neuron-hardware/neuron-core-v3',\n 'about-neuron/arch/neuron-hardware/neuron-core-v4',\n 'about-neuron/benchmarks/inf2/inf2-performance',\n 'about-neuron/models/training-inference-trn2-samples',\n]\n\nclear_trn2_tag = [\n 'archive/tensorflow/',\n 'libraries/transformers-neuronx/',\n 'about-neuron/arch/neuron-hardware/trn1-arch',\n 'about-neuron/arch/neuron-hardware/trainium',\n 'about-neuron/arch/neuron-hardware/neuron-core-v2',\n 'about-neuron/arch/neuron-hardware/neuron-core-v4',\n 'about-neuron/arch/neuron-hardware/trn3-arch',\n 'about-neuron/benchmarks/',\n 'about-neuron/benchmarks/trn1/',\n 'about-neuron/benchmarks/inf2/inf2-performance',\n 'about-neuron/models/inference-inf2-trn1-samples',\n 'about-neuron/models/training-trn1-samples',\n 'neuron-customops/programming-guide/custom-c++-operators-devguide'\n]\n\nclear_trn3_tag = [\n 'archive/tensorflow/',\n 'libraries/transformers-neuronx/',\n 'about-neuron/arch/neuron-hardware/trn1-arch',\n 'about-neuron/arch/neuron-hardware/trainium',\n 'about-neuron/arch/neuron-hardware/neuron-core-v2',\n 'about-neuron/arch/neuron-hardware/neuron-core-v3',\n 'about-neuron/benchmarks/',\n 'about-neuron/benchmarks/trn1/',\n 'about-neuron/benchmarks/inf2/inf2-performance',\n 'about-neuron/models/inference-inf2-trn1-samples',\n 'about-neuron/models/training-trn1-samples',\n 'libraries/neuronx-distributed/context_parallelism_overview',\n 'about-neuron/appnotes/',\n 'neuron-customops/programming-guide/custom-c++-operators-devguide'\n]\n\n# Neuron 1.x / NeuronCore v1 era content — clear all non-Inf1 tags\nclear_nc_v2_tag = [\n 'tools/tutorials/tutorial-neuron-check-model',\n 'tools/tutorials/tutorial-neuron-gatherinfo',\n 'tools/tutorials/getting-started-tensorboard-neuron-plugin',\n 'tools/tensorboard/getting-started-tensorboard-neuron-plugin',\n 'tools/helper-tools/tutorial-neuron-check-model',\n 'tools/helper-tools/tutorial-neuron-gatherinfo',\n 'about-neuron/appnotes/neuron-cc/mixed-precision',\n 'about-neuron/appnotes/perf/neuron-cc/',\n 'about-neuron/appnotes/neuron1x/',\n 'about-neuron/appnotes/torch-neuron/',\n 'about-neuron/arch/neuron-hardware/inf1-arch',\n 'about-neuron/arch/neuron-hardware/inferentia',\n 'about-neuron/arch/neuron-hardware/neuron-core-v1',\n 'about-neuron/arch/neuron-features/neuroncore-pipeline',\n 'about-neuron/announcements/neuron1.x/',\n 'about-neuron/quick-start/mxnet-neuron',\n 'about-neuron/benchmarks/inf1/',\n 'about-neuron/faq/inference/',\n 'about-neuron/models/inference-inf1-samples',\n 'containers/dlc-then-ec2-devflow',\n 'containers/dlc-then-ecs-devflow',\n 'containers/dlc-then-eks-devflow',\n 'containers/container-sm-hosting-devflow',\n 'containers/rn',\n 'containers/tutorials/k8s-neuron-scheduler',\n 'compiler/neuron-cc/',\n 'release-notes/mxnet-neuron/',\n 'release-notes/torch/torch-neuron/',\n 'release-notes/tensorflow/tensorflow-neuron/',\n 'release-notes/compiler/neuron-cc/',\n 'release-notes/neuron1/',\n 'archive/torch-neuron/',\n 'archive/torch-neuron/inference-torch-neuron',\n 'archive/tensorflow/tensorflow-neuron/',\n 'archive/tensorflow/tensorflow-neuron-inference',\n 'archive/mxnet-neuron/',\n 'setup/tensorflow-neuron',\n 'setup/torch-neuron',\n 'setup/mxnet-neuron',\n 'setup/neuron-setup/pytorch/neuron/',\n 'setup/neuron-setup/mxnet/neuron/ubuntu/',\n 'setup/neuron-setup/mxnet/neuron/amazon-linux/',\n 'setup/neuron-setup/tensorflow/neuron/ubuntu/',\n 'setup/neuron-setup/tensorflow/neuron/amazon-linux/',\n]\n\n# Top-level directories used for initial tag assignment\nNEURON1_DIRS = ['n1']\nCOMMON_DIRS = [\n 'tools', 'neuron-runtime', 'release-notes', 'containers', 'compiler',\n 'frameworks', 'src', 'about-neuron', 'setup', 'devflows', 'dlami', 'libraries',\n]\n\nTEXT_TEMPLATE = '**This document is relevant for**: '\n\n\n# =============================================================================\n# Hardware architecture page map (exact docname → instance list)\n# =============================================================================\n\nHW_ARCH_MAP = {\n 'about-neuron/arch/neuron-hardware/inf1-arch': ['Inf1'],\n 'about-neuron/arch/neuron-hardware/inf2-arch': ['Inf2'],\n 'about-neuron/arch/neuron-hardware/inferentia': ['Inf1'],\n 'about-neuron/arch/neuron-hardware/inferentia2': ['Inf2'],\n 'about-neuron/arch/neuron-hardware/neuron-core-v1': ['Inf1'],\n 'about-neuron/arch/neuron-hardware/neuron-core-v2': ['Inf2', 'Trn1'],\n 'about-neuron/arch/neuron-hardware/neuron-core-v3': ['Trn2'],\n 'about-neuron/arch/neuron-hardware/neuron-core-v4': ['Trn3'],\n 'about-neuron/arch/neuron-hardware/trainium': ['Trn1'],\n 'about-neuron/arch/neuron-hardware/trainium2': ['Trn2'],\n 'about-neuron/arch/neuron-hardware/trainium3': ['Trn3'],\n 'about-neuron/arch/neuron-hardware/trn1-arch': ['Trn1'],\n 'about-neuron/arch/neuron-hardware/trn2-arch': ['Trn2'],\n 'about-neuron/arch/neuron-hardware/trn3-arch': ['Trn3'],\n}\n\n# NxD Core training-specific pages (no Inf2)\nNXD_CORE_TRAINING_PAGES = [\n 'libraries/neuronx-distributed/index-training',\n 'libraries/neuronx-distributed/developer-guide-training',\n 'libraries/neuronx-distributed/api-reference-guide-training',\n 'libraries/neuronx-distributed/tp_developer_guide',\n 'libraries/neuronx-distributed/pp_developer_guide',\n 'libraries/neuronx-distributed/ptl_developer_guide',\n 'libraries/neuronx-distributed/save_load_developer_guide',\n 'libraries/neuronx-distributed/activation_memory_reduction',\n 'libraries/neuronx-distributed/activation_memory_reduction_developer_guide',\n 'libraries/neuronx-distributed/standard_mixed_precision',\n 'libraries/neuronx-distributed/tensor_parallelism_overview',\n 'libraries/neuronx-distributed/pipeline_parallelism_overview',\n 'libraries/neuronx-distributed/lora_finetune_developer_guide',\n 'libraries/neuronx-distributed/model_optimizer_wrapper_developer_guide',\n 'libraries/neuronx-distributed/context_parallelism_overview',\n]\n\n\ndef _in_list(cur_file, file_list):\n \"\"\"Return True if any entry in file_list is a substring of cur_file.\"\"\"\n return any(entry in cur_file for entry in file_list)\n\n\ndef _splitall(path):\n \"\"\"Split a path into all its components.\"\"\"\n parts = []\n while True:\n head, tail = os.path.split(path)\n if head == path:\n parts.insert(0, head)\n break\n elif tail == path:\n parts.insert(0, tail)\n break\n else:\n path = head\n parts.insert(0, tail)\n return parts, len(parts)\n\n\ndef _get_explicit_override(cur_file):\n \"\"\"Return (instances, True) if cur_file has an explicit CSV-based override,\n or (None, False) otherwise.\n\n Rules are evaluated top-to-bottom. More specific paths must come AFTER\n broader paths so they can override them (last match wins).\n \"\"\"\n\n # --- Libraries -----------------------------------------------------------\n\n # NxD Core = Inf2, Trn1, Trn2 (default for all neuronx-distributed pages)\n if cur_file.startswith('libraries/neuronx-distributed/'):\n result = ['Inf2', 'Trn1', 'Trn2']\n # Training-specific pages drop Inf2\n if cur_file in NXD_CORE_TRAINING_PAGES:\n result = ['Trn1', 'Trn2']\n if cur_file.startswith('libraries/neuronx-distributed/tutorials/training') or \\\n cur_file.startswith('libraries/neuronx-distributed/tutorials/finetune'):\n result = ['Trn1', 'Trn2']\n return result, True\n\n if cur_file.startswith('libraries/transformers-neuronx/'):\n return ['Inf2', 'Trn1'], True\n\n if cur_file.startswith('libraries/nxd-training/'):\n return ['Trn1', 'Trn2'], True\n\n # vLLM must come before general nxd-inference\n if cur_file.startswith('libraries/nxd-inference/vllm/'):\n return ['Trn2', 'Trn3'], True\n\n if cur_file.startswith('libraries/nxd-inference/'):\n return ['Inf2', 'Trn1', 'Trn2'], True\n\n if cur_file.startswith('libraries/nemo-megatron/'):\n return ['Trn1', 'Trn2'], True\n\n # --- NKI -----------------------------------------------------------------\n\n if cur_file.startswith('nki/'):\n return ['Trn2', 'Trn3'], True\n\n # --- CustomOps -----------------------------------------------------------\n\n if cur_file.startswith('neuron-customops/'):\n return ['Inf2', 'Trn1'], True\n\n # --- Frameworks ----------------------------------------------------------\n\n if cur_file.startswith('frameworks/jax/'):\n return ['Trn2', 'Trn3'], True\n\n # TensorFlow NeuronX (must come before TensorFlow Neuron check)\n if 'tensorflow/tensorflow-neuronx' in cur_file:\n return ['Inf2', 'Trn1'], True\n\n # TensorFlow Neuron (Inf1)\n if 'tensorflow/tensorflow-neuron' in cur_file and 'neuronx' not in cur_file:\n return ['Inf1'], True\n\n # TorchNeuron native PyTorch (must come before torch-neuronx check)\n if 'torch/pytorch-native' in cur_file:\n return ['Trn2', 'Trn3'], True\n\n # PyTorch NeuronX (Torch/XLA)\n if 'torch/torch-neuronx' in cur_file:\n return ['Inf2', 'Trn1', 'Trn2'], True\n\n # PyTorch NeuronX top-level pages (not in torch-neuronx/ subdir)\n if cur_file in ['frameworks/torch/inference-torch-neuronx',\n 'frameworks/torch/training-torch-neuronx',\n 'frameworks/torch/training',\n 'frameworks/torch/inference']:\n return ['Inf2', 'Trn1', 'Trn2'], True\n\n # PyTorch Neuron (Inf1)\n if 'torch/torch-neuron' in cur_file and 'neuronx' not in cur_file:\n return ['Inf1'], True\n\n if cur_file == 'archive/torch-neuron/inference-torch-neuron':\n return ['Inf1'], True\n\n # MXNet\n if 'mxnet-neuron' in cur_file:\n return ['Inf1'], True\n\n # --- Neuron Runtime ------------------------------------------------------\n\n # Collectives (more specific, must come after general runtime)\n if cur_file.startswith('neuron-runtime/about/collectives') or \\\n cur_file in ['neuron-runtime/explore/internode-collective-comm',\n 'neuron-runtime/explore/intranode-collective-comm',\n 'neuron-runtime/explore/compute-comm-overlap']:\n return ['Trn1', 'Trn2', 'Trn3'], True\n\n if cur_file.startswith('neuron-runtime/'):\n return ['Inf2', 'Trn1', 'Trn2', 'Trn3'], True\n\n # --- Compiler ------------------------------------------------------------\n\n if cur_file.startswith('compiler/error-codes/'):\n return ['Inf2', 'Trn1', 'Trn2', 'Trn3'], True\n\n if cur_file == 'compiler/neuron-cc' or cur_file.startswith('compiler/neuron-cc/'):\n return ['Inf1'], True\n\n if cur_file == 'compiler/neuronx-cc' or cur_file.startswith('compiler/neuronx-cc/'):\n return ['Inf2', 'Trn1', 'Trn2', 'Trn3'], True\n\n if cur_file == 'neuron-customops/programming-guide' or cur_file.startswith('neuron-customops/programming-guide'):\n return ['Inf2', 'Trn1'], True\n\n # --- Setup ---------------------------------------------------------------\n\n if cur_file.startswith('setup/install-templates/inf1/'):\n return ['Inf1'], True\n if cur_file.startswith('setup/install-templates/inf2/'):\n return ['Inf2'], True\n if cur_file.startswith('setup/install-templates/trn1/') or \\\n cur_file == 'setup/install-templates/launch-trn1-dlami':\n return ['Trn1'], True\n\n if cur_file in ['setup/setup-neuron', 'setup/torch-neuron', 'setup/torch-neuron-ubuntu20']:\n return ['Inf1'], True\n\n if cur_file.startswith('setup/neuron-setup/pytorch/neuronx/'):\n return ['Inf2', 'Trn1', 'Trn2'], True\n if cur_file.startswith('setup/neuron-setup/tensorflow/neuronx/'):\n return ['Inf2', 'Trn1'], True\n if cur_file.startswith('setup/neuron-setup/pytorch/neuron/'):\n return ['Inf1'], True\n if cur_file.startswith('setup/neuron-setup/tensorflow/neuron/'):\n return ['Inf1'], True\n\n if cur_file == 'setup/jax-neuronx':\n return ['Trn2', 'Trn3'], True\n if cur_file == 'setup/torch-neuronx':\n return ['Inf2', 'Trn1', 'Trn2'], True\n if cur_file == 'setup/tensorflow-neuronx':\n return ['Inf2', 'Trn1'], True\n if cur_file == 'setup/tensorflow-neuron':\n return ['Inf1'], True\n\n return None, False\n\n\ndef _get_page_override(cur_file):\n \"\"\"Return (instances, True) for page-specific overrides that don't fit\n neatly into _get_explicit_override (devflows, containers, tools, about-neuron, etc.).\n \"\"\"\n\n # --- Devflows ------------------------------------------------------------\n\n if cur_file == 'devflows/inference/byoc-hosting-devflow-inf2':\n return ['Inf2'], True\n if cur_file == 'devflows/inference/ec2-then-ec2-devflow-inf2':\n return ['Inf2'], True\n if cur_file == 'devflows/parallelcluster-flows':\n return ['Trn1', 'Trn2'], True\n\n if cur_file.startswith('devflows/training/batch/') or \\\n cur_file.startswith('devflows/training/ec2/') or \\\n cur_file.startswith('devflows/training/parallelcluster/') or \\\n cur_file.startswith('devflows/training/sm-devflow/'):\n return ['Trn1', 'Trn2', 'Trn3'], True\n\n if cur_file.startswith('devflows/plugins/npd'):\n return ['Inf2', 'Trn1', 'Trn2'], True\n\n # --- Containers ----------------------------------------------------------\n\n # OCI Hooks\n if 'tutorial-oci-hook' in cur_file:\n return ['Inf1', 'Inf2', 'Trn1', 'Trn2'], True\n\n # DRA\n if cur_file == 'containers/neuron-dra' or cur_file.startswith('containers/files/'):\n return ['Trn2', 'Trn3'], True\n\n if cur_file == 'containers/how-to/how-to-ultraserver':\n return ['Trn2', 'Trn3'], True\n\n # DLC quickstarts\n if cur_file == 'containers/get-started/quickstart-configure-deploy-dlc':\n return ['Trn2', 'Trn3'], True\n if cur_file == 'containers/get-started/quickstart-pytorch-inference-dlc':\n return ['Inf2', 'Trn1', 'Trn2', 'Trn3'], True\n\n # Inf1-era container content\n if cur_file == 'containers/tutorial-docker-runtime1.0':\n return ['Inf1'], True\n if cur_file == 'containers/container-deployment-flows' or \\\n cur_file.startswith('containers/docker-example/inference/') or \\\n cur_file.startswith('containers/docker-example/v1/') or \\\n cur_file == 'containers/ec2-then-ec2-devflow' or \\\n cur_file == 'containers/neo-then-hosting-devflow':\n return ['Inf1'], True\n\n # Container training/inference tutorials and docker examples\n if cur_file.startswith('containers/docker-example/training/'):\n return ['Trn1', 'Trn2', 'Trn3'], True\n if cur_file.startswith('containers/tutorials/inference/'):\n return ['Inf1'], True\n if cur_file.startswith('containers/tutorials/training/'):\n return ['Trn1', 'Trn2', 'Trn3'], True\n\n # Neuron Monitor Container\n if cur_file == 'containers/tutorials/k8s-neuron-monitor':\n return ['Inf2', 'Trn1', 'Trn2'], True\n\n # Node Problem Detector\n if cur_file.startswith('containers/tutorials/k8s-neuron-problem-detector'):\n return ['Inf2', 'Trn1', 'Trn2'], True\n\n # --- Tools ---------------------------------------------------------------\n\n # TensorBoard plugin (End Of Support)\n if cur_file.startswith('tools/tensorboard/getting-started-tensorboard-neuronx') or \\\n cur_file == 'tools/tutorials/tutorial-tensorboard-scalars-mnist' or \\\n cur_file == 'tools/tutorials/torch-neuronx-profiling-with-tb':\n return ['Inf2', 'Trn1'], True\n\n # --- Announcements -------------------------------------------------------\n\n if cur_file.startswith('about-neuron/announcements/'):\n return [], True\n\n # --- Hardware architecture -----------------------------------------------\n\n if cur_file in HW_ARCH_MAP:\n return HW_ARCH_MAP[cur_file], True\n\n # --- Arch features -------------------------------------------------------\n\n if cur_file == 'about-neuron/arch/neuron-features/custom-c++-operators':\n return ['Inf2', 'Trn1'], True\n if cur_file == 'about-neuron/arch/neuron-features/logical-neuroncore-config':\n return ['Trn2', 'Trn3'], True\n\n # --- Appnotes ------------------------------------------------------------\n\n if cur_file == 'about-neuron/appnotes/neuronx-distributed/introducing-nxd-inference':\n return ['Inf2', 'Trn1', 'Trn2'], True\n if cur_file == 'about-neuron/appnotes/neuronx-distributed/introducing-nxdt-training':\n return ['Trn1', 'Trn2'], True\n if cur_file.startswith('about-neuron/appnotes/torch-neuronx/'):\n return ['Inf2', 'Trn1', 'Trn2'], True\n if cur_file.startswith('about-neuron/appnotes/transformers-neuronx/'):\n return ['Inf2', 'Trn1'], True\n if cur_file == 'about-neuron/appnotes/neuronx-cc/neuronx-cc-training-mixed-precision':\n return ['Trn1', 'Trn2', 'Trn3'], True\n if cur_file.startswith('about-neuron/appnotes/neuron1x/'):\n return ['Inf1'], True\n\n # --- Benchmarks ----------------------------------------------------------\n\n if cur_file == 'about-neuron/benchmarks/index':\n return ['Inf1', 'Inf2', 'Trn1', 'Trn2', 'Trn3'], True\n\n # --- Quick-start ---------------------------------------------------------\n\n if cur_file == 'about-neuron/quick-start/tensorflow-neuron':\n return ['Inf1'], True\n if cur_file in ['about-neuron/quick-start/torch-neuron',\n 'about-neuron/quick-start/torch-neuron-tab-training']:\n return ['Inf1'], True\n\n if cur_file.startswith('about-neuron/quick-start/tab-inference-torch-neuronx'):\n return ['Inf2', 'Trn1', 'Trn2'], True\n if cur_file.startswith('about-neuron/quick-start/tab-inference-torch-neuron') and 'neuronx' not in cur_file:\n return ['Inf1'], True\n if cur_file.startswith('about-neuron/quick-start/tab-inference-tensorflow-neuronx'):\n return ['Inf2', 'Trn1'], True\n if cur_file.startswith('about-neuron/quick-start/tab-inference-tensorflow-neuron') and 'neuronx' not in cur_file:\n return ['Inf1'], True\n\n return None, False\n\n\nclass NeuronTag(SphinxDirective):\n\n def run(self):\n cur_file = self.env.docname\n path_split, path_len = _splitall(cur_file)\n\n # Landing page gets no tag\n if path_split[0] == 'index':\n return self._render('')\n\n # Step 1: Assign default instances based on top-level directory\n return_instances = []\n if path_split[0] in NEURON1_DIRS:\n return_instances = ['Inf1']\n elif path_split[0] in COMMON_DIRS:\n return_instances = ['Inf1', 'Inf2', 'Trn1', 'Trn2', 'Trn3']\n\n # Step 2: Check explicit overrides (CSV-based, highest priority)\n explicit_override = False\n\n result, matched = _get_explicit_override(cur_file)\n if matched:\n return_instances = result\n explicit_override = True\n\n if not explicit_override:\n result, matched = _get_page_override(cur_file)\n if matched:\n return_instances = result\n explicit_override = True\n\n # Step 3: Directory-based inference/training heuristic\n if not explicit_override:\n if path_len >= 2:\n parent_dir = path_split[path_len - 2]\n if parent_dir == 'inference':\n return_instances = ['Inf1']\n elif parent_dir == 'training':\n return_instances = ['Trn1', 'Trn2', 'Trn3']\n\n # Step 4: Legacy add/clear tag lists (only for non-overridden files)\n if not explicit_override:\n if _in_list(cur_file, add_trn1_tag):\n if 'Trn1' not in return_instances:\n return_instances.extend(['Trn1', 'Trn2', 'Trn3', 'Inf2'])\n\n if _in_list(cur_file, add_trn2_tag):\n if 'Trn2' not in return_instances:\n return_instances.extend(['Trn2', 'Trn3'])\n\n if _in_list(cur_file, add_trn3_tag):\n if 'Trn3' not in return_instances:\n return_instances.append('Trn3')\n\n if _in_list(cur_file, add_neuronx_tag):\n if 'Trn1' not in return_instances:\n return_instances.extend(['Trn1', 'Trn2', 'Trn3', 'Inf2'])\n\n if _in_list(cur_file, add_inf1_tag):\n if 'Inf1' not in return_instances:\n return_instances.append('Inf1')\n\n if _in_list(cur_file, clear_nc_v2_tag):\n for tag in ['Trn1', 'Trn2', 'Trn3', 'Inf2']:\n if tag in return_instances:\n return_instances.remove(tag)\n\n if _in_list(cur_file, clear_trn1_tag):\n if 'Trn1' in return_instances:\n return_instances.remove('Trn1')\n\n if _in_list(cur_file, clear_trn2_tag):\n if 'Trn2' in return_instances:\n return_instances.remove('Trn2')\n\n if _in_list(cur_file, clear_trn3_tag):\n if 'Trn3' in return_instances:\n return_instances.remove('Trn3')\n\n if _in_list(cur_file, clear_inf1_tag):\n if 'Inf1' in return_instances:\n return_instances.remove('Inf1')\n\n if _in_list(cur_file, clear_inf2_tag):\n if 'Inf2' in return_instances:\n return_instances.remove('Inf2')\n\n # Step 5: Generate output\n return_instances = sorted(set(return_instances))\n if return_instances:\n text = TEXT_TEMPLATE + ', '.join('``' + i + '``' for i in return_instances)\n else:\n text = ''\n\n return self._render(text)\n\n def _render(self, text):\n \"\"\"Parse RST text and return docutils nodes.\"\"\"\n rst = ViewList()\n rst.append(text, \"neuron-tag\", 1)\n node = nodes.section()\n node.document = self.state.document\n nested_parse_with_titles(self.state, rst, node)\n return node.children\n\n\ndef setup(app):\n app.add_directive(\"neuron-tag\", NeuronTag)\n return {\n 'version': '0.2',\n 'parallel_read_safe': True,\n 'parallel_write_safe': True,\n }\n" }, { "path": "_ext/release-notes-automation-spec.md", "content": "# Release Notes Review Automation Specification\n\n## Overview\n\nThis specification defines a GitHub Action that automatically reviews release notes files in pull requests using Amazon Q CLI to ensure they meet quality standards defined in the release notes writing guidelines.\n\n## Purpose\n\nAutomate the review of release notes changes to:\n- Ensure consistency and quality across all release notes\n- Catch common issues before human review\n- Provide immediate feedback to PR authors\n- Reduce manual review burden on documentation team\n\n## Scope\n\n### In Scope\n- PRs labeled with \"release-notes\"\n- RST files under `/release-notes/components/` directory\n- Files that have been modified in the PR (not just added to context)\n- Automated review using Q CLI with release notes guidelines\n- Posting review feedback as PR comments\n\n### Out of Scope\n- Release notes files outside `/release-notes/components/`\n- Non-RST files\n- PRs without the \"release-notes\" label\n- Manual approval/rejection of PRs (action only provides feedback)\n\n## Requirements\n\n### Functional Requirements\n\n#### FR1: PR Detection and Filtering\n- **FR1.1**: Action triggers on pull request events (opened, synchronize, labeled)\n- **FR1.2**: Action only runs when PR has \"release-notes\" label\n- **FR1.3**: Action identifies all changed RST files in `/release-notes/components/` directory\n\n#### FR2: File Analysis\n- **FR2.1**: Action reads content of each changed RST file\n- **FR2.2**: Action loads release notes guidelines from `_ext/release-notes-context.md`\n- **FR2.3**: Action processes files individually to provide file-specific feedback\n\n#### FR3: Q CLI Integration\n- **FR3.1**: Action invokes Amazon Q CLI with appropriate context\n- **FR3.2**: Action provides Q CLI with:\n - Release notes guidelines from `_ext/release-notes-context.md`\n - Content of the changed RST file\n - Instruction to review against guidelines\n- **FR3.3**: Action captures Q CLI output for each file\n\n#### FR4: Review Feedback\n- **FR4.1**: Action formats Q CLI feedback into readable PR comment\n- **FR4.2**: Action posts comment to PR with review results\n- **FR4.3**: Comment includes:\n - List of files reviewed\n - Issues found per file (using format from guidelines)\n - Suggested improvements\n - Link to full guidelines document\n- **FR4.4**: If no issues found, action posts positive confirmation\n\n#### FR5: Error Handling\n- **FR5.1**: Action handles Q CLI failures gracefully\n- **FR5.2**: Action reports when no RST files are found in scope\n- **FR5.3**: Action logs errors for debugging without failing the PR\n\n### Non-Functional Requirements\n\n#### NFR1: Performance\n- Action completes review within 5 minutes for typical PRs (1-5 files)\n- Action processes files in parallel when possible\n\n#### NFR2: Security\n- Action uses GitHub secrets for Q CLI credentials\n- Action has read-only access to repository\n- Action has write access only to PR comments\n\n#### NFR3: Maintainability\n- Action configuration is version controlled in `.github/workflows/`\n- Action uses official Q CLI container/action when available\n- Action logic is simple and well-documented\n\n## User Stories\n\n### US1: Automatic Review Trigger\n**As a** documentation contributor \n**I want** the review action to run automatically when I label my PR \n**So that** I get immediate feedback without manual intervention\n\n**Acceptance Criteria:**\n- Action triggers when \"release-notes\" label is added\n- Action runs on subsequent commits to labeled PR\n- Action does not run on PRs without the label\n\n### US2: Targeted File Review\n**As a** documentation contributor \n**I want** only my changed release notes files to be reviewed \n**So that** I get relevant feedback without noise from unchanged files\n\n**Acceptance Criteria:**\n- Only files in `/release-notes/components/*.rst` are reviewed\n- Only files modified in the PR are analyzed\n- Files in other directories are ignored\n\n### US3: Clear Feedback\n**As a** documentation contributor \n**I want** clear, actionable feedback on my release notes \n**So that** I know exactly what to improve\n\n**Acceptance Criteria:**\n- Feedback follows the format specified in guidelines\n- Each issue includes: original text, problem, example rewrite, action items\n- Feedback is posted as a PR comment\n- Comment includes link to full guidelines\n\n### US4: No False Failures\n**As a** documentation contributor \n**I want** the action to provide feedback without blocking my PR \n**So that** I can address issues without being blocked by automation\n\n**Acceptance Criteria:**\n- Action never fails the PR check\n- Action always succeeds even if issues are found\n- Issues are reported as comments, not check failures\n\n## Technical Design\n\n### GitHub Action Workflow\n\n**File Location:** `.github/workflows/release-notes-review.yml`\n\n**Trigger Events:**\n```yaml\non:\n pull_request:\n types: [opened, synchronize, labeled]\n paths:\n - 'release-notes/components/**/*.rst'\n```\n\n**Workflow Steps:**\n\n1. **Check Label**\n - Verify PR has \"release-notes\" label\n - Exit gracefully if label not present\n\n2. **Get Changed Files**\n - Use GitHub API to get list of changed files\n - Filter for `release-notes/components/**/*.rst`\n - Exit if no matching files found\n\n3. **Setup Q CLI**\n - Install/configure Amazon Q CLI\n - Authenticate using GitHub secrets\n\n4. **Load Guidelines**\n - Read `_ext/release-notes-context.md`\n - Prepare as context for Q CLI\n\n5. **Review Each File**\n - For each changed RST file:\n - Read file content\n - Invoke Q CLI with prompt:\n ```\n Review the following release notes file against the guidelines provided.\n \n Guidelines: [content from release-notes-context.md]\n \n File: [filename]\n Content: [file content]\n \n Provide feedback using the review format specified in the guidelines.\n Focus on: customer visibility, documentation links, impact clarity, \n specific conditions, and actionable information.\n ```\n - Capture Q CLI response\n\n6. **Format Feedback**\n - Combine all file reviews into single comment\n - Format as markdown with sections per file\n - Include summary at top\n\n7. **Post Comment**\n - Post formatted feedback as PR comment\n - Include link to guidelines\n - Tag PR author\n\n### Q CLI Prompt Template\n\n```markdown\nYou are reviewing release notes for the AWS Neuron SDK. Review the following \nfile against the release notes writing guidelines.\n\nGUIDELINES:\n[Full content of _ext/release-notes-context.md]\n\nFILE TO REVIEW: {filename}\n\nCONTENT:\n{file_content}\n\nINSTRUCTIONS:\n1. Review the content against all guidelines\n2. Identify issues using the review format from the guidelines\n3. For each issue, provide:\n - Issue number and title\n - Original text\n - Problem description\n - Phrasing problem (if applicable)\n - Example rewrite\n - Specific action items\n4. If no issues found, state \"No issues found - release notes meet guidelines\"\n\nFocus especially on:\n- Customer-visible language (no internal code names)\n- Documentation URLs for all new features\n- Specific conditions (not vague language)\n- Clear impact statements\n- Proper categorization (breaking changes vs bug fixes)\n- Migration guidance for breaking changes\n```\n\n### Comment Format Template\n\n```markdown\n## 🤖 Release Notes Review\n\nThis PR modifies {count} release notes file(s). Here's the automated review:\n\n### Files Reviewed\n- ✅ `release-notes/components/file1.rst` - {issue_count} issue(s)\n- ✅ `release-notes/components/file2.rst` - No issues found\n\n---\n\n### 📝 Review Feedback\n\n#### File: `release-notes/components/file1.rst`\n\n[Q CLI feedback for file1]\n\n---\n\n#### File: `release-notes/components/file2.rst`\n\n[Q CLI feedback for file2]\n\n---\n\n### 📚 Resources\n\n- [Release Notes Writing Guidelines](_ext/release-notes-context.md)\n- Need help? Tag @documentation-team\n\n---\n\n*This is an automated review. Please address the feedback and request human \nreview when ready.*\n```\n\n## Implementation Notes\n\n### GitHub Action Configuration\n\n**Required Secrets:**\n- `Q_CLI_TOKEN` or equivalent for Q CLI authentication\n\n**Required Permissions:**\n```yaml\npermissions:\n contents: read\n pull-requests: write\n```\n\n**Environment:**\n- Ubuntu latest runner\n- Node.js 18+ (if using JavaScript action)\n- Python 3.9+ (if using Python script)\n\n### Q CLI Integration Options\n\n**Option 1: Direct CLI Invocation**\n```bash\nq chat --prompt-file prompt.txt --context-file guidelines.md\n```\n\n**Option 2: Q CLI GitHub Action** (if available)\n```yaml\n- uses: aws/q-cli-action@v1\n with:\n prompt: ${{ steps.prepare.outputs.prompt }}\n context: ${{ steps.prepare.outputs.context }}\n```\n\n**Option 3: API Integration** (if Q provides API)\n```python\nimport q_cli\nresponse = q_cli.chat(prompt=prompt, context=guidelines)\n```\n\n## Testing Strategy\n\n### Unit Tests\n- Test file filtering logic\n- Test prompt generation\n- Test comment formatting\n\n### Integration Tests\n- Test with sample PR containing valid release notes\n- Test with sample PR containing issues\n- Test with PR without \"release-notes\" label\n- Test with PR modifying non-component files\n\n### Manual Testing\n- Create test PR with intentional issues\n- Verify action triggers correctly\n- Verify feedback is accurate and helpful\n- Verify comment formatting is readable\n\n## Success Criteria\n\n1. **Automation Works**: Action runs on 100% of labeled PRs\n2. **Accurate Detection**: Action correctly identifies changed RST files\n3. **Useful Feedback**: 80%+ of PR authors find feedback helpful\n4. **No False Blocks**: Action never blocks valid PRs\n5. **Performance**: Action completes within 5 minutes\n6. **Reliability**: Action succeeds 95%+ of the time\n\n## Future Enhancements\n\n### Phase 2 (Optional)\n- Support for reviewing other release notes files (not just components)\n- Severity levels for issues (critical, warning, suggestion)\n- Auto-fix suggestions as code suggestions\n- Integration with PR review status\n- Metrics dashboard for common issues\n\n### Phase 3 (Optional)\n- Pre-commit hook for local review\n- VS Code extension for real-time feedback\n- Training mode to help new contributors learn guidelines\n- Historical analysis of release notes quality trends\n\n## Dependencies\n\n- GitHub Actions infrastructure\n- Amazon Q CLI availability and access\n- Repository write access for bot account\n- `_ext/release-notes-context.md` guidelines file\n\n## Risks and Mitigations\n\n| Risk | Impact | Mitigation |\n|------|--------|------------|\n| Q CLI unavailable | High | Graceful failure with manual review fallback |\n| Q CLI rate limits | Medium | Implement retry logic and rate limiting |\n| False positives | Medium | Continuous refinement of guidelines and prompts |\n| Action performance | Low | Parallel processing and caching |\n| Cost of Q CLI usage | Low | Monitor usage and set budget alerts |\n\n## Rollout Plan\n\n1. **Phase 1**: Implement basic action with manual trigger\n2. **Phase 2**: Enable automatic trigger on label\n3. **Phase 3**: Gather feedback and refine prompts\n4. **Phase 4**: Expand to other release notes files if successful\n\n## Maintenance\n\n- **Owner**: Documentation team\n- **Review Frequency**: Quarterly\n- **Update Triggers**: \n - Changes to release notes guidelines\n - Q CLI updates\n - User feedback on accuracy\n - GitHub Actions platform changes\n" }, { "path": "_ext/release-notes-context.md", "content": "# Release Notes Writing Guidelines\n\n## Core Principles\n\n### Answer Three Questions for Every Item\n\n- **What?** — What feature/API is affected?\n- **When?** — Under what conditions does this occur?\n- **So what?** — What is the impact on the user?\n\n### All Content Must Be:\n\n- **Customer-visible** - Written from the customer's perspective about capabilities they can use\n- **Documented** - If documentation doesn't exist, exclude the feature. All new features must include documentation URLs.\n- **Actionable** - Include workarounds, timelines, or how to check if affected\n\n## DO:\n\n- **Write in customer-visible terms** - Describe what customers can now do, not how it was implemented\n- **State the impact clearly** - Use concrete language about what happens to users\n- **Be specific about conditions** - Replace vague phrases with precise conditions\n- **Quantify performance improvements** - Provide specific before/after metrics (e.g., \"improved from 2.164x to 3.654x speedup\") and state the conditions that trigger these improvements (e.g., \"for batch I/O operations with 1024 ops at 10KB\")\n- **Explain the impact of wrong defaults** - When fixing incorrect default values, state what the wrong default was and what impact it had on users\n- **Specify what was missing** - When fixing \"missing\" items, list what was missing and confirm they are now documented\n- **Describe previous behavior for bugs** - Always explain what the incorrect behavior was before the fix\n- **Categorize breaking changes correctly** - If a bug fix changes API behavior (e.g., renaming a parameter), list it under Breaking Changes, not Bug Fixes\n- **Provide actionable information** - Include workarounds if available, fix timelines if known, or how users can check if they're affected\n- **Provide migration guidance for breaking changes** - Tell users what they should do when behavior changes, with before/after examples\n- **Link to documentation** - Every feature must have corresponding documentation with URL\n- **Include documentation URLs for all new features** - If no URL exists, either create documentation first or remove the feature from release notes\n- **Use standard terminology** - Use terms your audience already knows\n- **Use clear, descriptive sentences** - Transform technical phrases into customer-understandable language\n- **Focus on customer-visible results** - Describe what customers will see, not internal mechanics\n- **Drop unnecessary words** - Remove \"when specified,\" \"may,\" \"is in progress\" when they add no value\n- **Remove empty sections** - Don't include placeholder text like \"None in this release\"\n- **Verify accuracy** - Check version numbers, dates, and technical details\n- **Run IP scanner** - Catch any internal code name leaks before publishing\n- **Use active voice** - Write \"The system ignores the parameter\" instead of \"The parameter is ignored\"\n- **Define abbreviations on first use** - Write \"time to first token (TTFT)\" before using \"TTFT\"\n- **Remove temporal qualifiers** - Replace \"for now\" with specific timelines or remove entirely\n- **Provide concrete examples** - Include calculation examples for complex parameters\n\n## DO NOT:\n\n- **Include internal code names** - Remove references like \"TRN3PDS\", \"Mariana\", \"Penguin\"\n- **Document undocumented features** - If documentation doesn't exist, exclude the feature\n- **Include features without documentation URLs** - Every new feature must have a documentation link\n- **List unreleased features** - Only include features available to customers\n- **Include internal-only metrics** - Remove metrics useful only internally\n- **Document bugs never released** - Only include fixes for publicly released issues\n- **Use internal API names** - Unless they're part of the public API\n- **Include debug variables** - Remove environment variables meant only for internal use\n- **Use vague language** - Avoid \"in certain cases,\" \"some patterns,\" \"may sometimes\"\n- **Use ambiguous phrasing** - Avoid phrases like \"Fixed dynamic for loop\" that could mean multiple things\n- **Leave impacts unexplained** - Don't just say \"fixed wrong default\" without explaining what the impact was\n- **Mix breaking changes with bug fixes** - Parameter renames or behavior changes belong in Breaking Changes, not Bug Fixes\n- **Create heavy noun chains** - Break up complex phrases (e.g., \"dtype override was ignored during reshape\" not \"reshape dtype override not being applied\")\n- **Write without context** - Every change needs metrics, conditions, or migration guidance\n- **Use hedging language** - Replace \"may result in\" with \"results in\" when deterministic\n- **Focus on internal implementation** - Avoid phrases like \"internally uses\" or internal platform identifiers\n- **Use passive voice without clear subject** - Avoid constructions where the actor is unclear\n- **Reference undefined versions** - Don't use \"V0\" or \"V1\" without defining them\n\n## Impact Statements\n\n| Avoid | Prefer |\n|-------|--------|\n| \"incorrectly interpret\" | \"produces incorrect results\" |\n| \"not being applied\" | \"is ignored\" |\n| \"failing check\" | \"crashes with validation error\" |\n| \"may incorrectly interpret tensor shapes\" | \"can produce incorrect results when transposing tensors\" |\n\n## Conditions - Be Specific\n\n| Avoid | Prefer |\n|-------|--------|\n| \"in certain cases\" | \"when reduction axis is not the last dimension\" |\n| \"some patterns\" | \"multi-dimensional transposes with more than 2 axes\" |\n| \"may sometimes\" | \"consistently occurs when...\" |\n| \"for now\" | \"Support is planned for version X.X.X\" or remove entirely |\n| \"small inputs\" | \"inputs under 512 tokens\" |\n| \"low batch sizes\" | \"batch sizes of 4 or less\" |\n\n## Phrasing Examples\n\n### Bug Fixes:\n\n| Avoid | Prefer |\n|-------|--------|\n| \"Fixed bug in nrt_vnc_usage_find_internal\" | \"Improved error handling to return a clear error instead of asserting during nrt_init\" |\n| \"Fixed dynamic for loop incorrectly incrementing the loop induction variable\" | \"Fixed: dynamic for loops now correctly increment the loop counter. Previously, the counter incremented incorrectly, causing [specific impact]\" |\n| \"Fixed reshape dtype override not being applied when specified\" | \"Fixed a bug where specifying a data type override during a reshape operation was ignored\" |\n| \"Fixed reshape of shared/private HBM tensors failing partition size check\" | \"Fixed a bug where reshaping tensors stored in shared or private HBM incorrectly failed the partition size check\" |\n| \"Fixed incorrect default value for on_false_value\" | \"Fixed incorrect default value for on_false_value in nki.isa.range_select. Previously defaulted to [X], now correctly defaults to [Y], which [impact]\" |\n\n### Performance Improvements:\n\n| Avoid | Prefer |\n|-------|--------|\n| \"Optimized zero-copy operations by enabling descriptor merging\" | \"Enhanced zero-copy operation performance: Write performance improved from 2.164x to 3.654x speedup for batch I/O operations(1_Batch_1024_Ops_10_KBs)\" |\n| \"Optimized mesh AllGather on TP8 configurations using destination routing\" | \"Optimized mesh AllGather: [X]% performance improvement on TP8 configurations when [specific conditions]\" |\n\n### New Features:\n\n| Avoid | Prefer |\n|-------|--------|\n| \"Added support for TRN3PDS platform\" | \"Added support for [public instance type name] with optimized topology configurations for distributed training. See [documentation URL]\" |\n| \"Added IOCTL to lookup Neuron device/HBM for a given virtual address\" | \"Added capability to lookup Neuron device for a given virtual address, enabling frameworks to identify which device holds a tensor. See [documentation link] for API details\" |\n\n### Known Issues:\n\n| Avoid | Prefer |\n|-------|--------|\n| \"may incorrectly interpret tensor shapes in certain multi-dimensional transpose patterns\" | \"can produce incorrect results when transposing tensors with certain multi-dimensional shapes\" |\n| \"Training, Inference, and Penguin kernels compilation and execution validation is in progress\" | Remove entirely (internal project name and not customer-actionable) |\n| \"Chunked prefill is not supported on Neuron for now\" | \"Chunked prefill is not supported. If you attempt to enable it with DISABLE_NEURON_CUSTOM_SCHEDULER='1', the system will fail to start with an error. Use standard prefill mode instead.\" |\n\n## Breaking Changes Checklist\n\nWhen documenting breaking changes, always include:\n\n1. **What changed** - The specific API, parameter, or behavior\n2. **Why it's breaking** - What will stop working\n3. **Migration path** - What users should do instead\n4. **Example (if helpful)** - Show old vs. new usage\n\n### Example:\n\n**Breaking:** NumPy synonyms (e.g., `np.add` for `nl.add`) are no longer accepted in NKI API calls.\n\n**Migration:** Replace all NumPy function calls with their NKI equivalents:\n- Replace `np.add(x, y)` with `nl.add(x, y)`\n- Replace `np.multiply(x, y)` with `nl.multiply(x, y)`\n\nAlways explain:\n- Why is this breaking?\n- What was the previous behavior?\n- What is the workaround or migration effort?\n\n## Quick Template\n\n```\n[Fixed/Known Issue]: [API/Feature] [impact] when [specific conditions]. [Optional: Workaround or timeline.]\n```\n\n### Example:\n\n```\nFixed: nki.isa.dma_copy causes a runtime timeout when copying FP32 from SBUF to BF16 in HBM with indirect addressing. Workaround: cast to BF16 in SBUF before copying.\n```\n\n## Quality Checks Before Publishing\n\n1. **No internal names** - Run IP scanner to catch code name leaks\n2. **Customer value** - Each item explains why customers should care\n3. **Documentation links** - New features link to relevant docs with URLs\n4. **Documentation exists** - Verify all features are documented before including; if no documentation URL exists, remove the feature from release notes\n5. **Accuracy** - Technical details are correct and verifiable\n6. **Clarity** - Phrasing is clear and professional\n7. **Completeness** - Previous behavior and migration paths explained\n8. **Impact explained** - Bug fixes describe what was broken and what the impact was\n9. **Active voice** - Sentences use active voice with clear subjects\n10. **Abbreviations defined** - All abbreviations spelled out on first use\n11. **No vague language** - All conditions and impacts are specific and quantified\n12. **Examples provided** - Complex parameters include calculation examples\n\n## Key Principles\n\n### All content must be:\n\n- **Customer-visible** (not internal implementation details)\n- **Documented with URLs** (if docs don't exist, exclude it)\n- **Impactful** (explain value, not just what changed)\n\n### Every bug fix must answer:\n\n- What was broken?\n- What was the impact?\n- What works now?\n\n### Every new feature must include:\n\n- Documentation URL\n- Customer benefit\n- Usage guidance or examples\n\n## How to Review Release Notes\n\nWhen reviewing release notes against these guidelines, provide feedback in the following format:\n\n### Issue [Number]: [Brief Issue Title]\n\n**Original Text:**\n```\n[Exact text from the release notes]\n```\n\n**Problem:**\n[Description of the content/completeness issue]\n\n**Phrasing Problem:**\n[Description of the language/clarity issue, if applicable]\n\n**Example Rewrite:**\n```\n[Suggested improved version showing correct phrasing and content]\n```\n\n**Action:**\n- [Specific action item 1]\n- [Specific action item 2]\n\n## Review Process:\n\n1. **Extract original text** - Include the exact text being reviewed\n2. **Identify problems** - Separate content issues from phrasing issues\n3. **Provide examples** - Show how to rewrite the text correctly\n4. **List actions** - Give specific, actionable steps to fix each issue\n5. **Check documentation** - Verify URLs exist for all new features; if not, recommend removal\n6. **Verify completeness** - Ensure all three questions (What? When? So what?) are answered\n7. **Check phrasing** - Identify vague language, passive voice, undefined terms, internal references\n8. **Validate breaking changes** - Ensure migration guidance and before/after examples are included\n" }, { "path": "_ext/sphinx_plotly_directive.py", "content": "\"\"\"\nCODE FROM: https://github.com/harupy/sphinx-plotly-directive\nLICENSE: MIT\n\nBased on: https://matplotlib.org/3.1.3/devel/plot_directive.html\n\nA directive for including a Plotly figure in a Sphinx document\n================================================================\n\nBy default, in HTML output, `plot` will include a .png file with a link to a\nhigh-res .png and .pdf. In LaTeX output, it will include a .pdf.\n\nThe source code for the plot may be included in one of three ways:\n\n1. **A path to a source file** as the argument to the directive::\n\n .. plot:: path/to/plot.py\n\n When a path to a source file is given, the content of the\n directive may optionally contain a caption for the plot::\n\n .. plot:: path/to/plot.py\n\n The plot's caption.\n\n Additionally, one may specify the name of a function to call (with\n no arguments) immediately after importing the module::\n\n .. plot:: path/to/plot.py plot_function1\n\n2. Included as **inline content** to the directive::\n\n .. plotly::\n\n import plotly.express as px\n px.scatter(x=[0, 1, 2, 3, 4], y=[0, 1, 4, 9, 16])\n\n3. Using **doctest** syntax::\n\n .. plotly::\n\n A plotting example:\n >>> import plotly.express as px\n >>> px.scatter(x=[0, 1, 2, 3, 4], y=[0, 1, 4, 9, 16])\n\n4. Using the `fig-vars` option. In the example below, `fig1` and `fig2` will be\n rendered::\n\n .. plotly::\n :fig-vars: fig1, fig2\n\n import plotly.express as px\n fig1 = px.scatter(x=[0, 1, 2, 3, 4], y=[0, 1, 4, 9, 16])\n fig2 = px.scatter(x=[4, 3, 2, 1, 0], y=[0, 1, 4, 9, 16])\n\nOptions\n-------\n\nThe ``plotly`` directive supports the following options:\n\n format : {'python', 'doctest'}\n The format of the input.\n\n include-source : bool\n Whether to display the source code. The default can be changed\n using the `plot_include_source` variable in :file:`conf.py`.\n\n encoding : str\n If this source file is in a non-UTF8 or non-ASCII encoding, the\n encoding must be specified using the ``:encoding:`` option. The\n encoding will not be inferred using the ``-*- coding -*-`` metacomment.\n\n context : bool or str\n If provided, the code will be run in the context of all previous plot\n directives for which the ``:context:`` option was specified. This only\n applies to inline code plot directives, not those run from files. If\n the ``:context: reset`` option is specified, the context is reset\n for this and future plots, and previous figures are closed prior to\n running the code. ``:context: close-figs`` keeps the context but closes\n previous figures before running the code.\n\n nofigs : bool\n If specified, the code block will be run, but no figures will be\n inserted. This is usually useful with the ``:context:`` option.\n\n caption : str\n If specified, the option's argument will be used as a caption for the\n figure. This overwrites the caption given in the content, when the plot\n is generated from a file.\n\n iframe-width\n The width of the iframe in which a plotly figure is rendered. The default can be changed\n using the `plotly_iframe_width` variable in :file:`conf.py`.\n\n iframe-height\n The height of the iframe in which a plotly figure is rendered. The default can be changed\n using the `plotly_iframe_height` variable in :file:`conf.py`.\n\nAdditionally, this directive supports all of the options of the `image`\ndirective, except for *target* (since plot will add its own target). These\ninclude *alt*, *height*, *width*, *scale*, *align* and *class*.\n\nConfiguration options\n---------------------\n\nThe plot directive has the following configuration options:\n\n plotly_include_source\n Default value for the include-source option\n\n plotly_html_show_source_link\n Whether to show a link to the source in HTML.\n\n plotly_pre_code\n Code that should be executed before each plot. If not specified or None\n it will default to a string containing::\n\n import numpy as np\n import plotly\n import plotly.graph_objects as go\n import plotly.express as px\n\n plotly_basedir\n Base directory, to which ``plot::`` file names are relative\n to. (If None or empty, file names are relative to the\n directory where the file containing the directive is.)\n\n plotly_formats\n File formats to generate. List of tuples or strings::\n\n [(suffix, dpi), suffix, ...]\n\n that determine the file format and the DPI. For entries whose\n DPI was omitted, sensible defaults are chosen. When passing from\n the command line through sphinx_build the list should be passed as\n suffix:dpi,suffix:dpi, ...\n\n plotly_html_show_formats\n Whether to show links to the files in HTML.\n\n plotly_working_directory\n By default, the working directory will be changed to the directory of\n the example, so the code can get at its data files, if any. Also its\n path will be added to `sys.path` so it can import any helper modules\n sitting beside it. This configuration option can be used to specify\n a central directory (also added to `sys.path`) where data files and\n helper modules for all code are located.\n\n plotly_iframe_width\n The width of the iframe in which a plotly figure is rendered. The default is \"100%\".\n\n plotly_iframe_height\n The height of the iframe in which a plotly figure is rendered. The default is \"500px\".\n\n plotly_template\n Provide a customized template for preparing restructured text.\n\"\"\"\n\nimport copy\nimport itertools\nimport os\nimport re\nimport shutil\nimport textwrap\nimport traceback\nfrom os.path import relpath\nfrom pathlib import Path\n\nimport jinja2 # Sphinx dependency.\nfrom docutils.parsers.rst import Directive, directives\nfrom docutils.parsers.rst.directives.images import Image\n\nimport re\nimport textwrap\n\nimport plotly\n\n\nINDENT_SPACES = \" \" * 3\n\n\ndef save_plotly_figure(fig, path):\n r\"\"\"\n Save a Plotly figure.\n Parameters\n ----------\n fig : plotly figure\n A plotly figure to save.\n path : str\n A file path.\n Returns\n -------\n None\n Examples\n --------\n >>> import plotly.express as px\n >>> import tempfile\n >>> fig = px.scatter(x=[0, 1, 2, 3, 4], y=[0, 1, 4, 9, 16])\n >>> path = tempfile.NamedTemporaryFile(suffix=\".html\").name\n >>> save_plotly_figure(fig, path)\n \"\"\"\n fig_html = plotly.offline.plot(fig, output_type=\"div\", include_plotlyjs=\"cdn\", auto_open=False)\n with open(path, \"w\") as f:\n f.write(fig_html)\n\n\ndef assign_last_line_into_variable(code, variable_name):\n r\"\"\"\n Save a Plotly figure.\n Parameters\n ----------\n code : str\n A string representing code.\n name : str\n A variable name.\n Returns\n -------\n str\n Mew code.\n Examples\n --------\n >>> code = \"a = 1\\nfunc(a)\"\n >>> new_code = assign_last_line_into_variable(code, \"b\")\n >>> print(new_code)\n a = 1\n b = func(a)\n \"\"\"\n lines = code.split(\"\\n\")\n for idx in range(len(lines) - 1, -1, -1):\n if lines[idx].strip() != \"\":\n lines[idx] = \"{} = \".format(variable_name) + lines[idx]\n break\n return \"\\n\".join(lines)\n\n\ndef create_directive_block(name, arguments, options, content):\n r\"\"\"\n Create a directive block.\n Parameters\n ----------\n name : str\n A directive name.\n arguments : list of str\n Arguments of the directive.\n option : dict\n Option of the directive.\n content : list of str\n Content of the directive.\n Returns\n -------\n str\n A directive block.\n Examples\n --------\n >>> block = create_directive_block(\n ... \"plotly\",\n ... [\"f1\", \"f2\"],\n ... {\"a\": 0, \"b\": 1},\n ... [\"l1\", \"l2\"],\n ... )\n >>> print(block)\n .. plotly:: f1 f2\n :a: 0\n :b: 1\n \n l1\n l2\n \"\"\"\n header = \".. {}:: \".format(name) + \" \".join(arguments)\n code = \"\\n\".join(map(str, content))\n\n lines = [header]\n\n if len(options.items()) > 0:\n\n def process_value(v):\n if isinstance(v, list):\n return \", \".join(v)\n return v\n\n options_block = \"\\n\".join(\":{}: {}\".format(k, process_value(v)) for k, v in options.items())\n lines.append(textwrap.indent(options_block, INDENT_SPACES))\n\n lines.append(\"\")\n lines.append(textwrap.indent(code, INDENT_SPACES))\n\n return \"\\n\".join(lines)\n\n\ndef create_code_block(code, language=None):\n return \"\\n\".join(\n [\n \".. code-block::{}\".format(\" \" + language if language else \"\"),\n \"\",\n textwrap.indent(code.strip(), INDENT_SPACES),\n \"\",\n ]\n )\n\n\ndef strip_last_line(code):\n r\"\"\"\n Strips the last line of the give code block\n Parameters\n ----------\n code : str\n Code to strip\n Returns\n -------\n str:\n Stripped code\n Examples\n --------\n >>> strip_last_line(\"a\")\n ''\n >>> strip_last_line(\"a\\nb\")\n 'a'\n >>> strip_last_line(\"a\\nb\\nc\")\n 'a\\nb'\n \"\"\"\n return \"\\n\".join(code.strip().split(\"\\n\")[:-1])\n\n\ndef ends_with_show(code):\n r\"\"\"\n Returns True if the last line of the given code block ends with `show()`\n Parameters\n ----------\n code : str\n Code that may contain a line that looks like `fig.show()`\n Returns\n -------\n str:\n Variable name of the object that calls `show()`\n Examples\n --------\n >>> ends_with_show(\"fig.show()\") # simple\n True\n >>> ends_with_show(\"fig.show(1, a=2)\") # show with arguments\n True\n >>> ends_with_show(\"fig = dummy\\nfig.show()\\n\") # multiline\n True\n >>> ends_with_show(\"foo\") # doesn't contains `show`\n False\n \"\"\"\n # TODO: Use a more strict regular expression\n pattern = r\"^(.+)\\.show\\(.*\\)$\"\n match = re.search(pattern, code.strip().split(\"\\n\")[-1], flags=re.DOTALL)\n return bool(match)\n\n\n# -----------------------------------------------------------------------------\n# Registration hook\n# -----------------------------------------------------------------------------\n\n\ndef _option_boolean(arg):\n if not arg or not arg.strip():\n # no argument given, assume used as a flag\n return True\n elif arg.strip().lower() in (\"no\", \"0\", \"false\"):\n return False\n elif arg.strip().lower() in (\"yes\", \"1\", \"true\"):\n return True\n else:\n raise ValueError('\"%s\" unknown boolean' % arg)\n\n\ndef _option_context(arg):\n if arg in [None, \"reset\", \"close-figs\"]:\n return arg\n raise ValueError(\"Argument should be None or 'reset' or 'close-figs'\")\n\n\ndef _option_format(arg):\n return directives.choice(arg, (\"python\", \"doctest\"))\n\n\ndef _option_fig_vars(arg):\n return [x.strip() for x in arg.split(\",\")]\n\n\ndef mark_plot_labels(app, document):\n \"\"\"\n To make plots referenceable, we need to move the reference from the\n \"htmlonly\" (or \"latexonly\") node to the actual figure node itself.\n \"\"\"\n for name, explicit in document.nametypes.items():\n if not explicit:\n continue\n labelid = document.nameids[name]\n if labelid is None:\n continue\n node = document.ids[labelid]\n if node.tagname in (\"html_only\", \"latex_only\"):\n for n in node:\n if n.tagname == \"figure\":\n sectname = name\n for c in n:\n if c.tagname == \"caption\":\n sectname = c.astext()\n break\n\n node[\"ids\"].remove(labelid)\n node[\"names\"].remove(name)\n n[\"ids\"].append(labelid)\n n[\"names\"].append(name)\n document.settings.env.labels[name] = (\n document.settings.env.docname,\n labelid,\n sectname,\n )\n break\n\n\nclass PlotlyDirective(Directive):\n \"\"\"The ``.. plotly::`` directive, as documented in the module's docstring.\"\"\"\n\n has_content = True\n required_arguments = 0\n optional_arguments = 2\n final_argument_whitespace = False\n option_spec = {\n \"alt\": directives.unchanged,\n \"height\": directives.length_or_unitless,\n \"width\": directives.length_or_percentage_or_unitless,\n \"scale\": directives.nonnegative_int,\n \"align\": Image.align,\n \"class\": directives.class_option,\n \"include-source\": _option_boolean,\n \"format\": _option_format,\n \"context\": _option_context,\n \"nofigs\": directives.flag,\n \"encoding\": directives.encoding,\n \"caption\": directives.unchanged,\n \"fig-vars\": _option_fig_vars,\n \"iframe-width\": directives.unchanged,\n \"iframe-height\": directives.unchanged,\n }\n\n def run(self):\n \"\"\"Run the plot directive.\"\"\"\n try:\n return run(\n self.arguments,\n self.content,\n self.options,\n self.state_machine,\n self.state,\n self.lineno,\n )\n except Exception as e:\n raise self.error(str(e))\n\n\ndef setup(app):\n setup.app = app\n setup.config = app.config\n setup.confdir = app.confdir\n app.add_directive(\"plotly\", PlotlyDirective)\n app.add_config_value(\"plotly_pre_code\", None, True)\n app.add_config_value(\"plotly_include_source\", False, True)\n app.add_config_value(\"plotly_html_show_source_link\", True, True)\n app.add_config_value(\"plotly_formats\", [\"html\"], True)\n app.add_config_value(\"plotly_basedir\", None, True)\n app.add_config_value(\"plotly_html_show_formats\", True, True)\n app.add_config_value(\"plotly_working_directory\", None, True)\n app.add_config_value(\"plotly_iframe_width\", \"100%\", True)\n app.add_config_value(\"plotly_iframe_height\", \"500px\", True)\n app.add_config_value(\"plotly_template\", None, True)\n\n app.add_config_value(\"plotly_include_directive_source\", None, False)\n\n app.connect(\"doctree-read\", mark_plot_labels)\n\n metadata = {\n \"parallel_read_safe\": True,\n \"parallel_write_safe\": True,\n \"version\": 0.1,\n }\n return metadata\n\n\n# -----------------------------------------------------------------------------\n# Doctest handling\n# -----------------------------------------------------------------------------\n\n\ndef contains_doctest(text):\n try:\n # check if it's valid Python as-is\n compile(text, \"\", \"exec\")\n return False\n except SyntaxError:\n pass\n r = re.compile(r\"^\\s*>>>\", re.M)\n m = r.search(text)\n return bool(m)\n\n\ndef unescape_doctest(text):\n \"\"\"\n Extract code from a piece of text, which contains either Python code\n or doctests.\n \"\"\"\n if not contains_doctest(text):\n return text\n\n code = \"\"\n for line in text.split(\"\\n\"):\n m = re.match(r\"^\\s*(>>>|\\.\\.\\.) (.*)$\", line)\n if m:\n code += m.group(2) + \"\\n\"\n elif line.strip():\n code += \"# \" + line.strip() + \"\\n\"\n else:\n code += \"\\n\"\n return code\n\n\ndef split_code_at_show(text):\n \"\"\"Split code at plt.show().\"\"\"\n parts = []\n is_doctest = contains_doctest(text)\n\n part = []\n for line in text.split(\"\\n\"):\n if (not is_doctest and line.strip() == \"plt.show()\") or (\n is_doctest and line.strip() == \">>> plt.show()\"\n ):\n part.append(line)\n parts.append(\"\\n\".join(part))\n part = []\n else:\n part.append(line)\n if \"\\n\".join(part).strip():\n parts.append(\"\\n\".join(part))\n return parts\n\n\n# -----------------------------------------------------------------------------\n# Template\n# -----------------------------------------------------------------------------\n\n\nTEMPLATE = \"\"\"\n{% if directive_source %}\nSource:\n\n{{ directive_source }}\n\nOutput:\n{% endif %}\n{{ source_code }}\n\n.. only:: html\n\n {% if source_link or (html_show_formats and not multi_image) %}\n (\n {%- if source_link -%}\n `Source code <{{ source_link }}>`__\n {%- endif -%}\n {%- if html_show_formats and not multi_image -%}\n {%- for fig in figures -%}\n {%- for fmt in fig.formats -%}\n {%- if source_link or not loop.first -%}, {% endif -%}\n `{{ fmt }} <{{ dest_dir }}/{{ fig.basename }}.{{ fmt }}>`__\n {%- endfor -%}\n {%- endfor -%}\n {%- endif -%}\n )\n {% endif %}\n\n {% for fig in figures %}\n .. raw:: html\n {% for option in options -%}\n {{ option }}\n {% endfor %}\n\n \n\n {% if html_show_formats and multi_figure -%}\n (\n {%- for fmt in fig.formats -%}\n {%- if not loop.first -%}, {% endif -%}\n `{{ fmt }} <{{ dest_dir }}/{{ fig.basename }}.{{ fmt }}>`__\n {%- endfor -%}\n )\n {%- endif -%}\n\n {{ caption }}\n {% endfor %}\n\n.. only:: not html\n\n {% for fig in figures %}\n .. raw:: html\n {% for option in options -%}\n {{ option }}\n {% endfor %}\n\n \n\n {{ caption }}\n {% endfor %}\n\n\"\"\"\n\nexception_template = \"\"\"\n.. only:: html\n\n [`source code <%(linkdir)s/%(basename)s.py>`__]\n\nException occurred rendering plot.\n\n\"\"\"\n\n# the context of the plot for all directives specified with the\n# :context: option\nplot_context = dict()\n\n\nclass FigureFile:\n def __init__(self, basename, dirname):\n self.basename = basename\n self.dirname = dirname\n self.formats = []\n\n def filename(self, format):\n return os.path.join(self.dirname, \"%s.%s\" % (self.basename, format))\n\n def filenames(self):\n return [self.filename(fmt) for fmt in self.formats]\n\n\ndef out_of_date(original, derived):\n \"\"\"\n Return whether *derived* is out-of-date relative to *original*, both of\n which are full file paths.\n \"\"\"\n return not os.path.exists(derived) or (\n os.path.exists(original) and os.stat(derived).st_mtime < os.stat(original).st_mtime\n )\n\n\nclass PlotError(RuntimeError):\n pass\n\n\ndef run_code(code, code_path, ns=None, function_name=None, fig_vars=None):\n \"\"\"\n Import a Python module from a path, and run the function given by\n name, if function_name is not None.\n \"\"\"\n\n # Change the working directory to the directory of the example, so\n # it can get at its data files, if any. Add its path to sys.path\n # so it can import any helper modules sitting beside it.\n pwd = os.getcwd()\n if setup.config.plotly_working_directory is not None:\n try:\n os.chdir(setup.config.plotly_working_directory)\n except OSError as err:\n raise OSError(\n str(err) + \"\\n`plot_working_directory` option in\"\n \"Sphinx configuration file must be a valid \"\n \"directory path\"\n ) from err\n except TypeError as err:\n raise TypeError(\n str(err) + \"\\n`plot_working_directory` option in \"\n \"Sphinx configuration file must be a string or \"\n \"None\"\n ) from err\n elif code_path is not None:\n dirname = os.path.abspath(os.path.dirname(code_path))\n os.chdir(dirname)\n\n try:\n code = unescape_doctest(code)\n if ns is None:\n ns = {}\n if not ns:\n if setup.config.plotly_pre_code is None:\n exec(\n \"\\n\".join(\n [\n \"import numpy as np\",\n \"import plotly\",\n \"import plotly.graph_objects as go\",\n \"import plotly.express as px\",\n ]\n ),\n ns,\n )\n else:\n exec(str(setup.config.plotly_pre_code), ns)\n if \"__main__\" in code:\n ns[\"__name__\"] = \"__main__\"\n\n variable_name = \"fig\"\n\n if ends_with_show(code):\n exec(strip_last_line(code), ns)\n figs = [ns[fig_var] for fig_var in fig_vars] if fig_vars else [ns[variable_name]]\n elif function_name is not None:\n exec(code, ns)\n exec(assign_last_line_into_variable(function_name + \"()\", variable_name), ns)\n figs = [ns[variable_name]]\n elif fig_vars:\n exec(code, ns)\n figs = [ns[fig_var] for fig_var in fig_vars]\n else:\n exec(assign_last_line_into_variable(code, variable_name), ns)\n figs = [ns[variable_name]]\n\n except (Exception, SystemExit) as err:\n raise PlotError(traceback.format_exc()) from err\n finally:\n os.chdir(pwd)\n\n return figs\n\n\ndef get_plot_formats(config):\n default_dpi = {\"html\": 0}\n formats = []\n plot_formats = config.plotly_formats\n for fmt in plot_formats:\n if isinstance(fmt, str):\n if \":\" in fmt:\n suffix, dpi = fmt.split(\":\")\n formats.append((str(suffix), int(dpi)))\n else:\n formats.append((fmt, default_dpi.get(fmt, 80)))\n elif isinstance(fmt, (tuple, list)) and len(fmt) == 2:\n formats.append((str(fmt[0]), int(fmt[1])))\n else:\n raise PlotError('invalid image format \"%r\" in plot_formats' % fmt)\n return formats\n\n\ndef render_figures(\n code,\n code_path,\n output_dir,\n output_base,\n context,\n function_name,\n config,\n context_reset=False,\n close_figs=False,\n fig_vars=None,\n):\n \"\"\"\n Run a pyplot script and save the images in *output_dir*.\n\n Save the images under *output_dir* with file names derived from\n *output_base*\n \"\"\"\n formats = get_plot_formats(config)\n\n # -- Try to determine if all images already exist\n\n code_pieces = split_code_at_show(code)\n\n # Look for single-figure output files first\n all_exists = True\n fig = FigureFile(output_base, output_dir)\n for format, dpi in formats:\n if out_of_date(code_path, fig.filename(format)):\n all_exists = False\n break\n fig.formats.append(format)\n\n if all_exists:\n return [(code, [fig])]\n\n # Then look for multi-figure output files\n results = []\n all_exists = True\n for i, code_piece in enumerate(code_pieces):\n figures = []\n for j in itertools.count():\n if len(code_pieces) > 1:\n fig = FigureFile(\"%s_%02d_%02d\" % (output_base, i, j), output_dir)\n else:\n fig = FigureFile(\"%s_%02d\" % (output_base, j), output_dir)\n for fmt, dpi in formats:\n if out_of_date(code_path, fig.filename(fmt)):\n all_exists = False\n break\n fig.formats.append(fmt)\n\n # assume that if we have one, we have them all\n if not all_exists:\n all_exists = j > 0\n break\n figures.append(fig)\n if not all_exists:\n break\n results.append((code_piece, figures))\n\n if all_exists:\n return results\n\n # We didn't find the files, so build them\n\n results = []\n if context:\n ns = plot_context\n else:\n ns = {}\n\n if context_reset:\n plot_context.clear()\n\n close_figs = not context or close_figs\n\n for i, code_piece in enumerate(code_pieces):\n\n if not context:\n pass\n elif close_figs:\n pass\n\n fig_objects = run_code(code_piece, code_path, ns, function_name, fig_vars)\n\n figures = []\n for j, fig_obj in enumerate(fig_objects):\n if len(fig_objects) == 1 and len(code_pieces) == 1:\n fig = FigureFile(output_base, output_dir)\n elif len(code_pieces) == 1:\n fig = FigureFile(\"%s_%02d\" % (output_base, j), output_dir)\n else:\n fig = FigureFile(\"%s_%02d_%02d\" % (output_base, i, j), output_dir)\n figures.append(fig)\n for fmt, dpi in formats:\n try:\n save_plotly_figure(fig_obj, fig.filename(fmt))\n except Exception as err:\n raise PlotError(traceback.format_exc()) from err\n fig.formats.append(fmt)\n\n results.append((code_piece, figures))\n\n if not context:\n pass\n\n return results\n\n\ndef run(arguments, content, options, state_machine, state, lineno):\n document = state_machine.document\n config = document.settings.env.config\n nofigs = \"nofigs\" in options\n\n formats = get_plot_formats(config)\n default_fmt = formats[0][0]\n\n options_copy = copy.deepcopy(options)\n\n options.setdefault(\"include-source\", config.plotly_include_source)\n options.setdefault(\"iframe-width\", config.plotly_iframe_width)\n options.setdefault(\"iframe-height\", config.plotly_iframe_height)\n keep_context = \"context\" in options\n context_opt = None if not keep_context else options[\"context\"]\n\n rst_file = document.attributes[\"source\"]\n rst_dir = os.path.dirname(rst_file)\n\n if len(arguments):\n if not config.plotly_basedir:\n source_file_name = os.path.join(setup.app.builder.srcdir, directives.uri(arguments[0]))\n else:\n source_file_name = os.path.join(\n setup.confdir, config.plotly_basedir, directives.uri(arguments[0])\n )\n\n # If there is content, it will be passed as a caption.\n caption = \"\\n\".join(content)\n\n # Enforce unambiguous use of captions.\n if \"caption\" in options:\n if caption:\n raise ValueError(\n \"Caption specified in both content and options.\" \" Please remove ambiguity.\"\n )\n # Use caption option\n caption = options[\"caption\"]\n\n # If the optional function name is provided, use it\n if len(arguments) == 2:\n function_name = arguments[1]\n else:\n function_name = None\n\n code = Path(source_file_name).read_text(encoding=\"utf-8\")\n output_base = os.path.basename(source_file_name)\n else:\n source_file_name = rst_file\n code = textwrap.dedent(\"\\n\".join(map(str, content)))\n counter = document.attributes.get(\"_plot_counter\", 0) + 1\n document.attributes[\"_plot_counter\"] = counter\n base, ext = os.path.splitext(os.path.basename(source_file_name))\n output_base = \"%s-%d.py\" % (base, counter)\n function_name = None\n caption = options.get(\"caption\", \"\")\n\n base, source_ext = os.path.splitext(output_base)\n if source_ext in (\".py\", \".rst\", \".txt\"):\n output_base = base\n else:\n source_ext = \"\"\n\n # ensure that LaTeX includegraphics doesn't choke in foo.bar.pdf filenames\n output_base = output_base.replace(\".\", \"-\")\n\n # is it in doctest format?\n is_doctest = contains_doctest(code)\n if \"format\" in options:\n if options[\"format\"] == \"python\":\n is_doctest = False\n else:\n is_doctest = True\n\n # determine output directory name fragment\n source_rel_name = relpath(source_file_name, setup.confdir)\n source_rel_dir = os.path.dirname(source_rel_name)\n while source_rel_dir.startswith(os.path.sep):\n source_rel_dir = source_rel_dir[1:]\n\n # build_dir: where to place output files (temporarily)\n build_dir = os.path.join(\n os.path.dirname(setup.app.doctreedir), \"plot_directive\", source_rel_dir\n )\n # get rid of .. in paths, also changes pathsep\n # see note in Python docs for warning about symbolic links on Windows.\n # need to compare source and dest paths at end\n build_dir = os.path.normpath(build_dir)\n\n if not os.path.exists(build_dir):\n os.makedirs(build_dir)\n\n # output_dir: final location in the builder's directory\n dest_dir = os.path.abspath(os.path.join(setup.app.builder.outdir, source_rel_dir))\n if not os.path.exists(dest_dir):\n os.makedirs(dest_dir) # no problem here for me, but just use built-ins\n\n # how to link to files from the RST file\n dest_dir_link = os.path.join(relpath(setup.confdir, rst_dir), source_rel_dir).replace(\n os.path.sep, \"/\"\n )\n try:\n build_dir_link = relpath(build_dir, rst_dir).replace(os.path.sep, \"/\")\n except ValueError:\n # on Windows, relpath raises ValueError when path and start are on\n # different mounts/drives\n build_dir_link = build_dir\n source_link = dest_dir_link + \"/\" + output_base + source_ext\n\n # make figures\n try:\n results = render_figures(\n code,\n source_file_name,\n build_dir,\n output_base,\n keep_context,\n function_name,\n config,\n context_reset=context_opt == \"reset\",\n close_figs=context_opt == \"close-figs\",\n fig_vars=options.get(\"fig-vars\"),\n )\n errors = []\n except PlotError as err:\n reporter = state.memo.reporter\n sm = reporter.system_message(\n 2,\n \"Exception occurred in plotting {}\\n from {}:\\n{}\".format(\n output_base, source_file_name, err\n ),\n line=lineno,\n )\n results = [(code, [])]\n errors = [sm]\n\n # Properly indent the caption\n caption = \"\\n\".join(\" \" + line.strip() for line in caption.split(\"\\n\"))\n\n # generate output restructuredtext\n total_lines = []\n for j, (code_piece, figures) in enumerate(results):\n if options[\"include-source\"]:\n if is_doctest:\n lines = [\"\", *code_piece.splitlines()]\n else:\n lines = [\n \".. code-block:: python\",\n \"\",\n *textwrap.indent(code_piece, \" \").splitlines(),\n ]\n source_code = \"\\n\".join(lines)\n else:\n source_code = \"\"\n\n if nofigs:\n figures = []\n\n opts = [\n \":%s: %s\" % (key, val)\n for key, val in options.items()\n if key in (\"alt\", \"height\", \"width\", \"scale\", \"align\", \"class\")\n ]\n\n # Not-None src_link signals the need for a source link in the generated\n # html\n if j == 0 and config.plotly_html_show_source_link:\n src_link = source_link\n else:\n src_link = None\n\n if config.plotly_include_directive_source:\n directive_source = create_directive_block(\"plotly\", arguments, options_copy, content)\n directive_source = create_code_block(directive_source, \"text\")\n else:\n directive_source = \"\"\n\n result = jinja2.Template(config.plotly_template or TEMPLATE).render(\n directive_source=directive_source,\n default_fmt=default_fmt,\n dest_dir=dest_dir_link,\n build_dir=build_dir_link,\n source_link=src_link,\n multi_figure=len(figures) > 1,\n options=opts,\n figures=figures,\n iframe_width=options[\"iframe-width\"],\n iframe_height=options[\"iframe-height\"],\n source_code=source_code,\n html_show_formats=config.plotly_html_show_formats and len(figures),\n caption=caption,\n )\n\n total_lines.extend(result.split(\"\\n\"))\n total_lines.extend(\"\\n\")\n\n if total_lines:\n state_machine.insert_input(total_lines, source=source_file_name)\n\n # copy image files to builder's output directory, if necessary\n Path(dest_dir).mkdir(parents=True, exist_ok=True)\n\n for code_piece, figures in results:\n for fig in figures:\n for fn in fig.filenames():\n destfig = os.path.join(dest_dir, os.path.basename(fn))\n if fn != destfig:\n shutil.copyfile(fn, destfig)\n\n # copy script (if necessary)\n Path(dest_dir, output_base + source_ext).write_text(\n unescape_doctest(code) if source_file_name == rst_file else code,\n encoding=\"utf-8\",\n )\n\n return errors" }, { "path": "_ext/symlink.py", "content": "from docutils import nodes\nfrom docutils.parsers.rst import Directive, directives\n\nimport os, sys\n\ndef remove_symlink_handler(app, exception):\n dst = './src'\n \n if os.path.exists(dst):\n if os.path.isdir(dst):\n if os.path.islink(dst):\n os.unlink(dst)\n else:\n shutil.rmtree(dst)\n else:\n if os.path.islink(dst):\n os.unlink(dst)\n else:\n os.remove(dst)\n\ndef setup(app):\n app.connect('build-finished', remove_symlink_handler)\n src = '../src'\n dst = './src'\n\n # This creates a symbolic link on python in tmp directory\n\n if os.path.exists(dst):\n if os.path.isdir(dst):\n if os.path.islink(dst):\n os.unlink(dst)\n else:\n shutil.rmtree(dst)\n else:\n if os.path.islink(dst):\n os.unlink(dst)\n else:\n os.remove(dst)\n\n os.symlink(src, dst)\n\n return {\n 'version': '1.0',\n 'parallel_read_safe': True,\n 'parallel_write_safe': True,\n }" }, { "path": "_static/css/custom.css", "content": ".xxtable-smaller-font-size p, strong {\n font-size:0.9em; \n}\n\n.ablog-post-title p {\n font-size:0.9em; \n}\n\n.ablog-post p {\n font-size:0.9em; \n}\n\n.sphinx-design-class-title-small {\n font-size:0.9em; \n}\n\n.sphinx-design-class-title-med {\n font-size:1em; \n}\n\n.sphinx-design-class-body-small {\n font-size:0.9em; \n}\n\n\nh1{font-size:2em;}\nh2{font-size:1.5em;}\nh3{font-size:1.3em;}\nh4{font-size:1.2em;}\ndiv.topic {\n font-size:0.85em; \n}\nli.toctree-l1 {\n font-size:0.95em; \n}\nth , tr, td {\n white-space: normal !important;\n}\nth {\n font-size:0.90em; \n}\n\n.ff th , tr, td{\n font-size:0.90em; \n white-space: normal !important;\n}\n\n.ff div.section.p {\n font-size:0.8em; \n}\n\nhr {\n border-color: #0000DD;\n height: 2px;\n}\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "_static/css/custom.css.new", "content": "\n.table-smaller-font-size p, strong {\n font-size: 90%; \n}\n\n \n\ntd, th , tr {\n white-space: normal !important;\n}\n\n/* Fixes the size of the RTD flyout */\n/* .rst-versions {\n width: 320px !important;\n} */\n\n/* Content area color */\n.wy-nav-content {\n background: #ffffff;\n}\n\n/* Scroll Bar*/\n.wy-side-scroll {\n width: auto;\n overflow-y: auto;\n margin-top: 0px;\n}\n\n/* width of the side panel */\n.wy-nav-side {\n width: 320px;\n}\n\n/* content section full screen */\n.wy-nav-content {\n max-width: none; \n}\n\n/* set color of left side bar */\n.wy-nav-side,.wy-side-nav-search,.wy-nav-top {\n /*background: #0079c1; /*005eb8 */\n background: #ffffff;\n}\n\n/* Change caption color to be more legible */\n.wy-menu > .caption > span.caption-text {\n color: #000000;\n font-size: 20px;\n}\n\n/* Change the version color to match caption color */\n.wy-side-nav-search>div.version {\n color: #000000;\n}\n\n/* Get rid of that ugly yellow highlight color and replace with something more appealing to the eye */\n.highlight .hll {\n background-color: #ffffff;\n}\n/* \n@media screen and (max-width: 768px) {\n .wy-nav-content-wrap {\n margin-left: 0px;\n }\n .wy-nav-side {\n width: 500px;\n }\n} */\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "_templates/recentposts.html", "content": "{% if ablog %}\n

\n {{ gettext('Recent Posts') }}\n

\n\n{% endif %}\n" }, { "path": "_templates/search-field.html", "content": "\n\n
\n Search Engine:\n Default \n \n Google\n
\n\n\n\n\n\n" }, { "path": "_templates/search-google.html", "content": "{%- extends \"page.html\" %}\n{# Over-ride the body to be custom search structure we want #}\n{% block docs_body %}\n
\n

{{ _(\"Search\") }}

\n \n\n

{{ _('Search Results') }}

\n\n \n
\n
\n
\n\n\n{% endblock docs_body %}\n{# Below sections just re-create the behavior of Sphinx default search #}\n{# Page metadata #}\n{%- block htmltitle -%}\n {{ _(\"Search\") }} - {{ title or docstitle }}\n{%- endblock htmltitle -%}\n{# Manually include the search JS that Sphinx includes #}\n{% block scripts -%}\n {{ super() }}\n{%- endblock scripts %}" }, { "path": "_templates/search.html", "content": "{%- extends \"page.html\" %}\n{# Over-ride the body to be custom search structure we want #}\n{% block docs_body %}\n
\n

{{ _(\"Search\") }}

\n \n
\n
\n \n{% endblock docs_body %}\n{# Below sections just re-create the behavior of Sphinx default search #}\n{# Page metadata #}\n{%- block htmltitle -%}\n {{ _(\"Search\") }} - {{ title or docstitle }}\n{%- endblock htmltitle -%}\n{# Manually include the search JS that Sphinx includes #}\n{% block scripts -%}\n {{ super() }}\n \n \n \n{%- endblock scripts %}" }, { "path": "_utilities/JIRA_SETUP_QUICKSTART.md", "content": "# Jira Integration Quick Start\n\n## Prerequisites Check\n\nRun these commands to verify you have everything installed:\n\n```bash\n# Check AWS CLI\naws --version\n\n# Check ada credentials tool\nada --version\n\n# Check Python 3\npython3 --version\n\n# Check if uvx is available (for MCP server)\nuvx --version\n```\n\nIf any are missing, install them:\n```bash\n# AWS CLI\nbrew install awscli\n\n# ada credentials tool\ntoolbox install ada\n\n# uv (includes uvx)\nbrew install uv\n```\n\n## One-Time Setup\n\n### 1. Configure Ada Credentials\n\n```bash\nada credentials setup\n```\n\nWhen prompted:\n- **Account**: 621547421844\n- **Role**: Admin\n- **Profile name**: kaena\n\n### 2. Add Kaena Profile to AWS Config\n\n```bash\necho '[profile kaena]\ncredential_process='$HOME'/.toolbox/bin/ada credentials print --profile=kaena' >> ~/.aws/config\n```\n\n### 3. Run the Setup Script\n\n```bash\ncd /path/to/aws-neuron-sdk-staging\nchmod +x _utilities/setup_jira_token.sh\n./_utilities/setup_jira_token.sh\n```\n\nThis script will:\n- Fetch the Jira API token from AWS Secrets Manager\n- Update your MCP configuration with the token\n- Verify everything is set up correctly\n\n### 4. Restart Kiro\n\nAfter running the setup script, restart Kiro CLI to load the new MCP server.\n\n## Using Jira in Kiro\n\nOnce set up, you can use Kiro Powers to interact with Jira:\n\n```bash\n# In Kiro CLI, check available powers\nkiro powers list\n\n# Look for Atlassian/Jira related tools\n```\n\n## Manual Verification\n\nTo manually verify the setup worked:\n\n```bash\n# Check MCP config has Jira server\ncat ~/.kiro/settings/mcp.json | grep -A 10 atlassian-jira\n\n# Test AWS Secrets Manager access\nexport AWS_PROFILE=kaena\naws secretsmanager get-secret-value \\\n --secret-id NKI_JIRA_API_TOKEN \\\n --region us-west-2 \\\n --query SecretString \\\n --output text\n```\n\n## Troubleshooting\n\n### \"Error: Failed to fetch Jira API token\"\n\n1. Verify ada credentials are set up:\n ```bash\n ada credentials list\n ```\n\n2. Check AWS profile is configured:\n ```bash\n cat ~/.aws/config | grep -A 2 kaena\n ```\n\n3. Test AWS access:\n ```bash\n export AWS_PROFILE=kaena\n aws sts get-caller-identity\n ```\n\n### \"MCP server not loading\"\n\n1. Check uvx is installed:\n ```bash\n uvx --version\n ```\n\n2. Manually test the MCP server:\n ```bash\n uvx mcp-server-atlassian\n ```\n\n3. Check Kiro MCP logs (location varies by installation)\n\n## What's Next\n\nAfter setup, you can:\n- Query NKI Jira tickets\n- Create new tickets\n- Update ticket status\n- Search and filter tickets\n- Generate reports\n\nSee the full guide at `.kiro/steering/jira.md` for detailed usage examples.\n" }, { "path": "_utilities/add_meta.py", "content": "#!/usr/bin/env python3\n\"\"\"Add missing .. meta:: blocks with :description:, :keywords:, and :date-modified: to .rst files.\"\"\"\n\nimport os\nimport re\nimport sys\nfrom pathlib import Path\n\nTODAY = \"2026-03-13\"\n\n# Map file paths to sensible descriptions/keywords based on content\ndef infer_meta(filepath: str, content: str) -> dict:\n \"\"\"Infer description and keywords from file path and content.\"\"\"\n rel = filepath.replace(\"frameworks/\", \"\")\n \n # Extract title from RST\n title = \"\"\n lines = content.split(\"\\n\")\n title_chars = set(\"=-~^\\\"'`#*+_.\")\n for i, line in enumerate(lines):\n stripped = line.rstrip()\n if (len(stripped) >= 3 and len(set(stripped)) == 1 \n and stripped[0] in title_chars and i > 0):\n candidate = lines[i-1].strip()\n if candidate and not candidate.startswith(\"..\"):\n title = candidate\n break\n \n # Build description from title or path\n if title:\n desc = f\"{title} - AWS Neuron SDK documentation\"\n else:\n desc = f\"AWS Neuron SDK documentation for {os.path.basename(filepath).replace('.rst', '').replace('-', ' ')}\"\n \n # Build keywords from path components\n kw_parts = set()\n if \"torch\" in rel:\n kw_parts.update([\"PyTorch\", \"AWS Neuron\"])\n if \"neuronx\" in rel:\n kw_parts.update([\"torch-neuronx\", \"Trainium\", \"Inferentia\"])\n if \"jax\" in rel:\n kw_parts.update([\"JAX\", \"AWS Neuron\", \"JAX NeuronX\"])\n if \"training\" in rel.lower():\n kw_parts.add(\"training\")\n if \"inference\" in rel.lower():\n kw_parts.add(\"inference\")\n if \"setup\" in rel.lower() or \"install\" in rel.lower() or \"update\" in rel.lower():\n kw_parts.add(\"setup\")\n if \"tutorial\" in rel.lower():\n kw_parts.add(\"tutorials\")\n if \"api\" in rel.lower():\n kw_parts.add(\"API reference\")\n if \"profil\" in rel.lower():\n kw_parts.add(\"profiling\")\n if \"troubleshoot\" in rel.lower():\n kw_parts.add(\"troubleshooting\")\n if \"debug\" in rel.lower():\n kw_parts.add(\"debugging\")\n if not kw_parts:\n kw_parts.update([\"AWS Neuron\", \"machine learning\"])\n \n keywords = \", \".join(sorted(kw_parts))\n \n return {\"description\": desc, \"keywords\": keywords}\n\n\ndef has_meta_field(content: str, field: str) -> bool:\n \"\"\"Check if a .. meta:: block contains a specific field.\"\"\"\n return bool(re.search(rf\"^\\s+:{field}:\", content, re.MULTILINE))\n\n\ndef process_file(filepath: str, dry_run: bool = False):\n \"\"\"Process a single .rst file to ensure it has complete meta block.\"\"\"\n with open(filepath, \"r\", encoding=\"utf-8\", errors=\"replace\") as f:\n content = f.read()\n \n # Skip include-only fragments (no title, very short)\n if len(content.strip()) < 50:\n print(f\" SKIP (fragment): {filepath}\")\n return False\n \n has_meta = \".. meta::\" in content\n has_desc = has_meta_field(content, \"description\")\n has_kw = has_meta_field(content, \"keywords\")\n has_date = has_meta_field(content, \"date-modified\")\n \n if has_desc and has_kw and has_date:\n print(f\" OK (complete): {filepath}\")\n return False\n \n meta = infer_meta(filepath, content)\n \n if has_meta:\n # Meta block exists but missing fields — add them\n missing = []\n if not has_desc:\n missing.append(f\" :description: {meta['description']}\")\n if not has_kw:\n missing.append(f\" :keywords: {meta['keywords']}\")\n if not has_date:\n missing.append(f\" :date-modified: {TODAY}\")\n \n insert_text = \"\\n\".join(missing)\n \n # Find the end of the existing meta block (last line starting with :field:)\n lines = content.split(\"\\n\")\n meta_start = -1\n meta_last_field = -1\n for i, line in enumerate(lines):\n if line.strip() == \".. meta::\":\n meta_start = i\n elif meta_start >= 0 and re.match(r\"\\s+:\\w\", line):\n meta_last_field = i\n elif meta_start >= 0 and meta_last_field >= 0 and not line.strip().startswith(\":\") and not (line.strip() and not line[0].isspace()):\n break\n \n if meta_last_field >= 0:\n lines.insert(meta_last_field + 1, insert_text)\n new_content = \"\\n\".join(lines)\n else:\n # Fallback: insert after .. meta:: line\n new_content = content.replace(\".. meta::\", f\".. meta::\\n{insert_text}\", 1)\n else:\n # No meta block at all — add one at the top (after any labels)\n lines = content.split(\"\\n\")\n insert_idx = 0\n \n # Skip leading labels (.. _label:) and blank lines\n for i, line in enumerate(lines):\n stripped = line.strip()\n if stripped.startswith(\".. _\") and stripped.endswith(\":\"):\n insert_idx = i + 1\n elif stripped == \"\" and i <= insert_idx + 1:\n insert_idx = i + 1\n else:\n break\n \n meta_block = (\n f\"\\n.. meta::\\n\"\n f\" :description: {meta['description']}\\n\"\n f\" :keywords: {meta['keywords']}\\n\"\n f\" :date-modified: {TODAY}\\n\\n\"\n )\n \n lines.insert(insert_idx, meta_block)\n new_content = \"\\n\".join(lines)\n \n action = \"UPDATE\" if has_meta else \"ADD\"\n fields = []\n if not has_desc: fields.append(\"description\")\n if not has_kw: fields.append(\"keywords\")\n if not has_date: fields.append(\"date-modified\")\n print(f\" {action} ({', '.join(fields)}): {filepath}\")\n \n if not dry_run:\n with open(filepath, \"w\", encoding=\"utf-8\") as f:\n f.write(new_content)\n \n return True\n\n\ndef main():\n import argparse\n parser = argparse.ArgumentParser(description=\"Add meta blocks to .rst files\")\n parser.add_argument(\"directory\", default=\"frameworks\", nargs=\"?\")\n parser.add_argument(\"--dry-run\", action=\"store_true\", help=\"Show what would change without writing\")\n args = parser.parse_args()\n \n root = Path(args.directory)\n rst_files = sorted(root.rglob(\"*.rst\"))\n \n print(f\"Scanning {len(rst_files)} .rst files in {root}/:\")\n changed = 0\n for f in rst_files:\n if process_file(str(f), dry_run=args.dry_run):\n changed += 1\n \n print(f\"\\n{'Would change' if args.dry_run else 'Changed'} {changed} file(s).\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "_utilities/audit_frameworks.py", "content": "#!/usr/bin/env python3\n\"\"\"\nAudit script for the /frameworks directory of the AWS Neuron SDK documentation.\n\nDetects orphaned pages (not referenced by any toctree, :doc:, :ref:, or\n.. include:: directive) and stale pages (containing outdated references).\n\nUsage:\n python3 _utilities/audit_frameworks.py --root . --output audit-report.md\n\"\"\"\n\nimport argparse\nimport os\nimport re\nfrom pathlib import Path\n\n\n# ---------------------------------------------------------------------------\n# Reference extraction helpers\n# ---------------------------------------------------------------------------\n\n# Regex patterns for RST directives and roles\nTOCTREE_BLOCK_RE = re.compile(r\"^\\.\\.\\s+toctree::\", re.MULTILINE)\nDOC_ROLE_RE = re.compile(r\":doc:`(?:[^<`]*<)?(/[^>`]+|[^>`/][^>`]*)`\")\nREF_ROLE_RE = re.compile(r\":ref:`(?:[^<`]*<)?([^>`]+)`\")\nINCLUDE_RE = re.compile(r\"^\\.\\.\\s+include::\\s+(.+)$\", re.MULTILINE)\nLABEL_RE = re.compile(r\"^\\.\\.\\s+_([a-zA-Z0-9_-]+)\\s*:\", re.MULTILINE)\n\n\ndef _resolve_path(ref: str, referencing_file: Path, root: Path) -> str | None:\n \"\"\"Resolve a toctree/doc/include reference to a repo-relative path.\"\"\"\n ref = ref.strip()\n if not ref:\n return None\n\n # Absolute path (starts with /)\n if ref.startswith(\"/\"):\n resolved = ref.lstrip(\"/\")\n else:\n # Relative to the directory of the referencing file\n ref_dir = referencing_file.parent.relative_to(root)\n resolved = str(ref_dir / ref)\n\n # Normalise (collapse ..)\n resolved = os.path.normpath(resolved)\n return resolved\n\n\ndef _resolve_to_files(base: str, root: Path) -> list[str]:\n \"\"\"Given a resolved base path, return candidate file paths that exist.\"\"\"\n candidates = []\n # Direct file match (already has extension)\n if (root / base).is_file():\n candidates.append(base)\n return candidates\n\n # Try common extensions\n for ext in (\".rst\", \".ipynb\", \".txt\"):\n p = base + ext\n if (root / p).is_file():\n candidates.append(p)\n\n # Could be a directory with index.rst\n idx = os.path.join(base, \"index.rst\")\n if (root / idx).is_file():\n candidates.append(idx)\n\n return candidates\n\n\ndef extract_toctree_entries(content: str, filepath: Path, root: Path) -> set[str]:\n \"\"\"Extract all file paths referenced in toctree directives.\"\"\"\n referenced: set[str] = set()\n lines = content.split(\"\\n\")\n i = 0\n while i < len(lines):\n if TOCTREE_BLOCK_RE.match(lines[i]):\n # Skip toctree options (lines starting with : or blank within indent)\n i += 1\n # Skip blank lines and option lines\n while i < len(lines):\n stripped = lines[i].strip()\n if stripped == \"\" or stripped.startswith(\":\"):\n i += 1\n continue\n break\n # Now read toctree entries (indented non-empty lines)\n while i < len(lines):\n line = lines[i]\n stripped = line.strip()\n if stripped == \"\":\n i += 1\n continue\n # Check if still indented (part of toctree body)\n if line[0] in (\" \", \"\\t\"):\n # Entry may have a title: \"Title \" or just \"path\"\n entry = stripped\n m = re.match(r\".*<(.+)>\", entry)\n if m:\n entry = m.group(1).strip()\n # Resolve the path\n resolved = _resolve_path(entry, filepath, root)\n if resolved:\n for f in _resolve_to_files(resolved, root):\n referenced.add(f)\n i += 1\n else:\n break\n else:\n i += 1\n return referenced\n\n\ndef extract_doc_refs(content: str, filepath: Path, root: Path) -> set[str]:\n \"\"\"Extract all file paths referenced via :doc: roles.\"\"\"\n referenced: set[str] = set()\n for m in DOC_ROLE_RE.finditer(content):\n ref = m.group(1).strip()\n resolved = _resolve_path(ref, filepath, root)\n if resolved:\n for f in _resolve_to_files(resolved, root):\n referenced.add(f)\n return referenced\n\n\ndef extract_include_refs(content: str, filepath: Path, root: Path) -> set[str]:\n \"\"\"Extract all file paths referenced via .. include:: directives.\"\"\"\n referenced: set[str] = set()\n for m in INCLUDE_RE.finditer(content):\n ref = m.group(1).strip()\n resolved = _resolve_path(ref, filepath, root)\n if resolved:\n for f in _resolve_to_files(resolved, root):\n referenced.add(f)\n return referenced\n\n\ndef extract_ref_labels(content: str) -> set[str]:\n \"\"\"Extract all :ref: label targets from content.\"\"\"\n return set(m.group(1) for m in REF_ROLE_RE.finditer(content))\n\n\ndef extract_label_definitions(content: str) -> set[str]:\n \"\"\"Extract all label definitions (.. _label:) from content.\"\"\"\n return set(m.group(1) for m in LABEL_RE.finditer(content))\n\n\n# ---------------------------------------------------------------------------\n# Orphan detection\n# ---------------------------------------------------------------------------\n\ndef find_all_framework_files(root: Path) -> tuple[set[str], set[str], set[str]]:\n \"\"\"Find all .rst, .ipynb, and .txt files under frameworks/.\n\n Returns (rst_files, ipynb_files, txt_files) as repo-relative paths.\n \"\"\"\n rst_files: set[str] = set()\n ipynb_files: set[str] = set()\n txt_files: set[str] = set()\n fw_dir = root / \"frameworks\"\n if not fw_dir.is_dir():\n return rst_files, ipynb_files, txt_files\n for p in fw_dir.rglob(\"*\"):\n if not p.is_file():\n continue\n rel = str(p.relative_to(root))\n if \"__pycache__\" in rel:\n continue\n if p.suffix == \".rst\":\n rst_files.add(rel)\n elif p.suffix == \".ipynb\":\n ipynb_files.add(rel)\n elif p.suffix == \".txt\":\n txt_files.add(rel)\n return rst_files, ipynb_files, txt_files\n\n\ndef collect_all_references(root: Path) -> tuple[set[str], set[str], set[str]]:\n \"\"\"Scan ALL .rst and .txt files in the repo to collect references.\n\n Returns (toctree_and_doc_refs, include_refs, ref_labels_used).\n We scan the entire repo (not just /frameworks) so that references\n from root index.rst, setup/, about-neuron/, etc. are captured.\n \"\"\"\n toctree_doc_refs: set[str] = set()\n include_refs: set[str] = set()\n ref_labels_used: set[str] = set()\n\n # Directories to skip entirely\n skip_dirs = {\"_build\", \".git\", \"venv\", \".venv\", \"__pycache__\", \".kiro\",\n \".vscode\", \".github\", \"node_modules\", \"_backup-rn\"}\n\n for ext in (\"*.rst\", \"*.txt\"):\n for p in root.rglob(ext):\n # Skip files in excluded directories\n rel = str(p.relative_to(root))\n parts = Path(rel).parts\n if any(part in skip_dirs for part in parts):\n continue\n try:\n content = p.read_text(encoding=\"utf-8\", errors=\"replace\")\n except Exception:\n continue\n\n toctree_doc_refs |= extract_toctree_entries(content, p, root)\n toctree_doc_refs |= extract_doc_refs(content, p, root)\n include_refs |= extract_include_refs(content, p, root)\n ref_labels_used |= extract_ref_labels(content)\n\n return toctree_doc_refs, include_refs, ref_labels_used\n\n\ndef build_label_to_file_map(root: Path) -> dict[str, str]:\n \"\"\"Build a mapping from :ref: label -> repo-relative file path.\n\n Only scans files under frameworks/ since we only need to know\n which framework files are referenced via :ref:.\n \"\"\"\n label_map: dict[str, str] = {}\n fw_dir = root / \"frameworks\"\n if not fw_dir.is_dir():\n return label_map\n for p in fw_dir.rglob(\"*.rst\"):\n rel = str(p.relative_to(root))\n try:\n content = p.read_text(encoding=\"utf-8\", errors=\"replace\")\n except Exception:\n continue\n for label in extract_label_definitions(content):\n label_map[label] = rel\n return label_map\n\n\ndef detect_orphans(root: Path) -> list[dict]:\n \"\"\"Detect orphaned pages under /frameworks.\n\n Returns a list of dicts with keys: path, type, reason, action.\n \"\"\"\n rst_files, ipynb_files, txt_files = find_all_framework_files(root)\n toctree_doc_refs, include_refs, ref_labels_used = collect_all_references(root)\n label_map = build_label_to_file_map(root)\n\n # Files referenced via :ref: labels\n ref_referenced_files: set[str] = set()\n for label in ref_labels_used:\n if label in label_map:\n ref_referenced_files.add(label_map[label])\n\n # All referenced content files (rst + ipynb)\n all_content_refs = toctree_doc_refs | ref_referenced_files\n # All referenced include files (txt)\n all_include_refs = include_refs\n\n orphans: list[dict] = []\n\n # Check .rst and .ipynb files against toctree/doc/ref references\n for f in sorted(rst_files | ipynb_files):\n if f not in all_content_refs and f not in all_include_refs:\n ext = Path(f).suffix\n orphans.append({\n \"path\": f,\n \"type\": ext,\n \"reason\": \"Not in any toctree or cross-reference\",\n \"action\": \"Delete\",\n })\n\n # Check .txt files against include references only\n for f in sorted(txt_files):\n if f not in all_include_refs:\n orphans.append({\n \"path\": f,\n \"type\": \".txt (include fragment)\",\n \"reason\": \"Not referenced by any .. include:: directive\",\n \"action\": \"Delete\",\n })\n\n return orphans\n\n\n# ---------------------------------------------------------------------------\n# Stale page detection\n# ---------------------------------------------------------------------------\n\n# Staleness indicator patterns\nSTALE_OS_RE = re.compile(\n r\"Ubuntu\\s+18\\.04|Ubuntu\\s+20\\.04|Amazon\\s+Linux\\s+2(?!\\s*023)(?!\\s*\\d{3})\\b\",\n re.IGNORECASE,\n)\nSTALE_PYTHON_RE = re.compile(\n r\"Python\\s+3\\.[0-9](?!\\d)\\b\", # matches Python 3.0 through 3.9\n)\nSTALE_SDK_RE = re.compile(r\"Neuron\\s+SDK\\s+2\\.(\\d+)\")\nTORCH_NEURON_SETUP_RE = re.compile(\n r\"torch-neuron.*(?:setup|install|update)\",\n re.IGNORECASE,\n)\nNEURON_CC_RE = re.compile(r\"\\bneuron-cc\\b\")\n\n\ndef _check_stale_python(content: str) -> list[str]:\n \"\"\"Find references to Python versions below 3.10.\"\"\"\n indicators = []\n for m in STALE_PYTHON_RE.finditer(content):\n ver_str = m.group(0)\n # Extract minor version\n minor = int(ver_str.split(\".\")[-1])\n if minor < 10:\n indicators.append(ver_str)\n return list(set(indicators))\n\n\ndef _check_stale_sdk(content: str) -> list[str]:\n \"\"\"Find references to Neuron SDK versions older than 2.20.\"\"\"\n indicators = []\n for m in STALE_SDK_RE.finditer(content):\n ver = int(m.group(1))\n if ver < 20:\n indicators.append(m.group(0))\n return list(set(indicators))\n\n\ndef _check_stale_os(content: str) -> list[str]:\n \"\"\"Find references to unsupported OS versions.\"\"\"\n return list(set(m.group(0) for m in STALE_OS_RE.finditer(content)))\n\n\ndef _check_torch_neuron_unsupported_os(content: str) -> list[str]:\n \"\"\"Flag torch-neuron setup/update instructions for unsupported OS.\"\"\"\n indicators = []\n if TORCH_NEURON_SETUP_RE.search(content):\n os_refs = _check_stale_os(content)\n if os_refs:\n indicators.append(\n f\"torch-neuron setup/update with unsupported OS: {', '.join(os_refs)}\"\n )\n return indicators\n\n\ndef _check_neuron_cc(content: str) -> list[str]:\n \"\"\"Flag deprecated neuron-cc references.\"\"\"\n if NEURON_CC_RE.search(content):\n return [\"References deprecated neuron-cc compiler\"]\n return []\n\n\ndef detect_stale_pages(root: Path) -> list[dict]:\n \"\"\"Detect stale pages under /frameworks.\n\n Returns a list of dicts with keys: path, indicators, recommendation.\n \"\"\"\n stale: list[dict] = []\n fw_dir = root / \"frameworks\"\n if not fw_dir.is_dir():\n return stale\n\n for p in fw_dir.rglob(\"*\"):\n if not p.is_file():\n continue\n if p.suffix not in (\".rst\", \".txt\"):\n continue\n rel = str(p.relative_to(root))\n try:\n content = p.read_text(encoding=\"utf-8\", errors=\"replace\")\n except Exception:\n continue\n\n indicators: list[str] = []\n indicators.extend(_check_stale_os(content))\n indicators.extend(_check_stale_python(content))\n indicators.extend(_check_stale_sdk(content))\n indicators.extend(_check_torch_neuron_unsupported_os(content))\n indicators.extend(_check_neuron_cc(content))\n\n if indicators:\n # Determine recommendation\n is_archival = (\n \"mxnet-neuron/\" in rel\n or \"tensorflow/\" in rel\n or (\"torch-neuron/\" in rel and \"torch-neuronx/\" not in rel)\n )\n if is_archival:\n rec = \"Will be archived\"\n else:\n rec = \"Update or archive\"\n stale.append({\n \"path\": rel,\n \"indicators\": \"; \".join(sorted(set(indicators))),\n \"recommendation\": rec,\n })\n\n return sorted(stale, key=lambda x: x[\"path\"])\n\n\n# ---------------------------------------------------------------------------\n# Report generation\n# ---------------------------------------------------------------------------\n\ndef generate_report(orphans: list[dict], stale: list[dict]) -> str:\n \"\"\"Generate the audit report as Markdown.\"\"\"\n lines: list[str] = []\n lines.append(\"# Frameworks Audit Report\\n\")\n\n # Orphaned pages\n lines.append(\"## Orphaned Pages\\n\")\n if orphans:\n lines.append(\"| File Path | Type | Reason | Action |\")\n lines.append(\"|---|---|---|---|\")\n for o in orphans:\n lines.append(\n f\"| {o['path']} | {o['type']} | {o['reason']} | {o['action']} |\"\n )\n else:\n lines.append(\"No orphaned pages detected.\\n\")\n\n lines.append(\"\")\n\n # Stale pages\n lines.append(\"## Stale Pages\\n\")\n if stale:\n lines.append(\"| File Path | Staleness Indicators | Recommendation |\")\n lines.append(\"|---|---|---|\")\n for s in stale:\n lines.append(\n f\"| {s['path']} | {s['indicators']} | {s['recommendation']} |\"\n )\n else:\n lines.append(\"No stale pages detected.\\n\")\n\n lines.append(\"\")\n return \"\\n\".join(lines)\n\n\n# ---------------------------------------------------------------------------\n# CLI\n# ---------------------------------------------------------------------------\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Audit /frameworks for orphaned and stale pages.\"\n )\n parser.add_argument(\n \"--root\",\n default=\".\",\n help=\"Repository root directory (default: current directory)\",\n )\n parser.add_argument(\n \"--output\",\n default=\"audit-report.md\",\n help=\"Output file path for the audit report (default: audit-report.md)\",\n )\n args = parser.parse_args()\n\n root = Path(args.root).resolve()\n print(f\"Auditing frameworks under: {root}\")\n\n orphans = detect_orphans(root)\n print(f\"Found {len(orphans)} orphaned page(s).\")\n\n stale = detect_stale_pages(root)\n print(f\"Found {len(stale)} stale page(s).\")\n\n report = generate_report(orphans, stale)\n output_path = Path(args.output)\n if not output_path.is_absolute():\n output_path = root / output_path\n output_path.write_text(report, encoding=\"utf-8\")\n print(f\"Audit report written to: {output_path}\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "_utilities/check_urls.sh", "content": "#!/bin/bash\n\n# Output file\noutput_file=\"url_check_results.txt\"\n\n# Initialize counters\ntotal=0\nworking=0\nnot_found=0\nother=0\n\n# Create output file with header\necho \"URL Status Check Results\" > $output_file\necho \"=========================\" >> $output_file\necho \"\" >> $output_file\n\n# Read each URL from the file\nwhile read url; do\n # Skip empty lines\n if [ -z \"$url\" ]; then\n continue\n fi\n \n # Increment total counter\n ((total++))\n \n # Print progress\n echo \"Checking $total: $url\"\n \n # Use curl to check the URL status\n status_code=$(curl -s -o /dev/null -w \"%{http_code}\" \"$url\")\n \n # Check status code\n if [ \"$status_code\" -eq 200 ]; then\n echo \"✓ WORKING: $url\" >> $output_file\n ((working++))\n elif [ \"$status_code\" -eq 404 ]; then\n echo \"✗ NOT FOUND (404): $url\" >> $output_file\n ((not_found++))\n else\n echo \"? OTHER STATUS ($status_code): $url\" >> $output_file\n ((other++))\n fi\n \n # Small delay to avoid overwhelming the server\n sleep 0.1\n \ndone < old-nki-apis.txt\n\n# Write summary\necho \"\" >> $output_file\necho \"\" >> $output_file\necho \"Summary\" >> $output_file\necho \"=======\" >> $output_file\necho \"Total URLs checked: $total\" >> $output_file\necho \"Working URLs: $working\" >> $output_file\necho \"Not found (404) URLs: $not_found\" >> $output_file\necho \"Other status URLs: $other\" >> $output_file\n\necho \"URL check completed. Results saved to $output_file\"\n" }, { "path": "_utilities/create_sitemap.py", "content": "# v1.0 by dougeric 2025-09-30\n# Script to create sitemap.xml for Sphinx-generated docs; must be run at the root of the docs repo with venv\n\nimport os\nfrom pathlib import Path\nfrom datetime import datetime\n\ndef create_sitemap(root_dir, base_url):\n \"\"\"\n This function generates a sitemap.xml file for the given root directory and base URL.\n It recursively scans all .rst files in the root directory, excluding those in directories\n starting with \"_\". For each .rst file, it calculates the last modification time, converts\n the .rst path to the corresponding HTML path, and adds a entry to the sitemap in the\n format required by Google Search Console.\n \"\"\"\n sitemap = ['',\n '']\n \n for path in Path(root_dir).rglob('*.rst'):\n # Skip directories starting with \"_\"\n if any(part.startswith('_') for part in path.parts):\n continue\n \n # Convert .rst path to expected html path\n rel_path = path.relative_to(root_dir)\n html_path = str(rel_path).replace('.rst', '.html')\n \n # Get file modification time\n mod_time = datetime.fromtimestamp(os.path.getmtime(path))\n \n sitemap.append(f' ')\n sitemap.append(f' {base_url}/{html_path}')\n sitemap.append(f' {mod_time.strftime(\"%Y-%m-%d\")}')\n sitemap.append(f' ')\n \n sitemap.append('')\n return '\\n'.join(sitemap)\n\n# Call the function and write the result to sitemap.xml\nsitemap_content = create_sitemap('./', 'https://awsdocs-neuron.readthedocs-hosted.com/en/latest')\nwith open('sitemap.xml', 'w') as f:\n f.write(sitemap_content)\nprint(\"\\nsitemap.xml has been created.\\n\")" }, { "path": "_utilities/format_build_logs.py", "content": "#!/usr/bin/env python3\n\"\"\"\nFormat Sphinx Build Logs\n\nThis script checks for Python 3.9 and pip, creates a virtual environment,\nruns sphinx-build, and formats the build log as Markdown with separate\nsections for errors and warnings.\n\"\"\"\n\nimport os\nimport sys\nimport subprocess\nimport re\nimport datetime\nimport platform\nimport shutil\nfrom collections import Counter\nfrom pathlib import Path\n\ndef check_python_version():\n \"\"\"Check if Python 3.9 is installed.\"\"\"\n python_version = sys.version_info\n \n if python_version.major != 3 or python_version.minor != 9:\n print(\"Error: Python 3.9 is required.\")\n \n if platform.system() == \"Darwin\": # macOS\n print(\"To install Python 3.9 on macOS, visit: https://www.python.org/downloads/release/python-3913/\")\n print(\"Or use Homebrew: brew install python@3.9\")\n elif platform.system() == \"Windows\":\n print(\"To install Python 3.9 on Windows, visit: https://www.python.org/downloads/release/python-3913/\")\n else:\n print(\"Please install Python 3.9 from: https://www.python.org/downloads/release/python-3913/\")\n \n sys.exit(1)\n \n return True\n\ndef check_pip_installed():\n \"\"\"Check if pip is installed.\"\"\"\n try:\n subprocess.run([sys.executable, \"-m\", \"pip\", \"--version\"], \n check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n return True\n except subprocess.CalledProcessError:\n print(\"Error: pip is not installed.\")\n print(\"Please install pip: https://pip.pypa.io/en/stable/installation/\")\n sys.exit(1)\n\ndef find_repo_root():\n \"\"\"Find the root of the private-aws-neuron-sdk-staging repo.\"\"\"\n # Start with the current directory\n current_dir = Path.cwd()\n \n # Check if we're already in the repo root\n if current_dir.name == \"private-aws-neuron-sdk-staging\":\n return current_dir\n \n # Check parent directory\n parent_dir = current_dir.parent\n if parent_dir.name == \"private-aws-neuron-sdk-staging\":\n return parent_dir\n \n # Look for the repo in the current directory\n for item in current_dir.iterdir():\n if item.is_dir() and item.name == \"private-aws-neuron-sdk-staging\":\n return item\n \n # Look for the repo in the parent directory\n for item in parent_dir.iterdir():\n if item.is_dir() and item.name == \"private-aws-neuron-sdk-staging\":\n return item\n \n print(\"Error: Repository 'private-aws-neuron-sdk-staging' not found on local machine.\")\n sys.exit(1)\n\ndef setup_venv(repo_parent):\n \"\"\"Create and activate a Python 3.9 virtual environment.\"\"\"\n venv_path = repo_parent / \"venv\"\n \n # Create venv if it doesn't exist\n if not venv_path.exists():\n print(f\"Creating virtual environment at {venv_path}...\")\n try:\n subprocess.run([sys.executable, \"-m\", \"venv\", str(venv_path)], check=True)\n except subprocess.CalledProcessError as e:\n print(f\"Error creating virtual environment: {e}\")\n sys.exit(1)\n \n # Determine the path to the activate script\n if platform.system() == \"Windows\":\n activate_script = venv_path / \"Scripts\" / \"activate.bat\"\n activate_cmd = str(activate_script)\n else:\n activate_script = venv_path / \"bin\" / \"activate\"\n activate_cmd = f\"source {activate_script}\"\n \n print(f\"Virtual environment created at {venv_path}\")\n print(f\"To activate manually, run: {activate_cmd}\")\n \n return venv_path\n\ndef get_venv_python(venv_path):\n \"\"\"Get the path to the Python executable in the virtual environment.\"\"\"\n if platform.system() == \"Windows\":\n return venv_path / \"Scripts\" / \"python.exe\"\n else:\n return venv_path / \"bin\" / \"python\"\n\ndef get_venv_pip(venv_path):\n \"\"\"Get the path to the pip executable in the virtual environment.\"\"\"\n if platform.system() == \"Windows\":\n return venv_path / \"Scripts\" / \"pip.exe\"\n else:\n return venv_path / \"bin\" / \"pip\"\n\ndef install_requirements(repo_root, venv_pip):\n \"\"\"Install requirements from requirements.txt.\"\"\"\n requirements_file = repo_root / \"requirements.txt\"\n \n if not requirements_file.exists():\n print(f\"Error: requirements.txt not found at {requirements_file}\")\n sys.exit(1)\n \n print(\"Installing requirements...\")\n try:\n subprocess.run([\n str(venv_pip), \"install\", \"-r\", str(requirements_file),\n \"--extra-index-url=https://pypi.org/simple\"\n ], check=True)\n except subprocess.CalledProcessError as e:\n print(f\"Error installing requirements: {e}\")\n sys.exit(1)\n \n print(\"Requirements installed successfully.\")\n\ndef run_sphinx_build(repo_root, venv_path):\n \"\"\"Run sphinx-build and capture the output.\"\"\"\n sphinx_build_path = venv_path / \"bin\" / \"sphinx-build\"\n if platform.system() == \"Windows\":\n sphinx_build_path = venv_path / \"Scripts\" / \"sphinx-build.exe\"\n \n if not sphinx_build_path.exists():\n print(f\"Error: sphinx-build not found at {sphinx_build_path}\")\n sys.exit(1)\n \n print(\"Running sphinx-build...\")\n \n # Create a log file to capture output\n log_file_path = repo_root / \"sphinx_build_output.log\"\n \n try:\n # Run sphinx-build with output redirected to both terminal and log file\n with open(log_file_path, 'w') as log_file:\n process = subprocess.Popen(\n [str(sphinx_build_path), \"-b\", \"html\", \".\", \"_build/html\", \"-w\", \"warnings.txt\"],\n cwd=str(repo_root),\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n bufsize=1\n )\n \n # Capture output in real-time\n output = []\n for line in process.stdout:\n print(line, end='') # Print to terminal\n log_file.write(line) # Write to log file\n output.append(line)\n \n process.wait()\n \n if process.returncode != 0:\n print(f\"sphinx-build exited with code {process.returncode}\")\n \n # Also read the warnings.txt file if it exists\n warnings_file = repo_root / \"warnings.txt\"\n if warnings_file.exists():\n with open(warnings_file, 'r') as f:\n warnings_content = f.read()\n output.append(\"\\n--- WARNINGS FILE CONTENT ---\\n\")\n output.append(warnings_content)\n \n return ''.join(output)\n except Exception as e:\n print(f\"Error running sphinx-build: {e}\")\n sys.exit(1)\n\ndef parse_build_log(log_text):\n \"\"\"Parse the build log to extract errors and warnings.\"\"\"\n # Save raw log for debugging\n with open(\"raw_build_log.txt\", \"w\") as f:\n f.write(log_text)\n \n # Check if warnings.txt exists and use it directly\n warnings_file = Path(\"warnings.txt\")\n if warnings_file.exists():\n print(f\"Found warnings.txt file with direct warnings from Sphinx\")\n with open(warnings_file, 'r') as f:\n warnings_content = f.read()\n \n # Parse warnings.txt which has format: path:line: WARNING: message\n warnings = []\n for line in warnings_content.split('\\n'):\n if not line.strip():\n continue\n \n # Try to match the standard format first\n match = re.match(r'(.*?):(\\d+): WARNING: (.*)', line)\n if match:\n file_path, line_num, message = match.groups()\n warnings.append({\n 'file': file_path,\n 'line': line_num,\n 'message': message.strip()\n })\n print(f\"Standard format match: file={file_path}, line={line_num}, message={message[:50]}...\")\n else:\n # Check for the \"document isn't included in any toctree\" pattern\n # Format: /path/to/file.rst: WARNING: document isn't included in any toctree\n toctree_match = re.match(r'(.*?): WARNING: (document isn\\'t included in any toctree.*)', line)\n if toctree_match:\n file_path, message = toctree_match.groups()\n warnings.append({\n 'file': file_path,\n 'line': '0', # No line number in this format\n 'message': message.strip()\n })\n print(f\"Toctree match: file={file_path}, message={message[:50]}...\")\n else:\n # If no match, just add as unknown\n warnings.append({\n 'file': 'unknown',\n 'line': '0',\n 'message': line.strip()\n })\n print(f\"No match: message={line[:50]}...\")\n else:\n print(\"No warnings.txt file found, parsing log output directly\")\n warnings = []\n lines = log_text.split('\\n')\n i = 0\n while i < len(lines):\n line = lines[i].strip()\n \n # Skip empty lines\n if not line:\n i += 1\n continue\n \n # Check for the \"document isn't included in any toctree\" pattern\n # Format: /path/to/file.rst: WARNING: document isn't included in any toctree\n toctree_match = re.match(r'(.*?): WARNING: (document isn\\'t included in any toctree.*)', line)\n if toctree_match:\n file_path, message = toctree_match.groups()\n warnings.append({\n 'file': file_path,\n 'line': '0', # No line number in this format\n 'message': message.strip()\n })\n i += 1\n continue\n \n # Check for warnings in the raw message\n # This is for warnings that are already in the log as complete messages\n raw_warning_match = re.match(r'(.*?): WARNING: (.*)', line)\n if raw_warning_match:\n file_path, message = raw_warning_match.groups()\n warnings.append({\n 'file': file_path,\n 'line': '0', # No line number in this format\n 'message': message.strip()\n })\n i += 1\n continue\n \n # Check for standard format: path:line: WARNING: message\n std_match = re.match(r'(.*?):(\\d+): WARNING: (.*)', line)\n if std_match:\n file_path, line_num, message = std_match.groups()\n warnings.append({\n 'file': file_path,\n 'line': line_num,\n 'message': message.strip()\n })\n i += 1\n continue\n \n # Check for alternative format: WARNING: message (path:line)\n alt_match = re.match(r'WARNING: (.*?) \\((.*?):(\\d+)\\)', line)\n if alt_match:\n message, file_path, line_num = alt_match.groups()\n warnings.append({\n 'file': file_path,\n 'line': line_num,\n 'message': message.strip()\n })\n i += 1\n continue\n \n # Check for simple warnings that start with \"WARNING:\"\n if line.startswith(\"WARNING:\"):\n message = line[8:].strip() # Remove \"WARNING: \" prefix\n \n # Collect continuation lines\n i += 1\n while i < len(lines) and lines[i].strip() and not lines[i].strip().startswith((\"WARNING:\", \"ERROR:\")):\n message += \" \" + lines[i].strip()\n i += 1\n \n warnings.append({\n 'file': 'unknown',\n 'line': '0',\n 'message': message\n })\n continue\n \n i += 1\n \n # Debug: Print the first few warnings to see what's being parsed\n print(f\"Parsed {len(warnings)} warnings\")\n for i, warning in enumerate(warnings[:5]):\n print(f\"Warning {i+1}: file={warning['file']}, line={warning['line']}, message={warning['message'][:50]}...\")\n \n # Debug: Print the warning categories\n categories = categorize_issues(warnings)\n print(f\"Warning categories: {categories}\")\n \n # Regular expressions for errors\n error_pattern = re.compile(r'(.*?):(\\d+): (?:ERROR|SEVERE): (.*?)(?:\\n|$)')\n \n errors = []\n lines = log_text.split('\\n')\n for line in lines:\n error_match = error_pattern.search(line)\n if error_match:\n file_path, line_num, message = error_match.groups()\n errors.append({\n 'file': file_path,\n 'line': line_num,\n 'message': message.strip()\n })\n \n return errors, warnings\n\ndef categorize_issues(issues):\n \"\"\"Categorize issues by type.\"\"\"\n categories = Counter()\n \n for issue in issues:\n # Extract the main category from the message\n message = issue['message'].lower()\n \n if \"undefined label\" in message:\n categories[\"Undefined Label\"] += 1\n elif \"unknown document\" in message:\n categories[\"Unknown Document\"] += 1\n elif \"duplicate label\" in message:\n categories[\"Duplicate Label\"] += 1\n elif \"image file not found\" in message:\n categories[\"Missing Image\"] += 1\n elif \"toctree contains reference to nonexisting document\" in message:\n categories[\"Missing Document\"] += 1\n elif \"document isn't included in any toctree\" in message:\n categories[\"Document Not in TOC\"] += 1\n else:\n categories[\"Other\"] += 1\n \n return categories\n\ndef format_markdown(errors, warnings, build_time):\n \"\"\"Format the build log as Markdown.\"\"\"\n timestamp = datetime.datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S\")\n \n error_categories = categorize_issues(errors)\n warning_categories = categorize_issues(warnings)\n \n markdown = f\"# Sphinx Build Log - {timestamp}\\n\\n\"\n \n # Build summary\n markdown += \"## Build Summary\\n\\n\"\n markdown += f\"- **Build Time**: {build_time:.2f} seconds\\n\"\n markdown += f\"- **Total Errors**: {len(errors)}\\n\"\n markdown += f\"- **Total Warnings**: {len(warnings)}\\n\\n\"\n \n # Error categories\n if error_categories:\n markdown += \"### Error Categories\\n\\n\"\n for category, count in error_categories.most_common():\n markdown += f\"- **{category}**: {count}\\n\"\n markdown += \"\\n\"\n \n # Warning categories\n if warning_categories:\n markdown += \"### Warning Categories\\n\\n\"\n for category, count in warning_categories.most_common():\n markdown += f\"- **{category}**: {count}\\n\"\n markdown += \"\\n\"\n \n # Errors section\n markdown += \"## Errors\\n\\n\"\n if errors:\n for i, error in enumerate(errors, 1):\n # Format the file path to be more readable\n file_path = error['file']\n if file_path.startswith('/Users/dougeric/git/private-aws-neuron-sdk-staging/'):\n file_path = file_path[len('/Users/dougeric/git/private-aws-neuron-sdk-staging/'):]\n \n # Create a more readable header with file and line info\n if error['file'] != 'unknown':\n markdown += f\"### Error {i}: {file_path} (line {error['line']})\\n\\n\"\n else:\n markdown += f\"### Error {i}\\n\\n\"\n \n markdown += f\"```\\n{error['message']}\\n```\\n\\n\"\n else:\n markdown += \"No errors found.\\n\\n\"\n \n # Warnings section\n markdown += \"## Warnings\\n\\n\"\n if warnings:\n for i, warning in enumerate(warnings, 1):\n # Format the file path to be more readable\n file_path = warning['file']\n if file_path.startswith('/Users/dougeric/git/private-aws-neuron-sdk-staging/'):\n file_path = file_path[len('/Users/dougeric/git/private-aws-neuron-sdk-staging/'):]\n \n # Create a more readable header with file and line info\n if warning['file'] != 'unknown':\n if warning['line'] != '0':\n markdown += f\"### Warning {i}: {file_path} (line {warning['line']})\\n\\n\"\n else:\n markdown += f\"### Warning {i}: {file_path}\\n\\n\"\n else:\n markdown += f\"### Warning {i}\\n\\n\"\n \n # Don't include the file path in the message if it's already in the header\n message = warning['message']\n if warning['file'] != 'unknown' and message.startswith(warning['file']):\n # Remove the file path from the message\n message = message[len(warning['file'])+2:] # +2 for \": \"\n \n markdown += f\"```\\n{message}\\n```\\n\\n\"\n else:\n markdown += \"No warnings found.\\n\\n\"\n \n return markdown\n\ndef main():\n \"\"\"Main function.\"\"\"\n print(\"Checking Python version...\")\n check_python_version()\n \n print(\"Checking pip installation...\")\n check_pip_installed()\n \n print(\"Finding repository root...\")\n repo_root = find_repo_root()\n repo_parent = repo_root.parent\n \n print(f\"Repository found at: {repo_root}\")\n \n print(\"Setting up virtual environment...\")\n venv_path = setup_venv(repo_parent)\n venv_python = get_venv_python(venv_path)\n venv_pip = get_venv_pip(venv_path)\n \n print(f\"Changing directory to {repo_root}...\")\n os.chdir(str(repo_root))\n \n print(\"Installing requirements...\")\n install_requirements(repo_root, venv_pip)\n \n print(\"Running sphinx-build...\")\n start_time = datetime.datetime.now()\n build_log = run_sphinx_build(repo_root, venv_path)\n end_time = datetime.datetime.now()\n build_time = (end_time - start_time).total_seconds()\n \n print(\"Parsing build log...\")\n errors, warnings = parse_build_log(build_log)\n \n print(\"Formatting build log as Markdown...\")\n markdown = format_markdown(errors, warnings, build_time)\n \n # Write the formatted log to a file\n timestamp = datetime.datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S\")\n output_file = repo_root / f\"build-log-{timestamp}.md\"\n \n with open(output_file, \"w\") as f:\n f.write(markdown)\n \n print(f\"Build log written to {output_file}\")\n print(f\"Found {len(errors)} errors and {len(warnings)} warnings.\")\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "_utilities/inject_archive_meta.py", "content": "#!/usr/bin/env python3\n\"\"\"Inject noindex/nofollow meta directives and deprecation banners into archived .rst files.\"\"\"\n\nimport os\nimport re\nimport sys\n\nMETA_BLOCK = \"\"\".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\"\"\"\n\nWARNING_TEMPLATE = \"\"\"\n.. warning::\n\n This document is archived. {framework} is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\"\"\"\n\n# Default for backward compatibility\nWARNING_BLOCK = WARNING_TEMPLATE.format(framework=\"MXNet\")\n\n\ndef find_title_end(lines):\n \"\"\"Find the line index after the RST title underline.\n \n RST titles look like:\n Title Text\n ==========\n \n or with overline:\n ==========\n Title Text\n ==========\n \n Returns the index of the line AFTER the title underline, or -1 if not found.\n \"\"\"\n title_chars = set('=-~^\"\\'`#*+_.')\n i = 0\n while i < len(lines):\n line = lines[i].rstrip()\n # Check if this line is an underline (all same char, at least 3 chars)\n if len(line) >= 3 and len(set(line)) == 1 and line[0] in title_chars:\n # Check if previous line is text (title) - this is an underline\n if i > 0 and lines[i-1].strip() and not (len(set(lines[i-1].rstrip())) == 1 and lines[i-1].rstrip()[0] in title_chars):\n return i + 1\n # Check if next line is text and line after that is underline (overline pattern)\n if i + 2 < len(lines) and lines[i+1].strip():\n next_next = lines[i+2].rstrip()\n if len(next_next) >= 3 and len(set(next_next)) == 1 and next_next[0] in title_chars:\n return i + 3\n i += 1\n return -1\n\n\ndef inject_meta_and_warning(filepath, framework=\"MXNet\"):\n \"\"\"Inject meta block at top and warning after title in an RST file.\"\"\"\n with open(filepath, 'r') as f:\n content = f.read()\n \n # Skip if already has noindex meta\n if ':noindex:' in content:\n print(f\" SKIP (already has meta): {filepath}\")\n return\n \n warning_block = WARNING_TEMPLATE.format(framework=framework)\n \n lines = content.split('\\n')\n \n # Separate any leading labels (.. _label:) and blank lines\n # These need to stay before the meta block\n label_lines = []\n content_start = 0\n for i, line in enumerate(lines):\n stripped = line.strip()\n if stripped.startswith('.. _') and stripped.endswith(':'):\n label_lines.append(line)\n content_start = i + 1\n elif stripped == '' and all(l.strip().startswith('.. _') for l in lines[:i] if l.strip()):\n label_lines.append(line)\n content_start = i + 1\n else:\n break\n \n # Build the content after labels\n remaining_lines = lines[content_start:]\n remaining_content = '\\n'.join(remaining_lines)\n \n # Find title end in remaining content\n title_end = find_title_end(remaining_lines)\n \n if title_end >= 0:\n # Insert warning after title\n before_title = '\\n'.join(remaining_lines[:title_end])\n after_title = '\\n'.join(remaining_lines[title_end:])\n \n new_remaining = before_title + '\\n' + warning_block + after_title\n else:\n # No title found, just add warning at the start of content\n print(f\" WARNING: No title found in {filepath}\")\n new_remaining = warning_block + remaining_content\n \n # Reconstruct: labels + meta + content with warning\n label_section = '\\n'.join(label_lines) + '\\n' if label_lines else ''\n new_content = label_section + META_BLOCK + new_remaining\n \n # Ensure file ends with newline\n if not new_content.endswith('\\n'):\n new_content += '\\n'\n \n with open(filepath, 'w') as f:\n f.write(new_content)\n \n print(f\" OK: {filepath}\")\n\n\ndef main():\n import argparse\n parser = argparse.ArgumentParser(description='Inject archive meta into .rst files')\n parser.add_argument('archive_dir', nargs='?', default='archive/mxnet-neuron',\n help='Directory containing .rst files to process')\n parser.add_argument('--framework', default='MXNet',\n help='Framework name for the deprecation warning (e.g., MXNet, TensorFlow)')\n args = parser.parse_args()\n\n archive_dir = args.archive_dir\n framework = args.framework\n \n rst_files = []\n for root, dirs, files in os.walk(archive_dir):\n for fname in files:\n if fname.endswith('.rst'):\n rst_files.append(os.path.join(root, fname))\n \n rst_files.sort()\n print(f\"Processing {len(rst_files)} .rst files in {archive_dir}:\")\n \n for filepath in rst_files:\n inject_meta_and_warning(filepath, framework=framework)\n \n print(f\"\\nDone. Processed {len(rst_files)} files.\")\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "_utilities/metadata_schema.yaml", "content": "# Metadata Schema for AWS Neuron SDK Setup Documentation\n# This schema defines the structured metadata fields used in setup documentation pages\n\nmetadata_fields:\n # Core identification fields\n description:\n type: string\n required: true\n description: \"SEO and AI agent description of the page content\"\n example: \"Install PyTorch Neuron using AWS Deep Learning AMI on Inf2, Trn1, Trn2, Trn3\"\n \n keywords:\n type: array[string]\n required: true\n description: \"Comma-separated search terms for discoverability\"\n example: \"pytorch, neuron, dlami, installation, inf2, trn1, trn2, trn3\"\n \n date-modified:\n type: date\n required: true\n format: \"YYYY-MM-DD\"\n description: \"ISO 8601 date of last modification\"\n example: \"2026-03-02\"\n \n content-type:\n type: enum\n required: true\n values:\n - navigation-hub\n - framework-setup-hub\n - installation-guide\n - troubleshooting\n - legacy-guide\n description: \"Type of documentation page\"\n \n # Setup-specific fields\n framework:\n type: enum\n required_for: [installation-guide, framework-setup-hub]\n values:\n - pytorch\n - jax\n - tensorflow\n - mxnet\n description: \"ML framework being documented\"\n validation: \"Must match parent directory name\"\n \n instance-types:\n type: array[enum]\n required_for: [installation-guide, framework-setup-hub, navigation-hub]\n values:\n - inf1\n - inf2\n - trn1\n - trn2\n - trn3\n description: \"Supported AWS instance types\"\n validation: \"Cannot mix inf1 with inf2/trn1/trn2/trn3\"\n \n installation-method:\n type: enum\n required_for: [installation-guide]\n values:\n - dlami\n - manual\n - container\n description: \"Installation approach documented\"\n \n os:\n type: array[enum]\n required_for: [installation-guide]\n values:\n - ubuntu-24.04\n - ubuntu-22.04\n - al2023\n - rocky-9\n description: \"Supported operating systems\"\n \n python-versions:\n type: array[string]\n required: false\n description: \"Supported Python versions\"\n example: \"3.10, 3.11, 3.12\"\n \n status:\n type: enum\n required: false\n values:\n - current\n - beta\n - legacy\n - deprecated\n description: \"Status of the documented feature/hardware\"\n validation: \"Must be 'legacy' when instance-types contains only inf1\"\n \n # AI agent hints\n task:\n type: string\n required: false\n description: \"Task-based description for AI agents\"\n example: \"Install PyTorch on Trn1 using DLAMI\"\n \n prerequisites:\n type: array[string]\n required: false\n description: \"List of required knowledge/resources\"\n \n estimated-time:\n type: string\n required: false\n description: \"Estimated completion time\"\n example: \"5 minutes\"\n\n# Validation Rules\nvalidation_rules:\n - rule: \"inf1_separation\"\n description: \"inf1 cannot be mixed with inf2, trn1, trn2, or trn3\"\n check: \"If 'inf1' in instance-types, then len(instance-types) == 1\"\n error_message: \"Cannot mix inf1 with other instance types\"\n \n - rule: \"framework_directory_match\"\n description: \"framework metadata must match parent directory\"\n check: \"framework value must equal parent directory name\"\n error_message: \"Framework '{framework}' does not match directory '{directory}'\"\n \n - rule: \"legacy_status_for_inf1\"\n description: \"Pages with only inf1 must have legacy status\"\n check: \"If instance-types == ['inf1'], then status == 'legacy'\"\n error_message: \"Inf1-only pages must have status: legacy\"\n \n - rule: \"legacy_directory_location\"\n description: \"Legacy content must be in legacy-inf1 directory\"\n check: \"If status == 'legacy', then path contains '/legacy-inf1/'\"\n warning_message: \"Legacy content should be in /setup/legacy-inf1/ directory\"\n \n - rule: \"installation_guide_completeness\"\n description: \"Installation guides must have complete metadata\"\n check: \"If content-type == 'installation-guide', then framework, instance-types, installation-method, and os must be present\"\n error_message: \"Installation guide missing required metadata: {missing_fields}\"\n \n - rule: \"content_type_requirements\"\n description: \"Each content type has specific required fields\"\n requirements:\n navigation-hub: [description, keywords, instance-types, content-type]\n framework-setup-hub: [description, keywords, framework, instance-types, content-type]\n installation-guide: [description, keywords, framework, instance-types, installation-method, os, content-type]\n troubleshooting: [description, keywords, content-type]\n legacy-guide: [description, keywords, instance-types, status, content-type]\n\n# Usage Examples\nexamples:\n installation_guide:\n description: \"Install PyTorch Neuron using AWS DLAMI on Inf2, Trn1, Trn2, Trn3\"\n keywords: \"pytorch, neuron, dlami, installation, inf2, trn1, trn2, trn3\"\n framework: \"pytorch\"\n instance-types: \"inf2, trn1, trn2, trn3\"\n installation-method: \"dlami\"\n os: \"ubuntu-24.04, ubuntu-22.04, al2023\"\n content-type: \"installation-guide\"\n date-modified: \"2026-03-02\"\n \n framework_hub:\n description: \"Install PyTorch for AWS Neuron on Inf2, Trn1, Trn2, Trn3 instances\"\n keywords: \"pytorch, neuron, installation, trn1, trn2, trn3, inf2\"\n framework: \"pytorch\"\n instance-types: \"inf2, trn1, trn2, trn3\"\n content-type: \"framework-setup-hub\"\n date-modified: \"2026-03-02\"\n \n legacy_guide:\n description: \"Legacy installation guide for AWS Inferentia 1 (Inf1) instances\"\n keywords: \"neuron, inf1, legacy, installation, inferentia\"\n instance-types: \"inf1\"\n status: \"legacy\"\n content-type: \"legacy-guide\"\n date-modified: \"2026-03-02\"\n" }, { "path": "_utilities/migrate_setup_content.py", "content": "#!/usr/bin/env python3\n\"\"\"\nSetup Content Migration Script\n\nMaps old setup file paths to new framework-first paths and generates\na migration report. This script does NOT move files — it produces a\nreport of what references exist and where they should point.\n\nUsage:\n python3 _utilities/migrate_setup_content.py [--dry-run] [--fix]\n\nOptions:\n --dry-run Show what would be changed without modifying files (default)\n --fix Apply changes to files\n\"\"\"\n\nimport argparse\nimport os\nimport re\nimport sys\nfrom pathlib import Path\n\n# Old path → new path mapping\nPATH_MAP = {\n \"/setup/torch-neuronx\": \"/setup/pytorch/index\",\n \"/setup/jax-neuronx\": \"/setup/jax/index\",\n \"/setup/tensorflow-neuronx\": \"/frameworks/tensorflow/index\",\n \"/setup/setup-neuronx\": \"/setup/index\",\n \"/setup/setup-neuron\": \"/setup/index\",\n \"/setup/mxnet-neuron\": \"/archive/mxnet-neuron/index\",\n}\n\n# External URL mapping (for hardcoded URLs in tutorials)\nURL_MAP = {\n \"setup/torch-neuronx.html\": \"setup/pytorch/index.html\",\n \"setup/jax-neuronx.html\": \"setup/jax/index.html\",\n}\n\n# Directories to scan\nSCAN_DIRS = [\n \"about-neuron\",\n \"frameworks\",\n \"libraries\",\n \"tools\",\n \"compiler\",\n \"containers\",\n \"devflows\",\n \"release-notes\",\n \"setup\",\n \"nki\",\n \"dlami\",\n]\n\n# Directories to skip\nSKIP_DIRS = {\"_build\", \".git\", \"__pycache__\", \".venv\", \"node_modules\"}\n\n\ndef find_rst_files(base_dir: str) -> list[Path]:\n \"\"\"Find all .rst files in scan directories.\"\"\"\n files = []\n for scan_dir in SCAN_DIRS:\n dir_path = Path(base_dir) / scan_dir\n if dir_path.exists():\n for rst_file in dir_path.rglob(\"*.rst\"):\n if not any(skip in rst_file.parts for skip in SKIP_DIRS):\n files.append(rst_file)\n return sorted(files)\n\n\ndef find_references(content: str, file_path: Path) -> list[dict]:\n \"\"\"Find old setup path references in file content.\"\"\"\n refs = []\n\n # Match :doc: references\n for old_path, new_path in PATH_MAP.items():\n pattern = re.compile(\n rf\":doc:`([^`]*<)?{re.escape(old_path)}(>)?`\", re.IGNORECASE\n )\n for match in pattern.finditer(content):\n line_num = content[: match.start()].count(\"\\n\") + 1\n refs.append(\n {\n \"file\": str(file_path),\n \"line\": line_num,\n \"old\": match.group(0),\n \"old_path\": old_path,\n \"new_path\": new_path,\n \"type\": \"doc_ref\",\n }\n )\n\n # Match :ref: references to old labels\n old_labels = {\n \"setup-torch-neuronx\": \"pytorch-setup\",\n \"setup-jax-neuronx\": \"jax-setup\",\n \"setup-tensorflow-neuronx\": \"tensorflow-setup\",\n }\n for old_label, new_label in old_labels.items():\n pattern = re.compile(rf\":ref:`([^`]*<)?{re.escape(old_label)}(>)?`\")\n for match in pattern.finditer(content):\n line_num = content[: match.start()].count(\"\\n\") + 1\n refs.append(\n {\n \"file\": str(file_path),\n \"line\": line_num,\n \"old\": match.group(0),\n \"old_label\": old_label,\n \"new_label\": new_label,\n \"type\": \"ref_label\",\n }\n )\n\n # Match hardcoded URLs\n for old_url, new_url in URL_MAP.items():\n if old_url in content:\n line_num = content[: content.index(old_url)].count(\"\\n\") + 1\n refs.append(\n {\n \"file\": str(file_path),\n \"line\": line_num,\n \"old_url\": old_url,\n \"new_url\": new_url,\n \"type\": \"url\",\n }\n )\n\n return refs\n\n\ndef apply_fix(file_path: Path, refs: list[dict]) -> bool:\n \"\"\"Apply reference fixes to a file.\"\"\"\n content = file_path.read_text()\n modified = False\n\n for ref in refs:\n if ref[\"type\"] == \"doc_ref\":\n old = ref[\"old_path\"]\n new = ref[\"new_path\"]\n new_content = content.replace(old, new)\n if new_content != content:\n content = new_content\n modified = True\n elif ref[\"type\"] == \"url\":\n old = ref[\"old_url\"]\n new = ref[\"new_url\"]\n new_content = content.replace(old, new)\n if new_content != content:\n content = new_content\n modified = True\n\n if modified:\n file_path.write_text(content)\n return modified\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Setup content migration script\")\n parser.add_argument(\n \"--fix\", action=\"store_true\", help=\"Apply changes (default is dry-run)\"\n )\n args = parser.parse_args()\n\n base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))\n rst_files = find_rst_files(base_dir)\n\n print(f\"Scanning {len(rst_files)} .rst files...\")\n print()\n\n all_refs = []\n for rst_file in rst_files:\n content = rst_file.read_text()\n refs = find_references(content, rst_file)\n all_refs.extend(refs)\n\n if not all_refs:\n print(\"No old setup references found. Migration complete.\")\n return\n\n # Group by file\n by_file = {}\n for ref in all_refs:\n by_file.setdefault(ref[\"file\"], []).append(ref)\n\n print(f\"Found {len(all_refs)} references in {len(by_file)} files:\")\n print()\n\n for file_path, refs in sorted(by_file.items()):\n print(f\" {file_path}:\")\n for ref in refs:\n if ref[\"type\"] == \"doc_ref\":\n print(f\" L{ref['line']}: {ref['old_path']} → {ref['new_path']}\")\n elif ref[\"type\"] == \"ref_label\":\n print(f\" L{ref['line']}: {ref['old_label']} → {ref['new_label']}\")\n elif ref[\"type\"] == \"url\":\n print(f\" L{ref['line']}: {ref['old_url']} → {ref['new_url']}\")\n print()\n\n if args.fix:\n fixed_count = 0\n for file_path, refs in by_file.items():\n if apply_fix(Path(file_path), refs):\n fixed_count += 1\n print(f\" ✓ Fixed: {file_path}\")\n print(f\"\\nFixed {fixed_count} files.\")\n else:\n print(\"Dry run — no files modified. Use --fix to apply changes.\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "_utilities/old-nki-apis.txt", "content": "https://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.benchmark.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.profile.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.baremetal.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.simulate_kernel.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.sbuf.alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.sbuf.mod_alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.sbuf.auto_alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.psum.alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.psum.mod_alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.psum.auto_alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.skip_middle_end_transformations.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.enable_stack_allocator.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.compiler.force_auto_alloc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.tensor.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.load.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.store.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.load_transpose2d.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.atomic_rmw.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.copy.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.broadcast_to.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.empty_like.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.zeros_like.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.ones.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.full.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.rand.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.random_seed.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.shared_constant.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.shared_identity_matrix.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.arange.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.mgrid.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.expand_dims.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.where.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.gather_flattened.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.all_reduce.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.par_dim.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.spmd_dim.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.nc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.device_print.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.loop_reduce.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.fp32.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.add.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.subtract.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.multiply.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.divide.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.power.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.maximum.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.minimum.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.max.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.min.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.mean.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.var.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.sum.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.prod.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.all.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.abs.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.negative.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.sign.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.trunc.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.floor.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.ceil.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.mod.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.fmod.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.exp.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.log.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.cos.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.sin.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.tan.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.tanh.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.arctan.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.sqrt.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.rsqrt.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.sigmoid.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.relu.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.gelu.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.gelu_dx.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.gelu_apprx_tanh.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.gelu_apprx_sigmoid.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.silu.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.silu_dx.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.erf.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.erf_dx.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.softplus.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.mish.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.square.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.softmax.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.rms_norm.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.dropout.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.matmul.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.transpose.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.reciprocal.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.bitwise_and.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.bitwise_or.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.bitwise_xor.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.invert.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.left_shift.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.right_shift.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.equal.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.not_equal.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.greater.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.greater_equal.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.less.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.less_equal.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.logical_and.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.logical_or.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.logical_xor.html\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/v2.26.1/nki/api/generated/nki.language.logical_not.html\n\n" }, { "path": "_utilities/setup_jira_token.sh", "content": "#!/bin/bash\n# Setup script to fetch Jira API token from AWS Secrets Manager\n# and configure it for the Atlassian MCP server\n\nset -e\n\necho \"Setting up Jira API token...\"\n\n# Check if AWS CLI is available\nif ! command -v aws &> /dev/null; then\n echo \"Error: AWS CLI is not installed\"\n echo \"Install with: brew install awscli\"\n exit 1\nfi\n\n# Check if ada is available\nif ! command -v ada &> /dev/null; then\n echo \"Error: ada credentials tool is not installed\"\n echo \"Install with: toolbox install ada\"\n exit 1\nfi\n\n# Set AWS profile to kaena\nexport AWS_PROFILE=kaena\n\necho \"Fetching Jira API token from AWS Secrets Manager...\"\nJIRA_TOKEN=$(aws secretsmanager get-secret-value \\\n --secret-id NKI_JIRA_API_TOKEN \\\n --region us-west-2 \\\n --query SecretString \\\n --output text 2>&1)\n\nif [ $? -ne 0 ]; then\n echo \"Error: Failed to fetch Jira API token\"\n echo \"Make sure you have:\"\n echo \" 1. Run 'ada credentials setup' with account 621547421844, role Admin, profile kaena\"\n echo \" 2. Added kaena profile to ~/.aws/config with ada credential_process\"\n echo \" 3. Have IAM permissions to access the secret\"\n echo \"\"\n echo \"Error details:\"\n echo \"$JIRA_TOKEN\"\n exit 1\nfi\n\necho \"✓ Successfully fetched Jira API token\"\n\n# Update the MCP config with the actual token\nMCP_CONFIG=\"$HOME/.kiro/settings/mcp.json\"\n\nif [ ! -f \"$MCP_CONFIG\" ]; then\n echo \"Error: MCP config not found at $MCP_CONFIG\"\n exit 1\nfi\n\n# Create a temporary file with the token substituted\npython3 << EOF\nimport json\nimport os\n\nconfig_path = os.path.expanduser('$MCP_CONFIG')\nwith open(config_path, 'r') as f:\n config = json.load(f)\n\n# Update the Jira API token\nif 'atlassian-jira' in config['mcpServers']:\n config['mcpServers']['atlassian-jira']['env']['JIRA_API_TOKEN'] = '''$JIRA_TOKEN'''\n \n with open(config_path, 'w') as f:\n json.dump(config, f, indent=2)\n \n print(\"✓ Updated MCP configuration with Jira API token\")\nelse:\n print(\"Error: atlassian-jira server not found in MCP config\")\n exit(1)\nEOF\n\necho \"\"\necho \"Setup complete! You can now use Jira tools in Kiro.\"\necho \"\"\necho \"To use Jira MCP tools:\"\necho \" 1. Restart Kiro CLI\"\necho \" 2. Use Jira tools through the MCP server\"\necho \"\"\necho \"Example queries:\"\necho \" - Search for NKI tickets\"\necho \" - Get ticket details\"\necho \" - Create new tickets\"\n" }, { "path": "about-neuron/amazonq-getstarted.rst", "content": "\n.. image:: /images/q-logo.png\n :scale: 30%\n :alt: Amazon Q\n :align: left\n :target: https://aws.amazon.com/q/\n\n.. _amazon-q-dev:\n\nAsk Amazon AI helper tools\n===========================\n\nUse Kiro, Quick, and Amazon Q in the AWS console as your Neuron Experts for general Neuron technical guidance and to jumpstart your NKI kernel developement.\n\n\n.. card:: Ask Q on AWS apps and websites\n :link: https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/q-on-aws.html\n\n.. card:: Ask Kiro IDE\n :link: https://kiro.dev/\n\n.. card:: Ask Kiro CLI\n :link: https://kiro.dev/cli\n\n.. card:: Ask Quick\n :link: https://aws.amazon.com/quick/\n\n.. card:: Guidelines for Quality Results\n :link: amazon-q-dev-guidelines\n :link-type: ref\n\n.. _amazon-q-dev-guidelines:\n\nGuidelines for Quality Results\n------------------------------\n\n1. Be Specific: Clearly state the task, desired output, and any\n constraints.\n2. Provide Context: Mention specific versions, strategies, and any relevant performance requirements.\n3. Request Complete Code: Ask for full implementations including\n imports, decorators, and main functions. Remember to always review and test the generated code before using it in\n production.\n4. Ask for Explanations: Request comments or separate explanations for\n complex parts of the code.\n5. Iterate: If the initial response isn’t satisfactory, refine your\n prompt based on the output. If you encounter issues or inaccuracies, consider rephrasing your\n prompt or breaking down complex tasks into smaller, more specific\n questions.\n6. Fact check: Use Q as a starting point and supplement its output with official documentation, AWS NKI Samples repository, and your own expertise.\n\nExample Prompts\n~~~~~~~~~~~~~~~~~\n\n.. note::\n Amazon AI helper tools may not be fully synched with the latest Neuron features. Therefore, they may not always produce optimal or fully accurate results.\n\n1. “Explain the key features and benefits of AWS Neuron Kernel Interface (NKI).”\n2. \"How do different parallelism strategies (data, pipeline, tensor) affect training performance on Neuron?\"\n3. “What are the best practices for optimizing matrix multiplication operations using Neuron Kernel Interface (NKI)?”\n4. “Provide complete Neuron Kernel Interface (NKI) code for a matrix multiplication kernel, including imports, decorators, and explanations of key optimizations. Focus on efficient tiling and data movement strategies.”\n" }, { "path": "about-neuron/announcements/index.rst", "content": ".. _announcements-main:\n\nAnnouncements\n=============\n\nThis page will be replaced by ABlog. It's here to make sure it's in the TOC.\n\n" }, { "path": "about-neuron/announcements/neuron1.x/announce-eol-mx-before-1-5.rst", "content": ".. post:: May 01, 2023 01:00\n :language: en\n :tags: announce-eol mxnet-neuron\n\n.. _announce-eol-mxnet-before-1-5:\n\nAnnouncing end of support for ``mxnet-neuron`` versions 1.5\n-----------------------------------------------------------\n\n:ref:`Neuron release 2.10 ` will be the last release that will include ``mxnet-neuron`` versions 1.5. Future Neuron releases will not include ``mxnet-neuron`` versions 1.5\n\nCurrent users of those versions are advised to migrate to latest ``mxnet-neuron`` version.\n" }, { "path": "about-neuron/announcements/neuron1.x/announce-eol-pt-1-5.rst", "content": ".. post:: Mar 25, 2022\n :language: en\n :tags: announce-eol torch-neuron\n\n.. _announce-eol-pt-1-5:\n\nAnnouncing end of support for torch-neuron version 1.5 starting with Neuron 1.19.0 release\n------------------------------------------------------------------------------------------\n\nStarting with *Neuron 1.19.0* release, *torch-neuron version 1.5* will no longer be supported. Last release of *torch-neuron version 1.5* will be issued\nas part of *Neuron 1.18.0* release. Current users of those versions are advised to migrate to latest *torch-neuron* version.\n" }, { "path": "about-neuron/announcements/neuron1.x/announce-eol-pt-before-1-8.rst", "content": ".. post:: Nov 22, 2022\n :language: en\n :tags: announce-eol torch-neuron\n\n.. _announce-eol-pt-before-1-8:\n\nAnnouncing end of support for ``torch-neuron`` versions 1.7 and 1.8\n-------------------------------------------------------------------\n\n:ref:`Neuron release 2.5 ` will be the last release that will include ``torch-neuron`` versions 1.7 and 1.8. Future Neuron releases will not include ``torch-neuron`` versions 1.7 and 1.8.\n\nCurrent users of those versions are advised to migrate to latest ``torch-neuron`` version.\n" }, { "path": "about-neuron/announcements/neuron1.x/announce-eol-tf-before-2-5.rst", "content": ".. post:: Nov 22, 2022 01:00\n :language: en\n :tags: announce-eol tensorflow-neuron\n\n.. _announce-eol-tf-before-2-5:\n\nAnnouncing end of support for ``tensorflow-neuron`` versions 2.5 and 2.6\n------------------------------------------------------------------------\n\n:ref:`Neuron release 2.5 ` will be the last release that will include ``tensorflow-neuron`` versions 2.5 and 2.6. Future Neuron releases will not include ``tensorflow-neuron`` versions 2.5 and 2.6.\n\nCurrent users of those versions are advised to migrate to latest ``tensorflow-neuron`` version.\n" }, { "path": "about-neuron/announcements/neuron1.x/announce-eol-tf-before-2-7.rst", "content": ".. post:: May 01, 2023 01:00\n :language: en\n :tags: announce-eol tensorflow-neuron\n\n.. _announce-eol-tf-before-2-7:\n\nAnnouncing end of support for ``tensorflow-neuron`` versions 2.7\n----------------------------------------------------------------\n\n:ref:`Neuron release 2.10 ` will be the last release that will include ``tensorflow-neuron`` versions 2.7. Future Neuron releases will not include ``tensorflow-neuron`` versions 2.7\n\nCurrent users of those versions are advised to migrate to latest ``tensorflow-neuron`` version.\n" }, { "path": "about-neuron/announcements/neuron1.x/announcements.rst", "content": ".. post:: Feb 17, 2022\n :language: en\n :tags: announcements\n\n.. _prev-announcements:\n\nPrevious Announcements\n======================\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 1\n\n.. _maintenance_tf21_tf24:\n\n02/17/2022 - tensorflow-neuron versions 2.1, 2.2, 2.3 and 2.4 enter maintenance mode\n------------------------------------------------------------------------------------\n\nStarting with *Neuron 1.17.2* release, *tensorflow-neuron versions 2.1, 2.2, 2.3 and 2.4* are entering maintenance mode. Future releases of \n*tensorflow-neuron versions 2.1, 2.2, 2.3 and 2.4* will address critical security issues only. Current users of those versions are advised to migrate to \nlatest *tensorflow-neuron* version.\n\n10/27/2021 - Introducing Neuron Runtime 2.x (libnrt.so) \n-------------------------------------------------------\n\nStarting with *Neuron 1.16.0* release, *Neuron Runtime 1.x* (``neuron-rtd``) is entering maintenance mode and is replaced by *Neuron Runtime 2.x*, a shared library named (``libnrt.so``). For more information on Runtime 1.x see :ref:`Neuron Runtime 1.x enters maintenance mode `.\n\nFor more information please see :ref:`introduce-libnrt`.\n\n.. _maintenance_rtd:\n\n10/27/2021 - Neuron Runtime 1.x (``neuron-rtd``) enters maintenance mode\n------------------------------------------------------------------------\n\nStarting with *Neuron 1.16.0* release, *Neuron Runtime 1.x* (``neuron-rtd``) is entering maintenance mode and replaced \nwith *Neuron Runtime 2.x*, a shared library named ``libnrt.so``. \nFuture releases of *Neuron Runtime 1.x* (``neuron-rtd``) will address critical bug fixes and security issues only. Previous releases of \n*Neuron Runtime 1.x* (``neuron-rtd``) will continue to be available via ``rpm`` and ``deb`` packages.\n\nFor more information please see:\n\n\t* :ref:`introduce-libnrt`\n\t* :ref:`install-guide-index`\n\t* :ref:`neuron-maintenance-policy`\n\n\n.. _maintenance_mxnet_1_5:\n\n10/27/2021 - Neuron support for *Apache MXNet 1.5* enters maintenance mode\n--------------------------------------------------------------------------\n\nStarting *Neuron release 1.16.0*, Neuron support for *MXNet 1.5* is entering maintenance mode.\nFuture releases of Neuron supporting *MXNet 1.5* will address critical bug fixes and security issues only.\nPrevious releases of *Apache MXNet 1.5* will continue to be available via ``pip`` packages.\n\nCurrent users of *MXNet Neuron 1.5* can migrate their applications to *MXNet Neuron 1.8*, for more information \nabout MXNet Neuron support and how to upgrade to latest *MXNet Neuron 1.8*, please see visit :ref:`neuron-mxnet`.\n\n\n.. _maintenance_neuron-cli:\n\n10/27/2021 - ``neuron-cli`` enters maintenance mode\n---------------------------------------------------\n\nStarting *Neuron release 1.16.0*, with the introduction of *Neuron Runtime 2.x*, ``neuron-cli`` is entering maintenance mode. ``neuron-cli`` \nfunctionality will be available only if *Neuron Runtime 1.x* (``neuron-rtd``) is being used by the application. If the application is using \n*Neuron Runtime 2.x* shared library(``libnrt.so``), ``neuron-cli`` functionality will not be available.\n\n\nIf you have used ``neuron-cli`` in previous releases, and you are migrating to\nnewer Neuron releases where applications require *Neuron Runtime 2.x* shared library, please see the below :ref:`neuron-cli-mntnce-faq`.\nFuture releases of ``neuron-cli`` will address \ncritical bug fixes and security issues only. Previous releases of ``neuron-cli`` will continue to be available via ``rpm`` and ``deb`` packages.\n\n\n.. _eol-ncg:\n\n10/27/2021 - End of support for NeuronCore Groups (NCG)\n-------------------------------------------------------\n\nBefore the introduction of *Neuron Runtime 2.x*, NeuronCore Group (NCG) has been used by Neuron Runtime 1.x \nto define an execution group of one or more NeuronCores where models can be loaded and executed. It also provided separation between processes.\n \nWith the introduction of *Neuron Runtime 2.x*, the strict separation of NeuronCores into groups is no longer needed and NeuronCore Groups (NCG) is \ndeprecated. *Neuron Runtime 2.x* enables each process to own a set of NeuronCores, and within each process, Neuron Runtime 2.x supports loading and \nexecuting multiple models on separate , different or overlapping sets of NeuronCores.\n\nPlease note that ``NEURONCORE_GROUP_SIZES`` environment variable is in the process of being :ref:`unsupported `, and for a transition period \n``NEURONCORE_GROUP_SIZES`` can be used to preserve the old NeuronCore Group behavior. The frameworks internally would convert ``NEURONCORE_GROUP_SIZES`` to \nuse runtime's new mode of mapping models to NeuronCores.\n\nFor more information see details about ``NEURON_RT_VISIBLE_CORES`` at :ref:`nrt-configuration` and and :ref:`neuron-migrating-apps-neuron-to-libnrt`.\n\n\n.. _eol-ncgs-env:\n\n10/27/2021 - Announcing end of support for ``NEURONCORE_GROUP_SIZES``\n---------------------------------------------------------------------\n\n``NEURONCORE_GROUP_SIZES`` environment variable is in the process of being deprecated, future Neuron releases may no longer support\nthe ``NEURONCORE_GROUP_SIZES`` environment variable. Please start\nusing ``NEURON_RT_VISIBLE_CORES`` instead.\n\nSee :ref:`eol-ncg`, :ref:`nrt-configuration` and :ref:`neuron-migrating-apps-neuron-to-libnrt` for more information.\n\n\n\n\n.. _neuron-cli-mntnce-faq:\n\nFrequently Asked questions (FAQ)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIs there another tool that provide the same functionality as ``neuron-cli list-model``?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYes, please see :ref:`neuron-ls-ug` or :ref:`neuron-monitor-ug`.\n\nIs there another tool that provide the same functionality as ``neuron-cli create-ncg``, ``neuron-cli destroy-ncg``, and ``neuron-cli list-ncg``?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNo, these functionalities are no longer needed with *Neuron Runtime 2.x*,NeuronCore Groups (NCG) :ref:`is deprecated ` and ``NEURONCORE_GROUP_SIZES`` environment variable :ref:`is in the process of being deprecated `, Please start using ``NEURON_RT_VISIBLE_CORES`` instead. See :ref:`nrt-configuration` and :ref:`neuron-migrating-apps-neuron-to-libnrt` \n\nfor more information.\n\nIs there another tool that provide the same functionality as ``neuron-cli reset``?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNo, this functionality is no longer needed with *Neuron Runtime 2.x*. Before introducing ``libnrt.so``, in certain cases after an application \ncrashed models had to be unloaded manually by calling neuron-cli reset.\n\nWith ``libnrt.so``, applications runs in the context of the ``libnrt.so`` shared library and when an application exits the Neuron driver will free all resources associated with the application.\n\n\nFor more information please see:\n\n\t* :ref:`introduce-libnrt`\n\t* :ref:`neuron-tools`\n\t* :ref:`install-guide-index`\n\t* :ref:`neuron-maintenance-policy`\n\n\n.. _eol-conda-packages:\n\n05/28/2021 - End of support for Neuron Conda packages in Deep Learning AMI starting Neuron 1.14.0\n-------------------------------------------------------------------------------------------------\n\n05/28/2021 - Starting with Neuron SDK 1.14.0, we will no longer support conda packages to install Neuron SDK framework in DLAMI and we will no longer update conda packages used to install Neuron SDK framework (Neuron conda packages) with new versions.\n\nStarting with Neuron SDK 1.14.0, pip packages (Neuron pip packages) will be used to install Neuron SDK framework in DLAMI conda environment. To upgrade Neuron SDK framework DLAMI users should use pip upgrade commands instead of conda update commands. Instructions are available in this blog and in Neuron SDK documentation (:ref:`setup-guide-index`).\n\n\nStarting with Neuron SDK 1.14.0, run one of the following commands to upgrade to latest Neuron framework of your choice:\n\n* To upgrade PyTorch Neuron:\n\n.. code-block::\n\n source activate aws_neuron_pytorch_p36\n pip install --upgrade torch-neuron neuron-cc[tensorflow] torchvision --extra-index-url https://pip.repos.neuron.amazonaws.com\n\n* To upgrade TensorFlow Neuron:\n\n.. code-block::\n\n source activate aws_neuron_tensorflow_p36\n pip install --upgrade torch-neuron neuron-cc[tensorflow] torchvision --extra-index-url https://pip.repos.neuron.amazonaws.com\n\n* To upgrade MXNet Neuron:\n\n.. code-block::\n\n source activate aws_neuron_mxnet_p36\n pip install --upgrade torch-neuron neuron-cc[tensorflow] torchvision --extra-index-url https://pip.repos.neuron.amazonaws.com\n\nFor more information please check the `blog `__.\n\n\n\n.. _eol-ubuntu16:\n\n05/01/2021 - End of support for Ubuntu 16 starting Neuron 1.14.0\n----------------------------------------------------------------\n\nUbuntu 16.04 entered end of life phase officially in April 2021 (see https://ubuntu.com/about/release-cycle) and will not receive any public software or security updates. Starting with Neuron SDK 1.14.0, Ubuntu 16 is no longer supported for Neuron, users who are using Ubuntu 16 are requested to migrate to Ubuntu18 or Amazon Linux 2.\n\nCustomers who choose to upgrade libc on Ubuntu 16 to work with Neuron v1.13.0 (or higher versions) are highly discouraged from doing that since Ubuntu 16 will no longer receive public security updates.\n\n.. _eol-classic-tensorboard:\n\n05/01/2021 - End of support for classic TensorBoard-Neuron starting Neuron 1.13.0 and introducing Neuron Plugin for TensorBoard \n-------------------------------------------------------------------------------------------------------------------------------\n\nStarting with Neuron SDK 1.13.0, we are introducing :ref:`Neuron Plugin for TensorBoard ` and we will no longer support classic TensorBoard-Neuron. Users are required to migrate to Neuron Plugin for TensorBoard.\n\nStarting with Neuron SDK 1.13.0, if you are using TensorFlow-Neuron within DLAMI Conda environment, attempting to run ``tensorboard`` with the existing version of TensorBoard will fail. Please update the TensorBoard version before installing the Neuron plugin by running ``pip install TensorBoard --force-reinstall``, for installation instructions see :ref:`neuron-plugin-tensorboard`.\n\nUsers who are using Neuron SDK releases before 1.13.0, can find classic TensorBoard-Neuron documentation at `Neuron 1.12.2 documentation `__.\n\n\nFor more information see see :ref:`neuron-tensorboard-rn` and :ref:`neuron-plugin-tensorboard`.\n\n.. _eol_python_3_5:\n\n02/24/2021 - End of support for Python 3.5 \n-------------------------------------------\n\nAs Python 3.5 reached end-of-life in October 2020, and many packages including TorchVision and Transformers have\nstopped support for Python 3.5, we will begin to stop supporting Python 3.5 for frameworks, starting with\nPyTorch-Neuron version :ref:`neuron-torch-11170` in this release. You can continue to use older versions with Python 3.5.\n\n\n11/17/2020 - End of support for ONNX \n------------------------------------\n\nONNX support is limited and from this version onwards we are not\nplanning to add any additional capabilities to ONNX. We recommend\nrunning models in TensorFlow, PyTorch or MXNet for best performance and\nsupport.\n\n\n07/16/2020 - End of support for PyTorch 1.3 \n--------------------------------------------\n\nStarting this release we are ending the support of PyTorch 1.3 and migrating to PyTorch 1.5.1, customers are advised to migrate to PyTorch 1.5.1.\n\n\n\n" }, { "path": "about-neuron/announcements/neuron1.x/eol-ncgs-env_2.rst", "content": ".. post:: Mar 25, 2022\n :language: en\n :tags: announce-eol\n\n\nAnnouncing end of support for ``NEURONCORE_GROUP_SIZES`` starting with Neuron 1.20.0 release\n--------------------------------------------------------------------------------------------\n\nStarting with Neuron SDK 1.20.0, ``NEURONCORE_GROUP_SIZES`` environment variable will no longer be supported. Setting \n``NEURONCORE_GROUP_SIZES`` environment variable will no longer affect applications behavior.\nCurrent customers using ``NEURONCORE_GROUP_SIZES`` environment variable are advised to use ``NEURON_RT_VISIBLE_CORES`` environment variable or ``NEURON_RT_NUM_CORES`` environment variable instead.\n\nSee :ref:`eol-ncg`, :ref:`nrt-configuration` and :ref:`neuron-migrating-apps-neuron-to-libnrt` for more information.\n" }, { "path": "about-neuron/announcements/neuron1.x/eol-pt-15.rst", "content": ".. post:: Apr 29, 2022\n :language: en\n :tags: eol\n\n.. _eol-pt-15:\n\n\nEnd of support for torch-neuron version 1.5\n-------------------------------------------\n\nStarting with *Neuron 1.19.0* release, *torch-neuron 1.5* will no longer be supported, and \nno further releases of *torch-neuron version 1.5* will be issued. Current users of torch-neuron version 1.5 are advised to migrate to \nlatest *torch-neuron* version." }, { "path": "about-neuron/announcements/neuron1.x/eol-tf-21-24.rst", "content": ".. post:: Mar 25, 2022\n :language: en\n :tags: eol\n\n.. _eol-tf-21-24:\n\nEnd of support for tensorflow-neuron versions 2.1, 2.2, 2.3 and 2.4\n--------------------------------------------------------------------\n\nStarting with *Neuron 1.18.0* release, *tensorflow-neuron versions 2.1, 2.2, 2.3 and 2.4* will no longer be supported, and \nno further releases of *tensorflow-neuron versions 2.1, 2.2, 2.3 and 2.4* will be issued. Current users of those versions are advised to migrate to \nlatest *tensorflow-neuron* version." }, { "path": "about-neuron/announcements/neuron2.x/announce-component-change.rst", "content": ".. post:: December 21, 2023\n :language: en\n :tags: announce-name-change, neuron-component\n\n.. _announce-component-name-change:\n\nAnnouncing Name Change for Neuron Components\n---------------------------------------------\n\nStarting with :ref:`Neuron release 2.16 `, the name of the following Neuron components will change as follows:\n\n======================= =================== ====================\nPackage name Current Name New Name\n======================= =================== ====================\ntorch-neuronx PyTorch Neuron PyTorch NeuronX\ntensorflow-neuronx TensorFlow Neuron TensorFlow NeuronX\nneuronx-cc Neuron Compiler NeuronX Compiler\naws-neuronx-runtime-lib Neuron Runtime NeuronX Runtime\ntensorflow-neuronx Transformers Neuron Transformers NeuronX\nneuronx-distributed Neuron Distributed NeuronX Distributed\n======================= =================== ====================\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-correction-neuron-driver-support-inf1.rst", "content": ".. post:: March 12, 2026\n :language: en\n :tags: announce-correction-neuron-driver-inf1, neuron-driver-version, inf1\n\n.. _announce-correction-neuron-driver-inf1-support:\n\n\nCorrection: Neuron Driver support for Inf1 — version 2.24 (not 2.21)\n---------------------------------------------------------------------\n\nWe are correcting a previous announcement regarding last Neuron Driver version to support Inf1. The last supported version is 2.24\n\nNeuron driver versions above 2.24 only support non-Inf1 instances (such as ``Trn1``, ``Inf2``, or other instance types).\nFor ``Inf1`` instance users, only Neuron driver version 2.24 will remain supported with regular security patches.\n\nAs part of this correction, Neuron Driver version **2.24.13.0** has been released as a patch for ``Inf1`` users, adding compatibility with Linux kernel 6.18.\n\n``Inf1`` instance users are advised to pin the Neuron driver version to ``2.24.*`` in their installation script:\n\nFor Ubuntu:\n\n.. code-block:: bash\n\n sudo apt-get install aws-neuronx-dkms=2.24.* -y\n\nFor Amazon Linux 2 / Amazon Linux 2023:\n\n.. code-block:: bash\n\n sudo yum install aws-neuronx-dkms-2.24.* -y\n\nRefer to the :ref:`Neuron Driver release notes ` for more details.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-deprecation-containers-rtd.rst", "content": ".. post:: December 20, 2023\n :language: en\n :tags: announce-deprecating-containers, runtime-rtd\n\n.. _announce-update-containers:\n\nAnnouncing end-of-support for Neuron Containers with Runtime 1.x\n-----------------------------------------------------------------\n\n:ref:`Neuron release 2.3 ` was the last release to support Neuron Runtime 1.x (neuron-rtd).\nCurrent users of Neuron DLC/DLAMI with Neuron Runtime 1.x are required to :ref:`update their image ` to support latest Neuron Runtime versions. For instructions, see the :ref:`Setup Guide `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-deprecation-nxd-path-trace-api.rst", "content": ".. post:: September 18, 2025\n :language: en\n :tags: announce-deprecation-nxd-path-trace-api, al2\n\n.. _announce-deprecation-nxd-path-trace-api:\n\nAnnouncing the deprecation of the NeuronX Deep Learning Inference API path_trace function\n-----------------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.26.0 ` is the last release supporting ``parallel_model_trace``. This NxD Inference function will be deprecated in the next version of the Neuron SDK in favor of the ``ModelBuilder.trace()`` method, which provides a more robust and flexible approach for tracing and compiling models for Neuron devices, enabling more advanced features such as weight layout optimization support, as well as other quality-of-life and stability improvements for SPMD tracing.\n\nFor customers directly invoking ``parallel_model_trace``, they can now use ModelBuilderV2 APIs. For more details on these APIS, see :ref:`nxd-core-model-builder-v2`. For customers that are directly using models in NxDI, there is no impact since NxDI models are already built on MBv1 which has no issues." }, { "path": "about-neuron/announcements/neuron2.x/announce-deprecation-transformer-flag.rst", "content": ".. post:: September 15, 2023\n :language: en\n :tags: announce-end-of-support, transformer-flag \n\n.. _announce-end-of-support-transformer-flag:\n\nAnnouncing end-of-support for ``--model-type=transformer-inference`` compiler flag\n-----------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron release 2.14 `, the ``--model-type=transformer-inference`` compiler flag is deprecated.\n\nNeuron SDK users using the ``--model-type=transformer-inference`` compiler flag are highly encouraged to migrate to the ``--model-type=transformer`` compiler flag.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eol-megatron-lm.rst", "content": ".. post:: Aug 8, 2023\n :language: en\n :tags: announce-eol, trn1, trn1n\n\n.. _announce-eol-megatronlm:\n\nAnnouncing end of support for AWS Neuron reference for Megatron-LM \n-------------------------------------------------------------------\n\n:ref:`Neuron release 2.12 ` will be the last release that will include support for `AWS Neuron reference for Megatron-LM `_. Future releases will not include Neuron support for Megatron-LM.\n\nCurrent Neuron Megatron-LM users are advised to migrate to `AWS Neuron reference for NeMo Megatron `_ or `Neuron Distributed `_.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eol-python-3-7.rst", "content": ".. post:: Jul 26, 2023 10:00\n :language: en\n :tags: announce-eol, python37\n\n.. _announce-eol-python37:\n\nAnnouncing end of support for ``Python 3.7`` \n---------------------------------------------\n\n:ref:`Neuron release 2.12 ` will be the last release that will include support for ``Python 3.7`` . Future Neuron releases will not include support for ``Python 3.7``\n\nCurrent users using ``Python 3.7`` are advised to migrate to latest supported Python version. (``Python 3.10`` )" }, { "path": "about-neuron/announcements/neuron2.x/announce-eol-ubuntu-18.rst", "content": ".. post:: Jul 13, 2023 11:00\n :language: en\n :tags: announce-eol, ubuntu18\n\n.. _announce-eol-ubuntu18:\n\nAnnouncing end of support for ``Ubuntu 18`` \n-------------------------------------------\n\n:ref:`Neuron release 2.12 ` will be the last release that will include support for ``Ubuntu 18`` . Future Neuron releases will not include support for ``Ubuntu 18``\n\nCurrent users using ``Ubuntu 18`` are advised to migrate to ``Ubuntu 20`` version.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-al2.rst", "content": ".. post:: June 28, 2024\n :language: en\n :tags: announce-eos-al2, al2\n\n.. _announce-eos-al2:\n\nAnnouncing end of support for Neuron Runtime support of Amazon Linux 2 (AL2)\n------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.19 ` will be the last release that will include Neuron Runtime support for ``Amazon Linux 2`` . Future Neuron releases will not include Neuron Runtime support for ``Amazon Linux 2``.\n\nCurrent users using ``Amazon Linux 2`` are advised to migrate to Amazon Linux 2023 (AL2023) or Ubuntu 20/22.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-beta-pytorch-neuroncore-placement-apis.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-no-longer-support-pytorch-neuroncore-placement\n\n.. _announce-no-longer-support-beta-pytorch-neuroncore-placement-apis:\n\nAnnouncing end of support for Beta PyTorch NeuronCore Placement APIs starting next release \n--------------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.24 ` is the last release to support the Beta PyTorch NeuronCore Placement APIs. \n\nCustomers using Beta PyTorch NeuronCore Placement APIs are recommended to migrate to using generally available (GA) PyTorch Neuron Core Placement APIs. Please refer to the :ref:`PyTorch Neuron documentation ` for guidance on using the supported functionality. Any models using the beta APIs will need to be updated to use the generally available APIs.\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-bf16-vars.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-no-longer-support-xla-env-vars\n\n.. _announce-eos-longer-support-xla-bf16-vars:\n\nAnnouncing end of support XLA_USE_BF16 and XLA_DOWNCAST_BF16 environment variables starting next release\n---------------------------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.24 ` will be the last release to support the following environment variables:\n\n- XLA_USE_BF16\n- XLA_DOWNCAST_BF16\n\n**I currently utilize these environment variables in my model code. What do I do?**\n\nCustomers are recommended to migrate to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to convert their model to BF16 format. For detailed migration guidance, please refer to :ref:`migration_from_xla_downcast_bf16`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-block-dimension-nki.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-eos-block-dimension-nki\n\n.. _announce-eos-block-dimension-nki:\n\nAnnouncing end of support for NKI block dimension starting next release\n--------------------------------------------------------------------------\n\n:ref:`Neuron release 2.24 ` will be the last release to include support for the NKI block dimension in NKI tensor creation routines. Starting with this release, using the block dimension will generate EOS warnings. In the next release (Neuron Release 2.25), these warnings will be upgraded to errors.\n\nCustomers are recommended to refer to the :ref:`nki_block_dimension_migration_guide` for detailed instructions on updating their code.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-dlami-ubuntu-22-04.rst", "content": ".. post:: December 18, 2025\n :language: en\n :tags: announce-eos-dlami-ubuntu-22-04\n\n.. _announce-eos-dlami-ubuntu-22-04:\n\nAnnouncing End of Support for Ubuntu 22.04 single framework DLAMIs for PyTorch and JAX in future release\n========================================================================================================\n\nUbuntu 22.04 single framework DLAMIs for PyTorch and JAX will reach end of support in a future release. Customers are advised to use multi-framework or previously released DLAMIs for Ubuntu 22.04." }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-dlami.rst", "content": ".. post:: April 24, 2024\n :language: en\n :tags: announce-eos-dlami, neuron-dlami\n\n.. _announce-eos-dlami:\n\nAnnouncing end of support for Neuron Release 2.18.0 Deep Learning AMIs \n------------------------------------------------------------------------\n\nWe are announcing end of support for :ref:`Neuron release 2.18.0 ` Deep Learning AMIs. DLAMIs released between March 26,2024 (2024-03-26) and April 10, 2024 (2024-04-10) were shipped without the audit package. The following are the affected DLAMIs:\n\n- Deep Learning AMI Neuron (Ubuntu 22.04) 20240401\n- Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) 20240328\n- Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) 20240402\n- Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) 20240409\n- Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) 20240328\n- Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) 20240402\n- Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) 20240409\n- Deep Learning AMI Neuron TensorFlow 2.10 (Amazon Linux 2) 20240328\n- Deep Learning AMI Neuron TensorFlow 2.10 (Amazon Linux 2) 20240402\n- Deep Learning AMI Neuron TensorFlow 2.10 (Amazon Linux 2) 20240409\n- Deep Learning AMI Neuron TensorFlow 2.10 (Ubuntu 20.04) 20240328\n- Deep Learning AMI Neuron TensorFlow 2.10 (Ubuntu 20.04) 20240402\n- Deep Learning AMI Neuron TensorFlow 2.10 (Ubuntu 20.04) 20240409\n- Deep Learning Base Neuron AMI (Amazon Linux 2) 20240401\n- Deep Learning Base Neuron AMI (Amazon Linux 2) 20240408\n- Deep Learning Base Neuron AMI (Ubuntu 20.04) 20240401\n- Deep Learning Base Neuron AMI (Ubuntu 20.04) 20240408\n\nCurrent users of the above :ref:`Neuron release 2.18 ` Deep Learning AMIs are required to upgrade to the latest DLAMIs in order to consume those with the audit package installed. For instructions to upgrade to the latest AMI, see the :ref:`DLAMI User Guide ` or find the specific DLAMI image id for the latest Neuron release with :ref:`SSM parameters `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-inf1-virtual-environments.rst", "content": ".. post:: December 18, 2025\n :language: en\n :tags: announce-eos-inf1-virtual-environments\n\n.. _announce-eos-inf1-virtual-environments:\n\nNeuron no longer supports Inf1 virtual environments and AMIs starting with Neuron 2.27\n======================================================================================\n\nStarting with Neuron release 2.27, Neuron no longer supports Inf1 virtual environments and AMIs. If you are a customer who is currently using Inf1 virtual environments or AMIs, use Neuron DLAMIs with Neuron version 2.26.1 or earlier." }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-jax-neuronx-nki-call.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-eos-jax-neuronx-features\n\n.. _announce-eos-jax-neuronx-features-2:\n\nAnnouncing end of support for ``jax_neuronx.nki_call`` API in ``jax-neuronx`` from starting next release\n------------------------------------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.23 `, Neuron will end support for ``jax_neuronx.nki_call`` API in ``jax-neuronx`` package.\n\nFor a full list of features that require ``jax-neuronx``, please see :ref:`jax-neuron-known-issues`. \n\nCustomers using ``jax_neuronx.nki_call`` API are recommended to switch invocations to directly call functions annotated with ``@nki.jit``.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-megatronlm-2-13.rst", "content": ".. post:: Aug 28, 2023\n :language: en\n :tags: announce-eos, trn1, trn1n\n\n.. _announce-eos-megatronlm:\n\nAWS Neuron reference for Megatron-LM no longer supported\n----------------------------------------------------------\n\n:ref:`Neuron release 2.13 ` no longer includes support for `AWS Neuron reference for Megatron-LM `_.\n\nCurrent Neuron Megatron-LM users are required to migrate to `AWS Neuron reference for NeMo Megatron `_ or `Neuron Distributed `_.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-mllama-checkpoint.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-eos-mllama-checkpoint\n\n.. _announce-eos-mllama-checkpoint:\n\nAnnouncing end of support for mllama 3.2 Meta Checkpoint API starting next release\n--------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.23 ` will be the last release to include support for the mllama 3.2 Meta checkpoint API. In the next release (Neuron 2.24), Neuron will end support.\n\nAll previously converted checkpoints will continue to function without disruption. Customers' existing workflows and converted models remain fully operational. For new checkpoint conversions, the HuggingFace solution provides equivalent functionality. Customers are recommended to use HuggingFace's official conversion script, available here:\n`Hugging Face Conversion Script `_\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-multiframework-dlamis-inf1.rst", "content": ".. post:: April 24, 2024\n :language: en\n :tags: announce-eos-dlamis-inf1, dlami-inf1\n\n.. _announce-update-multiframework-dlami:\n\nAnnouncing end of support for Neuron virtual environments in AWS Deep Learning AMI (Amazon Linux 2)\n----------------------------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.18.2 ` will be the last release that will include support for the following virtual environments in AWS Deep Learning AMI (Amazon Linux 2):\n\n- ``aws_neuron_pytorch_p38: PyTorch 1.13, Python 3.8``\n- ``aws_neuron_tensorflow2_p38: TensorFlow 2.10, Python 3.8``\n\nFuture releases will not include Neuron support for these virtual environments.\n\nCurrent users of Neuron virtual environments in `AWS Deep Learning AMI (Amazon Linux 2) `_ are required to migrate to the `Neuron multi framework DLAMI `_.\n\nTo see a list of Neuron supported virtual environments, please refer to :ref:`Neuron Multi Framework DLAMI User Guide `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-nemo.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-eos-nemo-megatron\n\n.. _announce-eos-nnm:\n\nAnnouncing end of support for Neuron support for NeMo Megatron starting next release\n-------------------------------------------------------------------------------------\n\nStarting with Neuron Release 2.23, Neuron will end support for :ref:`NeMo Megatron `. \n\nWe recommend all users of :ref:`NeMo Megatron ` to migrate their training workloads to :ref:`NxD Training `. Please refer to :ref:`Neuron NeMo Megatron to NeuronX Distributed Training Migration Guide ` for guidance." }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neuron-det.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-eos-neuron-det\n\n.. _announce-eos-neuron-det:\n\nAnnouncing end of support for Neuron DET tool starting next release\n-------------------------------------------------------------------\n\n:ref:`Neuron Release 2.21 ` will be the last release to support the Neuron Distributed Event Tracing (NDET/neuron-det) tool.\n\nWe recommend all customers using the NDET tool for debugging runtime hangs/issues in large-scale settings transition to the Neuron Profiler 2.0. This tool offers the same runtime function level traces with improved ease of use and optimized performance. For more information on Neuron Profiler 2.0, please refer to the :ref:`neuron-profiler-2-0-guide`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neuron-driver-support-inf1.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-eos-neuron-driver-2.21-version, neuron-driver-version, inf1\n\n.. _announce-upcoming-neuron-driver-2.21-version support changes for inf1 instance:\n\n\nUpcoming changes to Neuron driver 2.21 support for Inf1 starting Neuron 2.26 release\n------------------------------------------------------------------------------------\n\n.. note::\n\n This announcement has been superseded. The correct last supported Neuron driver version for ``Inf1`` is **2.24**, not 2.21. See :ref:`announce-correction-neuron-driver-inf1-support` for details.\n\nStarting with Neuron Release 2.26, Neuron driver versions above 2.21 will only support non-Inf1 instances (such as ``Trn1``, ``Inf2``, or other instance types). \nFor ``Inf1`` instance users, Neuron driver versions < 2.21 will remain supported with regular security patches. \n\n``Inf1`` instance users are advised to pin the Neuron driver version to ``2.21.*`` in their installation script. \nRefer to the :ref:`Neuron Driver release [2.22.2.0] ` for detailed instructions on pinning the Neuron Driver. \n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neuron-profiler-2.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-eos-neuron-profiler\n\n.. _announce-eos-neuron-profiler-2:\n\nNeuron Explorer Replaces Neuron Profiler, Starting with Neuron 2.29\n-------------------------------------------------------------------\n\nStarting with Neuron 2.29, **Neuron Profiler and Profiler 2.0 (UI and CLI) will reach end of support** and be replaced by Neuron Explorer. If you are currently using the Neuron Profiler, migrate to Neuron Explorer before the Neuron 2.29 release.\n\nFor migration guidance, see the :doc:`/tools/neuron-explorer/migration-faq`.\n\nWhat is Neuron Explorer?\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron Explorer is the next-generation suite of tools, guiding developers through their development journey on Trainium. It enables ML performance engineers to:\n\n* **Trace execution end-to-end** — from source code down to hardware operations.\n* **Analyze model behavior at every layer of the stack** — with detailed breakdowns per operation, per core, and per device.\n* **Profile distributed workloads** — with native support for multi-node and multi-worker analysis at scale.\n\nFor more details, see :doc:`/tools/neuron-explorer/index`.\n\nHow does this impact current Neuron Profiler users?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. important::\n\n Neuron strongly recommends migrating to Neuron Explorer **before** the Neuron 2.29 release.\n\nThere are two things to be aware of when migrating:\n\n* **Existing NTFF profile files are supported**, but must be reprocessed before they can be viewed in the Neuron Explorer UI.\n* **New features require new profiles.** To access the full set of Neuron Explorer capabilities, you must recapture your profiles using the updated tooling.\n\nFor detailed migration steps, see the :doc:`/tools/neuron-explorer/migration-faq` and the :ref:`Neuron Explorer FAQ `.\n\nWhat happens to Neuron Profiler after Neuron 2.29?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAfter Neuron 2.29, Neuron Profiler will:\n\n* **No longer receive** bug fixes, feature updates, or technical support.\n* **No longer be distributed** as part of the Neuron SDK.\n\nIf you need to continue using Neuron Profiler temporarily, you must pin your environment to Neuron 2.28 or earlier. This is **not recommended**, as you will not receive any SDK updates or security fixes.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neuron-profiler-v230.rst", "content": ".. post:: March 31, 2026\n :language: en\n :tags: announce-eos-neuron-profiler\n\n.. _announce-eos-neuron-profiler-v230:\n\nNeuron Explorer Replaces Neuron Profiler, Starting with Neuron 2.30.0\n----------------------------------------------------------------------\n\nStarting with Neuron 2.30.0, Neuron Profiler and Profiler 2.0 (UI and CLI) will reach end of support and be replaced by Neuron Explorer. If you are currently using the Neuron Profiler, migrate to Neuron Explorer before the Neuron 2.30.0 release.\n\nFor migration guidance, see the :doc:`/tools/neuron-explorer/migration-faq`.\n\nWhat is Neuron Explorer?\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron Explorer is the next-generation suite of tools, guiding developers through their development journey on Trainium. It enables ML performance engineers to:\n\n* **Trace execution end-to-end** — from source code down to hardware operations.\n* **Analyze model behavior at every layer of the stack** — with detailed breakdowns per operation, per core, and per device.\n* **Profile distributed workloads** — with native support for multi-node and multi-worker analysis at scale.\n\nFor more details, see :doc:`/tools/neuron-explorer/index`.\n\nHow does this impact current Neuron Profiler users?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. important::\n\n Neuron strongly recommends migrating to Neuron Explorer **before** the Neuron 2.30.0 release.\n\nThere are two things to be aware of when migrating:\n\n* **Existing NTFF profile files are supported**, but must be reprocessed before they can be viewed in the Neuron Explorer UI.\n* **New features require new profiles.** To access the full set of Neuron Explorer capabilities, you must recapture your profiles using the updated tooling.\n\nFor detailed migration steps, see the :doc:`/tools/neuron-explorer/migration-faq` and the :ref:`Neuron Explorer FAQ `.\n\nWhat happens to Neuron Profiler after Neuron 2.30.0?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAfter Neuron 2.30.0, Neuron Profiler will:\n\n* **No longer receive** bug fixes, feature updates, or technical support.\n* **No longer be distributed** as part of the Neuron SDK.\n\nIf you need to continue using Neuron Profiler temporarily, you must pin your environment to Neuron 2.28 or earlier. This is **not recommended**, as you will not receive any SDK updates or security fixes.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neuron-profiler.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announce-eos-neuron-profiler\n\n.. _announce-eos-neuron-profiler:\n\nEnd of Support for Neuron Profiler and Neuron Profiler 2.0 UI and CLI coming in a future Neuron release\n--------------------------------------------------------------------------------------------------------\n\nWhat's changing\n^^^^^^^^^^^^^^^^\nNeuron will end support for the legacy Neuron Profiler and Neuron Profiler 2.0 UI and CLI tools in a coming release (planned for v2.29.0). We launched Neuron Explorer in Neuron SDK 2.27, replacing these tools with a unified developer experience that will include device and system profiling in a single view, eager mode support, enhanced memory profiling, improved visualization capabilities, as well as support for the full developer lifecycle.\n\nWhy are we making this change\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nConsolidating to Neuron Explorer allows us to focus development efforts on a single, modern profiling solution while providing you with enhanced features and a better user experience.\n\nHow does this impact you\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIf you are currently using the legacy Neuron Profiler UI or CLI, please do the following before Neuron 2.29:\n\n* Begin using Neuron Explorer (available since Neuron 2.27). See https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/get-started.html#\n* Reprocess your existing NTFF files for the new UI: see https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/how-to-profile-workload.html\n\nNote: Neuron Explorer is backwards compatible with existing Profiler NTFF files, but they must be reprocessed to view in the new UI. For new features (eager mode, memory viewer, certain NKI tools), you'll need to recapture profiles.\n\nAfter Neuron 2.29.0 releases (planned):\n\n* Legacy UI will no longer receive bug fixes, updates, or technical support\n* To continue using legacy UI, you must pin to the last version that supports it (not recommended)\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neurondevice-version.rst", "content": ".. post:: June 28, 2024\n :language: en\n :tags: announce-eos-neuron-device-version, neuron-device-version\n\n.. _announce-eos-neuron-device-version:\n\nAnnouncing end of support for 'neuron-device-version' field in neuron-monitor\n-------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.19 ` will be the last release to include the field 'neuron-device-version' in neuron-monitor.\n\nIn future releases, customers who are using the field 'neuron-device-version' will instead need to use 'instance_type' field in the 'instance_info' section and the 'neuroncore_version' field to obtain neuron device information.\n\nPlease see :ref:`neuron-monitor-ug` for more details.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-neurondevice.rst", "content": ".. post:: June 28, 2024\n :language: en\n :tags: announce-eos-neuron-device, neuron-device\n\n.. _announce-eos-neurondevice:\n\nAnnouncing end of support for 'neurondevice' resource name in Neuron Device K8s plugin\n----------------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.19 ` will be the last release to include resource name 'neurondevice'. \n\nNeuron device plugin is a Neuron Software component that gets installed in Kubernetes environment. The resource name 'neurondevice' enables customers to allocate devices to the Neuron K8s container.\n\nIn future releases, we will rename resource name 'neurondevice' to 'neuron' to maintain consistency. Customers who are using the resource name 'neurondevice' in their YAML file will need to update to use 'neuron'.\n\nPlease see :ref:`k8s-neuron-device-plugin` for more details.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-nxd-examples.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-eos-nxd-examples\n\n.. _announce-eos-nxd-examples:\n\nAnnouncing migration of NxD Core examples from NxD Core repository to NxD Inference repository in next release\n--------------------------------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.21 ` will be the last release to include NxD Core repository inference examples under the NxD Core repository: https://github.com/aws-neuron/neuronx-distributed/tree/main/examples/inference. Starting with :ref:`Neuron Release 2.21 `, the models and modules in NxD Core inference examples are now available through NxD Inference package. We recommend customers to update their applications to use examples from the NxD Inference repository. See :ref:`nxdi-overview`\n\nIn Neuron Release 2.22, the NxD Core inference samples will only reside under the NxD Inference repository. Current users are advised to start using samples/tutorials under the NxD Inference repository: https://github.com/aws-neuron/neuronx-distributed-inference.\n\nI currently utilize an inference sample from the NxD Core repository in my model code. What do I do?\n======================================================================================================\n\nIf your applications depend on the inference examples from NxD Core, we recommend that you update your code to use the new NxD Inference package. With NxD Inference, you can import and use these models and modules in your applications. Any models compiled with inference code from the NxD Core repository will need to be re-compiled. Please refer to the :ref:`nxd-examples-migration-guide` for guidance.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-nxdt-nxd-core-training.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-eos-nxdt\n\n.. _announce-eos-nxdt-nxd-core-training:\n\nAnnouncing end of support for NxDT and NxD Core Training APIs starting with Neuron SDK release 2.29 (PyTorch 2.10)\n-------------------------------------------------------------------------------------------------------------------\n\nNeuron SDK release 2.28 (PyTorch 2.9) will be the last release to include the NeuronX Distributed Training (NxDT) library. Starting with Neuron SDK release 2.29 (PyTorch 2.10), the use of NxD Core training APIs and the PyTorch/XLA package for training will no longer be supported.\n\nHow does this impact you?\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nExisting NxDT/NxD Core users should stay on Neuron SDK 2.28 (PyTorch 2.9) until ready to migrate to native PyTorch on Neuron. Native PyTorch on Neuron uses standard distributed primitives (DTensor, FSDP, DDP). A migration guide will be published in a coming release.\n\nSee :doc:`Native PyTorch on Neuron Overview ` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-probuf.rst", "content": ".. post:: June 28, 2024\n :language: en\n :tags: announce-eos-probuf, probuf\n\n.. _announce-eos-probuf319:\n\nAnnouncing end of support for Probuf versions <= 3.19 for PyTorch NeuronX, NeuronX Distributed, and Transformers NeuronX libraries \n------------------------------------------------------------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.19 ` will be the last release that will include Probuf <= 3.19 support for PyTorch NeuronX, NeuronX Distributed, and Transformers NeuronX libraries. Future Neuron releases will not include Probuf <= 3.19 support for PyTorch NeuronX.\n\nCurrent PyTorch NeuronX, NeuronX Distributed, or Transformers NeuronX users using Probuf <= 3.19 are advised to migrate to latest supported Probuf version.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pt-versions.rst", "content": ".. post:: December 20, 2023\n :language: en\n :tags: announce-eos-pt, pt-versions\n\n.. _announce-eos_pytorch110:\n\nAnnouncing End of Support for PyTorch Neuron version 1.10\n-----------------------------------------------------------\n\n:ref:`Neuron release 2.16 ` will be the last release that will include support for PyTorch Neuron version 1.10. Future Neuron releases will not include support for PyTorch Neuron version 1.10.\n\nCurrent users of PyTorch Neuron version 1.10 are advised to migrate to latest supported PyTorch Neuron version.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pt2.rst", "content": ".. post:: December 20, 2023\n :language: en\n :tags: announce-eos-pt-two, pt-versions-two\n\n.. _announce-eos_pytorch2:\n\nAnnouncing End of Support for PyTorch NeuronX version 2.0 (beta)\n-----------------------------------------------------------------\n\n:ref:`Neuron release 2.16 ` will be the last release that will include beta support for PyTorch NeuronX version 2.0 (beta). Future Neuron releases will not include support for PyTorch NeuronX version 2.0.\n\nCurrent users of PyTorch NeuronX version 2.0 are advised to upgrade to PyTorch NeuronX 2.1 (beta).\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-python38.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-python-eos\n\n.. _announce-python-eos:\n\nAnnouncing end of support for Python 3.8 in future releases\n-----------------------------------------------------------\n\nDue to Python 3.8 reaching its end-of-life status, future Neuron releases will no longer include support for this version.\n\n=========================\nHow does this impact me?\n=========================\n\nI currently use Python 3.8.\n============================\n\nTo avoid security issues and bugs, current users of Python 3.8 are advised to migrate to a Neuron supported Python version (3.9, 3.10, or 3.11) as Neuron will no longer support Python 3.8. For a list of supported Python versions according to Neuron package, please see :ref:`latest-neuron-release-artifacts`.\n\nI currently use Ubuntu 20, which has Python 3.8 as the default version. Am I affected?\n=======================================================================================\n\nAlthough Python 3.8 is the default version of Ubuntu 20.04, Neuron will continue to support Ubuntu 20.04 until April 2025, due to extended standard support of Python 3.8 in Ubuntu 20. Please see the :ref:`sdk-maintenance-policy` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pytorch-1-1-3.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-eos-pytorch-version\n\n.. _announce-eos-pytorch-eos-113:\n\nAnnouncing end of support for PyTorch 1.13 starting next release\n----------------------------------------------------------------\n\n:ref:`Neuron Release 2.21 ` is the last release to support PyTorch 1.13, its associated Deep Learning Containers (DLCs), and Deep Learning AMIs (DLAMIS) for Trn1, Trn2, and Inf2 instances.\n\nWe recommend that all customers using torch-neuron 1.13, related DLCs, and DLAMIS on Trn2, Trn1, and Inf2 instances upgrade to the latest supported PyTorch version. For more information on supported versions, please refer to :ref:`latest-neuron-release-artifacts`.\n\nPlease note that PyTorch 1.13 will continue to be supported for Inf1 instances." }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pytorch-1-9.rst", "content": ".. post:: August 28, 2023\n :language: en\n :tags: announce-eol, torch-neuron \n\n.. _announce-eol-pytorch19:\n\nAnnouncing end of support for ``torch-neuron`` version 1.9 \n-----------------------------------------------------------\n\n:ref:`Neuron release 2.13 ` will be the last release that will include support for ``torch-neuron`` version 1.9. Future Neuron releases will not include support for ``torch-neuron`` version 1.9.\n\nCurrent users of ``torch-neuron`` version 1.9 are advised to migrate to latest supported ``torch-neuron`` version.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pytorch-2-1.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-eos-pytorch-version\n\n.. _announce-eos-pytorch-2-1:\n\nAnnouncing end of support for PyTorch 2.1 starting next release\n---------------------------------------------------------------\n\n:ref:`Neuron Release 2.21 ` is the last release to support PyTorch 2.1, its associated Deep Learning Containers (DLCs), and Deep Learning AMIs (DLAMIS).\n\nWe recommend that all customers using PyTorch 2.1, related DLCs, and DLAMIS upgrade to the latest supported PyTorch version. For more information on supported versions, please refer to :ref:`latest-neuron-release-artifacts`.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pytorch-2-7-2-8-v229.rst", "content": ".. post:: March 31, 2026\n :language: en\n :tags: announce-eos-pytorch-version\n\n.. _announce-eos-pytorch-2-7-2-8-v229:\n\nNeuron no longer supports PyTorch versions 2.7 and 2.8 starting with Neuron 2.29\n----------------------------------------------------------------------------------\n\nStarting with Neuron 2.29, Neuron no longer supports PyTorch versions 2.7 and 2.8. We recommend that all customers upgrade to the latest supported PyTorch version.\n\nCustomers currently using PyTorch versions 2.7 and 2.8 must upgrade to a newer supported PyTorch version. For more information on supported versions, refer to :ref:`latest-neuron-release-artifacts`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pytorch-2-7-2-8.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-eos-pytorch-version\n\n.. _announce-eos-pytorch-2-7-2-8:\n\nAnnouncing end of support for PyTorch versions 2.7 and 2.8 starting next release\n---------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.28 ` is the last release to support PyTorch versions 2.7 and 2.8. Future Neuron releases will not include support for PyTorch versions 2.7 and 2.8.\n\nCurrent users of PyTorch version 2.7 or 2.8 are advised to upgrade to PyTorch 2.9. For more information on supported versions, refer to :ref:`latest-neuron-release-artifacts`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-pytorch-profiling-api.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announce-eos-pytorch-profling-api\n\n.. _announce-eos-pytorch-profling-api:\n\nEnd of Support for PyTorch Experimental Profiling API starting in a future release\n------------------------------------------------------------------------------------\n\nWhat's changing\n^^^^^^^^^^^^^^^^\n\nNeuron will end support for the ``torch_neuronx.experimental.profiler.profile`` API in a future release of Neuron (planned for v2.29.0). This experimental API will be replaced by native PyTorch profiling support using the standard ``torch.profiler.profile()`` API.\n\nHow does this impact you\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIf you are using ``torch_neuronx.experimental.profiler.profile,`` before April/May 2026:\n\n* Update your code to use native PyTorch profiling API:\n\n.. code-block:: python\n\n # Before (Experimental API)\n from torch_neuronx.experimental import profiler\n with profiler.profile(output_path=\"/tmp/profile\") as prof:\n output = model(input)\n\n # After (Native API)\n import torch.profiler\n with torch.profiler.profile(\n activities=[torch.profiler.ProfilerActivity.NEURON],\n on_trace_ready=torch.profiler.tensorboard_trace_handler(\"/tmp/profile\")\n ) as prof:\n output = model(input)\n\nAfter Neuron 2.29.0 releases (planned):\n\n* Experimental API will no longer be supported\n* To continue using the experimental API, you must pin to Neuron SDK 2.28 or earlier (not recommended)\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-tensorboard-tools.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announce-eos-tensorboard-tools\n\n.. _announce-eos-tensorboard-tools:\n\nAnnouncing End of Support for TensorBoard Plugin for Neuron Profiler in Neuron 2.27\n-----------------------------------------------------------------------------------\n\nNeuron 2.27 will be the last release to support TensorBoard Plugin. Future Neuron releases will not include the support for TensorBoard plugin. All customers using the TensorBoard plugin to visualize and analyze model performance are recommended to migrate to Neuron Explorer. \n\nTo begin using Neuron Explorer (available since Neuron 2.27) for profiling, see :doc:`the Neuron Explorer documentation `. Neuron Explorer was introduced with :doc:`the release of the AWS Neuron SDK version 2.27.0 `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-tensorflow-2-8-9.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-tensorflow-versions-eos\n\n.. _announce-tfx-2-8-9-eos:\n\nAnnouncing end of support for Tensorflow 2.8 and 2.9 starting next release\n----------------------------------------------------------------------------\n\nStarting with Neuron Release 2.23, Neuron will end support for TensorFlow 2.8 and 2.9. Future Neuron releases will not include support for Tensorflow-Neuron 2.8 and 2.9 versions.\n\nCurrent users of those versions are advised to migrate to latest TensorFlow version (2.10). For a list of supported versions, please see :ref:`latest-neuron-release-artifacts`.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-tensorflow-inf2.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-eos-tensorflow\n\n.. _announce-eos-tensorflow-inf2:\n\nAnnouncing end of support for TensorFlow for Inferentia2 (Inf2) starting with Neuron 2.29\n------------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.28 ` is the last release to support TensorFlow for Inferentia2 (``Inf2``). Future Neuron releases will not include support for TensorFlow for ``Inf2`` instance users.\n\nCurrent Inf2 instance users are advised to use the latest PyTorch version 2.9. For a list of supported PyTorch versions, see :ref:`latest-neuron-release-artifacts`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-tensorflow1-x.rst", "content": ".. post:: June 28, 2024\n :language: en\n :tags: announce-tensorflow-eos, tf-versions-1-x\n\n.. _announce-tfx-eos:\n\nAnnouncing end of support for Tensorflow-Neuron 1.x\n-----------------------------------------------------\n\n:ref:`Neuron release 2.19 ` will be the last release to support Tensorflow-Neuron 1.x. \nFuture Neuron releases will not include support for Tensorflow-Neuron 1.x versions. Current users of those versions are advised to migrate to latest tensorflow-neuron version 2.10.1.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-torch-neuron.rst", "content": ".. post:: September 16, 2024\n :language: en\n :tags: announce-torch-neuron-eos, torch-neuron\n\n.. _announce-torch-neuron-eos:\n\nAnnouncing maintenance mode for torch-neuron 1.9 and 1.10 versions \n---------------------------------------------------------------------\n\nStarting with :ref:`Neuron release 2.20 `, torch-neuron 1.9 and 1.10 versions will enter maintenance mode.\nFuture Neuron releases will not include support for torch-neuron 1.9 and 1.10 versions. Current users of torch-neuron 1.9 and 1.10 versions are advised to migrate to torch-neuron 1.13.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-torch-neuronx-nki-jit.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-eos-torch-neuronx-nki-jit\n\n.. _announce-eos-torch-neuronx-nki-jit:\n\nAnnouncing end of support for ``torch_neuronx.nki_jit`` API in ``torch-neuronx`` starting next release\n---------------------------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.23 ` will be the last release to include support for ``torch_neuronx.nki_jit`` API in ``torch-neuronx`` package.\n\nCustomers using ``torch_neuronx.nki_jit`` API are recommended to switch invocations to directly call functions annotated with ``@nki.jit``.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-u20-dlamis.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-u20-dlami-dlc-eos\n\n.. _announce-u20-dlami-dlc-eos:\n\nAnnouncing end of support for Ubuntu20 DLCs and DLAMIs\n------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.21 `, AWS Neuron will begin phasing out support for Ubuntu20 Deep Learning Containers (DLCs) and Deep Learning AMIs (DLAMIs). Neuron 2.21 will be the last release to provide bug fixes, and by Neuron 2.22, these offerings will no longer be available.\n\nWe recommend that all customers using Ubuntu20 DLCs and DLAMIs migrate to newer versions based on Ubuntu22 or Amazon Linux 2023. For customers who need to continue using Ubuntu20, you can create custom AMIs based on the Ubuntu20 base image and install Neuron components manually. Please see :ref:`container-faq` and :ref:`neuron-dlami-overview`. \n\nPlease note that this does not affect support for the base Ubuntu20 operating system, which will continue to receive updates as per our standard support policy. For more information, please see :ref:`sdk-maintenance-policy`\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-eos-xla-bf16.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-eos-xla-bf\n\n.. _announce-eos-xla-bf:\n\nAnnouncing end of support for XLA_USE_BF16 and XLA_DOWNCAST_BF16 starting next release\n----------------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.23 `, Neuron will begin phasing out support for the ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` environment variables. In this release, usage of these variables will trigger warnings. Neuron will end support in a subsequent release, aligned with the torch-xla maintenance schedule.\n\nCustomers are recommended to migrate to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to convert their model to BF16 format. For detailed migration guidance, please refer to :ref:`migration_from_xla_downcast_bf16`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-eol-nemo-arg.rst", "content": ".. post:: Oct 26, 2023\n :language: en\n :tags: announce-intent-end-of-support-nemo-arg, nemo-arg\n\n.. _announce-intent-deprecate-nemo-arg:\n\nAnnouncing End of Support for ``nemo`` option-argument\n-------------------------------------------------------\n\n:ref:`Neuron release 2.15 ` will be the last release that will include support for ``nemo`` option-argument in the existing `--distribution_strategy` :ref:`compiler option `. Future releases will not include Neuron support for ``nemo`` option-argument.\nUsers are advised to migrate to the new ``llm-training`` option-argument.\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-eos-opt.rst", "content": ".. post:: Oct 26, 2023\n :language: en\n :tags: announce-intent-eos-opt, opt\n\n.. _announce-intent-eos-opt:\n\nAnnouncing End Of Support for OPT example in Transformers NeuronX\n------------------------------------------------------------------\n\n:ref:`Neuron release 2.15 ` will be the last release that will include OPT example in Transformers NeuronX.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-eos-pt-version.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-eos-pt-two-five\n\n.. _announce-eos_pytorch25:\n\nAnnouncing End of Support for PyTorch NeuronX version 2.5 starting next release\n---------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.24 ` will be the last release that will include support for PyTorch NeuronX version 2.5. Future Neuron releases will not include support for PyTorch NeuronX version 2.5.\n\nCurrent users of PyTorch NeuronX version 2.5 are advised to upgrade to PyTorch NeuronX 2.6 or 2.7. Please see release artifacts for more details on supported versions.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-eos-pt2-6.rst", "content": ".. post:: September 18, 2025\n :language: en\n :tags: announce-eos-pt2-6\n\n.. _announce-eos_pt2-6:\n\nAnnouncing End of Support for PyTorch NeuronX version 2.6 starting next release\n---------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.26 ` will be the last release that will include support for PyTorch NeuronX version 2.6. Future Neuron releases will not include support for PyTorch NeuronX version 2.6. Current users of PyTorch NeuronX version 2.6 are advised to upgrade to PyTorch NeuronX 2.7 or 2.8. See :ref:`Neuron release artifacts ` for more details on supported versions." }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-eos-tensorflow-tutorial-inf.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-eos-tensorflow-tutorial\n\n.. _announce-eos-tensorflow-tutorial:\n\nAnnouncing End of Support for Tensorflow Neuron Inf1 SSD300 tutorial starting next release\n--------------------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.24 ` will be the last release that will include support for :ref:`Tensorflow Neuron Inf1 SSD300 ` tutorial. Future Neuron releases will not include support for :ref:`Tensorflow Neuron Inf1 SSD300 ` tutorial due to security issues.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-eos-tnx.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-eos-tnx\n\n.. _announce-eos-tnx:\n\nAnnouncing end of support for Transformers NeuronX library starting in Neuron 2.26 release\n--------------------------------------------------------------------------------------------\n\nStarting from :ref:`Neuron Release 2.24 `, Transformers NeuronX library is in maintenance mode. ``transformers-neuronx`` releases will now only address critical security issues. In Neuron Release 2.26, Neuron will end support for ``transformers-neuronx``.\n\nCurrent users of ``transformers-neuronx`` are advised to migrate to :ref:`NeuronX Distributed Inference `.\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-intent-maintenance-tnx.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-transformers-neuronx-maintenance, tnx\n\n.. _announce-tnx-maintenance:\n\nAnnouncing maintenance mode for Transformers NeuronX library starting next release\n------------------------------------------------------------------------------------\n\nStarting from Neuron release 2.24, Transformers NeuronX library is entering maintenance mode. Future releases of ``transformers-neuronx`` will address critical security issues only and we will gradually end support. \n\nCurrent users of ``transformers-neuronx`` are advised to migrate to :ref:`NeuronX Distributed Inference `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-maintenance-mxnet.rst", "content": ".. post:: June 28, 2024\n :language: en\n :tags: announce-mxnet-maintenance, mxnet\n\n.. _announce-mxnet-maintenance:\n\nNeuron support for MxNet enters maintenance mode\n---------------------------------------------------\n\nStarting with :ref:`Neuron release 2.19 `, Neuron support for MxNet (``mxnet-neuron``) is entering maintenance mode.\n\nFuture releases of ``mxnet-neuron`` will address critical security issues only and we will gradually end support. Current users of ``mxnet-neuron`` are advised to migrate to PyTorch NeuronX or TensorFlow NeuronX.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-maintenance-nxdi-nxd-core-inference.rst", "content": ".. post:: March 31, 2026\n :language: en\n :tags: announce-maintenance-nxdi\n\n.. _announce-maintenance-nxdi-nxd-core-inference:\n\nAnnouncing maintenance mode for NxD Inference and NxD Core Inference APIs starting next release\n-----------------------------------------------------------------------------------------------\n\nStarting with Neuron 2.30.0, NxD Inference library and NxD Core Inference APIs are entering maintenance mode. Future releases will address critical security issues only and we will gradually end support.\n\nWe are actively investing in an enhanced vLLM Neuron plugin that will not require a separate NxD Inference library. More information about the vLLM Neuron plugin enhancements and migration guidance will be shared in the upcoming release.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-maintenance-nxdt-nxd-core-training.rst", "content": ".. post:: March 31, 2026\n :language: en\n :tags: announce-maintenance-nxdt\n\n.. _announce-maintenance-nxdt-nxd-core-training:\n\nAnnouncing maintenance mode for NxDT and NxD Core Training APIs starting next release\n-------------------------------------------------------------------------------------\n\nStarting with Neuron 2.30.0, NxDT and NxD Core Training APIs are entering maintenance mode. Future releases will address critical security issues only and we will gradually end support.\n\nHow does this impact you?\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nExisting NxDT/NxD Core users should stay on Neuron 2.28 and PyTorch 2.9 until ready to migrate to native PyTorch on Neuron (starting PyTorch 2.10). Customers are recommended to use native PyTorch with standard distributed primitives (DTensor, FSDP, DDP) and TorchTitan starting with Neuron 2.30.0 and PyTorch 2.10. A migration guide will be published in a coming release.\n\nSee :doc:`/frameworks/torch/pytorch-native-overview` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-maintenance-tf.rst", "content": ".. post:: April 1, 2024\n :language: en\n :tags: announce-tensorflow-maintenance, tf-versions\n\n.. _announce-tfx-maintenance:\n\nTensorflow-Neuron 1.x enters maintenance mode\n-----------------------------------------------\n\nStarting with :ref:`Neuron release 2.18 `, Tensorflow-Neuron 1.x is entering maintenance mode. Future releases of Tensorflow-Neuron 1.x will address critical security issues only and we will gradually end support. Current users of those versions are advised to migrate to latest tensorflow-neuron version 2.10.1.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-moving-samples.rst", "content": ".. post:: December 20, 2023\n :language: en\n :tags: announce-moving-nxd-samples, nxd-samples\n\n.. _announce-moving-samples:\n\nAnnouncing end-of-support for NeuronX Distributed Training Samples in Neuron Samples Repository \n------------------------------------------------------------------------------------------------\n\n:ref:`Neuron release 2.16 ` will be the last release to include support for NeuronX Distributed Training Samples (Llama-2, GPT-NeoX 20B, and GPT-NeoX 6.9B) under the `AWS Neuron Samples Github repository `_.\n\nIn future releases, NeuronX Distributed samples will reside under the `NeuronX Distributed Github repository `_. Current users are advised to start using samples under the NeuronX Distributed repository for all NeuronX Distributed tutorials.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-nki-library-namespace-changes-2-28.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-nki-library-changes\n\n.. _announce-nki-library-namespace-changes-2-28:\n\nNKI Library namespace changes starting with Neuron 2.28\n--------------------------------------------------------\n\nStarting with Neuron 2.28, the open source repository namespace has changed from ``nkilib_standalone.nkilib.*`` to ``nkilib.*``, providing a consistent namespace between the open source repository and the shipped version. If customers want to add or modify NKI Library kernels, they can build and install them to replace the default implementation without changing model imports.\n\nSee :ref:`NKI Library ` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-nki-namespace-migration.rst", "content": ".. post:: March 31, 2026\n :language: en\n :tags: announce-nki-namespace\n\n.. _announce-nki-namespace-migration:\n\nAnnouncing NKI Library Kernel Migration to New nki.* Namespace starting Neuron 2.29\n------------------------------------------------------------------------------------\n\nStarting with Neuron 2.29, all NKI Library kernels are migrated to the new ``nki.*`` namespace.\n\nThe new ``nki.*`` namespace introduces changes to NKI APIs and language constructs that improve usability and performance. This transition ensures consistency across all NKI kernels and allows us to focus development efforts on a single, modern namespace.\n\nSee the :doc:`/nki/deep-dives/nki-migration-guide` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-neuron-det.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-no-longer-support-neuron-det\n\n.. _announce-no-longer-support-neuron-det:\n\nNeuron no longer includes support for Neuron DET tool starting with this release \n---------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.22 `, Neuron no longer supports the Neuron Distributed Event Tracing (NDET/neuron-det) tool.\n\nWe recommend customers transition to the Neuron Profiler 2.0 for debugging runtime hangs and issues in large-scale settings. This tool offers the same runtime function level traces with improved ease of use and optimized performance. For more information about the Neuron Profiler 2.0, see :ref:`neuron-profiler-2-0-guide`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-nxd-examples.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-eol-nxd-examples\n\n.. _announce-eol-nxd-examples:\n\nAnnouncing migration of NxD Core inference examples from NxD Core repository to NxD Inference repository starting this release\n==================================================================================================================================\n\n\nStarting with :ref:`Neuron Release 2.23 `, the following models and modules in NxD Core inference examples are now only available through NxD Inference package:\n\n- Llama\n- Mixtral\n- DBRX\n\n\nI currently utilize one of the mentioned inference samples from the NxD Core repository in my model code. What do I do?\n------------------------------------------------------------------------------------------------------------------------\n\nFor customers who want to deploy models out of the box, please use the NxD Inference model hub, which is the recommended option. With NxD Inference, you can import and use these models and modules in your applications. \nCustomers will need to update their applications to use examples under the NxD Inference repository: https://github.com/aws-neuron/neuronx-distributed-inference.\nAny models compiled with inference code from the NxD Core repository will need to be re-compiled. Please refer to the :ref:`nxd-examples-migration-guide` for guidance and see :ref:`nxdi-overview` for more information.\n\nI would like to continue using NxD Core. What do I do?\n--------------------------------------------------------\n\nFor customers who want to continue using NxD Core without NxD Inference, please refer to the Llama3.2 1B sample as a reference implementation: https://github.com/aws-neuron/neuronx-distributed/tree/main/examples/inference/llama\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-pytorch-113.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-no-longer-support-pytorch-version\n\n.. _announce-no-longer-support-pytorch-113:\n\nNeuron no longer supports PyTorch 1.13 starting this release\n-------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.22 `, Neuron no longer supports PyTorch 1.13, its associated Deep Learning Containers (DLCs), and Deep Learning AMIs (DLAMIS) for Trn1, Trn2, and Inf2 instances.\n\nWe recommend that all customers using PyTorch 1.13, related DLCs, and DLAMIS on Trn2, Trn1, and Inf2 instances upgrade to the latest supported PyTorch version. For more information on supported versions, please refer to :ref:`latest-neuron-release-artifacts`.\n\nPlease note that PyTorch 1.13 will continue to be supported for Inf1 instances.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-pytorch-2-1.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-no-longer-support-pytorch-version\n\n.. _announce-no-longer-support-pytorch-2-1:\n\nNeuron no longer supports PyTorch 2.1 starting this release\n------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.22 `, Neuron no longer includes support for PyTorch 2.1, its associated Deep Learning Containers (DLCs), and Deep Learning AMIs (DLAMIS).\n\nWe recommend that all customers using PyTorch 2.1, related DLCs, and DLAMIS upgrade to the latest supported PyTorch version. For more information on supported versions, please refer to :ref:`latest-neuron-release-artifacts`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-pytorch-2-7-2-8.rst", "content": ".. post:: March 30, 2026\n :language: en\n :tags: announce-no-longer-support-pytorch-version\n\n.. _announce-no-longer-support-pytorch-2-7-2-8:\n\nNeuron no longer supports PyTorch versions 2.7 and 2.8 starting with Neuron 2.29\n----------------------------------------------------------------------------------\n\nStarting with Neuron 2.29, Neuron no longer supports PyTorch versions 2.7 and 2.8. We recommend that all customers upgrade to the latest supported PyTorch version.\n\nCustomers currently using PyTorch versions 2.7 and 2.8 must upgrade to a newer supported PyTorch version. For more information on supported versions, refer to :ref:`latest-neuron-release-artifacts`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-tensorflow-inf2.rst", "content": ".. post:: March 30, 2026\n :language: en\n :tags: announce-no-longer-support-tensorflow\n\n.. _announce-no-longer-support-tensorflow-inf2:\n\nNeuron no longer supports TensorFlow for Inferentia2 (Inf2) starting with Neuron 2.29\n---------------------------------------------------------------------------------------\n\nStarting with Neuron 2.29, Neuron no longer supports TensorFlow for Inferentia2 (Inf2). Current Inf2 instance users are advised to use the latest PyTorch version 2.9. For a list of supported PyTorch versions, see :doc:`/release-notes/releasecontent`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-longer-support-u20-dlc-dlami.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-u20-dlami-dlc-no-longer-support\n\n.. _announce-u20-dlami-dlc-eos:\n\nNeuron no longer includes support for Ubuntu20 DLCs and DLAMIs starting this release\n-------------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.22 `, Neuron no longer includes offerings for Ubuntu20 Deep Learning Containers (DLCs) and Deep Learning AMIs (DLAMIs). \n\nCustomers using Ubuntu20 DLCs and DLAMIs should migrate to newer versions based on Ubuntu22 or Amazon Linux 2023. For customers who need to continue using Ubuntu20, you can create custom AMIs based on the Ubuntu20 base image and install Neuron components manually. Please see :ref:`container-faq` and :ref:`neuron-dlami-overview`. \n\nPlease note that this does not affect support for the base Ubuntu20 operating system, which will continue to receive updates as per our standard support policy. For more information, please see :ref:`sdk-maintenance-policy`\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-al2.rst", "content": ".. post:: September 16, 2024\n :language: en\n :tags: end-support-al2\n\n.. _eos-al2:\n\nNeuron Runtime no longer supports Amazon Linux 2 (AL2)\n========================================================\n\nStarting from :ref:`Neuron release 2.20 `, the Neuron Runtime (``aws-neuronx-runtime-lib``) will no longer support Amazon Linux 2 (AL2). \nThe Neuron Driver (``aws-neuronx-dkms``) is now the only Neuron package that supports Amazon Linux 2. \nHowever, the Neuron Driver requires Linux kernel 5.10 or higher. Since default AL2 AMIs ship with kernel 4.14, you must upgrade your AL2 kernel to 5.10+ before installing driver versions 2.18 and later, or migrate to Amazon Linux 2023 or Ubuntu which include compatible kernels by default.\n\nThis change introduces the following constraint:\n\nCustomers cannot run their full Neuron-powered applications natively on an AL2-based Amazon Machine Image (AMI). To leverage Neuron functionality on an AL2 AMI, customers must containerize their applications using a Neuron supported container with non-AL2 Linux distribution (e.g., Ubuntu 22.04, Amazon Linux 2023, etc.) and then deploy those containers on an AL2-based AMI that has the Neuron Driver (``aws-neuronx-dkms``) installed.\n\nHow does this impact me?\n------------------------\n\n**I have an AL2 DLAMI**\n\nIf you are using one of the following Amazon\nLinux 2 DLAMIs, please migrate to a supported DLAMI (e.g., Ubuntu 22.04, Amazon Linux 2023 (AL2023), etc.). Please see :ref:`neuron-dlami-overview` for\na list of all supported DLAMIs to migrate to.\n\n+-----------------+------------------+-----------------------------------------------------------+\n| Framework | Operating System | DLAMI Name |\n+=================+==================+===========================================================+\n| PyTorch 1.13 | Amazon Linux 2 | Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) |\n+-----------------+------------------+-----------------------------------------------------------+\n| TensorFlow 2.10 | Amazon Linux 2 | Deep Learning AMI Neuron TensorFlow 2.10 (Amazon Linux 2) |\n+-----------------+------------------+-----------------------------------------------------------+\n\n**I am using my own AL2 Container**\n\nIf you using your own AL2 Container, please migrate to a Neuron supported container with non-AL2 Linux distribution (e.g., Ubuntu 22.04, Amazon Linux 2023, etc.)\n\n**I am using a base AL2 DLAMI**\n\nIf you are using a base Amazon Linux 2 DLAMI, please ensure the Neuron Driver (``aws-neuronx-dkms``) is the only Neuron package installed. Please use non AL2 (e.g., Ubuntu 22.04, Amazon Linux 2023, etc.) containers to run your Neuron applications.\n\n.. note::\n Neuron does not supports Linux kernel versions < 5.10. Customers using\n Linux kernel versions < 5.10 must migrate to >= 5.10.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-device-version.rst", "content": ".. post:: September 16, 2024\n :language: en\n :tags: eos-neuron-device, neuron-device-\n\n.. _eos-neurondevice:\n\n'neurondevice' resource name in Neuron Device K8s plugin no longer supported\n------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron release 2.20 `, Neuron no longer supports resource name 'neurondevice'. \n\nNeuron device plugin is a Neuron Software component that gets installed in Kubernetes environment. The resource name 'neurondevice' enables customers to allocate devices to the Neuron K8s container.\n\nIn this release, we renamed resource name 'neurondevice' to 'neuron' to maintain consistency. Customers who are using the resource name 'neurondevice' in their YAML file need to update to use 'neuron'.\n\nPlease see :ref:`k8s-neuron-device-plugin` for more details.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-jax-neuronx-nki-call.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: \n\n.. _announce-eos-jax-neuronx-features:\n\nNeuron no longer supports ``jax_neuronx.nki_call`` API in ``jax-neuronx`` starting this release\n-------------------------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.23 ` no longer supports ``jax_neuronx.nki_call`` API in ``jax-neuronx`` package.\n\nFor a full list of features that require ``jax-neuronx``, please see :ref:`jax-neuron-known-issues`. \n\nCustomers using ``jax_neuronx.nki_call`` API will need to switch invocations to directly call functions annotated with ``@nki.jit``.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-llama3-2-checkpoint.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-no-longer-support-llama-checkpoint\n\n.. _announce-no-longer-support-llama-32-meta-checkpoint:\n\nAnnouncing end of support for Llama 3.2 Meta checkpoint\n---------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.24 `, the mllama 3.2 Meta checkpoint API is no longer be supported.\n\n**I currently use the mllama 3.2 Meta checkpoint in my applications. What do I do?**\n\nAll previously converted checkpoints will continue to function without disruption. Customers' existing workflows and converted models remain fully operational. For new checkpoint conversions, customers are advised to use the Hugging Face solution which provides equivalent functionality. Hugging Face's official conversion script is available here:\n`HuggingFace Conversion Script `_\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-nemo-megatron.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-no-support-nemo-megatron\n\n.. _announce-no-support-nemo-megatron:\n\nNeuron no longer supports NeMo Megatron starting this release\n---------------------------------------------------------------\n\nStarting with :ref:`Neuron release 2.23 `, Neuron no longer supports :ref:`NeMo Megatron `. \n\nAll users of :ref:`nemo-megatron-index` are requested to migrate their training workloads to :ref:`NxD Training `. Please refer to :ref:`Neuron NeMo Megatron to NeuronX Distributed Training Migration Guide ` for guidance.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-neurondevice.rst", "content": ".. post:: September 16, 2024\n :language: en\n :tags: eos-neuron-device-version, neuron-device-version\n\n.. _eos-neuron-device-version:\n\n'neuron-device-version' field in neuron-monitor no longer supported\n--------------------------------------------------------------------\n\nStarting with :ref:`Neuron release 2.20 `, Neuron no longer supports the field 'neuron-device-version' in neuron-monitor.\n\nCustomers who are using the field 'neuron-device-version' will instead need to use 'instance_type' field in the 'instance_info' section and the 'neuroncore_version' field to obtain neuron device information.\n\nPlease see :ref:`neuron-monitor-ug` for more details.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-nki-jit-torch.rst", "content": ".. post:: June 24, 2025\n :language: en\n :tags: announce-no-longer-support-nki-jit\n\n.. _announce-no-longer-support-nki-jit:\n\nNeuron no longer supports nki_jit API in PyTorch Neuron starting this release\n--------------------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.24 `, ``torch_neuronx.nki_jit`` API in ``torch-neuronx`` package is no longer supported.\n\n**I currently use nki_jit in my PyTorch models. What do I do?**\n\nCustomers using ``torch_neuronx.nki_jit`` API are recommended to switch invocations to directly call functions annotated with ``@nki.jit``.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-tensorboard-plugin.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-no-support-tensorboard\n\n.. _announce-no-support-tensorboard-plugin:\n\nNeuron no longer supports TensorBoard Plugin for Neuron Profiler starting with Neuron 2.28\n-------------------------------------------------------------------------------------------\n\nStarting with Neuron 2.28, Neuron no longer supports TensorBoard Plugin for Neuron Profiler. All customers using TensorBoard Plugin to visualize and analyze model performance are recommended to migrate to Neuron Explorer.\n\nTo start using Neuron Explorer (available since Neuron 2.27) to profile your workloads, please see the :doc:`Neuron Explorer Getting Started guide `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-tensorflow1-x.rst", "content": ".. post:: September 16, 2024\n :language: en\n :tags: no-support-tensorflow-eos, tf-versions-1-x-no-support\n\n.. _announce-tfx-no-support:\n\nTensorflow-Neuron 1.x no longer supported\n------------------------------------------\n\nStarting with :ref:`Neuron release 2.20 `, Neuron no longer supports Tensorflow-Neuron 1.x. \nCurrent users of those versions are advised to migrate to latest tensorflow-neuron version 2.10.1. Please see :ref:`TensorFlow Neuron ` for more details.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-tensorflow2-10.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announce-no-support-tensorflow2-10\n\n.. _announce-no-support-tensorflow2-10:\n\nNeuron no longer supports tensorflow_2_10 single framework DLAMI and virtual environment in multi-framework DLAMIs starting with Neuron 2.27\n----------------------------------------------------------------------------------------------------------------------------------------------\n\nStarting with the release of Neuron 2.27.0, the ``tensorflow_2_10`` single framework Deep Learning AMI (DLAMI) and the TensorFlow 2.10 virtual environment in multi-framework DLAMIs are no longer supported.\n\nUsers are advised to use previously released DLAMIs for TensorFlow 2.10 support, or migrate to newer supported TensorFlow versions. For more information on supported versions, refer to :doc:`the list of current Neuron-supported package and library versions `.\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-tf-versions.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-no-support-tensorflow-eos\n\n.. _announce-no-support-tensorflow-eos:\n\nNeuron no longer supports Tensorflow 2.8 and 2.9 starting this release\n-----------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.23 `, Neuron no longer supports for TensorFlow-Neuron 2.8 and 2.9 versions. \n\nCurrent users of those versions are advised to migrate to latest TensorFlow version (2.10). For a list of supported versions, please see :ref:`latest-neuron-release-artifacts`." }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-torch-neuron-versions.rst", "content": ".. post:: December 20, 2024\n :language: en\n :tags: announce-no-support-torch-neuron\n\n.. _announce-no-support-torch-neuron:\n\nPyTorch Neuron versions 1.9 and 1.10 no longer supported\n----------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.21 `, Neuron no longer supports torch-neuron 1.9 and 1.10 versions. Current users of torch-neuron 1.9 and 1.10 versions are advised to migrate to the latest torch-neuron supported version. Please see :ref:`latest-neuron-release-artifacts`.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-ubuntu-20-base.rst", "content": ".. post:: May 15, 2025\n :language: en\n :tags: announce-u20-base-no-support\n\n.. _announce-u20-base-no-support:\n\nNeuron no longer supports base Ubuntu 20 operating system starting this release\n--------------------------------------------------------------------------------\n\n:ref:`Neuron Release 2.23 ` no longer includes support for base Ubuntu 20.04 operating system. \n\nCustomers using Ubuntu 20.04 are required to migrate their workloads to Ubuntu 22.04 or another supported operating system. Please refer to :ref:`neuron-dlami-overview` for guidance on Neuron supported operating systems. \n\nFor more information on the Neuron operating system support policy, please see :ref:`sdk-maintenance-policy`." }, { "path": "about-neuron/announcements/neuron2.x/announce-no-support-vllm-v0.rst", "content": ".. post:: February 26, 2026\n :language: en\n :tags: announce-no-support-vllm\n\n.. _announce-no-support-vllm-v0:\n\nNeuron no longer supports vLLM V0 starting with Neuron 2.28\n------------------------------------------------------------\n\nStarting with Neuron 2.28 release, vLLM V0 will no longer be supported. This includes the vLLM V0 Neuron forks in the AWS Neuron `upstreaming-to-vllm GitHub repo `__ and vLLM V0-based Neuron Inference Deep Learning Containers.\n\nCustomers are recommended to use vLLM V1-based inference containers as documented in the :doc:`vLLM V1 user guide `. Additionally, Neuron will be updating existing vLLM-based tutorials to use vLLM V1 in the coming release.\n\nSee :ref:`vLLM on Neuron ` for more information on vLLM V1 support.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-nxdi-changes.rst", "content": ".. post:: December 19, 2025\n :language: en\n :tags: announce-nxdi-changes\n\n.. _announce-nxdi-changes:\n\nAnnouncing changes to NxDI in the upcoming releases\n====================================================\n\nAs part of our transition to native PyTorch support, we are simplifying NxDI to provide a more streamlined developer experience.\n\n**What's changing:**\n\nIn the upcoming releases, we will introduce NxDI v2 that will not use NxDI ModelBuilder APIs. Instead, it will use ``torch.compile`` for model compilation. We will also simplify the NxDI APIs for modeling to align with native PyTorch primitives.\n\n**Timeline and migration:**\n\nWhile we introduce these changes, we will maintain both NxDI v1 and NxDI v2 simultaneously to ensure a smooth migration path for our customers. We will provide detailed migration guidance, timelines, and updated documentation as we approach the transition. More information about the migration path and specific release dates will be shared in the next release (Neuron 2.28).\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-package-change.rst", "content": ".. post:: September 16, 2024\n :language: en\n :tags: announce-nxdcore, neuron-component-nxdcore\n\n.. _announce-component-name-change-nxdcore:\n\nAnnouncing Name Change for Neuron Component \n---------------------------------------------\n\nStarting with :ref:`Neuron release 2.20 `, the name of the following Neuron component will change as follows:\n\n======================= ======================= ============================ ==================\nPackage name Current Name New Name Abbreviation\n======================= ======================= ============================ ==================\nneuronx-distributed NeuronX Distributed NeuronX Distributed Core NxD Core\n======================= ======================= ============================ ==================\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-python38-no-longer-support.rst", "content": ".. post:: April 3, 2025\n :language: en\n :tags: announce-python-version-no-longer-support\n\n.. _announce-python-no-longer-support:\n\nNeuron no longer includes Python 3.8 support starting this release\n-------------------------------------------------------------------\n\nStarting with :ref:`Neuron Release 2.22 `, Neuron no longer includes support for Python 3.8 as it has its reached end-of-life status.\n\n=========================\nHow does this impact me?\n=========================\n\nI currently use Python 3.8.\n============================\n\nTo avoid security issues and bugs, current users of Python 3.8 are advised to migrate to a Neuron supported Python version (3.9, 3.10, or 3.11) as Neuron no longer supports Python 3.8. For a list of supported Python versions according to Neuron package, please see :ref:`latest-neuron-release-artifacts`.\n\nI currently use Ubuntu 20, which has Python 3.8 as the default version. Am I affected?\n=======================================================================================\n\nAlthough Python 3.8 is the default version of Ubuntu 20.04, Neuron will continue to support Ubuntu 20.04 until April 2025, due to extended standard support of Python 3.8 in Ubuntu 20. Please see the :ref:`sdk-maintenance-policy` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announce-transition-pytorch-trainium.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announce-transition-pytorch-trainium\n\n.. _announce-transition-pytorch-trainium:\n\nAnnouncing Transition to PyTorch Native Support for AWS Trainium in the Next Neuron Release Supporting PyTorch 2.10\n------------------------------------------------------------------------------------------------------------------------\n\nStarting with the introduction of Neuron support for PyTorch 2.10, AWS Neuron will begin a transition from PyTorch/XLA to native PyTorch support via TorchNeuron. PyTorch 2.9 will be the last version based on PyTorch/XLA.\n\nWhat's changing\n^^^^^^^^^^^^^^^^\n\n* If you are using PyTorch 2.9, it will be the last version of it that uses the PyTorch/XLA backend in Neuron.\n* For PyTorch 2.10 and later users, Neuron will provide Native PyTorch support via TorchNeuron.\n\nCustomers using PyTorch/XLA-based training should migrate to native PyTorch with TorchNeuron, which provides:\n\n* Native PyTorch eager execution mode\n* Standard distributed primitives (DTensor, FSDP, DDP)\n* ``torch.compile`` support\n* Compatibility with frameworks like TorchTitan (PyTorch Training Library)\n\nFor more information about native PyTorch on Neuron and migration guidance, see :doc:`Native PyTorch for AWS Trainium `.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-end-of-support-neuronxcc-nki.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-end-of-support-neuronxcc-nki\n\n.. _announcement-end-of-support-neuronxcc-nki:\n\nAnnouncing End of Support for neuronxcc.nki Namespace Starting with Neuron 2.28\n--------------------------------------------------------------------------------\n\nNeuron 2.27 will be the last to include support for the neuronxcc.nki.* namespace. Starting with Neuron 2.28, this namespace will no longer be supported.\n\nThe new ``nki.*`` namespace introduces changes to NKI APIs and language constructs. \n\nExisting kernels using ``neuronxcc.nki.*`` must migrate to the new nki.* namespace. A kernel migration guide is available in the Neuron 2.27 documentation.\n\nSee :doc:`the NKI Kernel Migration Guide ` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-end-of-support-nxdt-nxd-core.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-end-of-support-nxdt-nxd-core\n\n.. _announcement-end-of-support-nxdt-nxd-core:\n\nAnnouncing End of Support for NxDT and NxD Core Training APIs Starting with PyTorch 2.10\n-----------------------------------------------------------------------------------------\n\nNeuron support for PyTorch 2.9 will be the last to include the NeuronX Distributed Training (NxDT) libraries, NxD Core training APIs, and PyTorch/XLA for training. Starting with Neuron support for PyTorch 2.10, these components will no longer be supported.\n\nHow does this impact you\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nExisting NxDT/NxD Core users should stay on PyTorch 2.9 until ready to migrate to native PyTorch on Neuron (starting PyTorch 2.10). Customers are recommended to use native PyTorch with standard distributed primitives (DTensor, FSDP, DDP) and TorchTitan starting with Neuron 2.28 and PyTorch 2.10. A migration guide will be published in a coming release.\n\nSee :doc:`Native PyTorch on Neuron Overview ` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-end-of-support-parallel-model-trace.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-end-of-support-parallel-model-trace\n\n.. _announcement-end-of-support-parallel-model-trace:\n\nNeuron no longer supports parallel_model_trace API starting with Neuron 2.27\n-----------------------------------------------------------------------------\n\nStarting with the Neuron 2.27 release, the :ref:`parallel_model_trace API ` is no longer supported for inference. We introduced the :doc:`Model Builder V2 API ` in Neuron 2.25 as an alternative to the tracing API, and it is now the default API in Neuron for model tracing.\n\nCustomers can migrate to the Model Builder V2 API by following the reference `Llama-3.2-1B inference sample `__.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-end-of-support-pytorch-2-6.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-end-of-support-pytorch-2-6\n\n.. _announcement-end-of-support-pytorch-2-6:\n\nNeuron no longer supports PyTorch 2.6 starting with Neuron 2.27\n---------------------------------------------------------------\n\nStarting with Neuron 2.27, Neuron no longer supports PyTorch 2.6. We recommend that all customers using PyTorch 2.6 to upgrade to the latest supported PyTorch version.\n\nCustomers currently using PyTorch 2.6 must upgrade to a newer supported PyTorch version. For more information on supported versions, refer to :doc:`the list of current Neuron-supported package and library versions `.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-end-of-support-vllm-v0.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-end-of-support-vllm-v0\n\n.. _announcement-end-of-support-vllm-v0:\n\nAnnouncing End of Support for vLLM V0 starting with Neuron 2.28\n----------------------------------------------------------------\n\nNeuron Release 2.27 will be the last release to support vLLM V0. In Neuron 2.27 release, vLLM V1 support is introduced for Neuron using the ``vllm-neuron`` plugin. Review the sources in the `Neuron vLLM GitHub Repository `__.\n\nStarting with the Neuron 2.28 release, vLLM V0 will not be supported. Support will be dropped for vLLM V0 Neuron forks of the `upstreaming-to-vllm `__ Neuron GitHub repo, along with vLLM V0-based Neuron Inference Deep Learning Containers.\n\nCustomers should migrate to vLLM V1 using the :doc:`vLLM V1 user guide `. Customers are recommended to start using vLLM V1 based inference containers that are released with Neuron v2.27.0. We plan to update the existing vLLM-based tutorials to use vLLM V1 in the coming release.\n\nSee :doc:`vLLM on Neuron ` for more information on vLLM V1.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-nki-library-kernel-migration.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-nki-library-kernel-migration\n\n.. _announcement-nki-library-kernel-migration:\n\nAnnouncing NKI Library Kernel Migration to New nki.* Namespace in Neuron 2.28\n------------------------------------------------------------------------------\n\nSome NKI Library kernels currently use the legacy ``neuronxcc.nki.*`` namespace. Starting with Neuron 2.28, all NKI Library kernels will migrate to the new ``nki.*`` namespace.\n\nThe new ``nki.*`` namespace introduces changes to NKI APIs and language constructs that improve usability and performance. This transition ensures consistency across all NKI kernels and allows us to focus development efforts on a single, modern namespace.\n\nSee :doc:`the NKI Kernel Migration Guide ` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-nki-library-namespace-changes.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-nki-library-namespace-changes\n\n.. _announcement-nki-library-namespace-changes:\n\nAnnouncing NKI Library Namespace Changes in Neuron 2.28\n--------------------------------------------------------\n\nNKI Library kernels are published in the `NKI Library GitHub repository `__. In Neuron 2.27, these kernels are also shipped as part of neuronx-cc using the nkilib.* namespace. To avoid namespace conflicts when customers use kernels from the open source repository, the repository uses the ``nkilib_standalone.nkilib.*`` namespace.\n\nStarting with Neuron 2.28 the open source repository namespace will change from ``nkilib_standalone.nkilib.*`` to ``nkilib.*``, providing a consistent namespace between the open source repository and the shipped version.\n\nSee :doc:`NKI Library ` for more information.\n" }, { "path": "about-neuron/announcements/neuron2.x/announcement-python-3-9-eol.rst", "content": ".. post:: December 16, 2025\n :language: en\n :tags: announcement-python-3-9-eol\n\n.. _announcement-python-3-9-eol:\n\nNeuron no longer supports Python 3.9 starting with Neuron version 2.27\n-----------------------------------------------------------------------\n\nStarting with Neuron Release 2.27, Neuron no longer includes support for Python 3.9 as it has reached its end-of-life status.\n\nIf you currently use Python 3.9, you are advised to migrate to a Neuron supported Python version (3.10, 3.11 or 3.12) to avoid security issues and bugs.\n\nFor a list of supported Python versions according to Neuron package, refer to :doc:`the list of current Neuron-supported package and library versions `." }, { "path": "about-neuron/announcements/neuron2.x/dlami-neuron-2.10.rst", "content": ".. post:: May 02, 2023 11:00\n :language: en\n :tags: dlami, pytorch, trn1, inf2, inf1\n\n.. _announce-dlc-sm-neuron-2.9.1:\n\nAWS Deep Learning AMIs now available with Neuron 2.10 version\n-------------------------------------------------------------\n\nWe are happy to announce that the following Deep Learning AMIs are now available with latest Neuron Version 2.10. These DLAMIs now support\nall the Neuron EC2 instances including Inf1, Inf2, Trn1/Trn1n.\n\nYou can access the AMIs at the following URLs\n\n* `AWS Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) `__\n* `AWS Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) `__\n* `AWS Deep Learning AMI Base Neuron (Ubuntu 20.04) `__\n* `AWS Deep Learning AMI Base Neuron (Amazon Linux 2) `__\n" }, { "path": "about-neuron/announcements/neuron2.x/dlami-neuron-2.12.rst", "content": ".. post:: July 26, 2023 11:00\n :language: en\n :tags: dlami, pytorch, trn1, inf2, inf1\n\n.. _announce-dlami-neuron-2.12:\n\nAWS Deep Learning AMIs now available with Neuron 2.12 version\n-------------------------------------------------------------\n\nWe are happy to announce that the following Deep Learning AMIs are now available with latest Neuron Version 2.12. \n\nYou can see more about the AMIs at the following URLs\n\n* `AWS Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) `__\n* `AWS Deep Learning AMI Neuron PyTorch 1.13 (Amazon Linux 2) `__\n* `AWS Deep Learning AMI Neuron TensorFlow 2.10 (Ubuntu 20.04) `__\n* `AWS Deep Learning AMI Neuron TensorFlow 2.10 (Amazon Linux 2) `__\n* `AWS Deep Learning AMI Base Neuron (Ubuntu 20.04) `__\n* `AWS Deep Learning AMI Base Neuron (Amazon Linux 2) `__\n" }, { "path": "about-neuron/announcements/neuron2.x/dlami-pytorch-introduce.rst", "content": ".. post:: Nov 02, 2022 00:01\n :language: en\n :tags: dlami, pytorch\n\n.. _announce-dlami-neuron-pytorch:\n\nIntroducing AWS Deep Learning AMI Neuron PyTorch\n------------------------------------------------\n\nWe are happy to announce that Deep Learning AMI (DLAMI) with pre-installed PyTorch Neuron (``torch-neuronx``) is now available, for more information see:\n\n* `AWS Deep Learning AMI Neuron PyTorch 1.11 \\(Amazon Linux 2\\) `_\n\n* `AWS Deep Learning AMI Neuron PyTorch 1.11 \\(Ubuntu 20.04\\) `_\n\nThe Neuron Setup Guide will be updated soon to include the DLAMI PyTorch Neuron." }, { "path": "about-neuron/announcements/neuron2.x/end-of-support-pt2.rst", "content": ".. post:: February 2, 2024\n :language: en\n :tags: eos-pt-two, pt-two\n\n.. _eos_pytorch2:\n\nPyTorch NeuronX version 2.0 (Beta) no longer supported\n-------------------------------------------------------\n\n:ref:`Neuron release 2.17 ` no longer supports PyTorch NeuronX version 2.0 (Beta). \n\nCurrent users of PyTorch NeuronX version 2.0 are advised to migrate to PyTorch NeuronX 2.1 (Beta).\n" }, { "path": "about-neuron/announcements/neuron2.x/github-changes.rst", "content": ".. post:: Oct 10, 2022 02:00\n :language: en\n :tags: github\n\n.. _announce-aws-neuron-github-org:\n\nIntroducing New Neuron GitHub Repositories\n------------------------------------------\n\nStarting with Neuron release 2.3, Neuron Github repositories will be migrated\nto the new `AWS Neuron GitHub Organization `_. \n\nThe new AWS Neuron GitHub Organization will include the `Neuron SDK GitHub `_ repository and will include the following additional new GitHub repositories:\n\n.. list-table:: AWS Neuron GitHub Organization \n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * \t- New GitHub repository\n \t- Description\n\n * \t- `AWS Neuron Samples `_\n \t- Repository that hosts examples and scripts used in the Neuron documentation tutorials\n\n * \t- `AWS Neuron Reference for Megatron-LM `_\n \t- Repository that hosts Neuron support for Megatron-LM\n\n * \t- `AWS Neuron Samples for AWS ParallelCluster `_\n \t- Repository that hosts Neuron support for AWS ParallelCluster\n" }, { "path": "about-neuron/announcements/neuron2.x/gpg-expiration.rst", "content": ".. post:: Nov 10, 2022 00:01\n :language: en\n :tags: dlami, pytorch\n\n.. _announce-dlami-neuron-pytorch:\n\nNeuron GPG key for Ubuntu installation has expired\n--------------------------------------------------\n\nGPG, or GNU Privacy Guard, is a public key cryptography implementation. This allows for the secure transmission of information between parties and can be used to verify that the origin of a message is genuine.\n\nThe GPG key for the Neuron repository (https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB) is installed on the Ubuntu (Canonical) server, the key was uploaded originally with an expiry date of three (3) years, which has expired on 11/10/22.\n\nPlease see :ref:`gpg_key_update` for instructions how to update the Neuron repository GPG keys." }, { "path": "about-neuron/announcements/neuron2.x/neuron-rtd-eol.rst", "content": ".. post:: Oct 10, 2022 01:00\n :language: en\n :tags: eol, neuron2.x\n\n.. _announce-neuron-rtd-eol:\n\nAnnouncing Neuron Runtime 1.x (``neuron-rtd``) end-of-support\n-------------------------------------------------------------\n\nStarting with Neuron release 2.3, Neuron components like Neuron System Tools\nand Neuron Driver will no longer support Neuron Runtime 1.x.\n\nIn addition, starting with Neuron release 2.3, the `AWS Neuron Runtime Proto GitHub `_ and `AWS Neuron Driver GitHub `_ repositories will no longer be supported.\n\nWhy are we removing support for Neuron Runtime 1.x?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNeuron Runtime 1.x (``neuron-rtd``) entered :ref:`maintenance mode ` when Neuron 1.16.0 \nwas released. While Neuron components like Neuron Driver and Neuron System Tools continued to support \nNeuron Runtime 1.x in addition to supporting Neuron Runtime 2.x, Neuron supported frameworks (e.g. PyTorch Neuron,\nTensorFlow Neuron, and MXNet Neuron) stopped supporting Neuron Runtime 1.x starting with Neuron 1.16.0. \nFor detailed information see :ref:`introduce-libnrt`.\n" }, { "path": "about-neuron/announcements/neuron2.x/neuron2-intro.rst", "content": ".. post:: Oct 10, 2022 04:00\n :language: en\n :tags: neuron2.x\n\n.. _neuron2-intro:\n\nIntroducing the first release of Neuron 2.x enabling EC2 Trn1 General Availability (GA)\n=======================================================================================\n\nNeuron release 2.3 is the first release of Neuron 2.x that enables GA of the new EC2 Trn1 instances.\nNeuron release 2.3 extends the latest release of Neuron 1.x (Neuron 1.19.2), adding support for Deep Learning training on the AWS Trainium chips.\n\nStarting with Neuron release 2.3, developers can run Deep Learning training workloads on Trn1 instances, saving training costs by up to \n50% over equivalent GPU-based EC2 instances, while achieving the highest training performance in the AWS cloud for popular NLP models. Neuron 2.x introduces new capabilities and major architectural updates to support training neural-networks with the Trn1 instances. \n\nIn addition, starting with this release, Neuron introduces new packages, renames several packages, \nand updates Neuron installation and update instructions. This release also ends support for Neuron Runtime 1.x.\n\nMore about the release\n----------------------\n\n.. include:: /release-notes/templates/n2.x-trn1-ga-quick.txt\n\n\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/neuron230-packages-changes.rst", "content": ".. post:: Oct 10, 2022 03:00\n :language: en\n :tags: neuron2.x\n\n.. _neuron-packages-changes:\n\nIntroducing Packaging and installation changes\n----------------------------------------------\n\nStarting with Neuron release 2.3, Neuron introduces changes in Neuron packages and installation instructions.\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n.. _neuron-new-packages:\n\nNew Neuron packages\n^^^^^^^^^^^^^^^^^^^\n\nStarting with Neuron release 2.3, Neuron introduces the following new packages:\n\n.. list-table:: New Neuron packages\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - New Package\n - Package Type\n - Description\n - Supported Instances \n \n (At the time of releasing Neuron release 2.3)\n\n * - ``torch-neuronx``\n - .whl (pip)\n - PyTorch Neuron package using `PyTorch XLA `_ \n - Trn1\n\n * - ``neuronx-cc``\n - .whl (pip)\n - Neuron Compiler with XLA front-end\n - Trn1\n\n * - ``aws-neuronx-runtime-lib``\n - .deb (apt), .rpm (yum)\n - Neuron Runtime library\n - Trn1\n\n * - ``aws-neuronx-collective``\n - .deb (apt), .rpm (yum)\n - Collective Communication library \n - Trn1\n\n * - ``aws-neuronx-tools``\n - .deb (apt), .rpm (yum)\n - Neuron System Tools\n - Trn1\n\n.. note::\n\n In next releases ``aws-neuronx-tools`` and ``aws-neuronx-runtime-lib`` will add support for Inf1.\n\nWhy are we introducing new Neuron packages?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo add Neuron support for training neural-networks, Neuron 2.x introduces new capabilities and major architectural updates. For example, Neuron adds support for Collective Communication Operations, in :ref:`new packages ` such as ``aws-neuron-collective``. \n\nIn addition, some of those updates and new capabilities are not backward compatible, for example the Pytorch Neuron package that adds support for training neural-networks uses `PyTorch XLA `_ as a backend. To reduce the possibility of customers using features that are not backward compatible, the new capabilities are introduced in new Neuron packages. For example, PyTorch Neuron and Neuron Compiler will support different packages for Inf1 and for Trn1: ``torch-neuron`` and ``neuron-cc`` will support Inf1 instances, and ``torch-neuronx`` and ``neuronx-cc`` will support Trn1 instances.\n\n.. _neuron-packages-renaming:\n\nRenamed Neuron Packages\n^^^^^^^^^^^^^^^^^^^^^^^\n\nStarting with Neuron release 2.3, the following Neuron packages will change names: \n\n\n.. list-table:: Neuron package with changed names\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size \n\n * - New name\n - Old name (deprecated package)\n - Package Type\n - Description\n - Supported Instances \n\n * - ``aws-neuronx-oci-hooks``\n - ``aws-neuron-runtime-base``\n - .deb (apt), .rpm (yum)\n - OCI Hooks support\n - Trn1, Inf1\n\n * - ``aws-neuronx-dkms``\n - ``aws-neuron-dkms``\n - .deb (apt), .rpm (yum)\n - Neuron Driver\n - Trn1, Inf1 \n\n\n\n\n * - ``aws-neuronx-k8-plugin``\n - ``aws-neuron-k8-plugin``\n - .deb (apt), .rpm (yum)\n - Neuron Kubernetes plugin\n - Trn1, Inf1\n\n * - ``aws-neuronx-k8-scheduler``\n - ``aws-neuron-k8-scheduler``\n - .deb (apt), .rpm (yum)\n - Neuron Scheduler plugin\n - Trn1, Inf1\n\nWhy are we changing package names?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo avoid situations where customers may accidentally install Neuron packages with features that are not backward compatible, we have introduced additional packages with different names for the same Neuron component. \n\n.. _neuron-installation-instruction-change:\n\nUpdated installation and update instructions \n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nStarting with Neuron release 2.3, Neuron installation and update instructions will include pinning of the major version of the Neuron package. For example, to install latest Neuron tools package, call ``sudo apt-get install aws-neuronx-tools=2.*`` and to install latest PyTorch Neuron package for Trn1, call ``pip install torch-neuronx==1.11.0.1.*``. \n\n\nWhy are we changing installation and update instructions?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe Neuron installation and update instructions now guide customers to pin the major version of the different Neuron packages as mentioned in :ref:`neuron-installation-instruction-change`. This is done to future-proof instructions for new, backwards-incompatible major version releases.\n\n.. note:: The change of the installation and update instructions will not include instruction to install or update ``torch-neuron`` and ``neuron-cc``.\n\nWhat do I need to do?\n~~~~~~~~~~~~~~~~~~~~~\n\nPlease follow the :ref:`Neuron setup guide ` to update to latest Neuron releases.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/neuron250-packages-changes.rst", "content": ".. post:: Nov 22, 2022 03:00\n :language: en\n :tags: neuron2.x\n\n.. _neuron250-packages-changes:\n\nIntroducing Neuron packaging and installation changes for Inf1 customers\n------------------------------------------------------------------------\n\nStarting with :ref:`Neuron release 2.5 `, Neuron introduces changes in Neuron packages and installation instructions for Inf1, the following Neuron packages will change names: \n\n\n.. list-table:: Neuron package with changed names for Inf1\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size \n\n * - New name\n - Old name (deprecated package)\n - Package Type\n - Description\n - Supported Instances \n\n * - ``aws-neuronx-tools``\n - ``aws-neuron-tools``\n - .deb (apt), .rpm (yum)\n - System Tools\n - Trn1, Inf1\n\n * - ``aws-neuronx-dkms``\n - ``aws-neuron-dkms``\n - .deb (apt), .rpm (yum)\n - Neuron Driver\n - Trn1, Inf1 \n\n * - ``aws-neuronx-k8-plugin``\n - ``aws-neuron-k8-plugin``\n - .deb (apt), .rpm (yum)\n - Neuron Kubernetes plugin\n - Trn1, Inf1\n\n * - ``aws-neuronx-k8-scheduler``\n - ``aws-neuron-k8-scheduler``\n - .deb (apt), .rpm (yum)\n - Neuron Scheduler plugin\n - Trn1, Inf1\n\n * - ``tensorflow-model-server-neuronx``\n - ``tensorflow-model-server-neuron``\n - .deb (apt), .rpm (yum)\n - tensorflow-model-server\n - Trn1, Inf1\n\n\nPlease follow the :ref:`Neuron setup guide ` to update to latest Neuron releases.\n\n" }, { "path": "about-neuron/announcements/neuron2.x/release-neuron2.4.rst", "content": "" }, { "path": "about-neuron/announcements/neuron2.x/sm-training-dlc-2.9.1.rst", "content": ".. post:: Apr 26, 2023 11:00\n :language: en\n :tags: sagemaker, pytorch, trn1, inf2\n\n.. _announce-dlc-sm-neuron-2.9.1:\n\nPyTorch 1.13 Deep Learning Container for Inf2 & Trn1/Trn1n now available for SageMaker \n--------------------------------------------------------------------------------------\n\nWe are happy to announce that an updated Deep Learning Container that supports PyTorch 1.13 and Neuron 2.9.1 versions is now available for Sagemaker Training.\n\nFor more information see `Neuron Containers `_\n\n\n" }, { "path": "about-neuron/announcements/neuron2.x/sm-training-trn1-introduce.rst", "content": ".. post:: Nov 03, 2022 00:01\n :language: en\n :tags: sagemaker, pytorch, trn1\n\n.. _announce-dlami-neuron-pytorch:\n\nAmazon SageMaker now supports Trn1 training jobs\n------------------------------------------------\n\nWe are happy to announce that Amazon SageMaker now supports running training jobs on ml.trn1 instance types.\n\nFor more information see `Distributed Training with PyTorch Neuron on Trn1 instances `_\n\nThe Neuron Developer Flows section will be updated soon.\n" }, { "path": "about-neuron/appnotes/index.rst", "content": ".. _neuron-appnotes-index:\n.. _neuron-appnotes:\n\n.. meta::\n :description: AWS Neuron SDK application notes for support announcements, performance optimization, migration guides, and framework-specific implementations.\n :date-modified: 2025-10-03\n\nNeuron application notes\n========================\n\n.. toctree:: \n :maxdepth: 2\n :hidden:\n\n Neuron Runtime Library \n Performance \n Parallel execution \n PyTorch for Neuron \n PyTorch for NeuronX \n\nApplication notes provide specific documentation for support announcements, migration guides, performance optimization techniques, and framework-specific implementations for AWS Neuron SDK components.\n\n\nFramework integration\n---------------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: torch-neuron-r-cnn-app-note\n :link-type: ref\n\n **PyTorch Neuron (Inf1)**\n ^^^\n R-CNN implementation and optimization techniques for PyTorch on ``Inf1``\n\n .. grid-item-card::\n :link: torch-neuronx-graph-partitioner-app-note\n :link-type: ref\n\n **PyTorch NeuronX Graph Partitioner**\n ^^^\n Advanced graph partitioning strategies for distributed training and inference\n\n .. grid-item-card::\n :link: torch-neuronx-dataparallel-app-note\n :link-type: ref\n\n **Data Parallel Inference on Torch NeuronX**\n ^^^\n Guide to using ``torch.neuronx.DataParallel`` for scalable inference on ``Inf1``\n\n .. grid-item-card::\n :link: torch-neuron-dataparallel-app-note\n :link-type: ref\n\n **Data Parallel Inference on Torch Neuron**\n ^^^\n Guide to using ``torch.neuron.DataParallel`` for scalable inference on ``Inf1``\n\n .. grid-item-card::\n :link: migration_from_xla_downcast_bf16\n :link-type: ref\n\n **Migrate from XLA_USE_BF16/XLA_DOWNCAST_BF16**\n ^^^\n Guide to migrating from deprecated XLA environment variables to recommended PyTorch mixed-precision options on NeuronX\n\n .. grid-item-card::\n :link: introduce-pytorch-2-9\n :link-type: ref\n\n **PyTorch 2.9 Support**\n ^^^\n New features and migration guide for PyTorch 2.9 on Neuron\n\n\n\n\n\n" }, { "path": "about-neuron/appnotes/mxnet-neuron/flex-eg.rst", "content": ".. _flexeg:\n\nFlexible Execution Group (FlexEG) in Neuron-MXNet\n=================================================\n\nIntroduction\n------------\n\nInf1 instances are available with a different number of Inferentia\nchips, each Inferentia chip is combined of 4 NeuronCores and an Inf1\ninstance includes 4 to 64 NeuronCores depending on the instance size.\nWith Neuron Runtime 1.x (neuron-rtd server), NeuronCores could be\ncombined into NeuronCore Groups (NCG),\nwhich were basic scheduling units of compiled neural network in Neuron.\nCreation of desired sized NCGs was done at the start of the application\nand could not be modified afterwards.\n\nStarting with Neuron SDK 1.16.0, and with the introduction of Neuron\nRuntime 2.x, MXNet Neuron 1.8 introduces Flexible Execution Groups\n(FlexEG) feature. With FlexEG, you do not have to create NCGs at the\nstart of the process, instead you will set the index of the first\nNeuronCore you want to load models onto, and FlexEG feature will enable\nthe flexibility of loading models onto any available NeuronCore on the\ninf1 instance starting from the first NeuronCore you set. This guide\nwill show you how to efficiently utilize NeuronCores using FlexEG\nfeature in NeuronMXNet.\n\nFlexEG\n------\n\nWith the introduction of FlexEG, you don’t need to create NCGs and can\nload models onto a group of consecutive NeuronCores by providing the\nindex of the first NeuronCore in the group. Neuron runtime takes care of\nfiguring out the number of NeuronCores required for the given compiled\nmodel and loads the model using the required number of cores\n(sequentially starting with the NeuronCore index provided by the user).\n\nFor example, assuming that you have an Inf1.6xl machine and there are 4\nmodels A, B, C, D compiled to 2, 4, 3, and 4 NeuronCores respectively,\nyou can map any model to any core by context\n``mx.neuron(neuron_core_index)`` where ``neuron_core_index`` is the\nNeuronCore index (0,1,2,3,4 … ).\n\nIn the example below, you map model A to ``mx.neuron(0)`` context, model\nB to ``mx.neuron(2)`` context, model C to ``mx.neuron(6)`` context and\nmodel D to ``mx.neuron(9)`` context. \n\n.. figure:: /images/mx_FlexEG_arch_1.png\n :scale: 80 %\n\nThe above configuration is achieved by using application code similar to\nbelow:\n\n.. code :: python\n\n # Load models (MXNet)\n # loaded into the 2 cores starting with core 0\n sym, args, aux = mx.model.load_checkpoint(mx_model0_file, 0)\n model0 = sym.bind(ctx=mx.neuron(0), args=args, aux_states=aux, grad_req='null')\n # loaded into the 4 cores starting with core 2\n sym, args, aux = mx.model.load_checkpoint(mx_model1_file, 0)\n model1 = sym.bind(ctx=mx.neuron(2), args=args, aux_states=aux, grad_req='null')\n # loaded into the 3 cores starting with core 6\n sym, args, aux = mx.model.load_checkpoint(mx_model2_file, 0)\n model2 = sym.bind(ctx=mx.neuron(6), args=args, aux_states=aux, grad_req='null')\n # loaded into the 4 cores starting with core 9\n sym, args, aux = mx.model.load_checkpoint(mx_model3_file, 0)\n model3 = sym.bind(ctx=mx.neuron(9), args=args, aux_states=aux, grad_req='null')\n\n # run inference by simply calling the loaded model\n results0 = model0.forward(data=inputs0)\n results1 = model1.forward(data=inputs1)\n results2 = model2.forward(data=inputs2)\n results3 = model3.forward(data=inputs3)\n\nSince there is no NCG creation at the start of the process, you can load\nthe same four models but in a different configuration by changing the\ncontext being used for inference. For example, you could map model C to\n``mx.neuron(0)`` context, model A to ``mx.neuron(3)`` context, model D\nto ``mx.neuron(5)`` context and model B to ``mx.neuron(9)`` context.\n\n.. figure:: /images/mx_FlexEG_arch_2.png\n :scale: 80 %\n\nMigration from NeuronCore Groups to FlexEG\n------------------------------------------\n\nNeuronCore Groups are defined by setting the environment variable\n``NEURONCORE_GROUP_SIZES`` with a comma separated list of number of\ncores in each group. In this mode of operation, number of devices\n(defined in ``NEURONCORE_GROUP_SIZES``) are grouped together to create a\nsingle entity.\n\n``NEURONCORE_GROUP_SIZES`` environment variable is set at runtime:\n\n.. code :: python\n\n #!/bin/bash\n export NEURONCORE_GROUP_SIZES=2,4,3,4 \n python your_neuron_application.py\n\nNeuronCore groups are created once at the start of the application and\ncannot be modified / re-created till the application process runs. The\nabove flow creates 4 neuron devices with 2,4,3 and 4 devices each. In\norder to get the same configuration as the example from before , you map\nmodel A to ``mx.neuron(0)`` context, model B to ``mx.neuron(1)``\ncontext, model C to ``mx.neuron(2)`` context and model D to\n``mx.neuron(3)`` context.\n\n\n.. figure:: /images/mx_FlexEG_arch_1.png\n :scale: 80 %\n\n\nThis can be achieved programmatically as shown below:\n\n.. code :: python\n\n # Set Environment \n os.environ['NEURONCORE_GROUP_SIZES']='2,4,3,4'\n\n # Load models (MXNet)\n # loaded into the first group of NC0-NC1\n sym, args, aux = mx.model.load_checkpoint(mx_model0_file, 0)\n model0 = sym.bind(ctx=mx.neuron(0), args=args, aux_states=aux, grad_req='null')\n # loaded into the second group of NC2-NC5\n sym, args, aux = mx.model.load_checkpoint(mx_model1_file, 0)\n model1 = sym.bind(ctx=mx.neuron(1), args=args, aux_states=aux, grad_req='null')\n # loaded into the third group of NC6-NC8\n sym, args, aux = mx.model.load_checkpoint(mx_model2_file, 0)\n model2 = sym.bind(ctx=mx.neuron(2), args=args, aux_states=aux, grad_req='null')\n # loaded into the fourth group of NC9-NC12\n sym, args, aux = mx.model.load_checkpoint(mx_model3_file, 0)\n model3 = sym.bind(ctx=mx.neuron(3), args=args, aux_states=aux, grad_req='null')\n\n # run inference by simply calling the loaded model\n results0 = model0.forward(data=inputs0)\n results1 = model1.forward(data=inputs1)\n results2 = model2.forward(data=inputs2)\n results3 = model3.forward(data=inputs3)\n\nSo comparing to FlexEG, we see that in case of NCGs neuron context\nrequires the index of the execution group, while in FlexEG\nneuron context requires the NeuronCore index of the first NeuronCore on which the\nmodel is supposed to be loaded and executed. For example, with\n``NEURONCORE_GROUP_SIZES='2,4,3,4'``, ``ctx=mx.neuron(1)`` loads the\nmodel on execution group 1 which effectively loads the model on the 2nd NCG group \nwhich has 4 NeuronCores.\n\nBest practices when using FlexEG\n--------------------------------\n\nFlexEG gives the user most flexibility in terms of accessing cores and\nloading models on specific cores. With this the users can effortlessly\nload and execute new models on NeuronCores without closing the\napplication. Here we shall outline some of the best practices that\nshould be kept in mind while using FlexEG.\n\nChoosing starting core\n~~~~~~~~~~~~~~~~~~~~~~\n\nFlexEG tries to use the required number of cores (based on the input\nmodel) starting with the core index provided by the user. Incase the\nsystem, doesnt have the required number of cores after the starting core\nindex, model load will fail. For example: We have a model X which needs\n2 cores and an inf1.xl machine with 4 NeuronCores (NeuronCore indexes\nare: 0, 1, 2 and 3). As the model needs at least 2 cores, valid start\nindexes for this model are: 0, 1, 2. However if the user gives 3 as the\nneuron context, then there are no 2 cores available starting from core\n3. So it will fail.\n\nPerformance vs. Flexibility tradeoff\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhile using data parallel model of operation (were models are executed\nin parallel), for optimal performance the user should make sure that the\nmodels are not sharing any cores. That is because NeuronCores can\nexecute one model at a time, when two or more models are executed on the\nsame core (assuming that they are already loaded), it executes the first model, stops it, starts the second\nmodel and then executes it. This is called model switiching and involves\nadditional overhead and prevents execution on model in parallel. For\nexample: assuming that you have an Inf1.6xl machine and there are 4\nmodels A, B, C, D compiled to 2, 4, 3, and 4 NeuronCores respectively.\nLoading model A to ``mx.neuron(0)`` context, model B to ``mx.neuron(2)``\ncontext, model C to ``mx.neuron(6)`` context and model D to\n``mx.neuron(9)`` context is a good configuration because no two models\nare sharing NeuronCores and thus can be executed in parallel. However,\nLoading model A to ``mx.neuron(0)`` context, model B to ``mx.neuron(2)``\ncontext, model C to ``mx.neuron(5)`` context and model D to\n``mx.neuron(9)`` context is a not a good configuration as models B and C\nshare NeuronCore 5 and thus cannot be executed in parallel.\n\n\n.. figure:: /images/mx_FlexEG_arch_bad.png\n :scale: 80 %\n\n" }, { "path": "about-neuron/appnotes/neuron-cc/mixed-precision.rst", "content": ".. _neuron-cc-training-mixed-precision:\n\nMixed precision and performance-accuracy tuning (``neuron-cc``)\n===============================================================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nThe Neuron Compiler supports machine learning models with FP32,\nFP16 and BF16 (Bfloat16) tensors and operators. The Neuron hardware supports a\nmix of 32 and 16 bit datatypes.\nThe available auto-cast methods and their performance / accuracy trade-offs\nare explained in this document.\n\nNeuron Hardware\n-------------------\n\nThe Neuron hardware supports matrix multiplication using FP16 or BF16 on its Matmult Engine, and\naccumulations using FP32.\nSimilarly, operators such as activations or vector operations\nare supported using FP16, BF16 and FP32.\nNeuron supports tensor transpose in two ways - by fast matrix\nmultiplication in FP16/BF16 or by slower byte-by-byte data movements.\n\n\nPerformance-accuracy tradeoffs for models trained in FP32\n---------------------------------------------------------\n\nModels that are trained using FP32 data types can be deployed on Neuron\nthrough ahead of time compilation using the :ref:`Neuron Compiler `.\n\n\n.. important::\n **By default**, the Neuron Compiler disables auto-casting and uses the data types defined within the model.\n This provides the best accuracy for FP32 trained models, but does not provide the best performance.\n\nweights and operations to BF16**. Only partial sums are left in FP32. The default, casting will generate the highest\nperformance for a FP32 trained model.\n\nUsing the ``--fast-math`` CLI option, you can choose the right \ntradeoff between performance and accuracy. The tradeoff usually is between achieving high performance or optimal accuracy, and decision what settings to use will be application specific.\n\nIt is recommended that the you start with compiling the model to achieve the high performance (default), you can then \ntest the accuracy of the application and, if needed, try the next higher precision casting option until the desired \naccuracy and performance are achieved. A typical flow can be:\n\n1. You can compile without options (default) or with ``--fast-math all`` which will optimize for performance.\n\n2. If accuracy is not sufficient you can try ``--fast-math fp32-cast-matmult`` \n\n3. If accuracy is not sufficient you can try ``--fast-math fp32-cast-matmult no-fast-relayout``\n\n4. If accuracy is not sufficient you can try ``--fast-math none`` which will optimize for accuracy .\n\n \nBetween step 2 and step 3, and between step 3 and step 4 you have additional options that can provide different level of accuracy and which are explained in the below section.\n\nNote that compiler has to preserve the input/output (i/o) tensor types requested by Framework, therefore no casting is done on the i/o tensors. Additional speedup can be obtained by casting them in the Framework prior compilation.\n\nTo learn how to use compiler command line interface (CLI) options with your application's framework, please see :ref:`torch_neuron_trace_api`, :ref:`tensorflow-ref-neuron-compile-api` and :ref:`tensorflow-ref-neuron-tracing-api`.\n\n\nCompiler casting options\n------------------------\n\n``--fast-math`` option\n^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe ``--fast-math`` option is intended to replace the ``--fp32-cast`` option. It is recommended to\nto start using or migrating to ``--fast-math`` option. The ``--fast-math`` option provides the same level of functionality\nas the ``--fp32-cast`` option in addition to the following:\n\n* The ``--fast-math`` option introduces the ``no-fast-relayout`` option to enable lossless transpose operation. This was not possible with the ``--fp32-cast`` option.\n* The ``--fast-math`` option provides finer control than the ``--fp32-cast`` option. The transpose operation and the cast operation are controlled independently:\n\n - ``no-fast-relayout`` and ``fast-relayout`` provide control for the transpose operation.\n - ``fp32-cast-*`` provide control for casting.\n\nSee the detailed list of the options in :ref:`/compiler/neuron-cc/command-line-reference.rst`.\n" }, { "path": "about-neuron/appnotes/neuron1x/important-neuronx-dkms.txt", "content": ".. important ::\n\n Starting with Neuron version 2.3, the ``aws-neuron-dkms`` package name has been changed to ``aws-neuronx-dkms``. See :ref:`neuron2-intro`\n" }, { "path": "about-neuron/appnotes/neuron1x/introducing-libnrt.rst", "content": ".. _introduce-libnrt:\n\nIntroducing Neuron Runtime 2.x (libnrt.so) \n==========================================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we changing?\n---------------------\n\nStarting with the *Neuron 1.16.0* release, *Neuron Runtime 1.x* (``neuron-rtd``) is entering maintenance mode and is being replaced by *Neuron Runtime 2.x*, a shared library named (``libnrt.so``). For more information on Runtime 1.x see :ref:`maintenance_rtd`.\n\nUpgrading to ``libnrt.so`` simplifies the Neuron installation and upgrade process, introduces new capabilities for allocating NeuronCores \nto applications, streamlines container creation, and deprecates tools that are no longer needed.\n\nThis document describes the capabilities of *Neuron Runtime 2.x* in detail, provides information needed for successful installation and upgrade, \nand provides information needed for successful upgrade of Neuron applications using *Neuron Runtime 1.x* (included in releases before *Neuron 1.16.0*)\nto *Neuron Runtime 2.x* (included in releases *Neuron 1.16.0* or newer).\n\n.. _introduce-libnrt-why:\n\nWhy are we making this change?\n------------------------------\n\nBefore *Neuron 1.16.0*, Neuron Runtime was delivered as a daemon (``neuron-rtd``), and communicated with Neuron framework extensions through a ``gRPC`` interface. \n``neuron-rtd`` was packaged as an ``rpm`` or ``debian`` package (``aws-neuron-runtime``) and required a separate installation step.\n\nStarting with *Neuron 1.16.0*, *Neuron Runtime 2.x* is delivered as a shared\nlibrary (``libnrt.so``) and is directly linked to Neuron framework extensions.\n``libnrt.so`` is packaged and installed as part of the Neuron framework extensions\n(e.g. TensorFlow Neuron, PyTorch Neuron or MXNet Neuron), and does not require a\nseparate installation step. Installing Neuron Runtime as part of the Neuron\nframework extensions simplifies installation and improves the user experience.\nIn addition, since ``libnrt.so`` is directly linked to the Neuron framework\nextensions, faster communication between the Neuron Runtime and\nNeuron Frameworks is enabled by eliminating the ``gRPC`` interface overhead.\n\nFor more information see :ref:`introduce-libnrt-how-sdk` and :ref:`neuron-migrating-apps-neuron-to-libnrt`.\n\n\n.. _libnrt-neuron-cmponents:\n\n.. _introduce-libnrt-how-sdk:\n\nHow will this change affect the Neuron SDK?\n-------------------------------------------\n\nNeuron Driver\n^^^^^^^^^^^^^\n\nUse the latest Neuron Driver. For successful installation and upgrade to *Neuron 1.16.0* or newer, \nyou must install or upgrade to Neuron Driver (``aws-neuron-dkms``) *version 2.1.5.0* or newer. Neuron applications using *Neuron 1.16.0* will fail if \nthey do not detect *Neuron Driver version 2.1.5.0* or newer. For installation and upgrade instructions see :ref:`install-guide-index`.\n\n\n.. include:: ./important-neuronx-dkms.txt\n\nTo see details of Neuron component versions please see :ref:`latest-neuron-release-artifacts`.\n\n.. important ::\n\n For successful installation or update to Neuron 1.16.0 and newer from previous releases:\n * Stop Neuron Runtime 1.x daemon (``neuron-rtd``) by running: ``sudo systemctl stop neuron-rtd``\n * Uninstall ``neuron-rtd`` by running: ``sudo apt remove aws-neuron-runtime`` or ``sudo dnf remove aws-neuron-runtime``\n * Install or upgrade to the latest Neuron Driver (``aws-neuron-dkms``) by following the :ref:`install-guide-index` instructions.\n * Starting with Neuron version 2.3, ``aws-neuron-dkms`` the package name has been changed to ``aws-neuronx-dkms``, see :ref:`neuron2-intro`\n\n\nNeuron Runtime\n^^^^^^^^^^^^^^\n\n* Installation\n Starting from *Neuron 1.16.0*, Neuron releases will no longer include the ``aws-neuron-runtime packages`` and Neuron Runtime will be part of the Neuron \n framework extension of choice (TensorFlow Neuron, PyTorch Neuron or MXNet Neuron). Installing any Neuron framework package will install the Neuron Runtime library \n (``libnrt.so``).\n\n * For installation and upgrade instructions see :ref:`install-guide-index`.\n\n* Configuring *Neuron Runtime*\n Before *Neuron 1.16.0*, *Neuron Runtime 1.x* was configured in configuration files (e.g. /opt/aws/neuron/config/neuron-rtd.config).\n Starting from *Neuron 1.16.0*, *Neuron Runtime 2.x* can be configured through environment variables. See :ref:`nrt-configuration` for details. \n\n* Starting and Stopping *Neuron Runtime*\n Before introducing ``libnrt.so``, ``neuron-rtd`` ran as a daemon that communicated through a ``gRPC`` interface. Whenever ``neuron-rtd`` took ownership of a Neuron device, \n it continued owning that device until it was stopped. This created the need to stop ``neuron-rtd`` in certain cases. With the introduction of ``libnrt.so``, *Neuron Runtime* as it runs inside the context of the application. With *Neuron Runtime 2.x*, the act of starting and stopping a Neuron application causes ``libnrt.so`` to automatically claim or release ownership of the required Neuron devices.\n \n\n* NeuronCore Groups (NCG) end-of-support\n Before the introduction of *Neuron Runtime 2.x*, NeuronCore Group (NCG) was used to define an execution group of one or more NeuronCores \n where models could be loaded and executed. It also provided separation between processes.\n \n With the introduction of *Neuron Runtime 2.x*, strict separation of NeuronCores into groups is no longer necessary and NeuronCore Groups (NCG) has been \n deprecated. See :ref:`eol-ncg` for more information.\n\n* Running multiple *Neuron Runtimes*\n Before the introduction of ``libnrt.so``, it was necessary to run multiple ``neuron-rtd`` daemons to allocate Neuron devices for each ``neuron-rtd``, \n using configuration files.\n After the introduction of ``libnrt.so``, it will no longer necessary to run multiple ``neuron-rtd`` daemons to allocate Neuron devices to a specific Neuron application. \n With ``libnrt.so`` NeuronCores (A Neuron device includes multiple NeuronCores) are allocated to a particular application by using ``NEURON_RT_VISIBLE_CORES`` or ``NEURON_RT_NUM_CORES``\n environment variables, for example:\n\n .. code ::\n\n NEURON_RT_VISIBLE_CORES=0-3 myapp1.py\n NEURON_RT_VISIBLE_CORES=4-11 myapp2.py\n\n Or\n\n .. code ::\n\n NEURON_RT_NUM_CORES=3 myapp1.py &\n NEURON_RT_NUM_CORES=4 myapp2.py &\n\n\n\n See :ref:`nrt-configuration` for details. \n\n* Logging\n Similar to *Neuron Runtime 1.x*, *Neuron Runtime 2.x* logs into syslog (verbose logging). To make debugging easier, *Neuron Runtime 2.x* also logs into the console (error-only logging). Refer to :ref:`nrt-configuration` to see how to increase or decrease logging verbosity.\n\n* Multi-process access to NeuronCores\n With the introduction of ``libnrt.so``, it is no longer possible to load models from multiple processes on the same NeuronCore. \n A NeuronCore can only be accessed from the same process. Instead you can load models on a specific NeuronCore, using multiple threads from the same process.\n\n .. note::\n\n For optimal performance of multi-model execution, each NeuronCore executes a single model.\n\n\n* Neuron Runtime architecture\n *Neuron Runtime 2.x* is delivered as a shared library (``libnrt.so``) and is directly linked to Neuron framework extensions.\n ``libnrt.so`` is packaged and installed as part of Neuron framework extensions \n (e.g. TensorFlow Neuron, PyTorch Neuron, or MXNet Neuron), and does not require a \n separate installation step. Installing Neuron Runtime as part of the Neuron \n framework extensions simplifies installation and improves the user experience. \n In addition, since ``libnrt.so`` is directly linked to Neuron framework \n extensions, it enables faster communication between Neuron Runtime and \n Neuron Frameworks by eliminating ``gRPC`` interface overhead.\n\n\nNeuron framework extensions\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nStarting from *Neuron 1.16.0*, Neuron framework extensions (TensorFlow Neuron, PyTorch Neuron, or MXNet Neuron) are packaged together with \n``libnrt.so``. It is required to install the ``aws-neuron-dkms`` Driver version 2.1.5.0 or newer for proper operation. The ``neuron-rtd`` daemon \nthat was installed in previous releases no longer works starting with Neuron 1.16.0.\n\nTo see details of Neuron component versions see :ref:`latest-neuron-release-artifacts`.\n\n.. :important:\n\n Starting Neuron version 2.3, the ``aws-neuron-dkms`` package name is changed to ``aws-neuronx-dkms``, see :ref:`neuron2-intro`\n\nTensorFlow model server\n^^^^^^^^^^^^^^^^^^^^^^^\n\nStarting from *Neuron 1.16.0*, the TensorFlow Neuron model server is packaged together with ``libnrt.so`` and expects ``aws-neuron-dkms`` \n*version 2.1.5.0* or newer for proper operation.\n\n\n.. note::\n\n The TensorFlow Neuron model server included in *Neuron 1.16.0* runs from the directory in which it was installed and will not run properly if copied to a different location, due to its dependency on ``libnrt.so``.\n\n.. include:: ./important-neuronx-dkms.txt\n\n\n\nNeuron tools\n^^^^^^^^^^^^\n\n* ``neuron-cli`` - Starting from *Neuron 1.16.0*, ``neuron-cli`` enters maintenance mode. See :ref:`maintenance_neuron-cli` for more information.\n* ``neuron-top`` - Starting from *Neuron 1.16.0*, ``neuron-top`` has a new user interface. See :ref:`neuron-top-ug` for more information.\n* ``neuron-monitor`` - ``neuron-monitor`` was updated to support Neuron Runtime 2.x (``libnrt.so``)\n\n * See :ref:`neuron-monitor-ug` for an updated user guide of ``neuron-monitor``.\n * See neuron-monitor upgrade notes for a list of changes between *Neuron Monitor 2.x* and *Neuron Monitor 1.0*\n * See neuron-monitor backward compatibility notes for instructions for using *Neuron Monitor 2.x* with *Neuron Runtime 1.x* (``neuron-rtd``) .\n\n\n\n.. _introduce-libnrt-how-user:\n\nHow will this change affect me?\n-------------------------------\n\nNeuron installation and upgrade\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\nAs explained in \":ref:`libnrt-neuron-cmponents`\", starting from *Neuron 1.16.0*, ``libnrt.so`` requires the latest Neuron Driver (``aws-neuron-dkms``). \nIn addition, it is no longer necessary to install ``aws-neuron-runtime``. To install Neuron or to upgrade to latest Neuron version, follow the \ninstallation and upgrade instructions below:\n\n* PyTorch Neuron\n * :ref:`install-neuron-pytorch`.\n * :ref:`update-neuron-pytorch`.\n\n* TensorFlow Neuron\n * :ref:`install-neuron-tensorflow`.\n * :ref:`update-neuron-tensorflow`.\n\n* MXNet Neuron\n * :ref:`install-neuron-mxnet`.\n * :ref:`update-neuron-mxnet`.\n\n\n.. include:: ./important-neuronx-dkms.txt\n\n\n.. _neuron-migrating-apps-neuron-to-libnrt:\n\nMigrate your application to Neuron Runtime 2.x (libnrt.so) \n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor a successful migration from previous releases of your application to *Neuron 1.16.0* or newer, make sure you perform the following:\n\n#. Prerequisite\n Read \":ref:`libnrt-neuron-cmponents`\".\n\n#. Make sure you are not using *Neuron Runtime 1.x* (``aws-neuron-runtime``) \n * Remove any code that installs ``aws-neuron-runtime`` from any CI/CD scripts.\n * Stop ``neuron-rtd`` by running ``sudo systemctl stop neuron-rtd``\n * Uninstall ``neuron-rtd`` by running ``sudo apt remove aws-neuron-runtime`` or ``sudo dnf remove aws-neuron-runtime``\n\n\n#. Upgrade to your Neuron Framework of choice:\n * :ref:`update-neuron-pytorch`.\n * :ref:`update-neuron-tensorflow`.\n * :ref:`update-neuron-mxnet`.\n\n\n#. If you have code that starts and/or stops ``neuron-rtd``\n Remove any code that starts or stops ``neuron-rtd`` from any CI/CD scripts.\n \n\n\n#. Application running multiple ``neuron-rtd``\n If your application runs multiple processes and requires running multiple ``neuron-rtd`` daemons:\n\n * Remove the code that runs multiple ``neuron-rtd`` daemons.\n * Instead of allocating Neuron devices to ``neuron-rtd`` through configuration files, use ``NEURON_RT_VISIBLE_CORES`` or ``NEURON_RT_NUM_CORES`` environment variables to\n allocate NeuronCores. See :ref:`nrt-configuration` for details.\n\n If you application uses ``NEURONCORE_GROUP_SIZES``, see the next item.\n\n\n .. note::\n\n ``NEURON_RT_VISIBLE_CORES`` and ``NEURON_RT_NUM_CORES`` environment variables enable you to allocate NeuronCores to an application. Allocating NeuronCores improves application granularity, because Neuron devices include multiple NeuronCores.\n\n#. Application running multiple processes using ``NEURONCORE_GROUP_SIZES``\n * Consider using ``NEURON_RT_VISIBLE_CORES`` or ``NEURON_RT_NUM_CORES`` environment variables instead of ``NEURONCORE_GROUP_SIZES``, which is being deprecated. See :ref:`nrt-configuration` for details.\n\n * If you are using TensorFlow Neuron (``tensorflow-neuron (TF2.x)``) and you are replacing ``NEURONCORE_GROUP_SIZES=AxB`` which enables auto multicore replication, see the new API :ref:`tensorflow-ref-auto-replication-python-api` for usage and documentation.\n \n * The behavior of your application will remain the same as before if you do not set ``NEURON_RT_VISIBLE_CORES`` and do not set ``NEURON_RT_NUM_CORES``.\n\n * If you are considering migrating to ``NEURON_RT_VISIBLE_CORES`` or ``NEURON_RT_NUM_CORES``:\n\n * ``NEURON_RT_VISIBLE_CORES`` takes precedence over ``NEURON_RT_NUM_CORES``.\n\n * If you are migrating to ``NEURON_RT_VISIBLE_CORES``:\n\n * For TensorFlow applications or PyTorch applications make sure that ``NEURONCORE_GROUP_SIZES`` is unset, or that ``NEURONCORE_GROUP_SIZES`` allocates the same or smaller number of NeuronCores as allocated by ``NEURON_RT_VISIBLE_CORES``.\n * For MXNet applications, setting ``NEURONCORE_GROUP_SIZES`` and ``NEURON_RT_VISIBLE_CORES`` environment variables at the same time is not supported. Use ``NEURON_RT_VISIBLE_CORES`` only.\n * See :ref:`nrt-configuration` for more details on how to use ``NEURON_RT_VISIBLE_CORES``.\n\n\n * If you are migrating to ``NEURON_RT_NUM_CORES``:\n\n * Make sure that ``NEURONCORE_GROUP_SIZES`` is unset.\n * See :ref:`nrt-configuration` for more details on how to use ``NEURON_RT_NUM_CORES``.\n\n\n#. Application running multiple processes accessing the same NeuronCore\n If your application accesses the same NeuronCore from multiple processes, this is no longer possible with ``libnrt.so``.\n Instead, modify your application to access the same NeuronCore from multiple threads.\n\n .. note::\n\n Optimal performance of multi-model execution is achieved when each NeuronCore executes a single model.\n\n\n#. Neuron Tools\n * If you are using Neuron Monitor, see the neuron-monitor upgrade notes for details.\n * If you are using ``neuron-cli`` remove any call to ``neuron-cli``. For more information, see :ref:`maintenance_neuron-cli`.\n\n\n\n#. Containers\n If your application is running within a container, and it previously executed ``neuron-rtd`` within the container, you need\n to re-build your container, so it will not include or install ``aws-neuron-runtime``. See :ref:`neuron-containers` for details.\n\n\n\nTroubleshooting\n---------------\n\nApplication fails to start\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDescription\n~~~~~~~~~~~\n\nStarting with the *Neuron 1.16.0* release, Neuron Runtime (``libnrt.so``) requires *Neuron Driver 2.0* or greater (``aws-neuron-dkms``). Neuron Runtime requires the Neuron Driver (``aws-neuron-dkms`` package) to access Neuron devices. \n\nIf ``aws-neuron-dkms`` is not installed, the application will fail with an error message on the console and syslog similar to the following:\n\n.. code::\n\n NRT:nrt_init Unable to determine Neuron Driver version. Please check aws-neuron-dkms package is installed.\n\nIf an old ``aws-neuron-dkms`` is installed, the application will fail with an error message on the console and syslog similar to the following:\n\n.. code::\n\n NRT:nrt_init This runtime requires Neuron Driver version 2.0 or greater. Please upgrade aws-neuron-dkms package.\n\n\n\n\nSolution\n~~~~~~~~\n\nFollow the installation steps in :ref:`install-guide-index` to install ``aws-neuron-dkms``.\n\n.. include:: ./important-neuronx-dkms.txt\n\n\nApplication fails to start although I installed latest ``aws-neuron-dkms``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDescription\n~~~~~~~~~~~\n\nStarting from the *Neuron 1.16.0* release, Neuron Runtime (``libnrt.so``) requires *Neuron Driver 2.0* or greater (``aws-neuron-dkms``). If an old ``aws-neuron-dkms`` is installed, the application will fail. You may try to install ``aws-neuron-dkms`` and still face application failure, because the ``aws-neuron-dkms`` installation failed as a result of ``neuron-rtd`` daemon that was still running.\n\n\nSolution\n~~~~~~~~\n\n* Stop ``neuron-rtd`` by running: ``sudo systemctl stop neuron-rtd``\n* Uninstall ``neuron-rtd`` by running: ``sudo apt remove aws-neuron-runtime`` or sudo ``dnf remove aws-neuron-runtime``\n* Install ``aws-neuron-dkms`` by following steps in :ref:`install-guide-index`\n\n.. include:: ./important-neuronx-dkms.txt\n\n\nApplication unexpected behavior when upgrading to release *Neuron 1.16.0* or newer \n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDescription\n~~~~~~~~~~~\n\nWhen upgrading to release *Neuron 1.16.0* or newer from previous releases, the OS may include two different versions of \n*Neuron Runtime*: the ``libnrt.so`` shared library and ``neuron-rtd`` daemon. This can happen if the user did not stop ``neuron-rtd`` daemon\nor did not make sure to uninstall the existing Neuron version before upgrade. \nIn this case the user application may behave unexpectedly.\n\nSolution\n~~~~~~~~\n\nIf the OS includes two different versions of *Neuron Runtime*, ``libnrt.so`` shared library and ``neuron-rtd`` daemon:\n\n * Before running applications that use ``neuron-rtd``, restart ``neuron-rtd`` by calling ``sudo systemctl restart neuron-rtd``.\n * Before running applications linked with ``libnrt.so``, stop ``neuron-rtd`` by calling ``sudo systemctl stop neuron-rtd``.\n\n\nApplication unexpected behavior when downgrading to releases before *Neuron 1.6.0* (from *Neuron 1.16.0* or newer)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDescription\n~~~~~~~~~~~\n\nWhen upgrading to release *Neuron 1.16.0* or newer from previous releases, and then downgrading back to releases before *Neuron 1.6.0*, \nthe OS may include two different versions of *Neuron Runtime*: the ``libnrt.so`` shared library and ``neuron-rtd`` daemon. This can happen \nif the user did not make sure to uninstall the existing Neuron version before the upgrade or downgrade.\nIn this case the user application may behave unexpectedly.\n\nSolution\n~~~~~~~~\n\nIf the OS include two different versions of *Neuron Runtime*, ``libnrt.so`` shared library and ``neuron-rtd`` daemon:\n\n * Before running applications that use ``neuron-rtd``, restart ``neuron-rtd`` by calling ``sudo systemctl restart neuron-rtd``.\n * Before running applications linked with ``libnrt.so``, stop ``neuron-rtd`` by calling ``sudo systemctl stop neuron-rtd``.\n\n\n\nNeuron Core is in use\n^^^^^^^^^^^^^^^^^^^^^\n\nDescription\n~~~~~~~~~~~\n\nA Neuron Core cannot be shared between two applications. If an application\nstarted using a Neuron Core all other applications trying to use the\nNeuronCore will fail during runtime initialization with the following\nmessage in the console and in syslog:\n\n.. code:: bash\n\n ERROR NRT:nrt_allocate_neuron_cores NeuronCore(s) not available - Requested:nc1-nc1 Available:0\n\nSolution\n~~~~~~~~\n\nTerminate the the process using NeuronCore and then try launching the application.\n\nFrequently Asked Questions (FAQ)\n--------------------------------\n\nDo I need to recompile my model to run it with Neuron Runtime 2.x (``libnrt.so``)?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNo. \n\nDo I need to change my application launch command?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNo.\n\n\nCan ``libnrt.so`` and ``neuron-rtd`` co-exist in the same environment?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAlthough we recommend upgrading to the latest Neuron release, we understand that for a transition period you may continue using ``neuron-rtd`` for old releases. If you are using Neuron Framework (PyTorch,TensorFlow or MXNet) from releases before *Neuron 1.16.0*: \n\n* Install the latest Neuron Driver (``aws-neuron-dkms``) \n\n.. include:: ./important-neuronx-dkms.txt\n\n* For development, we recommend using different environments for Neuron Framework (PyTorch,TensorFlow or MXNet) from releases before *Neuron 1.16.0* and for Neuron \n Framework (PyTorch,TensorFlow or MXNet) from *Neuron 1.16.0* and newer. If that is not possible, make sure to stop ``neuron-rtd`` before executing models using\n Neuron Framework (PyTorch,TensorFlow or MXNet) from *Neuron 1.16.0* and newer.\n\n* For deployment, when you are ready to upgrade, upgrade to Neuron Framework (PyTorch,TensorFlow or MXNet) from *Neuron 1.16.0* and newer. \n See :ref:`neuron-migrating-apps-neuron-to-libnrt` for more information.\n\n\n.. warning ::\n\n Executing models using Neuron Framework (PyTorch,TensorFlow or MXNet) from *Neuron 1.16.0* and newer in an environment where ``neuron-rtd`` is running may cause\n undefined behavior. Make sure to stop ``neuron-rtd`` before executing models using Neuron Framework (PyTorch,TensorFlow or MXNet) from *Neuron 1.16.0* and newer.\n\nAre there Neuron framework versions that will not support Neuron Runtime 2.x (``libnrt.so``)?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAll supported PyTorch Neuron and TensorFlow framework extensions, in addition to Neuron MXnet 1.8.0 framework extensions support Neuron Runtime 2.x.\n\nNeuron MxNet 1.5.1 does not support Neuron Runtime 2.x (``libnrt.so``) and has now entered maintenance mode. See :ref:`maintenance_mxnet_1_5` for details.\n" }, { "path": "about-neuron/appnotes/neuronx-cc/neuronx-cc-training-mixed-precision.rst", "content": ".. _neuronx-cc-training-mixed-precision:\n\nMixed Precision and Performance-accuracy Tuning (``neuronx-cc``)\n================================================================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nOverview\n--------\n\nThe Neuron Compiler supports machine learning models with FP32, TF32, FP16 and BF16 (Bfloat16) tensors and operators. The Neuron hardware supports a mix of 32, 16, and 8 bit datatypes. This guide explains how to apply the available auto-cast methods and their performance / accuracy trade-offs when compiling a model with Neuron.\n\n.. note:: Neuron Compiler support for INT8 is planned for a future Neuron SDK release. See `Neuron Compiler: Enable Neuron INT8 support `_ for details.\n\nNeuron Hardware\n---------------\n\nThe Neuron v2 hardware supports matrix multiplication using FP16, BF16, TF32, and FP32 on its matrix multiply (\"matmult\") engine, and accumulations using FP32. Operators such as activations or vector operations are supported using FP32, TF32, FP16, and BF16. Supporting FP16 and BF16 allows Neuron to have significantly higher performance than executing everything as FP32.\n\n\nPerformance-accuracy tradeoffs\n------------------------------\n\n**By default**, the Neuron Compiler will **automatically cast FP32 matrix multiplication operations to BF16**. The remaining operations are performed in the data type specified by the model. The Neuron Compiler provides CLI options that direct the compiler to cast to other data types, thereby giving the ability to choose an accuracy-to-performance tradeoff in model execution. Deciding what CLI settings to use will be application specific and may require some experimentation. See :ref:`Neuron Compiler CLI Reference Guide` for details.\n\n\nWhat is the difference between Data Types?\n-------------------------------------------\n\nThe NeuronCore v2 support multiple data types (see :ref:`NeuronCore v2 Data Types`). Each data type provides benefits and drawbacks due to its dynamic range and numeric precision.\n\n+------+-----------+----------+--------------------------------------------------------+---------------------------------------------------+\n| Type | Minimum | Maximum | Strength | Weakness |\n+======+===========+==========+========================================================+===================================================+\n| FP16 | -65504 | 65504 |\tNumeric Precision, High granularity, Mid-range numbers | Low range, medium precision |\n+------+-----------+----------+--------------------------------------------------------+---------------------------------------------------+\n| BF16 | -3.40E+38 | 3.40E+38 |\tDynamic Range, Extremely small/large numbers | Low precision |\n+------+-----------+----------+--------------------------------------------------------+---------------------------------------------------+\n| TF32 | -3.40E+38 | 3.40E+38 |\tDynamic Range, Extremely small/large numbers | Medium precision |\n+------+-----------+----------+--------------------------------------------------------+---------------------------------------------------+\n| FP32 | -3.40E+38 | 3.40E+38 | N/A | Larger model size, potentially slower computation |\n+------+-----------+----------+--------------------------------------------------------+---------------------------------------------------+\n\n* FP16 provides a high density of representable values that are neither extremely small or extremely large. The density of representable values within the range is approximately an order of magnitude greater than BF16.\n\n * Conversion from FP32 to FP16 will perform well when values are relatively small but non-extreme (either very small or very large).\n * Conversion from FP32 to FP16 will perform badly if the original FP32 values are outside of the range of FP16. This will produce inf/-inf values and may result in NaN depending on the operation.\n\n* BF16 provides a wider range of representable values which includes both very small and very large values. However, the overall density of representable values is usually lower than FP16 for more non-extreme values. The range is nearly identical to the range of FP32 but because the number of bits is halved, this means the individual values are sparse.\n\n * Conversion from FP32 to BF16 will perform well when the values are well-distributed throughout the range. Since BF16 covers the entire FP32 range, this means each original value can map to a relatively close downcast value.\n * Conversion from FP32 to BF16 will perform badly when fine granularity is needed. Since BF16 granularity is sacrificed for greater range it will almost always map worse to values that are within the FP16 range.\n\nShould I downcast operations to smaller Data Types?\n---------------------------------------------------\n\nThis choice here is driven entirely by accuracy vs performance tradeoff. Casting operations to smaller 16-bit data types will provide a significant performance benefit but may end up sacrificing accuracy.\n\nThe compiler uses BF16 casting **by default** for matrix multiplication operations. The speedup from casting operations gives a significant performance boost and the range of representable values in BF16 allows for more safety compared to FP16 when the possible numeric range of input values is unknown.\n\nThe Neuron Compiler's ``--auto-cast`` and ``--auto-cast-type`` CLI options are used to direct the compiler to perform alternate casting operations. See the detailed list of the options in :ref:`Neuron v2 Compiler CLI Reference Guide`. The default setting is ``--auto-cast=none``, which is applied if the ``--auto-cast`` flag is not provided.\n\n\nThe option combinations to consider in a typical flow are:\n\n\n+---------------------------------------------------------+--------------------------------------------------------------------------+-----------------------------------------------------+-------------------------------------------------+\n| Compiler autocast | Options Effect | Performance | Accuracy |\n+=========================================================+==========================================================================+=====================================================+=================================================+\n| ``--auto-cast none`` (default) | Disables all auto-casting, using the data types defined within the model | Lowest performance | Highest accuracy |\n+---------------------------------------------------------+--------------------------------------------------------------------------+-----------------------------------------------------+-------------------------------------------------+\n| ``--auto-cast matmult --auto-cast-type tf32`` | | Performance *increases* as you move down the table | Accuracy *decreases* as you move down the table |\n+---------------------------------------------------------+--------------------------------------------------------------------------+ | |\n| ``--auto-cast all —-auto-cast-type tf32`` | Balance of performance, dynamic range, and precision | | |\n+---------------------------------------------------------+--------------------------------------------------------------------------+ | |\n| ``--auto-cast matmult --auto-cast-type fp16`` | | | |\n+---------------------------------------------------------+--------------------------------------------------------------------------+ | |\n| ``--auto-cast all —-auto-cast-type fp16`` | Best performance at the expense of dynamic range | | |\n+---------------------------------------------------------+--------------------------------------------------------------------------+ | |\n| ``--auto-cast matmult --auto-cast-type bf16`` | Best performance at the expense of precision | | |\n+---------------------------------------------------------+ + ----------------------------------------------------+-------------------------------------------------+\n| ``--auto-cast all --auto-cast-type bf16`` | | Highest performance | Lowest accuracy |\n+---------------------------------------------------------+--------------------------------------------------------------------------+-----------------------------------------------------+-------------------------------------------------+\n\nNote that compiler has to preserve the input/output (i/o) tensor types requested by Framework, therefore no casting is done on the i/o tensors. Additional speedup can be obtained by casting them in the Framework prior to compilation.\n\nTo learn how to configure the compiler options from within your application’s framework, please see:\n\n* :ref:`Developer Guide for Training with PyTorch Neuron `\n" }, { "path": "about-neuron/appnotes/neuronx-distributed/introducing-nxd-inference.rst", "content": ".. _introduce-nxd-inference:\n\nIntroducing NeuronX Distributed (NxD) Inference\n=================================================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nWhat are we introducing?\n------------------------\n\n\nStarting with the Neuron SDK 2.21 release, we are introducing NxD Inference, an open-source PyTorch-based inference library that simplifies deep learning model deployment on AWS Inferentia and Trainium instances. NxD Inference is designed for optimized inference, enabling quick onboarding of PyTorch models with minimal changes. It features a modular architecture that facilitates easy integration of HuggingFace PyTorch models and is compatible with serving engines like vLLM.\n\nPlease see :ref:`nxdi-index` for NxD Inference overview and documentation.\n\n\nHow can I install NxD Inference library?\n-----------------------------------------\nPlease refer to :ref:`nxdi-setup` for installation instructions.\n\n\nI am currently using the Transformers NeuronX library for inference. How does the NxD Inference library affect me?\n--------------------------------------------------------------------------------------------------------------------\n\nIf you are using Transformers NeuronX (TNx) in production, you can continue doing so. However, if you are planning to onboard new models to Neuron for inference, NxD Inference offers several advantages to consider.\n\nNxD Inference is designed to enable easy on-boarding of PyTorch models and comes with new features and enhanced support:\n\n* **Hardware Support**: While TNx is not supported on Trn2, NxD Inference supports all platforms (Trn1, Inf2, and Trn2)\n* **Simplified interface**: To simplify model development with NxD Inference, you write modeling code using PyTorch with standard Python, rather than using PyHLO as in TNx.\n* **Easy Migration**: NxD Inference was designed to provide seamless migration from TNx, especially if you are using it with vLLM. You can migrate your existing TNx inference scripts using the :ref:`migration guide `\n* **Enhanced Capabilities**: NxD Inference offers more comprehensive support for MoE models and multimodal models (Llama 3.2) compared to TNx\n* **Future Development**: New inference features and support for advanced model architectures (like multi-modality/video models) will be focused on NxD Inference\n\n\n\nI am currently using vLLM with Transformers NeuronX library for inference. Does NxD Inference library support vLLM ?\n---------------------------------------------------------------------------------------------------------------------\n\nYes, NxD Inference library supports vLLM inference engine. Neuron vLLM integration in 2.21 release will start supporting both NxD Inference and Transformers NeuronX libraries. To use vLLM with NxD Inference library, you can refer to the :ref:`nxdi-vllm-user-guide-v1`.\n\n\n\nWhat features and models are available in Transformers NeuronX (TNx) but not yet in NeuronX Distributed Inference?\n-------------------------------------------------------------------------------------------------------------------\n\nWhile NxD Inference supports most features and models available in TNx, there are some differences in current support that users should be aware of.\n\n**Features that are not yet supported in NxD Inference**: The following TNx features aren't supported yet in the NxD Inference library.\n\n* Multi-Node Inference support\n\n\n**Models not part of NxD Inference Model Hub**: The following models are included in Transformers NeuronX but not currently in NxD Inference library:\n\n* Bloom\n* GPT2\n* GPT-J\n* GPT-NEOX\n\nIf you need to use these models with NxD Inference, we encourage you to follow the :ref:`onboarding models developer guide `. The onboarding process in NxD Inference is more straightforward compared to TNx due to its PyTorch-based architecture.\n\n\nI currently use Hugging Face TGI serving engine for deploying and serving Large Language Models (LLMs) on Neuron. How does NxD Inference library affect me?\n-----------------------------------------------------------------------------------------------------------------------------------------------------------\n\nIf you are currently using Hugging Face TGI serving engine to deploy models on Neuron, the introduction of NxD Inference library will not have any impact and you can continue to use your existing inference workloads. Hugging Face TGI integrates with Neuron SDK Inference libraries in a way that abstracts the underlying library for the users.\n\n\n\nI am new to Neuron and have inference workloads, what library should I use?\n----------------------------------------------------------------------------\n\nWe recommend you use NxD Inference for your model inference workloads. To learn how to get started using NxD Inference, see the :ref:`nxdi-index` documentation\n\n\n\n\n\n\n\n\nAdditional Resources\n--------------------\n\n* :ref:`nxdi-index`\n* :ref:`nxdi-overview`\n* :ref:`nxd-inference_rn`\n" }, { "path": "about-neuron/appnotes/neuronx-distributed/introducing-nxdt-training.rst", "content": ".. _introduce-nxd-training:\n\nIntroducing NxD Training\n===================================================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we introducing?\n------------------------\n\nStarting with the Neuron 2.20 release, we are introducing NxD Training. \nIn doing so, we are expanding NeuronX Distributed library (previously called NxD that will now be called NxD Core) to \nNxD Training with data science/engineering modules, and end to end examples. NxD Training is a PyTorch based \ndistributed training library that enables customers to train large-scale models. Some key distributed strategies \nsupported by NxD Training include 3D-parallelism (data parallelism, tensor parallelism and pipeline parallelism) and \nZeRO-1 (where optimizer states are partitioned across workers). \n \nNxD Training supports model training workflows like pretraining, supervised finetuning (SFT) and parameter efficient \nfinetuning (PEFT) using Low-Rank Adapter (LoRA) techniques [#f1]_. For developers, NxD Training offers both API level access \nthrough NxD Core and PyTorch Lightning and an intuitive interface via YAML based configuration files. NxD Training \noffers a flexible approach that enables customers to leverage only the functionalities that align with their unique \nworkflows and seamlessly integrate their machine learning training software at the appropriate level within NxD Training, \nensuring a user experience tailored to their specific requirements. This is a beta preview version of NxD Training \nand feedback from the developer community is strongly encouraged for upcoming releases.\n\n\n\n.. _how-nxd-core-user-affected:\n\nI currently use NeuronX Distributed (NxD Core). How does NxD Training release affect me?\n---------------------------------------------------------------------------------------------------------------\n\nExisting NxD Core customers can continue to use NxD Core APIs available under NxD Training. If workflows based on NxD Core \nmeet your needs, you do not need to do anything different with NxD Training’s introduction. NxD Core APIs and \nfunctionalities for NxD Core continue to be available to you as before. You can choose to \n:ref:`install NxD Core only ` and skip all subsequent installation steps for \nNxD Training. However, NxD Training has additional support for YAML based configuration, a model hub and integration with \nPyTorch Lightning. If these capabilities are of interest to you, you may choose to evaluate and start using NxD Training. \n\n.. _should_nnm_usage_continue:\n\nShould the current Neuron NeMo Megatron (NNM) users continue to use NNM?\n------------------------------------------------------------------------------------------------\n\nNxD Training offers same capabilities as Neuron NeMo Megatron (NNM). Additionally, NNM \nwill go into maintenance mode in the next release. If you are currently using NNM, the introduction of NxD Training \ntoolkit means that you should start evaluating NxD Training for your training needs. With its YAML interface, NxD \nTraining is very close in terms of usability to NNM and NeMo. Migrating from NNM to NxD Training \nshould involve a relatively minor effort and instructions for doing so are provided \n:ref:`here `.\n\n.. _what_to_use_as_new_user:\n\nI am new to Neuron and have training workloads, what toolkits or libraries should I use?\n----------------------------------------------------------------------------------------\n\nIf you are starting with Neuron and looking for solutions to your model pretraining or finetuning needs, then NxD Training \nis the recommended toolkit for you. Please start from :ref:`NxD Training page ` for overview, \ninstallation and usage instructions.\n\n\nAdditional Resources\n------------------------\n\nMultiple NxD Training resources on getting started, using it and getting required support are listed below. If you encounter issues \nor have product related questions, please refer to FAQs and troubleshooting guides. Additionally, please feel free to reach out to us \nusing resources in Support section.\n\n:ref:`How to get started `\n\n:ref:`Release notes `\n\n:ref:`Main section `\n\n:ref:`Troubleshooting ` \n\n:ref:`Support `\n\n.. [#f1] Supported through NxD Core." }, { "path": "about-neuron/appnotes/perf/neuron-cc/parallel-ncgs.rst", "content": ".. _parallel-exec-ncgs:\n\nParallel Execution using NEURON_RT_NUM_CORES\n===============================================\n\n.. important ::\n ``NEURONCORE_GROUP_SIZES`` will no longer be supported starting with the Neuron 1.19.0 release. If your application uses ``NEURONCORE_GROUP_SIZES``\n see :ref:`neuron-migrating-apps-neuron-to-libnrt` and :ref:`eol-ncgs-env_2` for more details.\n\n\nIntroduction\n------------\n\nInf1 instances are available with a different number of Inferentia\nchips. Each Inferentia chip consists of 4 NeuronCores and an Inf1\ninstance includes 4 to 64 NeuronCores, depending on the size of the instance.\nThis guide shows you how to load one or more compiled models into\ndifferent consecutive groups of NeuronCores using your framework of choice.\n\nData Parallel Execution\n-----------------------\n\nIn PyTorch and TensorFlow, the same compiled model can run in parallel on an Inf1 instance by loading it multiple times, up to the total number of NeuronCores specified in NEURON_RT_NUM_CORES or NEURON_RT_VISIBLE_CORES. For more information about NEURON_RT_NUM_CORES and NEURON_RT_VISIBLE_CORES, refer to :ref:`Neuron Runtime Configuration `.\n\n\nRunning multiple models using single process\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo run multiple models using a single process, set the environment\nvariable ``NEURON_RT_NUM_CORES`` with a list of the\nnumber of cores in each group, separated by commas.\n\nYou can set the ``NEURON_RT_NUM_CORES`` environment variable at runtime:\n\n.. code :: bash\n\n #!/bin/bash\n NEURON_RT_NUM_CORES=13 python your_neuron_application.py\n\nOr from within the Python process running your models (NOTE: You can\nonly set it once in the same process at the beginning of the script):\n\n.. code :: bash\n\n #!/usr/bin/env python\n import os\n\n # Set Environment\n os.environ['NEURON_RT_NUM_CORES']='13'\n\n # Load models and run inferences ...\n\nThe following examples allow you to load 4 models into 4 groups of NeuronCores\nwithin one process. For example, if there are 4 models A, B, C, D\ncompiled to 2, 4, 3, and 4 NeuronCores respectively, directly load\nthe models A, B, C, D in sequence within your TensorFlow or PyTorch\nNeuron process. This example requires an inf1.6xlarge instance with 16\nNeuronCores, as the total number of NeuronCores within the NeuronCore\nGroups is 13.\n\n\n\nIn MXNet, mapping from models to NeuronCores is controlled by\ncontext ``mx.neuron(neuron_core_index)`` where ``neuron_core_index`` is the NeuronCore\nindex at the start of the group. In the example above, map model A to ``mx.neuron(0)``\ncontext, model B to ``mx.neuron(2)`` context, model C to\n``mx.neuron(6)`` context and model D to ``mx.neuron(9)`` context. For\nfurther details, refer to :ref:`Flexible Execution Group (FlexEG) in Neuron-MXNet`.\n\nFor PyTorch\n\nSee :ref:`Data Parallel Inference on Torch Neuron` for more details.\n\nFor Tensorflow\n\n.. code :: python\n\n # Set Environment \n os.environ['NEURON_RT_NUM_CORES']='13'\n\n # Load models (TF2)\n model0 = tf.keras.models.load_model(model0_file) # loaded into the first group of NC0-NC1\n model1 = tf.keras.models.load_model(model1_file) # loaded into the second group of NC2-NC5\n model2 = tf.keras.models.load_model(model1_file) # loaded into the third group of NC6-NC8\n model3 = tf.keras.models.load_model(model1_file) # loaded into the fourth group of NC9-NC12\n\n # run inference by simply calling the loaded model\n results0 = model0(inputs0)\n results1 = model1(inputs1)\n results2 = model2(inputs2)\n results3 = model3(inputs3)\n\n\nFor MXNet 2.x:\n\n.. code :: python\n\n # Set Environment\n os.environ['NEURON_RT_NUM_CORES']='13'\n\n # Load models (MXNet)\n # loaded into the first group of NC0-NC1\n sym, args, aux = mx.model.load_checkpoint(mx_model0_file, 0)\n model0 = sym.bind(ctx=mx.neuron(0), args=args, aux_states=aux, grad_req='null')\n # loaded into the second group of NC2-NC5\n sym, args, aux = mx.model.load_checkpoint(mx_model1_file, 0)\n model1 = sym.bind(ctx=mx.neuron(2), args=args, aux_states=aux, grad_req='null')\n # loaded into the third group of NC6-NC8\n sym, args, aux = mx.model.load_checkpoint(mx_model2_file, 0)\n model2 = sym.bind(ctx=mx.neuron(6), args=args, aux_states=aux, grad_req='null')\n # loaded into the fourth group of NC9-NC12\n sym, args, aux = mx.model.load_checkpoint(mx_model3_file, 0)\n model3 = sym.bind(ctx=mx.neuron(9), args=args, aux_states=aux, grad_req='null')\n\n # run inference by simply calling the loaded model\n results0 = model0.forward(data=inputs0)\n results1 = model1.forward(data=inputs1)\n results2 = model2.forward(data=inputs2)\n results3 = model3.forward(data=inputs3)\n\nYou can identify the NeuronCores used by each application with the ``neuron-top`` command\nline tool. For more information about the neuron-top user interface, see :ref:`Neuron Top User Guide `.\n\n.. code :: bash\n\n $ neuron-top\n\n.. figure:: /images/multi_1core_models_multi_processes.png\n :scale: 80 %\n\nRunning multiple models using multiple processes\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYou can also run multiple models in parallel processes, when you set\n``NEURON_RT_NUM_CORES`` per process:\n\n.. code :: bash\n\n $ NEURON_RT_NUM_CORES=2 python your_1st_neuron_application.py\n $ NEURON_RT_NUM_CORES=2 python your_2nd_neuron_application.py\n\nThe first process automatically selects a first set of 2 unused\nNeuronCores for its new group. The second process automatically selects\na new set of 2 unused NeuronCores for its new group.\n\n.. figure:: /images/multi_2cores_models_multi_processes.png\n :scale: 80 %\n\nRunning multiple models on the same NeuronCore group\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYou can load more than one model in a NeuronCore group within one\nprocess. Neuron runtime handles switching from one model to the\nnext model within the NeuronCore group, when the next model is run within\nthe application. In TensorFlow or PyTorch, simply load the additional\nmodels after the initial number of models have been loaded, to fill the\nNeuronCore groups associated with the process.\n\nFor PyTorch:\n\n.. code :: python\n\n # Set Environment\n os.environ['NEURON_RT_NUM_CORES']='2'\n\n # Load models (PT)\n model0 = torch.jit.load(model0_file) # loaded into the first group of NC0-NC1\n model1 = torch.jit.load(model1_file) # loaded into the first group of NC0-NC1\n\n # run inference by simply calling the loaded model\n results0 = model0(inputs0)\n results1 = model1(inputs1)\n\nFor TensorFlow 2.x:\n\n.. code :: python\n\n # Set Environment\n os.environ['NEURON_RT_NUM_CORES']='2'\n\n # Load models (TF2)\n model0 = tf.keras.models.load_model(model0_file) # loaded into the first group of NC0-NC1\n model1 = tf.keras.models.load_model(model1_file) # loaded into the first group of NC0-NC1\n\n # run inference by simply calling the loaded model\n results0 = model0(inputs0)\n results1 = model1(inputs1)\n\nIn MXNet, use context ``mx.neuron(neuron_core_index)`` and use the\nsame NeuronCore start index for the additional models.\n\n.. code :: python\n\n # Set Environment\n os.environ['NEURON_RT_NUM_CORES']='2'\n\n # Load models (MXNet)\n # loaded into the first group of NC0-NC1\n sym, args, aux = mx.model.load_checkpoint(mx_model0_file, 0)\n model0 = sym.bind(ctx=mx.neuron(0), args=args, aux_states=aux, grad_req='null')\n # loaded into the first group of NC0-NC1\n sym, args, aux = mx.model.load_checkpoint(mx_model1_file, 0)\n model1 = sym.bind(ctx=mx.neuron(0), args=args, aux_states=aux, grad_req='null')\n\n # run inference by simply calling the loaded model\n results0 = model0.forward(data=inputs0)\n results1 = model1.forward(data=inputs1)\n\nThe total ``NEURON_RT_NUM_CORES`` across all processes cannot exceed\nthe number of NeuronCores available on the instance. For example,\non an inf1.xlarge with default configurations where the total number of\nNeuronCores visible to TensorFlow-Neuron is 4, you can launch one\nprocess with ``NEURON_RT_NUM_CORES=2`` (pipelined) and another\nprocess with ``NEURON_RT_NUM_CORES=2`` (data-parallel).\n\nExamples using ``NEURON_RT_NUM_CORES`` include:\n\n* :ref:`PyTorch example `\n* :ref:`MXNet example `\n\n\nAuto Model Replication in TensorFlow Neuron (``tensorflow-neuron``) (Beta)\n----------------------------------------------------------------------------------\n\nRefer to the following API documentation to see how to perform automatic replication on\nmultiple cores. Note auto-replication will only work on models compiled with pipeline size 1:\nvia ``--neuroncore-pipeline-cores=1``. If automatic replication is not enabled, the model will default to \nreplicate on up to 4 cores.\n\nPython API (TF 2.x only):\n\n:ref:`tensorflow-ref-auto-replication-python-api`\n\nCLI API (TF 1.x and TF 2.x):\n\n:ref:`tensorflow-ref-auto-replication-cli-api`\n\n\nAuto Model Replication (Being Deprecated)\n-----------------------------------------\n\nThe Auto Model Replication feature in TensorFlow-Neuron enables you to\nload the model once and the data parallel replication will occur\nautomatically. This reduces framework memory usage, as the same model is not loaded multiple times. This feature is beta and\navailable in TensorFlow-Neuron only.\n\nTo enable Auto Model Replication, set NEURONCORE_GROUP_SIZES to Nx1,\nwhere N is the desired replication count (the number of NeuronCore\ngroups, each group has size 1). For example, NEURONCORE_GROUP_SIZES=8x1\nwould automatically replicate the single-NeuronCore model 8 times.\n\n.. code :: python\n\n os.environ['NEURONCORE_GROUP_SIZES'] = '4x1'\n\nor\n\n.. code :: bash\n\n NEURONCORE_GROUP_SIZES=4x1 python3 application.py\n\nWhen NEURONCORE_GROUP_SIZES is not set, the default is 4x1, where a\nsingle-NeuronCore model is replicated 4 times on any size of inf1 machine.\n\nThis feature is only available for models compiled with\nneuroncore-pipeline-cores set to 1 (default).\n\nYou will still need to use threads in the scaffolding code, to feed the\nloaded replicated model instance, to achieve high throughput.\n\nExample of auto model replication: :ref:`/src/examples/tensorflow/openpose_demo/openpose.ipynb`\n\n\nFAQ\n---\n\nCan I mix data parallel and NeuronCore Pipelines?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYes. You can compile the model using the neuroncore-pipeline-cores option.\nThis tells the compiler to set compilation to the specified number of\ncores for :ref:`neuroncore-pipeline`.\nThe Neuron Compiler returns a NEFF that fits within this limit. See\nthe :ref:`neuron-compiler-cli-reference`\nfor instructions on how to use this option.\n\nFor example, on an inf1.2xlarge, you can load two model instances, each\ncompiled with neuroncore-pipeline-cores set to 2, so they can run\nin parallel. The model instances can be loaded from different saved\nmodels or from the same saved model.\n\nCan I have a mix of multiple models in one Neuroncore group and single model in another one Neuroncore group?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCurrently, you can do this in MXNet, by setting up two Neuroncore groups, then loading,\nfor example, multiple models in one NCG, using context mx.neuron(0), and\nloading a single model in the second NCG, using context mx.neuron(2). You can\nalso load a single model in the first NCG and multiple models in the\nsecond NCG. For example:\n\n.. code :: python\n\n\n # Set Environment\n os.environ['NEURON_RT_NUM_CORES']='6'\n\n # Load models (MXNet)\n # loaded into the first group of NC0-NC1\n sym, args, aux = mx.model.load_checkpoint(mx_model0_file, 0)\n model0 = sym.bind(ctx=mx.neuron(0), args=args, aux_states=aux, grad_req='null')\n # loaded into the second group of NC2-NC5\n sym, args, aux = mx.model.load_checkpoint(mx_model1_file, 0)\n model1 = sym.bind(ctx=mx.neuron(2), args=args, aux_states=aux, grad_req='null')\n # loaded into the second group of NC2-NC5\n sym, args, aux = mx.model.load_checkpoint(mx_model2_file, 0)\n model2 = sym.bind(ctx=mx.neuron(2), args=args, aux_states=aux, grad_req='null')\n # loaded into the second group of NC2-NC5\n sym, args, aux = mx.model.load_checkpoint(mx_model3_file, 0)\n model3 = sym.bind(ctx=mx.neuron(2), args=args, aux_states=aux, grad_req='null')\n\n # run inference by simply calling the loaded model\n results0 = model0.forward(data=inputs0)\n results1 = model1.forward(data=inputs1)\n results2 = model2.forward(data=inputs2)\n results3 = model3.forward(data=inputs3)\n\nLoading multiple models in one NCG and a single model in another NCG is\ncurrently not supported in TensorFlow and PyTorch.\n" }, { "path": "about-neuron/appnotes/perf/neuron-cc/performance-tuning.rst", "content": ".. _appnote-performance-tuning:\n\nPerformance Tuning\n==================\n\n.. important ::\n NeuronCore Groups (NCG) have been deprecated. See :ref:`eol-ncg` and :ref:`neuron-migrating-apps-neuron-to-libnrt` for more details.\n\nThis guide is intended to provide the reader with an in-depth\nunderstanding of how to optimize neural network performance on\nInferentia for both throughput and latency. For simplicity, the guide\nuses the TensorFlow and ResNet-50 models as teaching examples to show how\nto choose between different compile-time optimizations (e.g., Batching and\nNeuronCore Pipeline), as well as model-serving optimizations (e.g.,\nmulti-threading and dynamic-batching) to improve inference performance.\n\nThe following guides are considered to be prerequisites for this tutorial:\n\n- :ref:`/src/examples/tensorflow/tensorflow_resnet50/resnet50.ipynb`\n- TensorFlow Serving NeuronCore Group\n- :ref:`neuron-batching`\n- :ref:`neuroncore-pipeline`\n\nBatching and pipelining (technical background)\n----------------------------------------------\n\nNeuron provides developers with various performance optimization features.\n\nTwo of the most widely used features are batching and pipelining. Both\ntechniques aim to keep the data close to the compute engines, but they achieve\nthis data locality in different ways. In batching it is achieved by loading\nthe data into an on-chip cache and reusing it multiple times for multiple\ndifferent model-inputs, while in pipelining it is achieved by caching all\nmodel parameters into the on-chip cache across multiple NeuronCores and\nstreaming the calculation across them.\n\nAs a general rule of thumb, batching is preferred for applications that\naim to optimize throughput and cost at the expense of latency, while\npipelining is preferred for applications with a high-throughput\nrequirement under a strict latency budget.\n\nCompiling for batching optimization\n-----------------------------------\n\nTo enable batching optimization, the model must first be compiled\nfor a target batch-size. This is done by specifying the batch size in\nthe input tensor's batch dimension during compilation. Users are\nencouraged to evaluate multiple batch size, in order to determine the\noptimal latency/throughput deployment-point, which is application-dependent.\n\nFor example, the code snippet below enables batching on a ResNet50\nmodel, with a batch-size of 5:\n\n.. code:: python\n\n import numpy as np\n import tensorflow.neuron as tfn\n\n # To change the batch size, change the first dimension in example_input\n batch_size = 5\n example_input = np.zeros([batch_size,224,224,3], dtype='float16')\n\n tfn.saved_model.compile(\"rn50_fp16\",\n \"rn50_fp16_compiled/1\",\n model_feed_dict={'input_1:0': example_input },\n dynamic_batch_size=True)\n\n.. note::\n\n Depending on the size of the neural network, Neuron has a maximum\n batch size that works optimally on Inferentia. If\n an unsupported batch size is used, an internal compiler error message\n will be displayed.\n A simple way to explore optimal batch size for your specific model is to\n increment the batch size from 1 upward, one at a time, and test\n application performance.\n\nCompiling for pipeline optimization\n-----------------------------------\n\nIn NeuronCore Pipeline mode, Neuron stores the model parameters in\nInferentias' local cache and streams inference requests across\nthe available NeuronCores, as specified by the\n``--neuroncore-pipeline-cores`` compiler argument. For example, to\ncompile the model to fit a pipeline size of four Inferentia devices (16\nNeuronCores) avaliable in the inf1.6xlarge instance size:\n\n.. code:: python\n\n import numpy as np\n import tensorflow.neuron as tfn\n\n compiler_args = ['--neuroncore-pipeline-cores', '16']\n example_input = np.zeros([1,224,224,3], dtype='float16')\n tfn.saved_model.compile(\"rn50_fp16\",\n \"rn50_fp16_compiled/1\",\n model_feed_dict={'input_1:0': example_input },\n compiler_args=compiler_args)\n\nThe minimum number of NeuronCores needed to run a compiled model can be\nfound using the Neuron Check Model tool. See :ref:`neuron_check_model`.\n\nModel-serving inference optimizations\n-------------------------------------\n\nTo fully realize the maximum throughput of the compiled model\n(for either batching and pipelining), users need to launch multiple host\nCPU threads to feed inputs into the Neuron pipeline. The number of\nthreads needs to be larger than the specified maximum number of\nNeuronCores.\n\nAdditionally, dynamic batching can be used to process a larger\nclient-side inference batch-size and the framework automatically breaks\nup the user-batch into smaller batch sizes, to match the compiled\nbatch-size. This technique increases the achievable throughput by hiding\nthe framework-to-neuron overhead, and amortizing it over a larger batch\nsize. To use dynamic batching, set the argument\n``--dynamic_batch_size=True`` during compilation and send a larger\ninference batch size (user inference batch size) that is equal to a\nmultiple of the compiled batch size.\n\nBoth methods can be applied together if this improves\nperformance. However, multi-threading is always needed as a first step\nto achieve high throughput. You need to experiment to find\noptimal settings for your application.\n\nBy default the framework sets the number of outstanding inference\nrequests to the total number of NeuronCores plus three. This can be\nchanged by setting the NEURON_MAX_NUM_INFERS environment variable. For\nexample, if the compiled model includes CPU partitions (e.g., if the Neuron compiler decides that some operations are more efficient to execute on CPU), \nthe number of threads needs to be increased to account for the\nadditional compute performed on the CPU. Note that the available\ninstance host memory size needs to be taken into consideration to prevent\nout-of-memory errors. As above, you need to experiment in order to find\nthe optimal settings for your application.\n\n.. note::\n\n By default the framework allocates a NeuronCore Group size to\n match the size of the compiled model. The size of the model is the\n number of NeuronCores limit passed to compiler during compilation\n (``--neuroncore-pipeline-cores`` option). For more information see the\n TensorFlow Serving NeuronCore Group documentation.\n\nOther considerations\n--------------------\n\nMixed Precision\n~~~~~~~~~~~~~~~\n\nYou can find more information about performance and accuracy trade offs\nin :ref:`neuron-cc-training-mixed-precision`.\n\n\nOperator support\n~~~~~~~~~~~~~~~~\n\nThe Neuron Compiler maintains an evolving list of supported operators\nfor each framework: :ref:`neuron-supported-operators`\n\nAWS Neuron handles unsupported operators by partitioning the graph into\nsubgraphs and executing them on different targets (e.g., NeuronCore\npartition, CPU partition). If the entire model can run on Inferentia\n(i.e., all operators are supported), then it will be compiled into\na single subgraph, which will be executed by a NeuronCore Group.\n\nDebug\n~~~~~\n\nYou can examine the post-compiled model to view the compilation results\nusing the Neuron plugin for TensorBoard.\nSee :ref:`tensorboard-plugin-visualize-graph`.\n\nResNet-50 optimization example\n------------------------------\n\nFor an example demonstrating the concepts described here, see\n:ref:`/src/examples/tensorflow/keras_resnet50/keras_resnet50.ipynb`\n" }, { "path": "about-neuron/appnotes/torch-neuron/bucketing-app-note.rst", "content": ".. _bucketing_app_note:\n\nRunning inference on variable input shapes with bucketing\n=========================================================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nIntroduction\n------------\n\nWith Inferentia, the shape of every input must be fixed at compile time. For\napplications that require multiple input sizes, we recommend using padding or\nbucketing techniques. Padding requires you to compile your model with the\nlargest expected input size and pad every input to this maximum size. If the\nperformance of your model using padding is not within your targets, you can\nconsider implementing bucketing.\n\nThis guide introduces bucketing, a technique to run inference on inputs with\nvariable shapes on Inferentia. The following sections explain how bucketing can\nimprove the performance of inference workloads on Inferentia. It covers an\noverview of how bucketing works and provides examples of using bucketing in\n:ref:`computer vision ` and\n:ref:`natural language processing` applications.\n\nApplications that benefit from bucketing\n----------------------------------------\n\nBucketing refers to compiling your model multiple times with different target\ninput shapes to create “bucketed models.\" :ref:`creating_buckets` provides an\noverview on selecting the input shapes that you use to create bucketed models. At\ninference time, each input is padded until its shape matches the next largest\nbucket shape. The padded input is then passed into the corresponding bucketed model\nfor inference. By compiling the same model with multiple different input shapes,\nthe amount of input padding is reduced compared to padding every input to the\nmaximum size in your dataset. This technique minimizes the compute overhead\nand improves inference performance compared to padding every image to the\nmaximum shape in your dataset.\n\nBucketing works best when multiple different bucketed models are created to efficiently\ncover the full range of input shapes. You can fine-tune the model performance\nby experimenting with different bucket sizes that correspond to the\ndistribution of input shapes in your dataset.\n\nBucketing can only be used if there is an upper bound on the shape of the\ninputs. If necessary, an upper bound on the input shape can be enforced using\nresizing and other forms of preprocessing.\n\n.. _num_buckets:\n\nThe upper bound on the number of bucketed models that you use is dictated by the\ntotal size of the compiled bucketed models. Each Inferentia chip has 8GB of\nDRAM, or 2GB of DRAM per NeuronCore. An inf1.xlarge and inf1.2xlarge have\n1 Inferentia chip, an inf1.6xlarge has 4 Inferentia chips, and an inf1.24xlarge\nhas 16 Inferentia chips. Thus, you should limit the total size of all bucketed\nmodels to around 8GB per Inferentia chip or 2GB per NeuronCore.\nThe following formula provides an approximation for the number of\ncompiled bucketed models you can fit on each NeuronCore:\n\n::\n\n number-of-buckets = round(10^9 / number-of-weights-in-model)\n\nWe recommend using :ref:`neuron-top ` to monitor the\nmemory usage on your inf1 instance as you load multiple bucketed models.\n\nImplementing bucketing\n-----------------------\n\nImplementing bucketing consists of two main parts: creating multiple bucketed\nmodels at compile-time and running inference using the bucketed models on (padded)\ninputs. The following sections describe how to implement bucketing to run\ninference in applications that have variable input shapes.\n\n.. _creating_buckets:\n\nCreating bucketed models\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nBefore running inference, models should be compiled for different input shapes\nthat are representative of the input dataset. The input shapes that are used\nto compile the models determine the bucket shapes that are used during inference.\nThe bucket shapes should be chosen to minimize the amount of padding on each new input.\nAdditionally, there should always be a bucket that’s large enough to handle the\nmaximum input shape in the dataset. The limit on the number of compiled bucketed\nmodels that can be used is described in this :ref:`section`.\n\n\nRunning inference with bucketing\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAt inference time, each input should be padded to match the size of the next\nlargest bucket, such that the height and width (or sequence length) of the\npadded input equals the size of the bucket. Then, the padded input should\nbe passed into the corresponding bucket for inference. If necessary, it’s\nimportant to remove and/or crop any aberrant predictions that occur in the\npadded region. For example, in object detection applications, bounding box\npredictions that occur in the padded regions should be removed to avoid\nerroneous predictions. \n\n.. _bucketing_examples:\n\nExamples\n--------\n\nThe following sections provide examples of applying the bucketing technique\nto run inference in applications that have variable input shapes.\n\n.. _bucketing_example_cv:\n\nComputer vision bucketing\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAs an example of implementing bucketing for computer vision models, consider an\napplication where the height and width of images in dataset are uniformly\ndistributed between `[400, 400]` and `[800, 800]`. Given that every input\nshape between `[400, 400]` and `[800, 800]` is equally likely, it could\nmake sense to create bucketed models that divide up the range of input shapes into\nequally sized chunks. For example, we could create bucketed models for the input shapes\n`[500, 500]`, `[600, 600]`, `[700, 700]`, and `[800, 800]`. \n\nAs an example of running inference with bucketing, let’s assume that we created\nbucketed models for the input shapes `[500, 500]`, `[600, 600]`, `[700, 700]`, and\n`[800, 800]`. If we receive an input with shape `[640, 640]`, we would\npad the input to the next largest bucket, `[700, 700]`, and use this bucket\nfor inference. If we receive an input with shape `[440, 540]`, we would\nneed to pad the input to the bucket size, `[600, 600]`, and use this bucket\nfor inference.\n\nAs another example of creating bucketed models, consider a computer vision\napplication where the dataset is not uniformly distributed. As before, let’s\nassume the input shapes range between `[400, 400]` to `[800, 800]`. Now, let’s\nassume the data shape distribution is bimodal, such that `[540, 540]` and\n`[720, 720]` are the two most common input shapes. In this example, it might\nmake sense to create bucketed models for input shapes `[540, 540]`, `[720, 720]`, and\n`[800, 800]` to target the most common shapes while still including the\nentire range of input shapes.\n\n\nEnd-to-end computer vision bucketing example\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn this example, we run inference in a computer vision application that has\nvariable shaped images that range in shape from `[400, 400]` to\n`[800, 800]`. We create bucketed models for the input shapes `[500, 500]`,\n`[600, 600]`, `[700, 700]`, and `[800, 800]` to handle the variable input\nshapes.\n\n.. code-block:: python\n\n import numpy as np\n import torch\n from torchvision import models\n import torch_neuron\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Define the bucket sizes that will be used for compilation and inference\n bucket_sizes = [(500, 500), (600, 600), (700, 700), (800, 800)]\n\n # Create the bucketed models by compiling a model for each bucket size\n buckets = {}\n for bucket_size in bucket_sizes:\n # Create an example input that is the desired bucket size\n h, w = bucket_size\n image = torch.rand([1, 3, h, w])\n\n # Compile with the example input to create the bucketed model\n model_neuron = torch.neuron.trace(model, image)\n\n # Run a warm up inference to load the model into Inferentia memory\n model_neuron(image)\n\n # Add the bucketed model based on its bucket size\n buckets[bucket_size] = model_neuron\n\n\n def get_bucket_and_pad_image(image):\n # Determine which bucket size to use\n oh, ow = image.shape[-2:]\n target_bucket = None\n for bucket_size in bucket_sizes:\n # Choose a bucket that's larger in both the height and width dimensions\n if oh <= bucket_size[0] and ow <= bucket_size[1]:\n target_bucket = bucket_size\n break\n\n # Pad the image to match the size of the bucket\n h_delta = target_bucket[0] - oh\n w_delta = target_bucket[1] - ow\n\n b_pad = h_delta # Bottom padding\n l_pad = 0 # Left padding\n t_pad = 0 # Top padding\n r_pad = w_delta # Right padding\n\n # Pad the height and width of the image\n padding_amounts = (l_pad, r_pad, t_pad, b_pad)\n image_padded = torch.nn.functional.pad(image, padding_amounts, value=0)\n\n return image_padded, target_bucket\n\n\n # Run inference on inputs with different shapes\n for _ in range(10):\n # Create an image with a random height and width in range [400, 400] to [800, 800]\n h = int(np.random.uniform(low=400, high=800))\n w = int(np.random.uniform(low=400, high=800))\n image = torch.rand(1, 3, h, w)\n\n # Determine bucket and pad the image\n image_padded, target_bucket = get_bucket_and_pad_image(image)\n\n # Use the corresponding bucket to run inference\n output = buckets[target_bucket](image_padded)\n\n\n.. _bucketing_example_nlp:\n\nNatural language processing bucketing\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAs an example of implementing bucketing for natural language processing models,\nconsider an application where the lengths of tokenized sequences in a dataset are\nuniformly distributed between 0 and 128 tokens. Given that every tokenized sequence\nlength between 0 and 128 is equally likely, it might make sense to create\nbucketed models that divide up the range of tokenized sequence lengths into equally sized\nchunks. For example, we could create bucketed models for tokenized sequence lengths 64\nand 128.\n\nAs an example of running inference with bucketing, let's assume that we created\nbucketed models for the input tokenized sequence lengths 64 and 128. If we receive a\ntokenized sequence with length 55, we would need to pad it to the bucket size\n64 and use this bucket for inference. If we receive a tokenized sequence with\nlength 112, we would need to pad it to the bucket size 128 and use this bucket\nfor inference.\n\nEnd-to-end natural language processing bucketing example\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn this example, we run inference in a natural language processing application\nthat has variable length tokenized sequences that range from 0 to 128. We\ncreate bucketed models for lengths 64 and 128 to handle the variable input lengths.\n\n.. code-block:: python\n\n import numpy as np\n import torch\n from transformers import AutoTokenizer, AutoModelForSequenceClassification\n import torch_neuron\n\n # Build tokenizer and model\n tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased-finetuned-mrpc\")\n model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased-finetuned-mrpc\", return_dict=False)\n model.eval()\n\n # Define the bucket sizes that will be used for compilation and inference\n bucket_sizes = [64, 128]\n\n # Create the bucketed models by compiling a model for each bucket size\n buckets = {}\n for bucket_size in bucket_sizes:\n # Setup some example inputs\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n\n # Create an example input that is the desired bucket size\n paraphrase = tokenizer.encode_plus(sequence_0,\n sequence_1,\n max_length=bucket_size,\n padding='max_length',\n truncation=True,\n return_tensors=\"pt\")\n\n # Convert example inputs to a format that is compatible with TorchScript tracing\n example_inputs_paraphrase = paraphrase['input_ids'], paraphrase['attention_mask'], paraphrase['token_type_ids']\n\n # Compile with the example input to create the bucketed model\n model_neuron = torch.neuron.trace(model, example_inputs_paraphrase)\n\n # Run a warm up inference to load the model into Inferentia memory\n model_neuron(*example_inputs_paraphrase)\n\n # Add the bucketed model based on its bucket size\n buckets[bucket_size] = model_neuron\n\n\n def get_bucket_and_pad_paraphrase(paraphrase):\n # Determine which bucket size to use\n inputs = paraphrase['input_ids']\n attention = paraphrase['attention_mask']\n token_type = paraphrase['token_type_ids']\n paraphrase_len = inputs.shape[1]\n target_bucket = None\n for bucket_size in bucket_sizes:\n if paraphrase_len <= bucket_size:\n target_bucket = bucket_size\n break\n\n # Pad the paraphrase to match the size of the bucket\n delta = target_bucket - paraphrase_len\n zeros = torch.zeros([1, delta], dtype=torch.long)\n inputs = torch.cat([inputs, zeros], dim=1)\n attention = torch.cat([attention, zeros], dim=1)\n token_type = torch.cat([token_type, zeros], dim=1)\n\n paraphrase_padded = inputs, attention, token_type\n return paraphrase_padded, target_bucket\n\n\n # Create two sample sequences\n sequence_0 = (\"The only other bear similar in size to the polar bear is the \"\n \"Kodiak bear, which is a subspecies of the brown bear. Adult male \"\n \"polar bears weigh 350–700 kg and measure 2.4–3 meters in total \"\n \"length. All bears are short-tailed, the polar bear's tail is \"\n \"relatively the shortest amongst living bears.\")\n sequence_1 = (\"Around the Beaufort Sea, however, mature males reportedly \"\n \"average 450 kg. Adult females are roughly half the size of males \"\n \"and normally weigh 150–250 kg, measuring 1.8–2.4 meters in length. \"\n \"The legs are stocky and the ears and tail are small.\")\n\n # Run inference on inputs with different shapes\n # We create the variable shapes by randomly cropping the sequences\n for _ in range(10):\n # Get random sequence lengths between 0 and 128\n paraphrase_len = int(np.random.uniform(128))\n\n # Crop the paraphrase\n paraphrase_cropped = tokenizer.encode_plus(sequence_0,\n sequence_1,\n max_length=paraphrase_len,\n padding='max_length',\n truncation=True,\n return_tensors=\"pt\")\n\n # Determine bucket and pad the paraphrase\n paraphrase_padded, target_bucket = get_bucket_and_pad_paraphrase(paraphrase_cropped)\n\n # Use the corresponding bucket to run inference\n output = buckets[target_bucket](*paraphrase_padded)\n" }, { "path": "about-neuron/appnotes/torch-neuron/index.rst", "content": ".. _torch-neuron-appnotes:\n\nPyTorch Neuron Application Notes\n=================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n bucketing-app-note\n rcnn-app-note\n torch-neuron-dataparallel-app-note\n\nThis section contains application notes specific to PyTorch Neuron (``torch-neuron``) for ``Inf1`` instances. These guides cover advanced optimization techniques, implementation patterns, and best practices for deploying PyTorch models on AWS Inferentia.\n\nApplication Notes\n-----------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: bucketing-app-note\n :link-type: doc\n\n **Dynamic Batching with Bucketing**\n ^^^\n Optimize inference performance using dynamic batching and bucketing strategies\n\n .. grid-item-card::\n :link: rcnn-app-note\n :link-type: doc\n\n **R-CNN Implementation Guide**\n ^^^\n Comprehensive guide for implementing and optimizing R-CNN models on Inferentia\n\n .. grid-item-card::\n :link: torch-neuron-dataparallel-app-note\n :link-type: doc\n\n **Data Parallel Inference**\n ^^^\n Scale inference workloads using ``torch.neuron.DataParallel`` for multi-core execution" }, { "path": "about-neuron/appnotes/torch-neuron/rcnn-app-note.rst", "content": ".. _torch-neuron-r-cnn-app-note:\n\nRunning R-CNNs on Inf1\n======================\n\nThis application note demonstrates how to compile and run\n`Detectron2 `__-based\nR-CNNs on Inf1. It also provides guidance on how to use profiling to\nimprove performance of R-CNN models on Inf1.\n\n.. contents:: Table of contents\n :local:\n\n\nR-CNN Model Overview\n--------------------\n\nRegion-based CNN (R-CNN) models are commonly used for object detection\nand image segmentation tasks. A typical R-CNN architecture consists\nof the following components:\n\n- **Backbone:** The backbone extracts features from input images. In\n some models the backbone is a Feature Pyramid Network (FPN), which\n uses a top-down architecture with lateral connections to build an\n in-network feature pyramid from a single-scale input. The backbone is\n commonly a ResNet or Vision Transformer based network.\n- **Region Proposal Network (RPN):** The RPN predicts region proposals\n with a wide range of scales and aspect ratios. RPNs are constructed\n using convolutional layers and anchor boxes, which that serve as references\n for multiple scales and aspect ratios.\n- **Region of Interest (RoI):** The RoI component is used to resize the\n extracted features of varying size to the same size so that\n they can be consumed by a fully connected layer. RoI Align is\n typically used instead of RoI Pooling, because RoI Align provides\n better alignment.\n\nThe `Detectron2 `__\nlibrary provides many popular PyTorch R-CNN implementations, including\nR-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN. This application note \nfocuses on the Detectron2 R-CNN models.\n\nR-CNN Limitations and Considerations on Inferentia (NeuronCore-v1)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nR-CNN models may have limitations and considerations on Inferentia\n(NeuronCore-v1). See the Model Architecture Fit Guidelines\nfor more information. These limitations are not\napplicable to NeuronCore-v2.\n\nRequirements\n------------\n\nThe process described in this application note is intended to be run on an ``inf1.2xlarge``. In practice,\nR-CNN models can be run on any Inf1 instance size.\n\nVerify that this Jupyter notebook is running the Python kernel\nenvironment that was set up according to the `PyTorch Installation\nGuide `__.\nSelect the kernel from the “Kernel -> Change Kernel” option at\nthe top of the Jupyter notebook page.\n\nInstallation\n------------\n\nThis process requires the following pip packages:\n\n- ``torch==1.11.0``\n- ``torch-neuron``\n- ``neuron-cc``\n- ``opencv-python``\n- ``pycocotools``\n- ``torchvision==0.12.0``\n- ``detectron2==0.6``\n\nThe following section explains how to build ``torchvision`` from source and install\nthe ``Detectron2`` package. It also reinstalls the Neuron packages, to ensure\nversion compatibility.\n\nThe ``torchvision`` ``roi_align_kernel.cpp`` kernel is modified to\nuse OMP threading for a multi-threaded inference on the CPU. This significantly\nimproves the performance of RoI Align kernels on Inf1: OMP threading\nleads to a RoI Align latency reduction two to three times larger than the default\n``roi_align_kernel.cpp`` kernel configuration.\n\n.. code:: ipython3\n\n # Install python3.7-dev for pycocotools (a Detectron2 dependency)\n !sudo apt install python3.7-dev -y\n \n # Install Neuron packages\n !pip uninstall -y torchvision\n !pip install --force-reinstall \"protobuf==3.20.1\" ninja opencv-python\n !pip install --force-reinstall torch-neuron==1.11.0.* neuron-cc[tensorflow] --extra-index-url https://pip.repos.neuron.amazonaws.com\n\n # Change cuda to 10.2 for Detectron2\n !sudo rm /usr/local/cuda\n !sudo ln -s /usr/local/cuda-10.2 /usr/local/cuda\n \n # Install Torchvision 0.12.0 from source\n !git clone -b release/0.12 https://github.com/pytorch/vision.git\n \n # Update the RoI Align kernel to use OMP multithreading\n with open('vision/torchvision/csrc/ops/cpu/roi_align_kernel.cpp', 'r') as file:\n content = file.read()\n \n # Enable OMP Multithreading and set the number of threads to 4\n old = \"// #pragma omp parallel for num_threads(32)\"\n new = \"#pragma omp parallel for num_threads(4)\"\n content = content.replace(old, new)\n \n # Re-write the file\n with open('vision/torchvision/csrc/ops/cpu/roi_align_kernel.cpp', 'w') as file:\n file.write(content)\n \n # Build Torchvision with OMP threading\n !cd vision && CFLAGS=\"-fopenmp\" python setup.py bdist_wheel\n %pip install vision/dist/*.whl\n \n # Install Detectron2 release v0.6\n !python -m pip install 'git+https://github.com/facebookresearch/detectron2.git@v0.6'\n\nCompiling an R-CNN for Inf1\n---------------------------\n\nBy default, R-CNN models are not compilable on Inf1, because they cannot\nbe traced with ``torch.jit.trace``, which is a requisite for inference\non Inf1. The following section demonstrates techniques for compiling a\nDetectron2 R-CNN model for inference on Inf1.\n\nSpecifically, this section explains how to create a standard Detectron2 R-CNN model,\nusing a ResNet-101 backbone. It demonstrates how to use profiling to\nidentify the most compute-intensive parts of the R-CNN that need to be\ncompiled for accelerated inference on Inf1. It then explains how to\nmanually extract and compile the ResNet backbone (the dominant compute\ncomponent) and inject the compiled backbone back into the full model, for\nimproved performance.\n\nCreate a Detectron2 R-CNN Model\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCreate a Detectron2 R-CNN model using the\n``COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml`` pretrained weights and\nconfig file. Download a sample image from the COCO dataset and\nrun an example inference.\n\n.. code:: ipython3\n\n from detectron2 import model_zoo\n from detectron2.engine import DefaultPredictor\n from detectron2.config import get_cfg\n \n def get_model():\n \n # Configure the R-CNN model\n CONFIG_FILE = \"COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml\"\n WEIGHTS_FILE = \"COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml\"\n cfg = get_cfg()\n cfg.merge_from_file(model_zoo.get_config_file(CONFIG_FILE))\n cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(WEIGHTS_FILE)\n cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5\n cfg.MODEL.DEVICE = 'cpu' # Send to CPU for Neuron Tracing\n \n # Create the R-CNN predictor wrapper\n predictor = DefaultPredictor(cfg)\n return predictor\n\n.. code:: ipython3\n\n import os\n import urllib.request\n \n # Define a function to get a sample image\n def get_image():\n filename = 'input.jpg'\n if not os.path.exists(filename):\n url = \"http://images.cocodataset.org/val2017/000000439715.jpg\"\n urllib.request.urlretrieve(url, filename)\n return filename\n\n.. code:: ipython3\n\n import time\n import cv2\n \n # Create an R-CNN model\n predictor = get_model()\n \n # Get a sample image from the COCO dataset\n image_filename = get_image()\n image = cv2.imread(image_filename)\n \n # Run inference and print inference latency\n start = time.time()\n outputs = predictor(image)\n print(f'Inference time: {(time.time() - start):0.3f} s')\n\nProfile the Model\n~~~~~~~~~~~~~~~~~\n\nUse the `PyTorch\nProfiler `__ to identify\nwhich operators contribute the most to the model’s runtime on CPU.\nIdeally, you can compile these compute intensive operators onto Inf1 for\naccelerated inference.\n\n.. code:: ipython3\n\n import torch.autograd.profiler as profiler\n \n with profiler.profile(record_shapes=True) as prof:\n with profiler.record_function(\"model_inference\"):\n predictor(image)\n print(prof.key_averages().table(sort_by=\"cpu_time_total\", row_limit=30))\n\nWe see that convolution operators (``aten::convolution``) contribute the\nmost to inference time. By compiling these convolution operators to\nInf1, you can improve performance of the R-CNN model. Print the\nR-CNN model architecture to see which layers contain the\n``aten::convolution`` operators:\n\n.. code:: ipython3\n\n print(predictor.model)\n\nNote that the ResNet FPN backbone\n(`predictor.model.backbone `__ L17-L162)\ncontains the majority of convolution operators in the model. The RPN\n(`predictor.model.proposal_generator `__ L181-L533)\nalso contains several convolutions. Based on this,\ncompile the ResNet backbone and RPN onto Inf1 to maximize performance.\n\nCompiling the ResNet backbone to Inf1\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis section demonstrates how to compile the ResNet backbone to\nInf1 and use it for inference.\n\nEextract the backbone by accessing it with\n``predictor.model.backbone``. Compile the backbone using\n``strict=False``, because the backbone outputs a dictionary. Use a\nfixed input shape (``800 x 800``) for compilation, as all inputs will be resized to this shape during inference. This\nsection also defines a basic preprocessing function (mostly derived from\nthe Detectron2 R-CNN\n`DefaultPredictor `__\nmodule L308-L318) that reshapes inputs to ``800 x 800``.\n\nCreate a ``NeuronRCNN`` wrapper to inject the\ncompiled backbone back into the model by dynamically replacing the\n``predictor.model.backbone`` attribute with the compiled model.\n\n.. code:: ipython3\n\n import torch\n import torch_neuron \n \n example = torch.rand([1, 3, 800, 800])\n \n # Use `with torch.no_grad():` to avoid a jit tracing issue in the ResNet backbone\n with torch.no_grad():\n neuron_backbone = torch_neuron.trace(predictor.model.backbone, example, strict=False)\n \n backbone_filename = 'backbone.pt'\n torch.jit.save(neuron_backbone, backbone_filename)\n\n.. code:: ipython3\n\n from detectron2.modeling.meta_arch.rcnn import GeneralizedRCNN\n from torch.jit import ScriptModule\n\n class NeuronRCNN(torch.nn.Module):\n \"\"\"\n Creates a `NeuronRCNN` wrapper that injects the compiled backbone into\n the R-CNN model. It also stores the `size_divisibility` attribute from\n the original backbone.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN, neuron_backbone: ScriptModule) -> None:\n super().__init__()\n \n # Keep track of the backbone variables\n size_divisibility = model.backbone.size_divisibility\n \n # Load and inject the compiled backbone\n model.backbone = neuron_backbone\n \n # Set backbone variables\n setattr(model.backbone, 'size_divisibility', size_divisibility)\n \n self.model = model\n \n def forward(self, x):\n return self.model(x)\n\n.. code:: ipython3\n\n # Create the R-CNN with the compiled backbone\n neuron_rcnn = NeuronRCNN(predictor.model, neuron_backbone)\n neuron_rcnn.eval()\n\n # Print the R-CNN architecture to verify the backbone is now the\n # `neuron_backbone` (shows up as `RecursiveScriptModule`)\n print(neuron_rcnn)\n\n.. code:: ipython3\n\n def preprocess(original_image, predictor):\n \"\"\"\n A basic preprocessing function that sets the input height=800 and \n input width=800. The function is derived from the preprocessing\n steps in the Detectron2 `DefaultPredictor` module.\n \"\"\"\n \n height, width = original_image.shape[:2]\n resize_func = predictor.aug.get_transform(original_image)\n resize_func.new_h = 800 # Override height\n resize_func.new_w = 800 # Override width\n image = resize_func.apply_image(original_image)\n image = torch.as_tensor(image.astype(\"float32\").transpose(2, 0, 1))\n inputs = {\"image\": image, \"height\": height, \"width\": width}\n return inputs\n\n.. code:: ipython3\n\n # Get a resized input using the sample image\n inputs = preprocess(image, get_model())\n \n # Run inference and print inference latency\n start = time.time()\n for _ in range(10):\n outputs = neuron_rcnn([inputs])[0]\n print(f'Inference time: {((time.time() - start)/10):0.3f} s')\n\n.. code:: ipython3\n\n with profiler.profile(record_shapes=True) as prof:\n with profiler.record_function(\"model_inference\"):\n neuron_rcnn([inputs])\n print(prof.key_averages().table(sort_by=\"cpu_time_total\", row_limit=30))\n\nBy running the backbone on Inf1, the overall runtime is already\nsignificantly improved. The count and runtime of ``aten::convolution``\noperators is also decreased. We now see a ``neuron::forward_v2``\noperator that is the compiled backbone.\n\nOptimize the R-CNN model\n------------------------\n\nCompiling the RPN\n~~~~~~~~~~~~~~~~~\n\nExamine the profiling and note that there are still several\n``aten::convolution``, ``aten::linear``, and ``aten::addmm`` operators\nthat significantly contribute to the model’s overall latency. By\ninspecting the model's architecture and code, we can determine that the\nmajority of these operators are contained in the RPN module\n(`predictor.model.proposal_generator `__ L181-L533).\n\nTo improve the model's performance, extract the RPN Head and\ncompile it on Inf1 to increase the number of operators running\non Inf1. You need to compile the RPN Head, because the RPN Anchor Generator\ncontains objects that are not traceable with ``torch.jit.trace``.\n\nThe RPN Head contains five layers that run inference on multiple resized\ninputs. To compile the RPN Head, create a list of tensors\nthat contain the input (“``features``”) shapes used by RPN Head on\neach layer. These tensor shapes can be determined by printing the input\nshapes in the RPN Head ``forward`` function\n(``predictor.model.proposal_generator.rpn_head.forward``).\n\nCreate a new ``NeuronRCNN`` wrapper that injects both the\ncompiled backbone and RPN Head into the R-CNN model.\n\n.. code:: ipython3\n\n import math\n \n input_shape = [1, 3, 800, 800] # Overall input shape at inference time\n \n # Create the list example of RPN inputs using the resizing logic from the RPN Head\n features = list()\n for i in [0, 1, 2, 3, 4]:\n ratio = 1 / (4 * 2**i)\n x_i_h = math.ceil(input_shape[2] * ratio)\n x_i_w = math.ceil(input_shape[3] * ratio)\n feature = torch.zeros(1, 256, x_i_h, x_i_w)\n features.append(feature)\n\n.. code:: ipython3\n\n # Extract and compile the RPN Head\n neuron_rpn_head = torch_neuron.trace(predictor.model.proposal_generator.rpn_head, [features])\n rpn_head_filename = 'rpn_head.pt'\n torch.jit.save(neuron_rpn_head, rpn_head_filename)\n\n.. code:: ipython3\n\n class NeuronRCNN(torch.nn.Module):\n \"\"\"\n Creates a wrapper that injects the compiled backbone and RPN Head\n into the R-CNN model.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN, neuron_backbone: ScriptModule, neuron_rpn_head: ScriptModule) -> None:\n super().__init__()\n \n # Keep track of the backbone variables\n size_divisibility = model.backbone.size_divisibility\n \n # Inject the compiled backbone\n model.backbone = neuron_backbone\n \n # Set backbone variables\n setattr(model.backbone, 'size_divisibility', size_divisibility)\n \n # Inject the compiled RPN Head\n model.proposal_generator.rpn_head = neuron_rpn_head\n \n self.model = model\n \n def forward(self, x):\n return self.model(x)\n\n.. code:: ipython3\n\n # Create the R-CNN with the compiled backbone and RPN Head\n predictor = get_model()\n neuron_rcnn = NeuronRCNN(predictor.model, neuron_backbone, neuron_rpn_head)\n neuron_rcnn.eval()\n\n # Print the R-CNN architecture to verify the compiled modules show up\n print(neuron_rcnn)\n\n.. code:: ipython3\n\n # Run inference and print inference latency\n start = time.time()\n for _ in range(10):\n outputs = neuron_rcnn([inputs])[0]\n print(f'Inference time: {((time.time() - start)/10):0.3f} s')\n\n.. code:: ipython3\n\n with profiler.profile(record_shapes=True) as prof:\n with profiler.record_function(\"model_inference\"):\n neuron_rcnn([inputs])\n print(prof.key_averages().table(sort_by=\"cpu_time_total\", row_limit=30))\n\nBy running the compiled backbone and RPN Head on Inf1, overall\nruntime is improved. Once again, the number and runtime of\n``aten::convolution`` operators is also decreased. There are now two\n``neuron::forward_v2`` operators, which correspond to the compiled\nbackbone and RPN Head.\n\nFusing the Backbone and RPN Head\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIt is usually preferable to compile fewer independent models\n(“subgraphs”) on Inf1. Combining models and compiling them as a single\nsubgraph enables the Neuron compiler to perform additional optimizations\nand reduces I/O data transfer between CPU and NeuronCores between\neach subgraph.\n\nIn this section, the ResNet backbone and RPN Head are \"fused\" into a\nsingle model to compile on Inf1. Create the\n``NeuronFusedBackboneRPNHead`` wrapper as a compilable model that\ncontains both the ResNet backbone\n(`predictor.model.backbone `__ L17-L162)\nand RPN Head\n(`predictor.model.proposal_generator `__ L181-L533).\nOutput the ``features`` to be used downstream by the RoI\nHeads. Compile this ``NeuronFusedBackboneRPNHead`` wrapper as\n``neuron_backbone_rpn``, then create a separate ``BackboneRPN``\nwrapper to inject the ``neuron_backbone_rpn`` in place of\nthe original backbone and RPN Head. Copy the remainder of the\nRPN ``forward`` code\n(`predictor.model.proposal_generator.forward `__ L431-L480)\nto create a “fused” backbone + RPN module. Lastly, re-write the\n``NeuronRCNN`` wrapper to use the fused backbone + RPN module. The\n``NeuronRCNN`` wrapper also uses the ``predictor.model`` ``forward``\ncode to re-write the rest of the R-CNN model forward function.\n\n.. code:: ipython3\n\n class NeuronFusedBackboneRPNHead(torch.nn.Module):\n \"\"\"\n Wrapper to compile the fused ResNet backbone and RPN Head.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.backbone = model.backbone\n self.rpn_head = model.proposal_generator.rpn_head\n self.in_features = model.proposal_generator.in_features\n \n def forward(self, x):\n features = self.backbone(x)\n features_ = [features[f] for f in self.in_features]\n return self.rpn_head(features_), features\n\n.. code:: ipython3\n\n # Create the wrapper with the combined backbone and RPN Head\n predictor = get_model()\n backbone_rpn_wrapper = NeuronFusedBackboneRPNHead(predictor.model)\n backbone_rpn_wrapper.eval()\n \n # Compile the wrapper\n example = torch.rand([1, 3, 800, 800])\n \n with torch.no_grad():\n neuron_backbone_rpn_head = torch_neuron.trace(\n backbone_rpn_wrapper, example, strict=False)\n \n backbone_rpn_filename = 'backbone_rpn.pt'\n torch.jit.save(neuron_backbone_rpn_head, backbone_rpn_filename)\n\n.. code:: ipython3\n\n class BackboneRPN(torch.nn.Module):\n \"\"\"\n Wrapper that uses the compiled `neuron_backbone_rpn` instead\n of the original backbone and RPN Head. We copy the remainder\n of the RPN `forward` code (`predictor.model.proposal_generator.forward`)\n to create a \"fused\" backbone + RPN module.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.backbone_rpn_head = NeuronFusedBackboneRPNHead(model)\n self._rpn = model.proposal_generator\n self.in_features = model.proposal_generator.in_features\n \n def forward(self, images):\n preds, features = self.backbone_rpn_head(images.tensor)\n features_ = [features[f] for f in self.in_features]\n pred_objectness_logits, pred_anchor_deltas = preds\n anchors = self._rpn.anchor_generator(features_)\n \n # Transpose the Hi*Wi*A dimension to the middle:\n pred_objectness_logits = [\n # (N, A, Hi, Wi) -> (N, Hi, Wi, A) -> (N, Hi*Wi*A)\n score.permute(0, 2, 3, 1).flatten(1)\n for score in pred_objectness_logits\n ]\n pred_anchor_deltas = [\n # (N, A*B, Hi, Wi) -> (N, A, B, Hi, Wi) -> (N, Hi, Wi, A, B) -> (N, Hi*Wi*A, B)\n x.view(x.shape[0], -1, self._rpn.anchor_generator.box_dim,\n x.shape[-2], x.shape[-1])\n .permute(0, 3, 4, 1, 2)\n .flatten(1, -2)\n for x in pred_anchor_deltas\n ]\n \n proposals = self._rpn.predict_proposals(\n anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes\n )\n return proposals, features\n\n.. code:: ipython3\n\n class NeuronRCNN(torch.nn.Module):\n \"\"\"\n Wrapper that uses the fused backbone + RPN module and re-writes\n the rest of the R-CNN `model` `forward` function.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n \n # Use the fused Backbone + RPN\n self.backbone_rpn = BackboneRPN(model)\n \n self.roi_heads = model.roi_heads\n \n self.preprocess_image = model.preprocess_image\n self._postprocess = model._postprocess\n \n def forward(self, batched_inputs):\n images = self.preprocess_image(batched_inputs)\n proposals, features = self.backbone_rpn(images)\n results, _ = self.roi_heads(images, features, proposals, None)\n return self._postprocess(results, batched_inputs, images.image_sizes)\n\n.. code:: ipython3\n\n # Create the new NeuronRCNN wrapper with the combined backbone and RPN Head\n predictor = get_model()\n neuron_rcnn = NeuronRCNN(predictor.model)\n neuron_rcnn.eval()\n\n # Inject the Neuron compiled models\n neuron_rcnn.backbone_rpn.backbone_rpn_head = neuron_backbone_rpn_head\n\n # Print the R-CNN architecture to verify the compiled modules show up\n print(neuron_rcnn)\n\n.. code:: ipython3\n\n # Run inference and print inference latency\n start = time.time()\n for _ in range(10):\n outputs = neuron_rcnn([inputs])[0]\n print(f'Inference time: {((time.time() - start)/10):0.3f} s')\n\n.. code:: ipython3\n\n with profiler.profile(record_shapes=True) as prof:\n with profiler.record_function(\"model_inference\"):\n neuron_rcnn([inputs])\n print(prof.key_averages().table(sort_by=\"cpu_time_total\", row_limit=30))\n\nBy running the fused backbone + RPN Head on Inf1, overall runtime is\nimproved even more. We now see a single ``neuron::forward_v2`` operator with\na lower runtime than the previous combined runtime of the two separate\n``neuron::forward_v2`` operators.\n\nCompiling the RoI Heads\n~~~~~~~~~~~~~~~~~~~~~~~\n\nThis section describes how to extract and compile part of RoI Heads module\n(`predictor.model.roi_heads `__ L530-L778) which runs most of the remaining ``aten::linear`` and ``aten::addmm``\noperators on Inf1. The entire RoI Heads module cannot be extracted, because\nit contains unsupported operators. So you need to create a\n``NeuronBoxHeadBoxPredictor`` wrapper, extracts specific parts of\nthe ``roi_heads`` for compilation. The example input for compilation is\nthe shape of the input into the ``self.roi_heads.box_head.forward``\nfunction. Write another wrapper, ``ROIHead`` that combines the\ncompiled ``roi_heads`` into the rest of the RoI module. The\n``_forward_box`` and ``forward`` functions are from the\n``predictor.model.roi_heads`` module. Lastly, re-write the ``NeuronRCNN``\nwrapper to use the optimized RoI Heads wrapper as well as the fused\nbackbone + RPN module.\n\n.. code:: ipython3\n\n class NeuronBoxHeadBoxPredictor(torch.nn.Module):\n \"\"\"\n Wrapper that extracts the RoI Box Head and Box Predictor\n for compilation.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.roi_heads = model.roi_heads\n \n def forward(self, box_features):\n box_features = self.roi_heads.box_head(box_features)\n predictions = self.roi_heads.box_predictor(box_features)\n return predictions\n\n.. code:: ipython3\n\n # Create the NeuronBoxHeadBoxPredictor wrapper\n predictor = get_model()\n box_head_predictor = NeuronBoxHeadBoxPredictor(predictor.model)\n box_head_predictor.eval()\n\n # Compile the wrapper\n example = torch.rand([1000, 256, 7, 7])\n neuron_box_head_predictor = torch_neuron.trace(box_head_predictor, example)\n\n roi_head_filename = 'box_head_predictor.pt'\n torch.jit.save(neuron_box_head_predictor, roi_head_filename)\n\n.. code:: ipython3\n\n class ROIHead(torch.nn.Module):\n \"\"\"\n Wrapper that combines the compiled `roi_heads` into the\n rest of the RoI module. The `_forward_box` and `forward`\n functions are from the `predictor.model.roi_heads` module.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.roi_heads = model.roi_heads\n self.neuron_box_head_predictor = NeuronBoxHeadBoxPredictor(model)\n \n def _forward_box(self, features, proposals):\n features = [features[f] for f in self.roi_heads.box_in_features]\n box_features = self.roi_heads.box_pooler(\n features, [x.proposal_boxes for x in proposals])\n predictions = self.neuron_box_head_predictor(box_features)\n pred_instances, _ = self.roi_heads.box_predictor.inference(\n predictions, proposals)\n return pred_instances\n \n def forward(self, images, features, proposals, targets=None):\n pred_instances = self._forward_box(features, proposals)\n pred_instances = self.roi_heads.forward_with_given_boxes(\n features, pred_instances)\n return pred_instances, {}\n\n.. code:: ipython3\n\n class NeuronRCNN(torch.nn.Module):\n \"\"\"\n Wrapper that uses the fused backbone + RPN module and the optimized RoI\n Heads wrapper\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n \n # Create fused Backbone + RPN\n self.backbone_rpn = BackboneRPN(model)\n \n # Create Neuron RoI Head\n self.roi_heads = ROIHead(model)\n \n # Define pre and post-processing functions\n self.preprocess_image = model.preprocess_image\n self._postprocess = model._postprocess\n \n def forward(self, batched_inputs):\n images = self.preprocess_image(batched_inputs)\n proposals, features = self.backbone_rpn(images)\n results, _ = self.roi_heads(images, features, proposals, None)\n return self._postprocess(results, batched_inputs, images.image_sizes)\n\n.. code:: ipython3\n\n # Initialize an R-CNN on CPU\n predictor = get_model()\n\n # Create the Neuron R-CNN on CPU\n neuron_rcnn = NeuronRCNN(predictor.model)\n neuron_rcnn.eval()\n\n # Inject the Neuron compiled models\n neuron_rcnn.backbone_rpn.backbone_rpn_head = neuron_backbone_rpn_head\n neuron_rcnn.roi_heads.neuron_box_head_predictor = neuron_box_head_predictor\n\n.. code:: ipython3\n\n # Run inference and print inference latency\n start = time.time()\n for _ in range(10):\n outputs = neuron_rcnn([inputs])[0]\n print(f'CPU Inference time: {((time.time() - start)/10):0.3f} s')\n\n.. code:: ipython3\n\n with profiler.profile(record_shapes=True) as prof:\n with profiler.record_function(\"model_inference\"):\n neuron_rcnn([inputs])\n print(prof.key_averages().table(sort_by=\"cpu_time_total\", row_limit=30))\n\nAlthough the overall latency did not change significantly, running more\nof the model on Inf1 instead of CPU frees up CPU resources when\nmultiple models are running in parallel.\n\nEnd-to-end Compilation and Inference\n------------------------------------\n\nThis section provides standalone code that compiles and runs an\noptimized Detectron2 R-CNN on Inf1. Most of the code in this section is\nfrom the previous sections in this application note and is\nconsolidated here for easy deployment. This section has the following\nmain components:\n\n- Preprocessing and compilation functions\n- Wrappers that extract the R-CNN ResNet backbone, RPN Head, and RoI\n Head for compilation on Inf1.\n- A ``NeuronRCNN`` wrapper that creates an optimized end-to-end\n Detectron2 R-CNN model for inference on Inf1\n- Benchmarking code that runs parallelized inference for optimized\n throughput on Inf1\n\nBenchmarking\n~~~~~~~~~~~~\n\nThe benchmarking section explains how to load multiple optimized RCNN models and\nrun them in parallel, to maximize throughput.\n\nUse the beta NeuronCore placement API,\n``torch_neuron.experimental.neuron_cores_context()``, to ensure all\ncompiled models in an optimized RCNN model are loaded onto the same\nNeuronCore. Note that the functionality and API of\n``torch_neuron.experimental.neuron_cores_context()`` might change in\nfuture releases.\n\nDefine a simple benchmark function that loads four optimized RCNN\nmodels onto four separate NeuronCores, runs multithreaded inference, and\ncalculates the corresponding latency and throughput. Benchmark\nvarious numbers of loaded models, to show the impact of parallelism.\n\nNote that throughput increases (at the cost of latency) when more\nmodels are run in parallel on Inf1. Increasing the number of worker\nthreads also improves throughput.\n\nOther improvements\n~~~~~~~~~~~~~~~~~~\n\nThere are many additional optimizations that can be applied to RCNN\nmodels on Inf1 depending on the application:\n\nFor latency sensitive applications:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n- Each of the five layers in the RPN head can be parallelized to\n decrease overall latency.\n- The number of OMP Threads can be increased in the ROI Align kernel.\n Both of these optimizations improve latency, at the cost of\n decreasing throughput.\n\nFor throughput sensitive applications:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n- The input batch size can be increased to improve NeuronCore\n utilization.\n\n.. code:: ipython3\n\n import time\n import os\n import urllib.request\n from typing import Any, Union, Callable\n \n import cv2\n import numpy as np\n from concurrent.futures import ThreadPoolExecutor\n \n import torch\n import torch_neuron\n \n from detectron2 import model_zoo\n from detectron2.engine import DefaultPredictor\n from detectron2.config import get_cfg\n from detectron2.modeling.meta_arch.rcnn import GeneralizedRCNN\n \n \n # -----------------------------------------------------------------------------\n # Helper functions\n # -----------------------------------------------------------------------------\n \n def get_model():\n \n # Configure the R-CNN model\n CONFIG_FILE = \"COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml\"\n WEIGHTS_FILE = \"COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml\"\n cfg = get_cfg()\n cfg.merge_from_file(model_zoo.get_config_file(CONFIG_FILE))\n cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(WEIGHTS_FILE)\n cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5\n cfg.MODEL.DEVICE = 'cpu' # Send to CPU for Neuron Tracing\n \n # Create the R-CNN predictor wrapper\n predictor = DefaultPredictor(cfg)\n return predictor\n \n \n def get_image():\n \n # Get a sample image\n filename = 'input.jpg'\n if not os.path.exists(filename):\n url = \"http://images.cocodataset.org/val2017/000000439715.jpg\"\n urllib.request.urlretrieve(url, filename)\n return filename\n \n \n def preprocess(original_image, predictor):\n \"\"\"\n A basic preprocessing function that sets the input height=800 and \n input width=800. The function is derived from the preprocessing\n steps in the Detectron2 `DefaultPredictor` module.\n \"\"\"\n \n height, width = original_image.shape[:2]\n resize_func = predictor.aug.get_transform(original_image)\n resize_func.new_h = 800 # Override height\n resize_func.new_w = 800 # Override width\n image = resize_func.apply_image(original_image)\n image = torch.as_tensor(image.astype(\"float32\").transpose(2, 0, 1))\n inputs = {\"image\": image, \"height\": height, \"width\": width}\n return inputs\n \n \n # -----------------------------------------------------------------------------\n # Neuron modules\n # -----------------------------------------------------------------------------\n \n class NeuronFusedBackboneRPNHead(torch.nn.Module):\n \"\"\"\n Wrapper to compile the fused ResNet backbone and RPN Head.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.backbone = model.backbone\n self.rpn_head = model.proposal_generator.rpn_head\n self.in_features = model.proposal_generator.in_features\n \n def forward(self, x):\n features = self.backbone(x)\n features_ = [features[f] for f in self.in_features]\n return self.rpn_head(features_), features\n \n \n class BackboneRPN(torch.nn.Module):\n \"\"\"\n Wrapper that uses the compiled `neuron_backbone_rpn` instead\n of the original backbone and RPN Head. We copy the remainder\n of the RPN `forward` code (`predictor.model.proposal_generator.forward`)\n to create a \"fused\" backbone + RPN module.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.backbone_rpn_head = NeuronFusedBackboneRPNHead(model)\n self._rpn = model.proposal_generator\n self.in_features = model.proposal_generator.in_features\n \n def forward(self, images):\n preds, features = self.backbone_rpn_head(images.tensor)\n features_ = [features[f] for f in self.in_features]\n pred_objectness_logits, pred_anchor_deltas = preds\n anchors = self._rpn.anchor_generator(features_)\n \n # Transpose the Hi*Wi*A dimension to the middle:\n pred_objectness_logits = [\n # (N, A, Hi, Wi) -> (N, Hi, Wi, A) -> (N, Hi*Wi*A)\n score.permute(0, 2, 3, 1).flatten(1)\n for score in pred_objectness_logits\n ]\n pred_anchor_deltas = [\n # (N, A*B, Hi, Wi) -> (N, A, B, Hi, Wi) -> (N, Hi, Wi, A, B) -> (N, Hi*Wi*A, B)\n x.view(x.shape[0], -1, self._rpn.anchor_generator.box_dim,\n x.shape[-2], x.shape[-1])\n .permute(0, 3, 4, 1, 2)\n .flatten(1, -2)\n for x in pred_anchor_deltas\n ]\n \n proposals = self._rpn.predict_proposals(\n anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes\n )\n return proposals, features\n \n \n class NeuronBoxHeadBoxPredictor(torch.nn.Module):\n \"\"\"\n Wrapper that extracts the RoI Box Head and Box Predictor\n for compilation.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.roi_heads = model.roi_heads\n \n def forward(self, box_features):\n box_features = self.roi_heads.box_head(box_features)\n predictions = self.roi_heads.box_predictor(box_features)\n return predictions\n \n \n class ROIHead(torch.nn.Module):\n \"\"\"\n Wrapper that combines the compiled `roi_heads` into the\n rest of the RoI module. The `_forward_box` and `forward`\n functions are from the `predictor.model.roi_heads` module.\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n self.roi_heads = model.roi_heads\n self.neuron_box_head_predictor = NeuronBoxHeadBoxPredictor(model)\n \n def _forward_box(self, features, proposals):\n features = [features[f] for f in self.roi_heads.box_in_features]\n box_features = self.roi_heads.box_pooler(\n features, [x.proposal_boxes for x in proposals])\n predictions = self.neuron_box_head_predictor(box_features)\n pred_instances, _ = self.roi_heads.box_predictor.inference(\n predictions, proposals)\n return pred_instances\n \n def forward(self, images, features, proposals, targets=None):\n pred_instances = self._forward_box(features, proposals)\n pred_instances = self.roi_heads.forward_with_given_boxes(\n features, pred_instances)\n return pred_instances, {}\n \n \n class NeuronRCNN(torch.nn.Module):\n \"\"\"\n Wrapper that uses the fused backbone + RPN module and the optimized RoI\n Heads wrapper\n \"\"\"\n \n def __init__(self, model: GeneralizedRCNN) -> None:\n super().__init__()\n \n # Create fused Backbone + RPN\n self.backbone_rpn = BackboneRPN(model)\n \n # Create Neuron RoI Head\n self.roi_heads = ROIHead(model)\n \n # Define pre and post-processing functions\n self.preprocess_image = model.preprocess_image\n self._postprocess = model._postprocess\n \n def forward(self, batched_inputs):\n images = self.preprocess_image(batched_inputs)\n proposals, features = self.backbone_rpn(images)\n results, _ = self.roi_heads(images, features, proposals, None)\n return self._postprocess(results, batched_inputs, images.image_sizes)\n \n \n # -----------------------------------------------------------------------------\n # Compilation functions\n # -----------------------------------------------------------------------------\n \n def compile(\n model: Union[Callable, torch.nn.Module],\n example_inputs: Any,\n filename: str,\n **kwargs\n ) -> torch.nn.Module:\n \"\"\"\n Compiles the model for Inf1 if it doesn't already exist and saves it as the provided filename. \n \n model: A module or function which defines a torch model or computation.\n example_inputs: An example set of inputs which will be passed to the\n `model` during compilation.\n filename: Name of the compiled model\n kwargs: Extra `torch_neuron.trace` kwargs\n \"\"\"\n \n if not os.path.exists(filename):\n with torch.no_grad():\n compiled_model = torch_neuron.trace(model, example_inputs, **kwargs)\n torch.jit.save(compiled_model, filename)\n \n \n # -----------------------------------------------------------------------------\n # Benchmarking function\n # -----------------------------------------------------------------------------\n \n def benchmark(backbone_rpn_filename, roi_head_filename, inputs, \n n_models=4, batch_size=1, n_threads=4, iterations=200):\n \"\"\"\n A simple benchmarking function that loads `n_models` optimized\n models onto separate NeuronCores, runs multithreaded inference,\n and calculates the corresponding latency and throughput.\n \"\"\"\n \n # Load models\n models = list()\n for i in range(n_models):\n with torch_neuron.experimental.neuron_cores_context(i):\n # Create the RCNN with the fused backbone + RPN Head and compiled RoI Heads\n # Initialize an R-CNN on CPU\n predictor = get_model()\n\n # Create the Neuron R-CNN on CPU\n neuron_rcnn = NeuronRCNN(predictor.model)\n neuron_rcnn.eval()\n\n # Inject the Neuron compiled models\n neuron_rcnn.backbone_rpn.backbone_rpn_head = torch.jit.load(backbone_rpn_filename)\n neuron_rcnn.roi_heads.neuron_box_head_predictor = torch.jit.load(roi_head_filename)\n\n models.append(neuron_rcnn)\n \n # Warmup\n for _ in range(8):\n for model in models:\n model([inputs])\n \n latencies = []\n \n # Thread task\n def task(i):\n start = time.time()\n models[i]([inputs])\n finish = time.time()\n latencies.append((finish - start) * 1000)\n \n begin = time.time()\n with ThreadPoolExecutor(max_workers=n_threads) as pool:\n for i in range(iterations):\n pool.submit(task, i % n_models)\n end = time.time()\n \n # Compute metrics\n boundaries = [50, 95, 99]\n names = [f'Latency P{i} (ms)' for i in boundaries]\n percentiles = np.percentile(latencies, boundaries)\n duration = end - begin\n \n # Display metrics\n results = {\n 'Samples': iterations,\n 'Batch Size': batch_size,\n 'Models': n_models,\n 'Threads': n_threads,\n 'Duration (s)': end - begin,\n 'Throughput (inf/s)': (batch_size * iterations) / duration,\n **dict(zip(names, percentiles)),\n }\n \n print('-' * 80)\n pad = max(map(len, results))\n for key, value in results.items():\n if isinstance(value, float):\n print(f'{key + \":\" :<{pad + 1}} {value:0.3f}')\n else:\n print(f'{key + \":\" :<{pad + 1}} {value}')\n print()\n \n \n if __name__ == \"__main__\":\n \n # Create and compile the combined backbone and RPN Head wrapper\n backbone_rpn_filename = 'backbone_rpn.pt'\n predictor = get_model()\n backbone_rpn_wrapper = NeuronFusedBackboneRPNHead(predictor.model)\n backbone_rpn_wrapper.eval()\n example = torch.rand([1, 3, 800, 800])\n compile(backbone_rpn_wrapper, example, backbone_rpn_filename, strict=False)\n\n # Create and compile the RoI Head wrapper\n roi_head_filename = 'box_head_predictor.pt'\n predictor = get_model()\n box_head_predictor = NeuronBoxHeadBoxPredictor(predictor.model)\n box_head_predictor.eval()\n example = torch.rand([1000, 256, 7, 7])\n compile(box_head_predictor, example, roi_head_filename)\n\n # Download a sample image from the COCO dataset and read it\n image_filename = get_image()\n image = cv2.imread(image_filename)\n inputs = preprocess(image, get_model())\n \n # Benchmark the Neuron R-CNN model for various numbers of loaded models\n benchmark(backbone_rpn_filename, roi_head_filename, inputs, n_models=1, n_threads=1)\n benchmark(backbone_rpn_filename, roi_head_filename, inputs, n_models=1, n_threads=2)\n benchmark(backbone_rpn_filename, roi_head_filename, inputs, n_models=2, n_threads=2)\n benchmark(backbone_rpn_filename, roi_head_filename, inputs, n_models=2, n_threads=4)\n benchmark(backbone_rpn_filename, roi_head_filename, inputs, n_models=4, n_threads=4)\n benchmark(backbone_rpn_filename, roi_head_filename, inputs, n_models=4, n_threads=8)\n" }, { "path": "about-neuron/appnotes/torch-neuron/torch-neuron-dataparallel-app-note.rst", "content": ".. _torch-neuron-dataparallel-app-note:\n\nData Parallel Inference on Torch Neuron\n=======================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nIntroduction\n------------\n\nThis guide introduces :func:`torch.neuron.DataParallel`, a Python API that\nimplements data parallelism on :class:`~torch.jit.ScriptModule` models created by the\n:doc:`Trace API `.\nThe following sections explain how data parallelism can improve the performance of\ninference workloads on Inferentia, including how :func:`torch.neuron.DataParallel`\nuses dynamic batching to run inference on variable input sizes. It covers an\noverview of the :func:`torch.neuron.DataParallel` module and provides a few\n:ref:`example data parallel applications `.\n\nData parallel inference\n-------------------------\n\nData Parallelism is a form of parallelization across multiple devices or cores,\nreferred to as nodes. Each node contains the same model and parameters, but\ndata is distributed across the different nodes. By distributing the\ndata across multiple nodes, data parallelism reduces the total\nexecution time of large batch size inputs compared to sequential execution.\nData parallelism works best for smaller models in latency sensitive\napplications that have large batch size requirements.\n\n\ntorch.neuron.DataParallel\n-------------------------\n\nTo fully leverage the Inferentia hardware, we want to use all available\nNeuronCores. An inf1.xlarge and inf1.2xlarge have four NeuronCores, an\ninf1.6xlarge has 16 NeuronCores, and an inf1.24xlarge has 64 NeuronCores.\nFor maximum performance on Inferentia hardware, we can use\n:func:`torch.neuron.DataParallel` to utilize all available NeuronCores.\n\n:func:`torch.neuron.DataParallel` implements data parallelism at the module\nlevel by replicating the Neuron model on all available NeuronCores\nand distributing data across the different cores for parallelized inference.\nThis function is analogous to :class:`~torch.nn.DataParallel` in PyTorch.\n:func:`torch.neuron.DataParallel` requires PyTorch >= 1.8.\n\nThe following sections provide an overview of some of the features\nof :func:`torch.neuron.DataParallel` that enable maximum performance on\nInferentia.\n\nNeuronCore selection\n^^^^^^^^^^^^^^^^^^^^\n\nBy default, DataParallel will try to use all NeuronCores allocated to the\ncurrent process to fully saturate the Inferentia hardware for maximum performance.\nIt is more efficient to make the batch dimension divisible by the number of\nNeuronCores. This will ensure that NeuronCores are not left idle during\nparallel inference and the Inferentia hardware is fully utilized.\n\nIn some applications, it is advantageous to use a subset of the\navailable NeuronCores for DataParallel inference. DataParallel has a\n``device_ids`` argument that accepts a list of :obj:`int` or ``'nc:#'``\nthat specify the NeuronCores to use for parallelization. See\n:ref:`Specifying NeuronCores `\nfor an example of how to use ``device_ids`` argument.\n\nBatch dim\n^^^^^^^^^\n\nDataParallel accepts a ``dim`` argument that denotes the batch dimension used\nto split the input data for distributed inference. By default,\nDataParalell splits the inputs on ``dim = 0`` if the ``dim`` argument is not\nspecified. For applications with a non-zero batch dim, the ``dim`` argument\ncan be used to specify the inference-time input batch dimension.\n:ref:`DataParallel with dim ! = 0 ` provides an\nexample of data parallel inference on inputs with batch dim = 2.\n\n.. _dynamic_batching_description:\n\nDynamic batching\n^^^^^^^^^^^^^^^^\n\nBatch size has a direct impact on model performance. The Inferentia chip is optimized\nto run with small batch sizes. This means that a Neuron compiled model can outperform\na GPU model, even if running single digit batch sizes.\n\nAs a general best practice, we recommend optimizing your model's throughput by\ncompiling the model with a small batch size and gradually increasing it to\nfind the peak throughput on Inferentia.\n\nDynamic batching is a feature that allows you to use tensor batch sizes that the\nNeuron model was not originally compiled against. This is necessary because the\nunderlying Inferentia hardware will always execute inferences with the batch\nsize used during compilation. Fixed batch size execution allows tuning the\ninput batch size for optimal performance. For example, batch size 1 may be\nbest suited for an ultra-low latency on-demand inference application, while\nbatch size > 1 can be used to maximize throughput for offline inferencing.\nDynamic batching is implemented by slicing large input tensors into chunks\nthat match the batch size used during the :func:`torch_neuron.trace` compilation call.\n\nThe :func:`torch.neuron.DataParallel` class automatically enables dynamic batching on\neligible models. This allows us to run inference in applications that have\ninputs with a variable batch size without needing to recompile the model. See\n:ref:`Dynamic batching ` for an example\nof how DataParallel can be used to run inference on inputs with a dynamic batch\nsize without needing to recompile the model.\n\nDynamic batching using small batch sizes can result in sub-optimal throughput\nbecause it involves slicing tensors into chunks and iteratively sending data\nto the hardware. Using a larger batch size at compilation time can use the\nInferentia hardware more efficiently in order to maximize throughput. You can\ntest the tradeoff between individual request latency and total throughput by\nfine-tuning the input batch size.\n\nAutomatic batching in the DataParallel module can be disabled using the\n``disable_dynamic_batching()`` function as follows:\n\n.. code-block:: python\n\n >>> model_parallel = torch.neuron.DataParallel(model_neuron)\n >>> model_parallel.disable_dynamic_batching()\n\nIf dynamic batching is disabled, the compile-time batch size must be equal to\nthe inference-time batch size divided by the number of NeuronCores.\n:ref:`DataParallel with dim != 0 ` and\n:ref:`Dynamic batching disabled `\nprovide examples of running DataParallel inference with dynamic batching\ndisabled.\n\n\nPerformance optimizations\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe DataParallel module has a ``num_workers`` attribute that can be used to\nspecify the number of worker threads used for multithreaded inference. By\ndefault, ``num_workers = 2 * number of NeuronCores``. This value can be\nfine tuned to optimize DataParallel performance.\n\nDataParallel has a ``split_size`` attribute that dictates the size of the input\nchunks that are distributed to each NeuronCore. By default,\n``split_size = max(1, input.shape[dim] // number of NeuronCores)``. This value\ncan be modified to optimally match the inference input chunk size with the\ncompile-time batch size.\n\n.. _data_paraellel_examples:\n\nExamples\n--------\n\nThe following sections provide example usages of the\n:func:`torch.neuron.DataParallel` module.\n\n\n.. _dataparallel_example_default:\n\nDefault usage\n^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-default.rst\n\n.. _dataparallel_example_specify_ncs:\n\nSpecifying NeuronCores\n^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-specify-ncs.rst\n\n\n.. _dataparallel_example_dim_neq_zero:\n\nDataParallel with dim != 0\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-dim-neq-zero.rst\n\n\n.. _dataparallel_example_dynamic_batching:\n\nDynamic batching\n^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-dynamic-batching.rst\n\n\n.. _dataparallel_example_disable_dynamic_batching:\n\nDynamic batching disabled\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-disable-dynamic-batching.rst\n\n\nFull tutorial with torch.neuron.DataParallel\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor an end-to-end tutorial that uses DataParallel, see the\n:ref:`PyTorch Resnet Tutorial `.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/index.rst", "content": ".. _torch-neuronx-appnotes:\n\nPyTorch NeuronX Application Notes\n==================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n introducing-pytorch-2-6\n introducing-pytorch-2-7\n introducing-pytorch-2-8\n introducing-pytorch-2-9\n introducing-pytorch-2-x\n migration-from-xla-downcast-bf16\n torch-neuronx-dataparallel-app-note\n torch-neuronx-graph-partitioner-app-note\n\nThis section contains application notes specific to PyTorch NeuronX (``torch-neuronx``) for ``Trn1`` and ``Inf2`` instances. These guides cover PyTorch version migrations, advanced features, optimization techniques, and best practices for training and inference on AWS Trainium and Inferentia2.\n\nPyTorch Version Support\n-----------------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: introducing-pytorch-2-9\n :link-type: doc\n\n **PyTorch 2.9 Support**\n ^^^\n New features and migration guide for PyTorch 2.9 on Neuron\n\n .. grid-item-card::\n :link: introducing-pytorch-2-8\n :link-type: doc\n\n **PyTorch 2.8 Support**\n ^^^\n New features and migration guide for PyTorch 2.8 on Neuron\n\n .. grid-item-card::\n :link: introducing-pytorch-2-7\n :link-type: doc\n\n **PyTorch 2.7 Support**\n ^^^\n Features and improvements introduced with PyTorch 2.7 support\n\n .. grid-item-card::\n :link: introducing-pytorch-2-x\n :link-type: doc\n\n **PyTorch 2.x Overview**\n ^^^\n General guide to PyTorch 2.x series support and features\n\nAdvanced Features\n-----------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: torch-neuronx-graph-partitioner-app-note\n :link-type: doc\n\n **Graph Partitioner**\n ^^^\n Advanced graph partitioning strategies for distributed training and inference\n\n .. grid-item-card::\n :link: torch-neuronx-dataparallel-app-note\n :link-type: doc\n\n **Data Parallel Inference**\n ^^^\n Scale inference workloads using ``torch.neuronx.DataParallel`` for multi-core execution\n\n .. grid-item-card::\n :link: migration-from-xla-downcast-bf16\n :link-type: doc\n\n **XLA Migration Guide**\n ^^^\n Migrate from deprecated XLA environment variables to PyTorch mixed-precision options\n" }, { "path": "about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-6.rst", "content": ".. _introduce-pytorch-2-6:\n\nIntroducing PyTorch 2.6 Support\n===============================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we introducing?\n------------------------\n\nStarting with the :ref:`Neuron 2.23 ` release, customers can now upgrade to PyTorch NeuronX (``torch-neuronx``) with specific support for PyTorch version 2.6.\n\n:ref:`setup-torch-neuronx` is updated to include installation instructions for PyTorch NeuronX 2.6 for Amazon Linux 2023 and Ubuntu 22.04. Note that PyTorch NeuronX 2.6 is supported on Python 3.9, 3.10, and 3.11.\n\nReview :ref:`migration guide ` for possible changes to training scripts. No code changes are required for inference scripts.\n\n\n.. _how-pytorch-2.6-different:\n\nHow is PyTorch NeuronX 2.6 different compared to PyTorch NeuronX 2.5?\n---------------------------------------------------------------------\n\nPyTorch NeuronX 2.6 uses Torch-XLA 2.6 which has improved support for Automatic Mixed Precision and buffer aliasing. Additionally:\n\n* Reintroduced ``XLA_USE_32BIT_LONG`` to give customers the flexibility to use INT32 for their workloads. This flag was removed in v2.5.\n* Added xm.xla_device_kind() to return the XLA device kind string ('NC_v2' for Trainium1, 'NC_v3' and 'NC_v3d' for Trainium2). See :ref:`logical-neuroncore-config` for more info.\n\nSee `Torch-XLA 2.6 release `__ for a full list.\n\nSee :ref:`migrate_to_pytorch_2.6` for changes needed to use PyTorch NeuronX 2.6.\n\n.. note::\n\n GSPMD and Torch Dynamo (torch.compile) support in Neuron will be available in a future release.\n\n.. _install_pytorch_neuron_2.6:\n\nHow can I install PyTorch NeuronX 2.6?\n--------------------------------------------\n\nTo install PyTorch NeuronX 2.6, follow the :ref:`setup-torch-neuronx` guides for Amazon Linux 2023 and Ubuntu 22.04 AMI. Refer to the Neuron Multi-Framework DLAMI :ref:`setup guide ` for Ubuntu 22.04 with a pre-installed virtual environment for PyTorch NeuronX 2.6 that you can use to get started. PyTorch NeuronX 2.6 can be installed using the following:\n\n.. code::\n\n python -m pip install --upgrade neuronx-cc==2.* torch-neuronx==2.6.* torchvision\n\n.. note::\n\n PyTorch NeuronX 2.6 is currently available for Python 3.9, 3.10, 3.11.\n\n.. _migrate_to_pytorch_2.6:\n\nMigrate your application to PyTorch 2.6\n---------------------------------------\n\nFirst, install the PyTorch NeuronX 2.6 as described above in :ref:`installation guide `\n\n\nMigrating training scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo migrate the training scripts from PyTorch NeuronX 2.5 to PyTorch NeuronX 2.6, implement the following changes: \n\n.. note::\n\n ``xm`` below refers to ``torch_xla.core.xla_model``, ``xr`` refers to ``torch_xla.runtime``, and ``xmp`` refers to ``torch_xla.distributed.xla_multiprocessing``\n\n* The environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warning when used) and will be removed in an upcoming release. Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to convert model to BF16 format. (see :ref:`migration_from_xla_downcast_bf16`)\n* The functions ``xm.xrt_world_size()``, ``xm.get_ordinal()``, and ``xm.get_local_ordinal()`` are deprecated (warnings are shown when used). Switch to ``xr.world_size()``, ``xr.global_ordinal()``, and ``xr.local_ordinal()`` respectively as replacements.\n* The default behavior of ``torch.load`` parameter ``weights_only`` is changed from ``False`` to ``True``. Setting ``weights_only`` to ``True`` may cause issues with pickling custom objects.\n* If using ``xmp.spawn``, the ``nprocs`` argument is limited to 1 or None since v2.1. Previously, passing a value > 1 would result in a warning. In torch-xla 2.6, passing a value > 1 will result in an error with an actionable message to use ``NEURON_NUM_DEVICES`` to set the number of NeuronCores to use.\n\nSee :ref:`v2.5 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.1.\n\nMigrating inference scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThere are no code changes required in the inference scripts.\n\n\nTroubleshooting and Known Issues\n--------------------------------\n\nTensor split on second dimension of 2D array not working\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, when using the tensor split operation on a 2D array in the second dimension, the resulting tensors do not contain the expected data (https://github.com/pytorch/xla/issues/8640). The workaround is to set ``XLA_DISABLE_FUNCTIONALIZATION=0``. Another workaround is to use ``torch.tensor_split``.\n\n\nLower BERT pretraining performance with torch-neuronx 2.6 compared to torch-neuronx 2.5\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, BERT pretraining performance is ~10% lower with torch-neuronx 2.6 compared to torch-neuronx 2.5. This is due to a known regression in the torch-xla library https://github.com/pytorch/xla/issues/9037 and may affect other models with high graph tracing overhead. To work around this issue, build the ``r2.6_aws_neuron`` branch of torch-xla as follows (see :ref:`pytorch-neuronx-install-cxx11` for C++11 ABI version):\n\n.. code:: bash\n\n # Setup build env (make sure you are in a python virtual env). Replace \"apt\" with \"yum\" on AL2023.\n sudo apt install cmake\n pip install yapf==0.30.0\n wget https://github.com/bazelbuild/bazelisk/releases/download/v1.20.0/bazelisk-linux-amd64\n sudo cp bazelisk-linux-amd64 /usr/local/bin/bazel\n # Clone repos\n git clone --recursive https://github.com/pytorch/pytorch --branch v2.6.0\n cd pytorch/\n git clone --recursive https://github.com/pytorch/xla.git --branch r2.6_aws_neuron\n _GLIBCXX_USE_CXX11_ABI=0 python setup.py bdist_wheel\n # The pip wheel will be present in ./dist\n cd xla/\n CXX_ABI=0 python setup.py bdist_wheel\n # The pip wheel will be present in ./dist and can be installed instead of the torch-xla released in pypi.org\n\nLower BERT pretraining performance when switch to using ``model.to(torch.bfloat16)``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, BERT pretraining performance is approximately 11% lower when switching to using ``model.to(torch.bfloat16)`` as part of migration away from the deprecated environment variable ``XLA_DOWNCAST_BF16`` due to https://github.com/pytorch/xla/issues/8545. As a workaround to recover the performance, you can set ``XLA_DOWNCAST_BF16=1``, which will still work in torch-neuronx 2.5 and 2.6 although there will be end-of-support warnings (as noted below).\n\n\nWarning \"XLA_DOWNCAST_BF16 will be deprecated after the 2.6 release, please downcast your model directly\"\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEnvironment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warning when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\n\nWARNING:root:torch_xla.core.xla_model.xrt_world_size() will be removed in release 2.7. is deprecated. Use torch_xla.runtime.world_size instead.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a warning that ``torch_xla.core.xla_model.xrt_world_size()`` will be removed in a future release. Switch to using ``torch_xla.runtime.world_size`` instead.\n\n\nWARNING:torch_xla.core.xla_model.get_ordinal() will be removed in release 2.7. is deprecated. Use torch_xla.runtime.global_ordinal instead.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a warning that ``torch_xla.core.xla_model.get_ordinal()`` will be removed in a future release. Switch to using ``torch_xla.runtime.global_ordinal`` instead.\n\nWARNING:torch_xla.core.xla_model.get_local_ordinal() will be removed in release 2.7. is deprecated. Use torch_xla.runtime.local_ordinal instead.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\n.. warning::\n ``torch_xla.core.xla_model.get_local_ordinal()`` will be removed in a future release. Use ``torch_xla.runtime.local_ordinal`` instead.\n \n\n\nSocket Error: Socket failed to bind\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn PyTorch 2.6, there must be a socket available for both torchrun and the ``init_process_group`` to bind. By default, both \nwill be set to use unused sockets. If you plan to use a ``MASTER_PORT`` environment variable then this error may occur if the port you set it to\nis already in use.\n\n.. code:: \n\n [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:2.600 (errno: 98 - Address already in use).\n [W socket.cpp:426] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).\n [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.\n RuntimeError: The server socket has failed to listen on any local network address. \n The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).\n\nTo resolve the issue, ensure you are setting ``MASTER_PORT`` to a port value that is not used anywhere else in your scripts. Otherwise,\nyou can leave ``MASTER_PORT`` unset and torchrun will set the default port for you.\n\n\n``AttributeError: module 'torch' has no attribute 'xla'`` Failure\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn PyTorch 2.6, training scripts might fail during activation checkpointing with the error shown below.\n\n.. code::\n\n AttributeError: module 'torch' has no attribute 'xla'\n\n\nThe solution is to use ``torch_xla.utils.checkpoint.checkpoint`` instead of ``torch.utils.checkpoint.checkpoint`` as the checkpoint function while wrapping pytorch modules for activation checkpointing.\nRefer to the pytorch/xla discussion regarding this `issue `_.\nAlso set ``use_reentrant=True`` while calling the torch_xla checkpoint function. Failure to do so will lead to ``XLA currently does not support use_reentrant==False`` error.\nFor more details on checkpointing, refer the `documentation `_.\n\n\nError ``Attempted to access the data pointer on an invalid python storage`` when using HF Trainer API\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nWhile using HuggingFace Transformers Trainer API to train (i.e. :ref:`HuggingFace Trainer API fine-tuning tutorial`), you may see the error \"Attempted to access the data pointer on an invalid python storage\". This is a known `issue `_ and has been fixed in the version ``4.37.3`` of HuggingFace Transformers.\n\n\n``ImportError: libcrypt.so.1: cannot open shared object file: No such file or directory`` on Amazon Linux 2023\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\ntorch-xla version 2.6+ now requires ``libcrypt.so.1`` shared library. Currently, Amazon Linux 2023 includes ``libcrypt.so.2`` shared library by default so you may see ``ImportError: libcrypt.so.1: cannot open shared object file: No such file or directory`` when using torch-neuronx 2.1+ on Amazon Linux 2023. To install ``libcrypt.so.1`` on Amazon Linux 2023, run the following installation command (see also https://github.com/amazonlinux/amazon-linux-2023/issues/182 for more context):\n\n.. code::\n\n sudo dnf install libxcrypt-compat\n\n\n``FileNotFoundError: [Errno 2] No such file or directory: 'libneuronpjrt-path'`` Failure\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nIn PyTorch 2.6, users might face the error shown below due to incompatible ``libneuronxla`` and ``torch-neuronx`` versions being installed.\n\n.. code::\n\n FileNotFoundError: [Errno 2] No such file or directory: 'libneuronpjrt-path'\n\nCheck that the version of ``libneuronxla`` that support PyTorch NeuronX 2.6 is ``2.2.*``. If not, then uninstall ``libneuronxla`` using ``pip uninstall libneuronxla`` and then reinstall the packages following the installation guide :ref:`installation guide `\n\n\n``Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` error during Neuron Parallel Compile\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running Neuron Parallel Compile with HF Trainer API, you may see the errors ``Status: INVALID_ARGUMENT: Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` in Accelerator's ``pad_across_processes`` function. This is due to data-dependent operation in evaluation metrics computation. Data-dependent operations would result in undefined behavior with Neuron Parallel Compile trial execution (execute empty graphs with zero outputs). To work-around this error, disable compute_metrics when NEURON_EXTRACT_GRAPHS_ONLY is set to 1:\n\n.. code:: python\n\n compute_metrics=None if os.environ.get(\"NEURON_EXTRACT_GRAPHS_ONLY\") else compute_metrics\n\nCompiler assertion error when running Stable Diffusion training\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWith PyTorch 2.6 (torch-neuronx), you may encounter the following compiler assertion error with Stable Diffusion training when gradient accumulation is enabled. This will be fixed in an upcoming release. For now, if you want to run Stable Diffusion training, disable gradient accumulation in torch-neuronx 2.6 by keeping the `default gradient accumulation steps of 1 `__.\n\n.. code:: bash\n\n ERROR 222163 [NeuronAssert]: Assertion failure in usr/lib/python3.9/concurrent/futures/process.py at line 239 with exception:\n too many partition dims! {{0,+,960}[10],+,10560}[10]\n\n\nFrequently Asked Questions (FAQ)\n--------------------------------\n\nDo I need to recompile my models with PyTorch 2.6?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes.\n\nDo I need to update my scripts for PyTorch 2.6?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSee the :ref:`migration guide `\n\nWhat environment variables will be changed with PyTorch NeuronX 2.6 ?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warning when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\nWhat features will be missing with PyTorch NeuronX 2.6?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch NeuronX 2.6 has all of the supported features in PyTorch NeuronX 2.5, with known issues listed above, and unsupported features as listed in :ref:`pytorch_rn`.\n\nCan I use Neuron Distributed and Transformers Neuron libraries with PyTorch NeuronX 2.6?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes, NeuronX Distributed, and Transformers NeuronX, and AWS Neuron Reference for NeMo Megatron libraries will work with PyTorch NeuronX 2.6.\n\nCan I still use PyTorch 2.5 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.5 is supported for releases 2.21/2.22/2.23 and will reach end-of-life in a future release. Additionally, the CVE `CVE-2025-32434 `_ affects PyTorch version 2.5. We recommend upgrading to the new version of Torch-NeuronX by following :ref:`setup-torch-neuronx`.\n\nCan I still use PyTorch 2.1 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.1 is supported for release 2.21 and has reached end-of-life in release 2.22. Additionally, the CVEs `CVE-2024-31583 `_ and `CVE-2024-31580 `_ affect PyTorch versions 2.1 and earlier. We recommend upgrading to the new version of Torch-NeuronX by following :ref:`setup-torch-neuronx`.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-7.rst", "content": ".. _introduce-pytorch-2-7:\n\nIntroducing PyTorch 2.7 Support\n===============================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we introducing?\n------------------------\n\nStarting with the :ref:`Neuron 2.24 ` release, customers can now upgrade to PyTorch NeuronX (``torch-neuronx``) with specific support for PyTorch version 2.7.\n\n:ref:`setup-torch-neuronx` is updated to include installation instructions for PyTorch NeuronX 2.7 for Amazon Linux 2023 and Ubuntu 22.04. Note that PyTorch NeuronX 2.7 is supported on Python 3.9, 3.10, and 3.11.\n\nReview :ref:`migration guide ` for possible changes to training scripts. No code changes are required for inference scripts.\n\n\n.. _how-pytorch-2.7-different:\n\nHow is PyTorch NeuronX 2.7 different compared to PyTorch NeuronX 2.5?\n---------------------------------------------------------------------\n\nPyTorch NeuronX 2.7 uses Torch-XLA v2.7 and PyTorch v2.7 which have C++11 ABI enabled by default. \n\nAdditionally, Torch-XLA v2.7 includes a fix for the training performance issue https://github.com/pytorch/xla/issues/9037.\n\nSee `Torch-XLA 2.7 release `__ for a full list.\n\nSee :ref:`migrate_to_pytorch_2.7` for changes needed to use PyTorch NeuronX 2.7.\n\n.. note::\n\n GSPMD and Torch Dynamo (torch.compile) support in Neuron will be available in a future release.\n\n.. _install_pytorch_neuron_2.7:\n\nHow can I install PyTorch NeuronX 2.7?\n--------------------------------------------\n\nTo install PyTorch NeuronX 2.7, follow the :ref:`setup-torch-neuronx` guides for Amazon Linux 2023 and Ubuntu 22.04 AMI. Refer to the Neuron Multi-Framework DLAMI :ref:`setup guide ` for Ubuntu 22.04 with a pre-installed virtual environment for PyTorch NeuronX 2.7 that you can use to get started. PyTorch NeuronX 2.7 can be installed using the following:\n\n.. code::\n\n python -m pip install --upgrade neuronx-cc==2.* torch-neuronx==2.7.* torchvision\n\n.. note::\n\n PyTorch NeuronX 2.7 is currently available for Python 3.9, 3.10, 3.11.\n\n.. _migrate_to_pytorch_2.7:\n\nMigrate your application to PyTorch 2.7\n---------------------------------------\n\nFirst, install the PyTorch NeuronX 2.7 as described above in :ref:`installation guide `\n\n\nMigrating training scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo migrate the training scripts from PyTorch NeuronX 2.5/2.6 to PyTorch NeuronX 2.7, implement the following changes: \n\n.. note::\n\n ``xm`` below refers to ``torch_xla.core.xla_model``, ``xr`` refers to ``torch_xla.runtime``, and ``xmp`` refers to ``torch_xla.distributed.xla_multiprocessing``\n\n* The environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used) and will be removed in an upcoming release. Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to convert model to BF16 format. (see :ref:`migration_from_xla_downcast_bf16`)\n* The functions ``xm.xrt_world_size()``, ``xm.get_ordinal()``, and ``xm.get_local_ordinal()`` are deprecated and removed so there are errors when used. Switch to ``xr.world_size()``, ``xr.global_ordinal()``, and ``xr.local_ordinal()`` respectively as replacements.\n* The default behavior of ``torch.load`` parameter ``weights_only`` is changed from ``False`` to ``True``. Setting ``weights_only`` to ``True`` may cause issues with pickling custom objects.\n* If using ``xmp.spawn``, the ``nprocs`` argument is limited to 1 or None since v2.1. Previously, passing a value > 1 would result in a warning. In torch-xla 2.6+, passing a value > 1 will result in an error with an actionable message to use ``NEURON_NUM_DEVICES`` to set the number of NeuronCores to use.\n\nSee :ref:`v2.6 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.5.\nSee :ref:`v2.5 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.1.\n\nMigrating inference scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThere are no code changes required in the inference scripts.\n\n\nTroubleshooting and Known Issues\n--------------------------------\n\nUsing the latest torch-xla v2.7 may result in increase in host memory usage\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing the latest torch-xla v2.7 may result in an increase in host memory usage compared to torch-xla v2.6. In one example, LLama2 pretraining with ZeRO1 and sequence length 16k could see an increase of 1.6% in host memory usage.\n\nTypeError: AdamW.__init__() got an unexpected keyword argument 'decoupled_weight_decay'\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAdamW now has an additional argument “decoupled_weight_decay” which defaults to False. If you get “TypeError: AdamW.__init__() got an unexpected keyword argument 'decoupled_weight_decay'” with NeuronX Distributed, update to the latest version.\n\n\nTensor split on second dimension of 2D array not working\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, when using the tensor split operation on a 2D array in the second dimension, the resulting tensors do not contain the expected data (https://github.com/pytorch/xla/issues/8640). The workaround is to set ``XLA_DISABLE_FUNCTIONALIZATION=0``. Another workaround is to use ``torch.tensor_split``.\n\nLower BERT pretraining performance when switch to using ``model.to(torch.bfloat16)``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, BERT pretraining performance is approximately 11% lower when switching to using ``model.to(torch.bfloat16)`` as part of migration away from the deprecated environment variable ``XLA_DOWNCAST_BF16`` due to https://github.com/pytorch/xla/issues/8545. As a workaround to recover the performance, you can set ``XLA_DOWNCAST_BF16=1``, which will still work in torch-neuronx 2.5 and 2.7 although there will be end-of-support warnings (as noted below).\n\n\nWarning \"XLA_DOWNCAST_BF16 will be deprecated after the 2.6 release, please downcast your model directly\"\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEnvironment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\n\nAttributeError: `_.\nAlso set ``use_reentrant=True`` while calling the torch_xla checkpoint function. Failure to do so will lead to ``XLA currently does not support use_reentrant==False`` error.\nFor more details on checkpointing, refer the `documentation `_.\n\n\nError ``Attempted to access the data pointer on an invalid python storage`` when using HF Trainer API\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nWhile using HuggingFace Transformers Trainer API to train (i.e. :ref:`HuggingFace Trainer API fine-tuning tutorial`), you may see the error \"Attempted to access the data pointer on an invalid python storage\". This is a known `issue `_ and has been fixed in the version ``4.37.3`` of HuggingFace Transformers.\n\n\n``ImportError: libcrypt.so.1: cannot open shared object file: No such file or directory`` on Amazon Linux 2023\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\ntorch-xla version 2.5+ now requires the ``libcrypt.so.1`` shared library. Currently, Amazon Linux 2023 includes ``libcrypt.so.2`` shared library by default so you may see `ImportError: libcrypt.so.1: cannot open shared object file: No such file or directory`` when using torch-neuronx 2.1+ on Amazon Linux 2023. To install ``libcrypt.so.1`` on Amazon Linux 2023, run the following installation command (see also https://github.com/amazonlinux/amazon-linux-2023/issues/182 for more context):\n\n.. code::\n\n sudo dnf install libxcrypt-compat\n\n\n``FileNotFoundError: [Errno 2] No such file or directory: 'libneuronpjrt-path'`` Failure\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nIn PyTorch 2.7, users might face the error shown below due to incompatible ``libneuronxla`` and ``torch-neuronx`` versions being installed.\n\n.. code::\n\n FileNotFoundError: [Errno 2] No such file or directory: 'libneuronpjrt-path'\n\nCheck that the version of ``libneuronxla`` that supports PyTorch NeuronX 2.7 is ``2.2.*``. If not, then uninstall ``libneuronxla`` using ``pip uninstall libneuronxla`` and then reinstall the packages following the installation guide :ref:`installation guide `\n\n\n``Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` error during Neuron Parallel Compile\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running Neuron Parallel Compile with HF Trainer API, you may see the errors ``Status: INVALID_ARGUMENT: Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` in Accelerator's ``pad_across_processes`` function. This is due to data-dependent operations in evaluation metrics computation. Data-dependent operations would result in undefined behavior with Neuron Parallel Compile trial execution (execute empty graphs with zero outputs). To work around this error, disable compute_metrics when NEURON_EXTRACT_GRAPHS_ONLY is set to 1:\n\n.. code:: python\n\n compute_metrics=None if os.environ.get(\"NEURON_EXTRACT_GRAPHS_ONLY\") else compute_metrics\n\nCompiler assertion error when running Stable Diffusion training\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWith PyTorch 2.7 (torch-neuronx), you may encounter the following compiler assertion error with Stable Diffusion training when gradient accumulation is enabled. This will be fixed in an upcoming release. For now, if you want to run Stable Diffusion training, disable gradient accumulation in torch-neuronx 2.7 by keeping the `default gradient accumulation steps of 1 `__.\n\n.. code:: bash\n\n ERROR 222163 [NeuronAssert]: Assertion failure in usr/lib/python3.9/concurrent/futures/process.py at line 239 with exception:\n too many partition dims! {{0,+,960}[10],+,10560}[10]\n\n\nFrequently Asked Questions (FAQ)\n--------------------------------\n\nDo I need to recompile my models with PyTorch 2.7?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes.\n\nDo I need to update my scripts for PyTorch 2.7?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSee the :ref:`migration guide `\n\nWhat environment variables will be changed with PyTorch NeuronX 2.7 ?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\nWhat features will be missing with PyTorch NeuronX 2.7?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch NeuronX 2.7 has all of the supported features in PyTorch NeuronX 2.6, with known issues listed above, and unsupported features as listed in :ref:`pytorch_rn`.\n\nCan I use Neuron Distributed and Transformers Neuron libraries with PyTorch NeuronX 2.7?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes, NeuronX Distributed and Transformers NeuronX are supported by PyTorch NeuronX 2.7. AWS Neuron Reference for NeMo Megatron has reached end-of-support in release 2.23.\n\nCan I still use PyTorch 2.6 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.6 is supported since release 2.23.\n\nCan I still use PyTorch 2.5 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.5 is supported for releases 2.21 to 2.24 and will reach end-of-life in a future release. Additionally, the CVE `CVE-2025-32434 `_ affects PyTorch version 2.5. We recommend upgrading to the new version of Torch-NeuronX by following :ref:`setup-torch-neuronx`.\n\nCan I still use PyTorch 2.1 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.1 is supported for release 2.21 and has reached end-of-life in release 2.22. Additionally, the CVEs `CVE-2024-31583 `_ and `CVE-2024-31580 `_ affect PyTorch versions 2.1 and earlier. We recommend upgrading to the new version of Torch-NeuronX by following :ref:`setup-torch-neuronx`.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-8.rst", "content": ".. _introduce-pytorch-2-8:\n\nIntroducing PyTorch 2.8 Support\n===============================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we introducing?\n------------------------\n\nStarting with the :ref:`Neuron 2.26 ` release, customers can now upgrade to PyTorch NeuronX (``torch-neuronx``) with specific support for PyTorch version 2.8.\n\n:ref:`setup-torch-neuronx` is updated to include installation instructions for PyTorch NeuronX 2.8 for Ubuntu 22.04. Note that PyTorch NeuronX 2.8 is supported on Python 3.10 and 3.11, with 3.12+ support coming in a future release.\n\nReview :ref:`migration guide ` for possible changes to training scripts. No code changes are required for inference scripts.\n\n\n.. _how-pytorch-2.8-different:\n\nHow is PyTorch NeuronX 2.8 different compared to PyTorch NeuronX 2.7?\n---------------------------------------------------------------------\n\nSee `Torch-XLA 2.8 release `__ for a full list of changes.\n\nSee :ref:`migrate_to_pytorch_2.8` for changes needed to use PyTorch NeuronX 2.8.\n\n.. note::\n\n GSPMD and Torch Dynamo (torch.compile) support in Neuron will be available in a future release.\n\n.. _install_pytorch_neuron_2.8:\n\nHow can I install PyTorch NeuronX 2.8?\n--------------------------------------------\n\nTo install PyTorch NeuronX 2.8, follow the :ref:`setup-torch-neuronx` guides for Ubuntu 22.04 AMI. Refer to the Neuron Multi-Framework DLAMI :ref:`setup guide ` for Ubuntu 22.04 with a pre-installed virtual environment for PyTorch NeuronX 2.8 that you can use to get started. PyTorch NeuronX 2.8 can be installed using the following:\n\n.. code::\n\n python -m pip install --upgrade neuronx-cc==2.* torch-neuronx==2.8.* torchvision\n\n.. note::\n\n PyTorch NeuronX 2.8 is currently available for Python 3.10 and 3.11, with 3.12+ support coming in a future release.\n\n.. note::\n\n To use Amazon Linux 2023, you will need to install Python 3.10 or 3.11 to use PyTorch NeuronX 2.8.\n\n.. _migrate_to_pytorch_2.8:\n\nMigrate your application to PyTorch 2.8\n---------------------------------------\n\nFirst, install the PyTorch NeuronX 2.8 as described above in :ref:`installation guide `\n\n\nMigrating training scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThere are no code changes required in the training scripts to move from PyTorch NeuronX 2.7 to PyTorch NeuronX 2.8.\n\nSee :ref:`v2.7 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.6.\nSee :ref:`v2.6 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.5.\n\nMigrating inference scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThere are no code changes required in the inference scripts.\n\n\nTroubleshooting and Known Issues\n--------------------------------\n\n[v2.8] Lower BERT/LLaMA performance with torch-xla 2.8.0\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing the publicly released version of torch-xla 2.8.0 from public PyPI repositories would result in lower performance for models like BERT and LLaMA (https://github.com/pytorch/xla/issues/9605). To fix this, switch to using the updated torch-xla version 2.8.1 from public PyPI repositories.\n\nUsing the latest torch-xla 2.7/2.8 may result in increase in host memory usage\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing torch-xla 2.7/2.8 may result in an increase in host memory usage compared to torch-xla 2.6. In one example, LLama2 pretraining with ZeRO1 and sequence length 16k could see an increase of 1.6% in host memory usage.\n\nTypeError: AdamW.__init__() got an unexpected keyword argument 'decoupled_weight_decay'\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAdamW now has an additional argument ``decoupled_weight_decay`` which defaults to False. If you get ``TypeError: AdamW.__init__() got an unexpected keyword argument 'decoupled_weight_decay'`` with NeuronX Distributed, update to the latest version.\n\n\nTensor split on second dimension of 2D array not working\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, when using the tensor split operation on a 2D array in the second dimension, the resulting tensors do not contain the expected data (https://github.com/pytorch/xla/issues/8640). The workaround is to set ``XLA_DISABLE_FUNCTIONALIZATION=0``. Another workaround is to use ``torch.tensor_split``.\n\nLower BERT pretraining performance when switch to using ``model.to(torch.bfloat16)``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, BERT pretraining performance is approximately 11% lower when switching to using ``model.to(torch.bfloat16)`` as part of migration away from the deprecated environment variable ``XLA_DOWNCAST_BF16`` due to https://github.com/pytorch/xla/issues/8545. As a workaround to recover the performance, you can set ``XLA_DOWNCAST_BF16=1``, which will still work in torch-neuronx 2.5 to 2.8 although there will be end-of-support warnings (as noted below).\n\n\nDeprecationWarning: Use torch_xla.device instead\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a end-of-support warning when using ``torch_xla.core.xla_model.xla_device()``. Switch to ``torch_xla.device()`` instead.\n\nDeprecationWarning: Use torch_xla.sync instead\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a end-of-support warning when using ``torch_xla.core.xla_model.mark_step()``. Switch to ``torch_xla.sync()`` instead.\n\nWarning \"XLA_DOWNCAST_BF16 will be deprecated after the 2.6 release, please downcast your model directly\"\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEnvironment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\n\nAttributeError: `_.\nAlso set ``use_reentrant=True`` while calling the torch_xla checkpoint function. Failure to do so will lead to ``XLA currently does not support use_reentrant==False`` error.\nFor more details on checkpointing, refer the `documentation `_.\n\n\nError ``Attempted to access the data pointer on an invalid python storage`` when using HF Trainer API\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nWhile using HuggingFace Transformers Trainer API to train (i.e. :ref:`HuggingFace Trainer API fine-tuning tutorial`), you may see the error \"Attempted to access the data pointer on an invalid python storage\". This is a known `issue `_ and has been fixed in the version ``4.37.3`` of HuggingFace Transformers.\n\n``Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` error during Neuron Parallel Compile\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running Neuron Parallel Compile with HF Trainer API, you may see the errors ``Status: INVALID_ARGUMENT: Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` in Accelerator's ``pad_across_processes`` function. This is due to data-dependent operations in evaluation metrics computation. Data-dependent operations would result in undefined behavior with Neuron Parallel Compile trial execution (execute empty graphs with zero outputs). To work around this error, disable compute_metrics when NEURON_EXTRACT_GRAPHS_ONLY is set to 1:\n\n.. code:: python\n\n compute_metrics=None if os.environ.get(\"NEURON_EXTRACT_GRAPHS_ONLY\") else compute_metrics\n\nCompiler assertion error when running Stable Diffusion training\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWith PyTorch 2.8 (torch-neuronx), you may encounter the following compiler assertion error with Stable Diffusion training when gradient accumulation is enabled. This will be fixed in an upcoming release. For now, if you want to run Stable Diffusion training, disable gradient accumulation in torch-neuronx 2.8 by keeping the `default gradient accumulation steps of 1 `__.\n\n.. code:: bash\n\n ERROR 222163 [NeuronAssert]: Assertion failure in usr/lib/python3.9/concurrent/futures/process.py at line 239 with exception:\n too many partition dims! {{0,+,960}[10],+,10560}[10]\n\n\nFrequently Asked Questions (FAQ)\n--------------------------------\n\nDo I need to recompile my models with PyTorch 2.8?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes.\n\nDo I need to update my scripts for PyTorch 2.8?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSee the :ref:`migration guide `\n\nWhat environment variables will be changed with PyTorch NeuronX 2.8 ?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\nWhat features will be missing with PyTorch NeuronX 2.8?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch NeuronX 2.8 has all of the supported features in PyTorch NeuronX 2.7, with known issues listed above, and unsupported features as listed in :ref:`pytorch_rn`.\n\nCan I use Neuron Distributed libraries with PyTorch NeuronX 2.8?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes, NeuronX Distributed libraries are supported by PyTorch NeuronX 2.8. Transformers NeuronX has reached end-of-support in release 2.26. AWS Neuron Reference for NeMo Megatron has reached end-of-support in release 2.23.\n\nCan I still use PyTorch 2.7 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.7 is supported since release 2.24.\n\nCan I still use PyTorch 2.6 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.6 is supported since release 2.23.\n\nCan I still use PyTorch 2.5 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.5 reached end-of-support in release 2.25.\n\nCan I still use Amazon Linux 2023?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes. You will need to install Python 3.10 or 3.11 to use PyTorch NeuronX 2.8.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-9.rst", "content": ".. _introduce-pytorch-2-9:\n\nIntroducing PyTorch 2.9 Support\n===============================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we introducing?\n------------------------\n\nStarting with the :ref:`Neuron 2.27 ` release, customers can now upgrade to PyTorch NeuronX (``torch-neuronx``) with specific support for PyTorch version 2.9.\n\nPyTorch NeuronX 2.9 adds support for AWS Trainium 3 (Trn3) instances, in addition to existing support for Trainium (Trn2/Trn1/Trn1n) and Inferentia (Inf2) instances.\n\n:ref:`setup-torch-neuronx` is updated to include installation instructions for PyTorch NeuronX 2.9 for Ubuntu 24.04. Note that PyTorch NeuronX 2.9 is supported on Python 3.10, 3.11 and 3.12.\n\nReview :ref:`migration guide ` for possible changes to training scripts. No code changes are required for inference scripts.\n\n\n.. _how-pytorch-2.9-different:\n\nHow is PyTorch NeuronX 2.9 different compared to PyTorch NeuronX 2.8?\n---------------------------------------------------------------------\n\nSee `Torch-XLA 2.9 release `__ for a full list of changes.\n\nSee :ref:`migrate_to_pytorch_2.9` for changes needed to use PyTorch NeuronX 2.9.\n\n.. note::\n\n Torch Dynamo (torch.compile) support in Neuron will be available in a future release.\n\n.. _install_pytorch_neuron_2.9:\n\nHow can I install PyTorch NeuronX 2.9?\n--------------------------------------------\n\nTo install PyTorch NeuronX 2.9, follow the :ref:`setup-torch-neuronx` guides for Ubuntu 24.04 AMI. Refer to the Neuron Multi-Framework DLAMI :ref:`setup guide ` for Ubuntu 24.04 with a pre-installed virtual environment for PyTorch NeuronX 2.9 that you can use to get started. PyTorch NeuronX 2.9 can be installed using the following:\n\n.. code::\n\n python -m pip install --upgrade neuronx-cc==2.* torch-neuronx==2.9.* torchvision\n\n.. note::\n\n PyTorch NeuronX 2.9 is currently available for Python 3.10, 3.11 and 3.12.\n\n.. note::\n\n To use Amazon Linux 2023, you will need to install Python 3.10, 3.11 or 3.12 to use PyTorch NeuronX 2.9. See `Amazon Linux 2023 Python documentation `_ for installation instructions.\n\n.. _migrate_to_pytorch_2.9:\n\nMigrate your application to PyTorch 2.9\n---------------------------------------\n\nFirst, install the PyTorch NeuronX 2.9 as described above in :ref:`installation guide `\n\n\nMigrating training scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThere are no code changes required in the training scripts to move from PyTorch NeuronX 2.8 to PyTorch NeuronX 2.9.\n\nSee :ref:`v2.8 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.7.\nSee :ref:`v2.7 migration guide ` for additional changes needed if you are migrating from PyTorch NeuronX 2.6.\n\nMigrating inference scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThere are no code changes required in the inference scripts.\n\n\nTroubleshooting and Known Issues\n--------------------------------\n\nGLIBC compatibility issue on Amazon Linux 2023\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running PyTorch NeuronX 2.9 on Amazon Linux 2023, you may encounter the following error:\n\n.. code::\n\n ImportError: /usr/lib64/libm.so.6: version `GLIBC_2.35' not found (required by /opt/conda/lib/python3.12/site-packages/_XLAC.cpython-312-x86_64-linux-gnu.so)\n\nThis occurs because the PyTorch NeuronX 2.9 binaries require GLIBC 2.35, but Amazon Linux 2023 ships with an older version of GLIBC. Use Ubuntu 24.04 AMI instead, which has the required GLIBC version. Follow the :ref:`setup-torch-neuronx` installation guide for Ubuntu 24.04.\n\nUsing the latest torch-xla 2.7/2.8/2.9 may result in increase in host memory usage\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing the latest torch-xla v2.7/2.8/2.9 may result in an increase in host memory usage compared to torch-xla v2.6. In one example, LLama2 pretraining with ZeRO1 and sequence length 16k could see an increase of 1.6% in host memory usage.\n\nTypeError: AdamW.__init__() got an unexpected keyword argument 'decoupled_weight_decay'\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAdamW now has an additional argument ``decoupled_weight_decay`` which defaults to False. If you get ``TypeError: AdamW.__init__() got an unexpected keyword argument 'decoupled_weight_decay'`` with NeuronX Distributed, update to the latest version.\n\n\nTensor split on second dimension of 2D array not working\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, when using the tensor split operation on a 2D array in the second dimension, the resulting tensors do not contain the expected data (https://github.com/pytorch/xla/issues/8640). The workaround is to set ``XLA_DISABLE_FUNCTIONALIZATION=0``. Another workaround is to use ``torch.tensor_split``.\n\nLower BERT pretraining performance when switch to using ``model.to(torch.bfloat16)``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, BERT pretraining performance is approximately 11% lower when switching to using ``model.to(torch.bfloat16)`` as part of migration away from the deprecated environment variable ``XLA_DOWNCAST_BF16`` due to https://github.com/pytorch/xla/issues/8545. As a workaround to recover the performance, you can set ``XLA_DOWNCAST_BF16=1``, which will still work in torch-neuronx 2.5 through 2.9 although there will be end-of-support warnings (as noted below).\n\n\nDeprecationWarning: Use torch_xla.device instead\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a end-of-support warning when using ``torch_xla.core.xla_model.xla_device()``. Switch to ``torch_xla.device()`` instead.\n\nDeprecationWarning: Use torch_xla.sync instead\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a end-of-support warning when using ``torch_xla.core.xla_model.mark_step()``. Switch to ``torch_xla.sync()`` instead.\n\nWarning \"XLA_DOWNCAST_BF16 will be deprecated after the 2.6 release, please downcast your model directly\"\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEnvironment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\n\nAttributeError: `_.\nAlso set ``use_reentrant=True`` while calling the torch_xla checkpoint function. Failure to do so will lead to ``XLA currently does not support use_reentrant==False`` error.\nFor more details on checkpointing, refer the `documentation `_.\n\n\nError ``Attempted to access the data pointer on an invalid python storage`` when using HF Trainer API\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nWhile using HuggingFace Transformers Trainer API to train (i.e. :ref:`HuggingFace Trainer API fine-tuning tutorial`), you may see the error \"Attempted to access the data pointer on an invalid python storage\". This is a known `issue `_ and has been fixed in the version ``4.37.3`` of HuggingFace Transformers.\n\n``Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` error during Neuron Parallel Compile\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running Neuron Parallel Compile with HF Trainer API, you may see the errors ``Status: INVALID_ARGUMENT: Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` in Accelerator's ``pad_across_processes`` function. This is due to data-dependent operations in evaluation metrics computation. Data-dependent operations would result in undefined behavior with Neuron Parallel Compile trial execution (execute empty graphs with zero outputs). To work around this error, disable compute_metrics when NEURON_EXTRACT_GRAPHS_ONLY is set to 1:\n\n.. code:: python\n\n compute_metrics=None if os.environ.get(\"NEURON_EXTRACT_GRAPHS_ONLY\") else compute_metrics\n\n\nFrequently Asked Questions (FAQ)\n--------------------------------\n\nDo I need to recompile my models with PyTorch 2.9?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes.\n\nDo I need to update my scripts for PyTorch 2.9?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSee the :ref:`migration guide `\n\nWhat environment variables will be changed with PyTorch NeuronX 2.9 ?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warnings are shown when used). Switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\nWhat features will be missing with PyTorch NeuronX 2.9?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch NeuronX 2.9 has all of the supported features in PyTorch NeuronX 2.8, with known issues listed above, and unsupported features as listed in :ref:`pytorch_rn`.\n\nCan I use Neuron Distributed libraries with PyTorch NeuronX 2.9?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes, NeuronX Distributed libraries are supported by PyTorch NeuronX 2.9. Transformers NeuronX has reached end-of-support in release 2.26. AWS Neuron Reference for NeMo Megatron has reached end-of-support in release 2.23.\n\nCan I still use PyTorch 2.8 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.8 is supported since release 2.26.\n\nCan I still use PyTorch 2.7 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.7 is supported since release 2.24.\n\n.. note::\n\n PyTorch NeuronX 2.7 supports Python 3.10, and 3.11. Python 3.12 is not supported for PyTorch 2.7 and earlier versions.\n\nCan I still use PyTorch 2.6 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.6 has reached end-of-support since release 2.27.\n\nCan I still use Amazon Linux 2023?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes. You will need to install Python 3.10, 3.11 or 3.12 to use PyTorch NeuronX 2.9.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-x.rst", "content": ".. _introduce-pytorch-2-5:\n\nIntroducing PyTorch 2.5 Support\n===============================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nWhat are we introducing?\n------------------------\n\nStarting with the :ref:`Neuron 2.21 ` release, customers will be able to upgrade to ``PyTorch NeuronX(torch-neuronx)`` supporting ``PyTorch 2.5``.\n\n:ref:`setup-torch-neuronx` is updated to include installation instructions for PyTorch NeuronX 2.5 for Amazon Linux 2023 and Ubuntu 22. Note that PyTorch NeuronX 2.5 does not support Python 3.8 which is default in Ubuntu 20. To use Ubuntu 20, customers will need to install Python 3.9+.\n\nPlease review :ref:`migration guide ` for possible changes to training scripts. No code changes are required for inference scripts.\n\n\n.. _how-pytorch-2-5-different:\n\nHow is PyTorch NeuronX 2.5 different compared to PyTorch NeuronX 2.1?\n---------------------------------------------------------------------\n\nPyTorch NeuronX 2.5 uses Torch-XLA 2.5 which has improved support for eager debug mode, Automatic Mixed Precission, PJRT device auto-detection, FP8, and others. See `Torch-XLA 2.5 release `__ for a full list.\n\nSee :ref:`migrate_to_pytorch_2_5` for changes needed to use PyTorch NeuronX 2.5.\n\n.. note::\n\n GSPMD and Torch Dynamo (torch.compile) support in Neuron will be available in a future release.\n\n.. _install_pytorch_neuron_2_5:\n\nHow can I install PyTorch NeuronX 2.5?\n--------------------------------------------\n\nTo install PyTorch NeuronX 2.5 please follow the :ref:`setup-torch-neuronx` guides for Amazon Linux 2023 and Ubuntu 22 AMI. Please also refer to the Neuron multi-framework DLAMI :ref:`setup guide ` for Ubuntu 22 with a pre-installed virtual environment for PyTorch NeuronX 2.5 that you can use to get started. PyTorch NeuronX 2.5 can be installed using the following:\n\n.. code::\n\n python -m pip install --upgrade neuronx-cc==2.* torch-neuronx==2.5.* torchvision\n\n.. note::\n\n PyTorch NeuronX 2.5 is currently available for Python 3.9, 3.10, 3.11.\n\n.. _migrate_to_pytorch_2_5:\n\nMigrate your application to PyTorch 2.5\n---------------------------------------\n\nPlease make sure you have first installed the PyTorch NeuronX 2.5 as described above in :ref:`installation guide `\n\n\nMigrating training scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo migrate the training scripts from PyTorch NeuronX 2.1 to PyTorch NeuronX 2.5, implement the following changes: \n\n.. note::\n\n ``xm`` below refers to ``torch_xla.core.xla_model`` and ``xr`` refers to ``torch_xla.runtime``\n\n* The environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warning when used). Please switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to convert model to BF16 format. (see :ref:`migration_from_xla_downcast_bf16`)\n* The ``torch_xla.experimental.pjrt`` module which was replaced by ``torch_xla.runtime`` in Torch-XLA 2.1, has been removed in Torch-XLA 2.5. Users should now utilize the ``torch_xla.runtime`` module as a replacement.\n* ``torch_xla.runtime.using_pjrt`` is removed because PJRT is the sole Torch-XLA runtime.\n* ``xm.all_reduce`` no longer operates in-place for single tensors. To fix this, please convert the single tensor to an array (e.g.. ``[single_tensor]``) or assign the output of ``xm.all_reduce`` to a variable.\n* The functions ``xm.xrt_world_size()``, ``xm.xla_model.get_ordinal()``, and ``xm.xla_model.get_local_ordinal()`` are deprecated (warning when used). Please switch to ``xr.world_size``, ``xr.global_ordinal``, and ``xr.local_ordinal`` respectively as replacements.\n* ``torch_xla.experimental.xla_sharding`` is now replaced by ``torch_xla.distributed.spmd.xla_sharding``.\n* Class ``ZeroRedundancyOptimizer`` now has two new arguments that replaces the optional boolean argument ``coalesce_cc``:\n * ``bucket_cap_mb_all_gather`` (int, Optional): Number of MegaBytes of the tensor bucket to fill before doing all-gather. Default: 0 (disable all gather coalescing).\n * ``bucket_cap_mb_reduce_scatter`` (int, Optional): Number of MegaBytes of the tensor bucket to fill before doing reduce-scatter. Default: 0 (disable reduce scatter coalescing).\n\nMigrating inference scripts\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThere are no code changes required in the inference scripts.\n\n\nTroubleshooting and Known Issues\n--------------------------------\n\nNeuronx-Distributed Training Llama 3.1 70B 8-node tutorial failed with OSError when the Neuron Cache is placed on FSx mount\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nCurrently, the Neuronx-Distributed Training Llama 3.1 70B 8-node tutorial failed with OSError (Errno 61) when the Neuron Cache is placed on FSx mount:\n\n.. code:: bash\n\n [rank197]: RuntimeError: Bad StatusOr access: INVALID_ARGUMENT: RunNeuronCCImpl: error condition !(error != 400): : [Errno 61] No data available: '/fsxl/neuron_cache/neuronxcc-2.16.372.0+4a9b2326/MODULE_3540044791706521849+4eb52b03/model.neff' -> '/tmp/tmpx7bvfpmm/model.neff'\n\nWe found that the error is due to FSx failing during file copy when there are multiple readers (13 workers fail to copy out of 256). This issue doesn’t affect simpler models like BERT.\n\nTo work-around the issue, please use the shared NFS mount (/home directory on a Parallel Cluster) instead of FSx to store Neuron Cache. This will be fixed in an upcoming release.\n\nRunning in-place update operations (e.g. all_reduce) on 0-dimensional tensors result in buffer aliasing errors in torch 2.5 and earlier\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nTorch's lazy tensor core has a feature where 0-dimensional tensors are stored in a device cache, so scalar constant values can be transferred once and then reused. The values in the device cache are supposed to be marked read-only and never participate in parameter aliasing. However, due to a bug in torch-xla 2.5 (`#8499 `_), sometimes the read-only flag can be dropped, allowing these tensors to be donated, resulting in aliasing errors later when the cached value is used again.\n\nA work-around is to avoid using 0-dimensional tensors by changing them to be 1d tensor of length 1 (`example `_).\nIf modifying library code is not possible, disable XLA parameter aliasing by setting environment variable XLA_ENABLE_PARAM_ALIASING=0\n\nTensor split on second dimension of 2D array not working\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, when using tensor split operation on a 2D array in the second dimension, the resulting tensors don't have the expected data (https://github.com/pytorch/xla/issues/8640). The work-around is to set ``XLA_DISABLE_FUNCTIONALIZATION=0``. Another work-around is to use ``torch.tensor_split``.\n\nImport torch_xla crashed with ``TypeError: must be called with a dataclass type or instance`` with torch-xla 2.5 and torch 2.5.1+cpu (CPU flavor)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen using torch 2.5.1+cpu (CPU flavor) on python 3.10, importing torch_xla crashed with ``TypeError: must be called with a dataclass type or instance`` due to installed triton version 3.2.0 (https://github.com/pytorch/xla/issues/8560). To work-around, please remove the installed triton package or downgrade to triton==3.1.0 or use the regular torch 2.5.1 (GPU flavor).\n\nCertain sequence of operations with ``xm.save()`` could corrupt tensors\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen using the ``xm.save`` function to save tensors, please use ``xm.mark_step()`` before ``xm.save`` to avoid the error described in https://github.com/pytorch/xla/issues/8422 where parameter aliasing could corrupt other tensor values. This issue will be fixed in a future release.\n\n(Here ``xm`` is ``torch_xla.core.xla_model`` following PyTorch/XLA convention)\n\nLower BERT pretraining performance when switch to using ``model.to(torch.bfloat16)``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, BERT pretraining performance is ~11% lower when switching to using ``model.to(torch.bfloat16)`` as part of migration away from the deprecated environment variable ``XLA_DOWNCAST_BF16`` due to https://github.com/pytorch/xla/issues/8545. As a work-around to recover the performance, you can set ``XLA_DOWNCAST_BF16=1`` which would still work in torch-neuronx 2.5 and 2.6 although there will be end-of-support warnings (as noted below).\n\nWarning \"XLA_DOWNCAST_BF16 will be deprecated after the 2.5 release, please downcast your model directly\"\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEnvironment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warning when used). Please switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\n\nWARNING:root:torch_xla.core.xla_model.xrt_world_size() will be removed in release 2.7. is deprecated. Use torch_xla.runtime.world_size instead.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a warning that ``torch_xla.core.xla_model.xrt_world_size()`` will be removed in a future release. Please switch to using ``torch_xla.runtime.world_size`` instead.\n\n\nWARNING:torch_xla.core.xla_model.xla_model.get_ordinal() will be removed in release 2.7. is deprecated. Use torch_xla.runtime.global_ordinal instead.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis is a warning that ``torch_xla.core.xla_model.xla_model.get_ordinal()`` will be removed in a future release. Please switch to using ``torch_xla.runtime.global_ordinal`` instead.\n\n\nAttributeError: module 'torch_xla.runtime' has no attribute 'using_pjrt'\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn Torch-XLA 2.5, ``torch_xla.runtime.using_pjrt`` is removed because PJRT is the sole Torch-XLA runtime.\nSee `commit PR `__.\n\n\nSocket Error: Socket failed to bind\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn PyTorch 2.5, there needs to be a socket available for both torchrun and the ``init_process_group`` to bind. Both of these, by default,\nwill be set to unused sockets. If you plan to use a ``MASTER_PORT`` environment variable then this error may occur, if the port you set it to\nis already in use.\n\n.. code:: \n\n [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use).\n [W socket.cpp:426] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).\n [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.\n RuntimeError: The server socket has failed to listen on any local network address. \n The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).\n\nTo resolve the issue, please ensure if you are setting ``MASTER_PORT`` that the port you're setting it to is not used anywhere else in your scripts. Otherwise,\nyou can leave ``MASTER_PORT`` unset, and torchrun will set the default port for you.\n\n\n``AttributeError: module 'torch' has no attribute 'xla'`` Failure\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn PyTorch 2.5, training scripts might fail during activation checkpointing with the error shown below.\n\n.. code::\n\n AttributeError: module 'torch' has no attribute 'xla'\n\n\nThe solution is to use ``torch_xla.utils.checkpoint.checkpoint`` instead of ``torch.utils.checkpoint.checkpoint`` as the checkpoint function while wrapping pytorch modules for activation checkpointing.\nRefer to the pytorch/xla discussion regarding this `issue `_.\nAlso set ``use_reentrant=True`` while calling the torch_xla checkpoint function. Failure to do so will lead to ``XLA currently does not support use_reentrant==False`` error.\nFor more details on checkpointing, refer the `documentation `_.\n\n\nError ``Attempted to access the data pointer on an invalid python storage`` when using HF Trainer API\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhile using HuggingFace Transformers Trainer API to train (i.e. :ref:`HuggingFace Trainer API fine-tuning tutorial`), you may see the error \"Attempted to access the data pointer on an invalid python storage\". This is a known `issue `_ and has been fixed in the version ``4.37.3`` of HuggingFace Transformers.\n\n``ImportError: libcrypt.so.1: cannot open shared object file: No such file or directory`` on Amazon Linux 2023\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\ntorch-xla version 2.5+ now requires ``libcrypt.so.1`` shared library. Currently, Amazon Linux 2023 includes ``libcrypt.so.2`` shared library by default so you may see `ImportError: libcrypt.so.1: cannot open shared object file: No such file or directory`` when using torch-neuronx 2.1+ on Amazon Linux 2023. To install ``libcrypt.so.1`` on Amazon Linux 2023, please run the following installation command (see also https://github.com/amazonlinux/amazon-linux-2023/issues/182 for more context):\n\n.. code::\n\n sudo dnf install libxcrypt-compat\n\n\n``FileNotFoundError: [Errno 2] No such file or directory: 'libneuronpjrt-path'`` Failure\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nIn PyTorch 2.5, users might face the error shown below due to incompatible ``libneuronxla`` and ``torch-neuronx`` versions being installed.\n\n.. code::\n\n FileNotFoundError: [Errno 2] No such file or directory: 'libneuronpjrt-path'\n\nCheck that the version of ``libneuronxla`` that support PyTorch NeuronX 2.5 is ``2.1.*``. If not, then uninstall ``libneuronxla`` using ``pip uninstall libneuronxla`` and then reinstall the packages following the installation guide :ref:`installation guide `\n\n\nGlibC error on Amazon Linux 2\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nIf using Torch-NeuronX 2.5 on Amazon Linux 2, you will see a GlibC error below. Please switch to a newer supported OS such as Ubuntu 22 or Amazon Linux 2023.\n\n.. code:: bash\n\n ImportError: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /tmp/debug/_XLAC.cpython-38-x86_64-linux-gnu.so)\n\n``Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` error during Neuron Parallel Compile\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running Neuron Parallel Compile with HF Trainer API, you may see the error ``Status: INVALID_ARGUMENT: Input dimension should be either 1 or equal to the output dimension it is broadcasting into`` or ``IndexError: index out of range`` in Accelerator's ``pad_across_processes`` function. This is due to data-dependent operation in evaluation metrics computation. Data-dependent operations would result in undefined behavior with Neuron Parallel Compile trial execution (execute empty graphs with zero outputs). To work-around this error, please disable compute_metrics when NEURON_EXTRACT_GRAPHS_ONLY is set to 1:\n\n.. code:: python\n\n compute_metrics=None if os.environ.get(\"NEURON_EXTRACT_GRAPHS_ONLY\") else compute_metrics\n\nCompiler assertion error when running Stable Diffusion training\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCurrently, with PyTorch 2.5 (torch-neuronx), we are seeing the following compiler assertion error with Stable Diffusion training when gradient accumulation is enabled. This will be fixed in an upcoming release. For now, if you would like to run Stable Diffusion training with Neuron SDK release 2.21/2.22, please disable gradient accumulation in torch-neuronx 2.5.\n\n.. code:: bash\n\n ERROR 222163 [NeuronAssert]: Assertion failure in usr/lib/python3.9/concurrent/futures/process.py at line 239 with exception:\n too many partition dims! {{0,+,960}[10],+,10560}[10]\n\n\nFrequently Asked Questions (FAQ)\n--------------------------------\n\nDo I need to recompile my models with PyTorch 2.5?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes.\n\nDo I need to update my scripts for PyTorch 2.5?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPlease see the :ref:`migration guide `\n\nWhat environment variables will be changed with PyTorch NeuronX 2.5 ?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe environment variables ``XLA_DOWNCAST_BF16`` and ``XLA_USE_BF16`` are deprecated (warning when used). Please switch to automatic mixed-precision or use ``model.to(torch.bfloat16)`` command to cast model to BF16. (see :ref:`migration_from_xla_downcast_bf16`)\n\nWhat features will be missing with PyTorch NeuronX 2.5?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch NeuronX 2.5 now has most of the supported features in PyTorch NeuronX 2.1, with known issues listed above, and unsupported features as listed in :ref:`pytorch_rn`.\n\nCan I use Neuron Distributed and Transformers Neuron libraries with PyTorch NeuronX 2.5?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nYes, NeuronX Distributed, and Transformers NeuronX, and AWS Neuron Reference for NeMo Megatron libraries will work with PyTorch NeuronX 2.5.\n\nCan I still use PyTorch 2.1 version?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nPyTorch 2.1 is supported for release 2.21 and will reach end-of-life in a future release. Additionally, the CVEs `CVE-2024-31583 `_ and `CVE-2024-31580 `_ affect PyTorch versions 2.1 and earlier. We recommend upgrading to the new version of Torch-NeuronX by following :ref:`setup-torch-neuronx`.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/migration-from-xla-downcast-bf16.rst", "content": ".. _migration_from_xla_downcast_bf16:\n\nMigration From ``XLA_USE_BF16``/``XLA_DOWNCAST_BF16``\n=====================================================\n\nIntroduction\n------------\n\nThe environmental variables ``XLA_USE_BF16`` and ``XLA_DOWNCAST_BF16`` were created to provide an easy cast-to-bf16 option before automatic mixed-precision or ``model.to(torch.bfloat16)`` as available in Torch-XLA. Now that both automatic mixed precision and ``model.to(torch.bfloat16)`` are available in Torch-XLA, ``XLA_USE_BF16`` and ``XLA_DOWNCAST_BF16`` are redundant and can be replaced with these options as a more familiar experience as on other platforms such as CPUs and GPUs. Using them in Torch-XLA 2.5+ would cause warnings to be displayed about their end-of-support. While they are still functional, their functionality will be removed in a future release (Torch-XLA 2.8) so the recommended changes below are available as replacement.\n\nNeuronX Distributed Training has been updated to use some of the options below. Please see :ref:`standard_mixed_precision` for more information.\n\nThe changes recommended below can best be made to scripts running with Torch-XLA 2.5+. The same recommendations are also available in :ref:`pytorch-neuronx-programming-guide`.\n\n.. note::\n\n This guide recommends the options below as replacement for ``XLA_USE_BF16`` and ``XLA_DOWNCAST_BF16``. Do not set ``XLA_USE_BF16=1`` or ``XLA_DOWNCAST_BF16=1`` when using the options below on Neuron devices. Using them will override the per-operator precision settings provided by the options and thus cause more operators to execute in bfloat16.\n\nFull BF16 with stochastic rounding enabled\n------------------------------------------\n\nPreviously, on torch-neuronx 2.1 and earlier, the environmental variables ``XLA_USE_BF16`` or ``XLA_DOWNCAST_BF16`` provided full casting to BF16 with stochastic rounding enabled by default. These environmental variables are deprecated in torch-neuronx 2.5, although still functional with warnings. To replace ``XLA_USE_BF16`` or ``XLA_DOWNCAST_BF16`` with stochastic rounding on Neuron, set ``NEURON_RT_STOCHASTIC_ROUNDING_EN=1`` and use the ``torch.nn.Module.to`` method to cast model floating-point parameters and buffers to data-type BF16 as follows:\n\n.. code:: python\n\n os.environ[\"NEURON_RT_STOCHASTIC_ROUNDING_EN\"] = \"1\"\n\n # model is created\n model.to(torch.bfloat16)\n\nStochastic rounding is needed to enable faster convergence for full BF16 model.\n\nIf the loss is to be kept in FP32, initialize it with ``dtype=torch.float`` as follows:\n\n.. code:: python\n\n running_loss = torch.zeros(1, dtype=torch.float).to(device)\n\nSimilarly, if the optimizer states are to be kept in FP32, convert the gradients to FP32 before optimizer computations:\n\n.. code:: python\n\n grad = p.grad.data.float()\n\nFor a full example, please see the :ref:`PyTorch Neuron BERT Pretraining Tutorial (Data-Parallel) `, which has been updated to use ``torch.nn.Module.to`` instead of ``XLA_DOWNCAST_BF16``.\n\nBF16 in GPU-compatible mode without stochastic rounding enabled\n---------------------------------------------------------------\n\nFull BF16 training in GPU-compatible mode would enable faster convergence without the need for stochastic rounding, but would require a FP32 copy of weights/parameters to be saved and used in the optimizer. To enable BF16 in GPU-compatible mode without stochastic rounding enabled, use the ``torch.nn.Module.to`` method to cast model floating-point parameters and buffers to data-type bfloat16 as follows without setting ``NEURON_RT_STOCHASTIC_ROUNDING_EN=1``:\n\n.. code:: python\n\n # model is created\n model.to(torch.bfloat16)\n\nIn the initializer of the optimizer, for example AdamW, you can add code like the following code snippet to make a FP32 copy of weights:\n\n.. code:: python\n\n # keep a copy of weights in highprec\n self.param_groups_highprec = []\n for group in self.param_groups:\n params = group['params']\n param_groups_highprec = [p.data.float() for p in params]\n self.param_groups_highprec.append({'params': param_groups_highprec})\n\nFrom then, you can use the usual gradients but updating the FP32 copy of weights instead:\n\n.. code:: python\n\n for group, group_highprec in zip(self.param_groups, self.param_groups_highprec):\n for p, p_highprec in zip(group['params'], group_highprec['params']):\n # convert gradients to FP32 before computing exponential average\n grad = p.grad.data.float()\n\n # compute the exponential average and denominator using grad\n ...\n\n # Update FP32 copy of weights\n p_highprec.data.addcdiv_(exponential_avg, denominator, value=-step_size)\n\n\nIn the :ref:`PyTorch Neuron BERT Pretraining Tutorial (Data-Parallel) `, this mode can be enabled by pasing ``--optimizer=AdamW_FP32ParamsCopy`` option to ``dp_bert_large_hf_pretrain_hdf5.py`` and setting ``NEURON_RT_STOCHASTIC_ROUNDING_EN=0`` (or leave it unset).\n\nBF16 automatic mixed precision using PyTorch Autocast\n-----------------------------------------------------\n\nBy default, the compiler automatically casts internal FP32 operations to\nBF16. You can disable this and allow PyTorch's BF16 automatic mixed precision function (``torch.autocast``) to\ndo the casting of certain operations to operate in BF16.\n\nTo enable PyTorch's BF16 mixed-precision, first turn off the Neuron\ncompiler auto-cast:\n\n.. code:: python\n\n os.environ[\"NEURON_CC_FLAGS\"] = \"--auto-cast=none\"\n\nNext, per recommendation from official PyTorch `torch.autocast documentation `__, place only\nthe forward-pass of the training step in the ``torch.autocast`` scope with ``xla`` device type:\n\n.. code:: python\n\n with torch.autocast(dtype=torch.bfloat16, device_type='xla'):\n # forward pass\n\nThe device type is XLA because we are using PyTorch-XLA's autocast backend. The PyTorch-XLA `autocast mode source code `_ lists which operations are casted to lower precision BF16 (\"lower precision fp cast policy\" section), which are maintained in FP32 (\"fp32 cast policy\"), and which are promoted to the widest input types (\"promote\" section).\n\n.. note::\n\n If an operation is not part of any policy in `autocast mode source code `_, the data type of the inputs will be used for the computation of the operation.\n\n\nExample showing the original training code snippet:\n\n.. code:: python\n\n def train_loop_fn(train_loader):\n for i, data in enumerate(train_loader):\n inputs = data[0]\n labels = data[3]\n outputs = model(inputs, labels=labels)\n loss = outputs.loss/ flags.grad_acc_steps\n loss.backward()\n optimizer.step()\n xm.mark_step()\n\nThe following shows the training loop modified to use BF16 autocast:\n\n.. code:: python\n\n os.environ[\"NEURON_CC_FLAGS\"] = \"--auto-cast=none\"\n\n def train_loop_fn(train_loader):\n for i, data in enumerate(train_loader):\n torch.cuda.is_bf16_supported = lambda: True\n with torch.autocast(dtype=torch.bfloat16, device_type='xla'):\n inputs = data[0]\n labels = data[3]\n outputs = model(inputs, labels=labels)\n loss = outputs.loss/ flags.grad_acc_steps\n loss.backward()\n optimizer.step()\n xm.mark_step()\n\nFor a full example of BF16 mixed-precision, see :ref:`PyTorch Neuron BERT Pretraining Tutorial (Data-Parallel) `.\n\nSee official PyTorch documentation for more details about\n`torch.autocast `__\n.\n" }, { "path": "about-neuron/appnotes/torch-neuronx/torch-neuronx-dataparallel-app-note.rst", "content": ".. _torch-neuronx-dataparallel-app-note:\n\nData Parallel Inference on torch_neuronx\n=======================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nIntroduction\n------------\n\nThis guide introduces :func:`torch_neuronx.DataParallel`, a Python API that\nimplements data parallelism on :class:`~torch.jit.ScriptModule` models created by the\n:ref:`torch_neuronx_trace_api`.\nThe following sections explain how data parallelism can improve the performance of\ninference workloads on Inferentia, including how :func:`torch_neuronx.DataParallel`\nuses dynamic batching to run inference on variable input sizes. It covers an\noverview of the :func:`torch_neuronx.DataParallel` module and provides a few\n:ref:`example data parallel applications `.\n\nData parallel inference\n-------------------------\n\nData Parallelism is a form of parallelization across multiple devices or cores,\nreferred to as nodes. Each node contains the same model and parameters, but\ndata is distributed across the different nodes. By distributing the\ndata across multiple nodes, data parallelism reduces the total\nexecution time of large batch size inputs compared to sequential execution.\nData parallelism works best for smaller models in latency sensitive\napplications that have large batch size requirements.\n\n\ntorch_neuronx.DataParallel\n-------------------------\n\nTo fully leverage the Inferentia hardware, we want to use all available\nNeuronCores. An inf2.xlarge and inf2.8xlarge have two NeuronCores, an\ninf2.24xlarge has 12 NeuronCores, and an inf2.48xlarge has 24 NeuronCores.\nFor maximum performance on Inferentia hardware, we can use\n:func:`torch_neuronx.DataParallel` to utilize all available NeuronCores.\n\n:func:`torch_neuronx.DataParallel` implements data parallelism at the module\nlevel by replicating the Neuron model on all available NeuronCores\nand distributing data across the different cores for parallelized inference.\nThis function is analogous to :class:`~torch.nn.DataParallel` in PyTorch.\n:func:`torch_neuronx.DataParallel` requires PyTorch >= 1.8.\n\nThe following sections provide an overview of some of the features\nof :func:`torch_neuronx.DataParallel` that enable maximum performance on\nInferentia.\n\nNeuronCore selection\n^^^^^^^^^^^^^^^^^^^^\n\nBy default, DataParallel will try to use all NeuronCores allocated to the\ncurrent process to fully saturate the Inferentia hardware for maximum performance.\nIt is more efficient to make the batch dimension divisible by the number of\nNeuronCores. This will ensure that NeuronCores are not left idle during\nparallel inference and the Inferentia hardware is fully utilized.\n\nIn some applications, it is advantageous to use a subset of the\navailable NeuronCores for DataParallel inference. DataParallel has a\n``device_ids`` argument that accepts a list of :obj:`int` or ``'nc:#'``\nthat specify the NeuronCores to use for parallelization. See\n:ref:`Specifying NeuronCores `\nfor an example of how to use ``device_ids`` argument.\n\nBatch dim\n^^^^^^^^^\n\nDataParallel accepts a ``dim`` argument that denotes the batch dimension used\nto split the input data for distributed inference. By default,\nDataParalell splits the inputs on ``dim = 0`` if the ``dim`` argument is not\nspecified. For applications with a non-zero batch dim, the ``dim`` argument\ncan be used to specify the inference-time input batch dimension.\n:ref:`DataParallel with dim ! = 0 ` provides an\nexample of data parallel inference on inputs with batch dim = 2.\n\n.. _dynamic_batching_description_torch_neuronx:\n\nDynamic batching\n^^^^^^^^^^^^^^^^\n\nBatch size has a direct impact on model performance. The Inferentia chip is optimized\nto run with small batch sizes. This means that a Neuron compiled model can outperform\na GPU model, even if running single digit batch sizes.\n\nAs a general best practice, we recommend optimizing your model's throughput by\ncompiling the model with a small batch size and gradually increasing it to\nfind the peak throughput on Inferentia.\n\nDynamic batching is a feature that allows you to use tensor batch sizes that the\nNeuron model was not originally compiled against. This is necessary because the\nunderlying Inferentia hardware will always execute inferences with the batch\nsize used during compilation. Fixed batch size execution allows tuning the\ninput batch size for optimal performance. For example, batch size 1 may be\nbest suited for an ultra-low latency on-demand inference application, while\nbatch size > 1 can be used to maximize throughput for offline inferencing.\nDynamic batching is implemented by slicing large input tensors into chunks\nthat match the batch size used during the :func:`torch_neuronx.trace` compilation call.\n\nThe :func:`torch_neuronx.DataParallel` class automatically enables dynamic batching on\neligible models. This allows us to run inference in applications that have\ninputs with a variable batch size without needing to recompile the model. See\n:ref:`Dynamic batching ` for an example\nof how DataParallel can be used to run inference on inputs with a dynamic batch\nsize without needing to recompile the model.\n\nDynamic batching using small batch sizes can result in sub-optimal throughput\nbecause it involves slicing tensors into chunks and iteratively sending data\nto the hardware. Using a larger batch size at compilation time can use the\nInferentia hardware more efficiently in order to maximize throughput. You can\ntest the tradeoff between individual request latency and total throughput by\nfine-tuning the input batch size.\n\nAutomatic batching in the DataParallel module can be disabled using the\n``disable_dynamic_batching()`` function as follows:\n\n.. code-block:: python\n\n >>> model_parallel = torch_neuronx.DataParallel(model_neuron)\n >>> model_parallel.disable_dynamic_batching()\n\nIf dynamic batching is disabled, the compile-time batch size must be equal to\nthe inference-time batch size divided by the number of NeuronCores.\n:ref:`DataParallel with dim != 0 ` and\n:ref:`Dynamic batching disabled `\nprovide examples of running DataParallel inference with dynamic batching\ndisabled.\n\n\nPerformance optimizations\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe DataParallel module has a ``num_workers`` attribute that can be used to\nspecify the number of worker threads used for multithreaded inference. By\ndefault, ``num_workers = 2 * number of NeuronCores``. This value can be\nfine tuned to optimize DataParallel performance.\n\nDataParallel has a ``split_size`` attribute that dictates the size of the input\nchunks that are distributed to each NeuronCore. By default,\n``split_size = max(1, input.shape[dim] // number of NeuronCores)``. This value\ncan be modified to optimally match the inference input chunk size with the\ncompile-time batch size.\n\n.. _data_parallel_examples_torch_neuronx:\n\nExamples\n--------\n\nThe following sections provide example usages of the\n:func:`torch_neuronx.DataParallel` module.\n\n\n.. _dataparallel_example_default_torch_neuronx:\n\nDefault usage\n^^^^^^^^^^^^^\n\n.. include:: /frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-default.rst\n\n.. _dataparallel_example_specify_ncs_torch_neuronx:\n\nSpecifying NeuronCores\n^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-specify-ncs.rst\n\n\n.. _dataparallel_example_dim_neq_zero_torch_neuronx:\n\nDataParallel with dim != 0\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-dim-neq-zero.rst\n\n\n.. _dataparallel_example_dynamic_batching_torch_neuronx:\n\nDynamic batching\n^^^^^^^^^^^^^^^^\n\n.. include:: /frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-dynamic-batching.rst\n\n\n.. _dataparallel_example_disable_dynamic_batching_torch_neuronx:\n\nDynamic batching disabled\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-disable-dynamic-batching.rst\n\n" }, { "path": "about-neuron/appnotes/torch-neuronx/torch-neuronx-graph-partitioner-app-note.rst", "content": ".. _torch-neuronx-graph-partitioner-app-note:\n\nGraph Partitioner on torch_neuronx\n=======================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nIntroduction\n------------\n\nThis guide introduces the graph partitioner for torch-neuronx.\nThe following sections explain the purpose of the graph partitioner,\nhow it works, and go over a few examples.\n\nThe Purpose of the Graph Partitioner\n------------------------------------\n\nWhile ``neuronx-cc`` is very sophisticated and can compile most operators,\nthere are some operator configurations that are not supported by the compiler.\nUsually in a model that contains unsupported operators, these are only a few\noperators while the supported parts of the model can benefit from the acceleration\nbenefits that Neuron offers. With this in mind, we developed a graph partitioner\nthat will partition out unsupported operators to be executed on CPU, while \ncompiling and executing the supported operators on Neuron.\n\nHow it Works\n------------\n\nDetermining Unsupported Operators\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOperator support is determined by the ``neuronx-cc`` compiler frontend. This is done\nbecause this gives us more flexibility than a static list. This is evident\nin cases where a specific operator configuration is supported but another\nconfiguration is not supported. For example, we support the square root operator,\nbut do not support it with a ``C64`` data type for example.\n\nTo check operator support, we use the :func:`torch_neuronx.analyze` API, which\nqueries the compiler for device placement: Neuron or CPU, which gives the graph\npartitioner a base graph to start partitioning.\n\nThe below image shows the flow of the graph partitioner:\n\n|torch-neuronx-graph-partitioner-flow-diagram|\n\n.. |torch-neuronx-graph-partitioner-flow-diagram| image:: /images/torch-neuronx-graph-partitioner-flow-diagram.png\n\nCustomizability\n^^^^^^^^^^^^^^^\n\nThe graph partitioner has a wide range of customizability\nfor a variety of situations. The customization options include:\n\n1. **Minimum Operator Support:** Only partition the model if a minimum percentage of operators are supported.\n2. **Minimum Subgraph Size:** The minimum number of operators in any given subgraph. This can be useful if having compute chokepoints with single operator subgraphs is not desired.\n3. **Maximum Subgraph Count:** The maximum number of subgraphs. Too many subgraphs can fragment the computation graph causing performance degredation.\n4. **Ops to Partition:** Additional operators to partition to CPU beyond the unsupported operators. This can be useful to suggest to the graph partitioner to partition to create a more balanced graph.\n\nFurthermore, compiler flags/args can be passed into all Neuron subgraphs through the graph partitioner.\n\nFor the API Reference, visit :func:`torch_neuronx.trace` and :class:`torch_neuronx.PartitionerConfig`\n\n.. note::\n Dynamic batching has a case-by-case support with partitioned\n models, because it is highly dependent on how the\n final partition scheme looks like.\n\nExamples\n--------\n\nThe following sections provide example usages of the graph partitioner.\n\nDefault Usage\n^^^^^^^^^^^^^\n\nThe below model is a simple MLP model with sorted log softmax output.\nThe sort operator, ``torch.sort()`` or ``aten::sort``, is not supported\nby ``neuronx-cc`` at this time, so the graph partitioner will partition\nout the sort operator to CPU.\n\n.. code-block:: python\n\n import torch\n import torch_neuronx\n import torch.nn as nn\n\n import logging\n \n # adjust logger level to see what the partitioner is doing\n logger = logging.getLogger(\"Neuron\")\n\n class MLP(nn.Module):\n def __init__(\n self, input_size=28 * 28, output_size=10, layers=[4096, 2048]\n ):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n self.relu = nn.ReLU()\n\n def forward(self, x):\n f1 = self.fc1(x)\n r1 = self.relu(f1)\n f2 = self.fc2(r1)\n r2 = self.relu(f2)\n f3 = self.fc3(r2)\n out = torch.log_softmax(f3, dim=1)\n sort_out,_ = torch.sort(out)\n return sort_out\n\n n = MLP()\n n.eval()\n\n inputs = torch.rand(32,784)\n\n # Configure the graph partitioner with the default values\n partitioner_config = torch_neuronx.PartitionerConfig()\n\n # Trace a neural network with graph partitioner enabled\n neuron_net = torch_neuronx.trace(n, inputs, partitioner_config=partitioner_config)\n\n # Run inference on the partitioned model\n output = neuron_net(inputs)\n\n\nSpecifying requirements\n^^^^^^^^^^^^^^^^^^^^^^^\n\nThis example is very similar to the previous example, but\nhas two differences. The unsupported sort operator is sandwiched\nbetween the ReLU activation function after the first linear layer\nand the second linear layer. The second difference is that we are\nspecifying a max subgraph count of 2.\n\n.. code-block:: python\n\n import torch\n import torch_neuronx\n import torch.nn as nn\n\n import logging\n \n # adjust logger level to see what the partitioner is doing\n logger = logging.getLogger(\"Neuron\")\n\n class MLP(nn.Module):\n def __init__(\n self, input_size=28 * 28, output_size=10, layers=[4096, 2048]\n ):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n self.relu = nn.ReLU()\n\n def forward(self, x):\n f1 = self.fc1(x)\n r1 = self.relu(f1)\n sort_r1,_ = torch.sort(r1)\n f2 = self.fc2(sort_r1)\n r2 = self.relu(f2)\n f3 = self.fc3(r2)\n out = torch.log_softmax(f3, dim=1)\n return out\n\n n = MLP()\n n.eval()\n\n inputs = torch.rand(32,784)\n\n # Configure the graph partitioner with the default values\n partitioner_config = torch_neuronx.PartitionerConfig(max_subgraph_count=2)\n\n # This trace will fail since the min_subgraph_size requirement can't be satisfied by the graph partitioner\n neuron_net = torch_neuronx.trace(n, inputs, partitioner_config=partitioner_config)\n\nOutput:\n\n.. code-block::\n\n ValueError: The partitioner has found 3 subgraphs which exceeds the specified max subgraph count of 2.\n\n\nThis example fails because the sort operator placement generates 3 subgraphs, which is more than 2.\n\nSpecifying additional operators to partition\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis example shows a situation where we want to partition out\nthe log_softmax operator despite it being supported. We also specify\nan 80% support percentage threshold.\n\n.. code-block:: python\n\n import torch\n import torch_neuronx\n import torch.nn as nn\n\n import logging\n \n # adjust logger level to see what the partitioner is doing\n logger = logging.getLogger(\"Neuron\")\n logger.setLevel(logging.INFO)\n\n class MLP(nn.Module):\n def __init__(\n self, input_size=28 * 28, output_size=10, layers=[4096, 2048]\n ):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n self.relu = nn.ReLU()\n\n def forward(self, x):\n f1 = self.fc1(x)\n r1 = self.relu(f1)\n f2 = self.fc2(r1)\n r2 = self.relu(f2)\n f3 = self.fc3(r2)\n out = torch.log_softmax(f3, dim=1)\n sort_out,_ = torch.sort(out)\n return sort_out\n\n n = MLP()\n n.eval()\n\n inputs = torch.rand(32,784)\n\n # Configure the graph partitioner with the default values\n partitioner_config = torch_neuronx.PartitionerConfig(min_operator_percentage_threshold=0.8,ops_to_partition=set([\"aten::log_softmax\"]))\n\n # This trace succeeds\n neuron_net = torch_neuronx.trace(n, inputs, partitioner_config=partitioner_config)\n\nKey Output logs:\n\n.. code-block::\n\n ...\n Neuron: The following operations are currently supported:\n Neuron: aten::linear\n Neuron: aten::relu\n Neuron: aten::log_softmax\n Neuron: The following operations are currently not supported:\n Neuron: aten::sort, unsup.py(28): \n ...\n Neuron: 85.71% of arithmetic operations (6 of 7) are supported\n Neuron: Num Partitions: 2\n\n Neuron: Creating Partition #1 for device: Device.NEURON\n Neuron: The following operators will be included in this partition:\n Neuron: prim::GetAttr:9\n Neuron: aten::linear:3\n Neuron: aten::relu:2\n ...\n Neuron: Creating Partition #2 for device: Device.CPU\n Neuron: The following operators will be included in this partition:\n Neuron: prim::Constant:4\n Neuron: aten::sort:1\n Neuron: aten::log_softmax:1\n\n\nNotice that we still report that ``aten::log_softmax`` is still supported, but also\nreport that ``aten::log_softmax`` is in Partition #2 which is for ``Device.CPU``." }, { "path": "about-neuron/appnotes/transformers-neuronx/generative-llm-inference-with-neuron.rst", "content": ".. _neuron_llm_inference:\n\nGenerative LLM inference with Neuron\n====================================\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nBackground\n----------\n\nLarge Language Models (LLMs) generate human-like text through a\nprocess known as generative inference. Fundamentally, given an input prompt, generative LLM\ninference generates text outputs, by\niteratively predicting the next token in a sequence.\n\nThese models typically take a sequence of integers as input, which\nrepresent a sequence of tokens (words/subwords), and generate a\nprediction for the next token to be emitted. Below is a simple example\nthat illustrates this in code:\n\n.. code-block:: python\n\n # Vocabulary of tokens the model can parse. The position of each token in the \n # vocabulary is used as the token_id (an integer representing that token)\n vocab = [\"having\", \"I\", \"fun\", \"am\", \"learning\", \".\", \"Neuron\"]\n\n # input token_ids: list of integers that represent the input tokens in this\n # case: \"I\", \"am\", \"having\", \"fun\"\n input_token_ids = [1, 3, 0, 2] \n \n\n # The LLM gets a vector of input token_ids, and generates a probability-distribution\n # for what the output token_id should be (with a probability score for each token_id\n # in the vocabulary)\n output = LLM(input_token_ids) \n \n\n # by taking argmax on the output, we effectively perform a 'greedy sampling' process,\n # i.e. we choose the token_id with the highest probability. Other sampling techniques\n # also exist, e.g. Top-K. By choosing a probabilistic sampling method we enable the model\n # to generate different outputs when called multiple times with the same input.\n next_token_id = np.argmax(output) \n\n\n # map the token_id back into an output token\n next_token = vocab[next_token_id] \n\n\nTo generate entire sentences, the application iteratively invokes the\nLLM to generate the next token's prediction, and at each iteration we\nappend the predicted token back into the input:\n\n\n.. code-block:: python\n\n def generate(input_token_ids, n_tokens_to_generate):\n for _ in range(n_tokens_to_generate): # decode loop\n output = LLM(input_token_ids) # model forward pass\n \n next_token_id = np.argmax(output) # greedy sampling\n \n if (next_token_id == EOS_TOK_ID)\n break # break if generated End Of Sentence (EOS)\n \n # append the prediction to the input, and continue to the next out_token\n input_token_ids.append(int(next_token_id)) \n\n return input_token_ids[-n_tokens_to_generate :] # only return generated token_ids\n\n input_token_ids = [1, 3] # \"I\" \"am\"\n output_token_ids = generate(input_tokens_ids, 4) # output_token_ids = [0, 2, 4, 6]\n output_tokens = [vocab[i] for i in output_token_ids] # \"having\" \"fun\" \"learning\" “Neuron”\n\n\nThis process, of predicting a future value (regression) and adding\nit back into the input (auto), is sometimes referred to as\nautoregression. For more details, Jay Mody’s \\ `GPT in 60 Lines of\nNumPy `__\\ is an\nexcellent writeup on GPTs (Generative Pre-trained Transformers).\n\n\nPerformance optimizations\n-------------------------\n\nThe sheer size of state-of-the-art LLMs, as well as the sequential\nnature of text generation, poses multiple challenges for efficient\ngenerative LLM deployment.\n\nFirst, the model is typically sharded across multiple devices, in order to fit the model\nin device memory. This creates communication overhead and complexity among devices.\nSecondly, certain deployments have strict application-level latency bounds, thus requiring\nsubstantial latency optimizations. This is especially challenging, due to the sequential nature\nof token-by-token generation. Finally, generating one token at a time often leads to poor \ndevice utilization, due to low arithmetic intensity, which can be improved via batching (see :ref:`what_batch_size_to_use`).\n\nThe Neuron SDK provides several built-in\noptimizations, allowing you to extract optimal performance when\ndeploying LLM models, including:\n\nKV-caching:\n^^^^^^^^^^^\n\nThe `transformers-neuronx `__\nlibrary implements KV-cache optimization, which saves compute\nresources by reusing previously calculated SelfAttention key-value\npairs, instead of recalculating them for each generated token.\n\nTo illustrate this concept, see the\ninner workings of the MaskedSelfAttention operator in the figure below.\n\nAt each token generation step, the Query vector of a single current token is multiplied by the Key vectors of all \nprevious tokens in the sequence to create attention scores and these scores are further multiplied by the Value\nvectors of all previous tokens.\n\n\n.. image:: /images/masked-self-attention-operator.png\n\n\nThe core idea behind this optimization is that instead of re-computing the Key and Value vectors\nfor all previous tokens at each token generation step, Neuron can perform only incremental\ncomputation for the current token and re-use previously computed Key/Value vectors from the KV-cache. \nThe Key/Value vector of the current token is also appended to the KV-cache, for the next token generation step.\n\n\n\n.. image:: /images/kv-cache-optimization.png\n\n\n\nNote that the first token in the\noutput sequence is unique in two ways:\n\n.. container::\n\n - No KV-cache is available at this point.\n - Neuron needs to compute the entire KV-cache for tokens (the\n input prompt), rather than one incremental KV-cache entry.\n\nThis means that first-token latency is typically higher\nthan the following tokens.\n\nModel sharding:\n^^^^^^^^^^^^^^^\n\nNeuron enables you to shard the model across devices via Tensor\nParallelism, Pipeline Parallelism (coming soon), or a combination of the two (coming soon).\n\nTensor Parallelism shards each layer across multiple devices,\nenabling you to achieve the optimal latency.\n\nPipeline Parallelism places different layers on different devices and\ncreates a pipeline between them (as the name suggests) and is\nuseful mainly when optimizing throughput and/or cost-per-inference.\n\nTo find the optimal Tensor/Pipeline parallelism configuration for your\nmodel, see the :ref:`model_partitioning` section.\n \nComputation/communication overlap:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThe Neuron compiler automatically fuses Collective Communication\nprimitives (e.g., AllReduce) with the following computation (e.g.,\nGEMM) in the compute graph. This helps minimize any overhead caused by sharding the\nmodel across devices.\n\nCompact data-types:\n^^^^^^^^^^^^^^^^^^^\nNeuron supports INT8 and FP8 (coming soon), which can significantly reduce the model's memory bandwidth and capacity requirements. \nThis is especially useful for Generative LLM inference, which is typically memory-bound. Therefore, using a compact data-type can improve the overall\nLLM inference performance with lower latency and higher throughput.\n\n\nBucketing:\n^^^^^^^^^^\nThe transformers-neuronx library automatically uses bucketing to process the input prompt and output tokens. Bucketing makes\nit possible to handle variable sequence lengths, without requiring support for dynamic shapes. Using multiple progressively \nlarger buckets helps minimize the portion of the KV-cache that needs to be read for each token.\n\n.. _model_partitioning:\n\nModel partitioning\n------------------\n\nHow many NeuronCores do I need?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nTransformer models are typically defined via a hyper-parameter configuration, such\nas the following:\n\n.. code-block:: python\n\n {\n \"n_vocab\": 50257, # number of tokens in our vocabulary\n \"n_ctx\": 2048, # maximum possible sequence length of the input\n \"n_embd\": 9216, # embedding dimension (determines the \"width\" of the network)\n \"n_head\": 72, # number of attention heads (n_embd must be divisible by n_head)\n \"n_layer\": 64 # number of layers (determines the \"depth\" of the network)\n }\n\nTo determine the number of NeuronCores needed to fit the model,\nperform the following calculation:\n\n.. code-block:: python\n\n weight_mem_footprint = 12 x x ^2 x \n KV_cache_mem_footprint = x x x x 2 x \n # is 2 for BF16/FP16, or 1 for FP8/INT8\n\n mem_footprint = weight_mem_footprint + KV_cache_mem_footprint\n\n\nAnd from here, determining the number of NeuronCores is straightforward:\n\n\n.. code-block:: python\n\n num_neuron_cores = ceil_to_closest_supported_size (mem_footprint / , ) # 16GiB per Inferentia2/Trainium1 NeuronCore\n\n\n\nFor example, when running OPT-66B on Inf2, with a batch-size of 16, \nthe number of required NeuronCores can be computed as follows.\n\n\n.. code-block:: python\n\n # OPT-66B example (BF16, Inf2)\n # n_layer=64, n_ctx=2048, n_embd=9216, batch=16\n weight_mem_footprint = 12 x 64 x 9216^2 x 2 = 121.5 GiB\n KV_cache_mem_footprint = 16 x 64 x 2048 x 9216 x 2 x 2 = 72 GiB \n\n mem_footprint = 121.5GiB + 72GiB = 193.5 GiB\n\n num_neuron_cores = ceil_to_closest_supported_size (193.5GiB / 16GiB, Inf2)\n = ceil_to_closest_supported_size (12.1) = 24\n ## Currently, the Neuron runtime supports tensor-parallelism degrees 2, 8, and 32 on Trn1\n ## and supports tensor-parallelism degrees 2, 4, 8, 12 and 24 on Inf2.\n\n\nUse the :ref:`neuron_calculator` to compute the number of cores needed for a custom hyper-parameter configuration.\n\nWhich parallelism technique should I use?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nTensor parallelism improves latency, at the expense of increased\nintra-layer communication. Thus, as a general rule, it is recommended to use\nthe smallest tensor parallelism degree that meets your latency\nrequirement and then use pipeline/data parallelism from that point on.\n\nIf latency is not a major concern in your application (e.g., model evaluation)\nand the primary goal is to maximize throughput (i.e., minimize total cost per token),\nthen it is most efficient to use pipeline parallelism and increase the batch-size\nas much as possible.\n\n\n.. _what_batch_size_to_use:\n\nWhat batch-size should I use?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nDue to the serial token generation nature of generative LLM inference,\nthis workload tends to be extremely memory bound. This means that\nthroughput (and thus cost per inference) improves significantly by\nbatching.\n\nAs a general rule, we recommend increasing the batch-size to the\nmaximum amount that fits within the latency budget (up to batch=256.\nA larger batch-size typically does not help with performance.)\n\nNote that the KV-cache grows linearly with the batch-size and can\ngrow until it runs out of memory (typically referred to as\nOOM). If the latency budget allows, we recommend increasing the\nbatch-size to the maximum value that does not result in OOM.\n\nUsers may also consider pipelining the model beyond what is necessary\nto fit model parameters / KV-cache on devices, in order to free up\ndevice-memory space and thus allow the batch-size to increase\nwithout causing OOM issues.\n\n\n" }, { "path": "about-neuron/arch/glossary.rst", "content": ".. _neuron_hw_glossary:\n\nNeuron Glossary\n===============\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nTerms\n-----\n\nNeuron Devices (Accelerated Machine Learning chips)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n \n\n * - Term\n - Description\n\n * - .. glossary::\n Inferentia\n - AWS first generation accelerated machine learning chip supporting inference only\n\n * - .. glossary::\n Trainium/Inferentia2\n - AWS second generation accelerated machine learning chip supporting training and inference\n\n * - .. glossary::\n Trainium2\n - AWS second generation accelerated machine learning chip supporting training and inference\n\n * - .. glossary::\n Neuron Device\n - Accelerated machine learning chip (e.g. Inferentia or Trainium)\n\nNeuron powered Instances\n^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n \n\n * - Term\n - Description\n\n\n * - .. glossary::\n Inf1\n - Inferentia powered accelerated compute EC2 instance\n\n * - .. glossary::\n Trn1\n - Trainium powered accelerated compute EC2 instance\n\n * - .. glossary::\n Inf2\n - Inferentia2 powered accelerated compute EC2 instance\n\n * - .. glossary::\n Trn2\n - Trainium2 powered accelerated compute EC2 instance\n\n\nNeuronCore terms\n^^^^^^^^^^^^^^^^\n\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n \n\n * - Term\n - Description\n\n\n * - .. glossary::\n NeuronCore\n - The machine learning compute cores within Inferentia/Trainium\n\n * - .. glossary::\n NeuronCore-v1\n - Neuron Core within Inferentia\n\n * - .. glossary::\n NeuronCore-v2\n - Neuron Core within Trainium1/Inferentia2\n\n * - .. glossary::\n NeuronCore-v3\n - Neuron Core within Trainium2\n\n * - .. glossary::\n Tensor Engine\n - 2D systolic array (within the NeuronCore), used for matrix computations\n\n * - .. glossary::\n Scalar Engine\n - A scalar-engine within each NeuronCore, which can accelerate element-wise operations (e.g. GELU, ReLU, reciprocal, etc)\n\n * - .. glossary::\n Vector Engine\n - A vector-engine with each NeuronCore, which can accelerate spatial operations (e.g. layerNorm, TopK, pooling, etc)\n\n * - .. glossary::\n GPSIMD Engine\n - Embedded General Purpose SIMD cores, within each NeuronCore, to accelerate custom-operators\n\n * - .. glossary::\n Sync Engine\n - The SP engine, which is integrated inside NeuronCore. Used for synchronization and DMA triggering.\n\n * - .. glossary::\n Collective Communication Engine\n - Dedicated engine for collective communication, allows for overlapping computation and communication\n\n * - .. glossary::\n High Bandwidth Memory\n - `High Bandwidth Memory `_, used as device memory for NeuronCore-v2 and beyond.\n \n * - .. glossary::\n State Buffer\n - The main software-managed on-chip memory in NeuronCore-v1 and beyond.\n\n * - .. glossary::\n Partial Sum Buffer\n - A second software-managed on-chip memory in NeuronCore-v1 and beyond, with near-memory accumulation support for TensorE output data.\n \n * - .. glossary::\n NeuronLink\n - Interconnect between NeuronCores\n\n * - .. glossary::\n NeuronLink-v1\n - Interconnect between NeuronCores in Inferentia device\n\n * - .. glossary::\n NeuronLink-v2\n - Interconnect between NeuronCores in Trainium1/Inferentia2 device\n\n * - .. glossary::\n NeuronLink-v3\n - Interconnect between NeuronCores in Trainium2 device\n\nNeuron SDK terms\n^^^^^^^^^^^^^^^^\n\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n \n\n * - Term\n - Description\n\n\n * - .. glossary::\n Neuron Kernel Interface\n - A bare-metal language and compiler for directly programming Neuron devices available on AWS Trainium/Inferentia2 and beyond devices.\n\n\nAbbreviations\n-------------\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n \n\n * - Abbreviation\n - Description\n\n * - .. glossary::\n NxD Core\n - NeuronX Distributed Core Library\n\n * - .. glossary::\n NxD Training\n - NeuronX Distributed Training Library\n\n * - .. glossary::\n NxD Inference\n - NeuronX Distributed Inference Library\n\n * - .. glossary::\n NC\n - Neuron Core\n\n * - .. glossary::\n NeuronCore\n - Neuron Core\n \n * - .. glossary::\n ND\n - Neuron Device\n\n * - .. glossary::\n NeuronDevice\n - Neuron Device\n\n * - .. glossary::\n TensorE\n - Tensor Engine\n\n * - .. glossary::\n ScalarE\n - Scalar Engine\n\n * - .. glossary::\n VectorE\n - Vector Engine\n\n * - .. glossary::\n GpSimdE\n - GpSimd Engine\n\n * - .. glossary::\n CCE\n - Collective Communication Engine\n\n * - .. glossary::\n HBM\n - High Bandwidth Memory\n\n * - .. glossary::\n SBUF\n - State Buffer \n\n * - .. glossary::\n PSUM\n - Partial Sum Buffer\n\n * - .. glossary::\n FP32\n - Float32\n\n * - .. glossary::\n TF32\n - TensorFloat32\n\n * - .. glossary::\n FP16\n - Float16\n\n * - .. glossary::\n BF16\n - Bfloat16\n\n * - .. glossary::\n cFP8\n - Configurable Float8\n\n * - .. glossary::\n RNE\n - Round Nearest Even\n\n * - .. glossary::\n SR\n - Stochastic Rounding\n\n * - .. glossary::\n NKI\n - Neuron Kernel Interface\n\n * - .. glossary::\n CustomOps\n - Custom Operators\n\n * - .. glossary::\n RT\n - Neuron Runtime\n\n * - .. glossary::\n DP\n - Data Parallel\n\n * - .. glossary::\n DPr\n - Data Parallel degree\n\n * - .. glossary::\n TP\n - Tensor Parallel\n\n * - .. glossary::\n TPr\n - Tensor Parallel degree\n\n * - .. glossary::\n PP\n - Pipeline Parallel\n\n * - .. glossary::\n PPr\n - Pipeline Parallel degree\n" }, { "path": "about-neuron/arch/index.rst", "content": ".. _neuron-architecture-index:\n\n.. meta::\n :description: Explore the hardware architecture of AWS Neuron instances, including EC2 Trn and Inf instance types, AWS Inferentia and Trainium chips, and NeuronCore processing units. Learn about system specifications, memory hierarchies, interconnect topologies, and architectural considerations for machine learning workloads.\n :date-modified: 2025-10-03\n\nAWS Neuron architecture guides\n==============================\n\nReview and understand the hardware architecture of AWS Neuron instances, including AWS Elastic Compute Cloud (EC2) ``Trn`` and ``Inf`` instance types, AWS Inferentia and Trainium chips, and NeuronCore processing units. The documentation covers system specifications, memory hierarchies, interconnect topologies, and architectural considerations for machine learning workloads.\n\nAbout Neuron Hardware\n----------------------\n\nAWS Neuron hardware consists of custom-designed machine learning accelerators optimized for deep learning workloads. This section covers the architecture and capabilities of AWS Inferentia and Trainium chips, their NeuronCore processing units, and the EC2 instances that host them.\n\nTrainium Architecture\n----------------------\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: AWS Trainium3\n :link: neuron-hardware/trainium3\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Third-generation training accelerator chip\n\n .. grid-item-card:: AWS Trainium2\n :link: neuron-hardware/trainium2\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Second-generation training accelerator chip\n\n .. grid-item-card:: AWS Trainium\n :link: neuron-hardware/trainium\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n First-generation training accelerator chip\n\nInferentia Architecture\n------------------------\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: AWS Inferentia2\n :link: neuron-hardware/inferentia2\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Second-generation inference accelerator chip\n\n .. grid-item-card:: AWS Inferentia\n :link: neuron-hardware/inferentia\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n First-generation inference accelerator chip\n\nNeuronCore Architecture\n------------------------\n\nNeuronCores are fully-independent heterogenous compute-units that power Tranium, Tranium2, Inferentia, and Inferentia2 chips.\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: NeuronCore v4\n :link: neuron-hardware/neuron-core-v4\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Processing unit architecture for Trainium3\n\n .. grid-item-card:: NeuronCore v3\n :link: neuron-hardware/neuron-core-v3\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Processing unit architecture for Trainium2\n\n .. grid-item-card:: NeuronCore v2\n :link: neuron-hardware/neuron-core-v2\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Processing unit architecture for Inferentia2 and Trainium\n\n\n .. grid-item-card:: NeuronCore v1\n :link: neuron-hardware/neuron-core-v1\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Processing unit architecture for Inferentia\n\nNeuron AWS EC2 Platform Architecture\n-------------------------------------\n\nOverviews of the AWS Inf and Trn instance and UltraServer architectures.\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: Inf1 Architecture\n :link: neuron-hardware/inf1-arch\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Inf1 instance architecture and specifications\n\n .. grid-item-card:: Inf2 Architecture\n :link: neuron-hardware/inf2-arch\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Inf2 instance architecture and specifications\n\n .. grid-item-card:: Trn1 Architecture\n :link: neuron-hardware/trn1-arch\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Trn1 instance architecture and specifications\n\n .. grid-item-card:: Trn2 Architecture\n :link: neuron-hardware/trn2-arch\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Trn2 instance architecture and specifications\n\n .. grid-item-card:: Trn3 Architecture\n :link: neuron-hardware/trn3-arch\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Trn3 instance architecture and specifications\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n AWS Inferentia \n AWS Inferentia2 \n AWS Trainium \n AWS Trainium2 \n AWS Trainium3 \n NeuronCore v1 \n NeuronCore v2 \n NeuronCore v3 \n NeuronCore v4 \n Inf1 Architecture \n Inf2 Architecture \n Trn1 Architecture \n Trn2 Architecture \n Trn3 Architecture \n" }, { "path": "about-neuron/arch/neuron-features/custom-c++-operators.rst", "content": ".. _feature-custom-c++-operators:\n\nNeuron Custom C++ Operators\n===========================\n\n.. include:: /neuron-customops/customops-intro.txt\n\n\nFor more details see :ref:`neuron_c++customops`" }, { "path": "about-neuron/arch/neuron-features/data-types.rst", "content": ".. _neuron-data-types:\n\nData Types\n==========\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nIntroduction\n------------\n\nInferentia and Trainium NeuronDevices include different NeuronCore versions, which support different data-types. This section describes what data-types are supported in each NeuronCore version.\n\nNeuronCore v1 Data Types\n------------------------\n\nNeuron Data-Types\n^^^^^^^^^^^^^^^^^\n\nNeuron enables developers to choose from multiple data-types. The\nsupported data-types are FP32, FP16, and BF16. Developers can\ntrain their models on their platform of choice (e.g. EC2 P3 instances),\nand then easily move their trained models to EC2 Inf1 for execution.\n\n.. raw:: html\n\n \n \n \t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \n \n \t\t\n \t\n
Data TypeSRangePrecision
FP3218 bits23 bits
BF1618 bits\n \t\t\t\n \t\t\t7 bits
FP161\n 5 bits\n \t\t\t10 bits
\n

\n\nFP16/BF16 models\n~~~~~~~~~~~~~~~~\n\nModels natively trained in FP16/BF16 will be executed in their trained\ndata-types. This is a straightforward migration from the training\nplatform to Inf1.\n\nFP32 models\n~~~~~~~~~~~\n\nNeuron SDK supports **automatic model conversion** from FP32 to BF16 by\ndefault. This capability allows developers to train their models using\nFP32 format for the highest accuracy, and achieve performance benefits\nwithout having to worry about low-precision training (e.g. no need for\nloss-scaling during training). ML models are typically robust to FP32 to\nBF16 conversion, with minimal to no impact on accuracy. The conversion\naccuracy is model dependent; therefore, users are encouraged to\nbenchmark the accuracy of the auto-converted model against the original\nFP32 trained model.\n\nWhen the compiler is supplied with an unmodified FP32 model input it\nwill automatically compile the model to run as BF16 on Inferentia. During\ninference the FP32 input data will be auto-converted internally by\nInferentia to BF16 and the output will be converted back to FP32\ndata-type. For explicit FP16 inferencing, either use an FP16 trained\nmodel, or use an external tool (like AMP) to make the explicit\nconversions.\n\n.. _neuron-data-types-v2:\n\nNeuronCore v2 Data Types\n------------------------\n\nThe NeuronCore v2 supports the following data types:\n\n* 32 and 16-bit Floating Point (FP32 / FP16)\n* TensorFloat-32 (TF32)\n* Brain Floating Point (BFloat16)\n* 8-bit Floating point with configurable range and precision (cFP8)\n* Unsigned 8-bit integer (UINT8)\n\nThe layout for these is as follows:\n\n.. raw:: html\n\n \n \n \t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \t\t\t\n \n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\t\n \n \t\t\t\n \n \n \t\t\t\n \t\t\t\n \n \n \t\t\n \n \t\t\t\n \t\t\t\n \n \n \t\t\n \n \t\t\t\n \t\t\t\n \n \n \t\t\n \n \t\t\t\n \t\t\t\n \n
Data TypeSRangePrecision
FP3218 bits23 bits
TF3218 bits\n \t\t\t10 bits
BF1618 bits\n \t\t\t\n \t\t\t7 bits
FP161\n \t\t\t5 bits\n \t\t\t10 bits
FP8_e5m21\n \t\t\t5 bits\n \n \t\t\t2 bits
FP8_e4m31\n \n \t\t\t4 bits\n \t\t\t3 bits
FP8_e3m41\n \n \t\t\t3 bits\n \t\t\t4 bits
UINT8\n \t\t\t8 bits\n \t\t
\n

\n\n\n\nModel Type Conversion\n^^^^^^^^^^^^^^^^^^^^^\n\nThe Neuron SDK supports automatic model conversion from FP32 to BF16 by default. This capability allows developers to train their models using FP32 format for the highest accuracy, and then achieve run-time performance benefits without having to worry about low-precision training (e.g. no need for loss-scaling during training). ML models are typically robust to FP32 to BF16 conversion, with minimal to no impact on accuracy. Since conversion accuracy is model dependent, users are encouraged to benchmark the accuracy of the auto-converted model against the original FP32 trained model.\n\nSee :ref:`Mixed Precision and Performance-accuracy Tuning for Training` for more details on supported data types and their properties.\n\nThe Neuron compiler offers the ``--auto-cast`` and ``--auto-cast-type`` options to specify automatic casting of FP32 tensors to other data types to address performance and accuracy tradeoffs. See the :ref:`Neuron Compiler CLI Reference Guide` for a description of these options.\n\n\nNeuronCore v2 Rounding Modes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nBecause floating point values are represented by a finite number of bits, they cannot represent all real numbers accurately. Floating point calculations that exceed their defined data type size are rounded. The NeuronCore v2 performs a Round-to-Nearest (RNE) algorithm with ties to Even by default. It also provides a new Stochastic Rounding mode. When Stochastic Rounding is enabled, the hardware will round the floating point value up or down using a proportional probability. This could lead to improved model convergence. Use the environment variable NEURON_RT_STOCHASTIC_ROUNDING_EN to select a rounding mode.\n" }, { "path": "about-neuron/arch/neuron-features/index.rst", "content": ".. _neuron-features-index:\n\nNeuron Features\n===============\nNeuron features provide insights into Neuron capabilities that enable high-performance and improve usability of developing and deploying deep learning acceleration on top of Inferentia and Trainium based instances.\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: Custom C++ operators\n :link: custom-c++-operators\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Framework for implementing custom operators in C++ to extend Neuron's built-in operation support.\n\n .. grid-item-card:: Data types\n :link: data-types\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Supported numerical data types including FP32, FP16, BF16, and INT8 for efficient model execution.\n\n .. grid-item-card:: Logical NeuronCore configuration\n :link: logical-neuroncore-config\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Configuration options for grouping and managing NeuronCores as logical units for workload distribution.\n\n .. grid-item-card:: Neuron persistent cache\n :link: neuron-caching\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Persistent caching system for compiled models to reduce compilation time across sessions.\n\n .. grid-item-card:: NeuronCore batching\n :link: neuroncore-batching\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Batching strategies to maximize throughput by processing multiple inputs simultaneously on NeuronCores.\n\n .. grid-item-card:: NeuronCore pipeline\n :link: neuroncore-pipeline\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Pipeline execution model that overlaps computation and data movement for improved performance.\n\n .. grid-item-card:: Rounding modes\n :link: rounding-modes\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Configurable numerical rounding modes for controlling precision and accuracy in computations. \n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Custom C++ operators \n Data types \n Logical NeuronCore configuration \n Neuron persistent cache \n NeuronCore batching \n NeuronCore pipeline \n Rounding modes \n\n\n\n \n" }, { "path": "about-neuron/arch/neuron-features/logical-neuroncore-config.rst", "content": ".. _logical-neuroncore-config:\n\n################################\nLogical NeuronCore configuration\n################################\n\nLogical NeuronCore configuration (LNC) is a set of compiler and runtime settings for instances powered by AWS Trainium2 that \ndetermines the number of NeuronCores exposed to your machine learning (ML) applications. LNC configuration works by combining \nthe compute and memory resources of multiple physical NeuronCores into a single logical NeuronCore. You can configure these settings \nto reduce the number of worker process needed for training and deployment of large-scale models. \n\n.. important::\n\n LNC can only be set to **1** or **2**. These are the only supported values. On Trn2, each chip has 8 physical NeuronCores. With LNC=2 (default), these are grouped into 4 Logical NeuronCores. With LNC=1, all 8 physical cores are treated as individual logical NeuronCores. LNC applies only to Trn2 and Trn3 instances.\n\n\n\n.. contents:: Concepts\n :depth: 1\n :local:\n :backlinks: none\n\n===================\nLogical NeuronCores\n===================\n\nA logical NeuronCore is a grouping of physical NeuronCores that the Neuron Compiler, Neuron Runtime, Neuron Tools, and Frameworks \nhandle as a single unified NeuronCore. Every Trainium2 device contains eight physical NeuronCore-v3. \n\n=============================\nCompiler and runtime settings\n=============================\n \nLNC configuration is controlled with the following runtime and compiler settings:\n\n| **Neuron Runtime**\n| The ``NEURON_LOGICAL_NC_CONFIG`` runtime environment variable controls how many physical NeuronCores are grouped to make up a logical NeuronCore.\n\n\n| **Neuron compiler flags** \n| The ``--logical-nc-config`` or ``-lnc`` command-line options control the degree of model sharding the compiler performs on an input graph. You must compile your Models to use the LNC configuration set by the Neuron Runtime environment variable. AWS Neuron currently doesn't support setting the compiler flag to a different LNC configuration than the Neuron Runtime environment variable. \n\n=================================\nLogical NeuronCore configurations\n=================================\n\nAWS Neuron supports the following Logical NeuronCore configurations:\n\n.. tab-set::\n\n .. tab-item:: LNC = 2\n\n A Logical NeuronCore configuration (LNC) of two is the default setting on Trainium2 devices. It combines two physical \n NeuronCore-v3 into a logical NeuronCore with the software id ``NC_V3d``. When you set Logical NeuronCore configuration to \n two, it directs Trainium2 devices to expose four ``NC_v3d`` to your machine learning applications. On this setting, \n a ``Trn2.48xlarge`` instance presents 64 available NeuronCores. The folowing high-level diagram shows a ``Trn2.48xlarge`` \n instance, connected in a 2D torus topology, with the Logical NeuronCore configuration set to two.\n\n .. image:: /images/architecture/Trn2/trn2_lnc2.png\n :align: center\n :width: 750\n |\n\n Trainium2 devices contain four 24GB HBM banks. Each bank is shared by two physical NeuronCore-v3. \n When LNC=2, the two physical NeuronCores share a single address space. Workers on each of the \n two physical NeuronCores can access tensors and perform local collective operations without \n accessing the network. The following diagram shows how a logical NeuronCore is presented to the \n software under this configuration.\n\n .. image:: /images/architecture/NeuronCore/lnc_2.png\n :align: center\n :width: 450\n |\n\n To set the Logical NeuronCore configuration to two, use the following runtime and compiler flag combination:\n\n | **Runtime environment variable:**\n | ``NEURON_LOGICAL_NC_CONFIG`` = 2\n\n | **Compiler flag:**\n | ``-lnc`` = 2 \n |\n\n .. tab-item:: LNC = 1\n\n When you set the Logical NeuronCore configuration to one, it assigns each physical NeuronCore-v3 to a single logical \n NeuronCore with the software id ``NC_V3``. This directs Trainium2 devices to expose eight ``NC_v3`` to your machine learning \n applications. On this setting, a ``Trn2.48xlarge`` instance presents 128 available NeuronCores. \n The following high-level diagram shows a ``Trn2.48xlarge`` instance, connected in a 2D torus topology, \n with the Logical NeuronCore configuration set to one.\n\n .. image:: /images/architecture/Trn2/trn2_lnc1.png\n :align: center\n :width: 750\n |\n\n Trainium2 devices contain four 24GB HBM banks. Each bank is shared by two physical NeuronCore-v3. \n When the Logical NeuronCore configuration is set to one, both physical NeuronCores have access to the entire 24GB HBM bank. The following \n diagram shows how logical NeuronCores are presented to the software under this configuration.\n\n .. image:: /images/architecture/NeuronCore/lnc_1.png\n :align: center\n :width: 475\n |\n\n To set the Logical NeuronCore configuration to one, use the following runtime and compiler flag combination:\n\n | **Runtime environment variable:**\n | ``NEURON_LOGICAL_NC_CONFIG`` = 1\n\n | **Compiler flag:**\n | ``-lnc`` = 1\n |\n\n \n \n\n\n " }, { "path": "about-neuron/arch/neuron-features/neuron-caching.rst", "content": ".. _neuron-caching:\n\nNeuron Persistent Cache\n=======================\n\nPyTorch Neuron (``torch-neuronx``) uses ``torch-xla``, and ``torch-xla`` operates in lazy mode. In other words, every operation in training script\nis recorded in a graph. The graph is executed only when the results are requested by \nthe user when they use ``print`` or ``xm.mark_step``. Requesting results tells \n``torch-xla`` that the recorded graph needs to be executed. \n\nBefore executing the graph on a Neuron device, ``torch-xla`` would call Neuron Compiler (``neuronx-cc``) to compile the graph into Neuron specific \ngraph. Then the graph is executed on the NeuronCores. Compiling the graph involves \nrunning optimizations that can make use of the NeuronCores efficiently. Running these \noptimizations can be expensive and can result in long compile times. To save the \nusers from compiling these graphs at every iteration, ``torch-xla`` maintains an \nin-memory cache called Just in Time (JIT) cache. When the user re-runs the same graph (eg. 2nd \niteration of the training run), torch-xla would check in this JIT cache and re-use \nthe cached compilation result, thereby avoiding the wait times.\n\nSince the JIT cache is an in-memory cache, it needs to be constructed every time the training script is \nrun. Hence, if the user re-runs the training script, a new JIT cache is created. This causes a compilation for the first training graph.\nTo avoid such compilations across training runs, PyTorch Neuron (``torch-neuronx``) has built an on-disk \n``Neuron Persistent Cache``. Since this cache is on-disk, its persistent across training runs. So \nnow, when a graph is compiled for the fist time, the compilation result is saved in \n``Neuron Persistent Cache``. When the user re-runs the training script, since the JIT cache is not \nready, it would send the graph for compilation. PyTorch Neuron (``torch-neuronx``) would then check if \nthe compiled result is present in the ``Neuron Persistent Cache``, if yes, it would return with the \ncompiled result. This on-disk cache thereby avoids compilations across training runs. \nThis cache is enabled by default for Neuron's PyTorch/XLA flow (training) as well as\ntransformers-neuronx LLM inference package.\nThe default cache path is the directory ``/var/tmp/neuron-compile-cache``.\n\nLook at the diagram below on the end to end flow:\n\n|Image:|\n\nAs seen from the diagram, the operations are recorded in a graph in lazy mode and only \nwhen a mark_step is hit, the graph is executed. Before execution, the graph passes through\ntwo caches to check if we have compiled the graph sometime in the past. If yes, we reuse \nthe compilation result and execute with it. This avoid duplicate compilations.\nOne thing to note, both JIT cache and Neuron Cache are complementary to each other.\nJIT cache prevents duplicate compilation within a run and Neuron Cache prevents duplicate \ncompilations across training runs. For example, within a training script, we have a training \nloop that iterates through the dataset. The first iteration would trace a unique graph \nand the following iteration would trace a graph that is similar to the first one. In this case,\nthe subsequent iterations would hit the JIT cache and reuse the result. However, to save \nusers from compiling for the first iteration graph, ``Neuron Persistent Cache`` would be used. In this case,\nthe very first time when the script is run, the ``Neuron Persistent Cache`` would be updated. Going forward \nwhen we re-run the training script, compilation results from ``Neuron Persistent Cache`` would be used.\n\nTo better understand how ``Neuron Persistent Cache`` works, consider the example below:\n\n.. code:: python\n\n import torch\n import torch_xla\n import torch_xla.core.xla_model as xm\n device = xm.xla_device()\n t1 = torch.randn(3, 3).to(device)\n t2 = t1 / 0.5\n x = t2.cpu()\n\nRunning the above example produces the following logs:\n\n.. code:: bash\n\n 2023-08-25 21:51:36.000433: INFO ||NCC_WRAPPER||: Compile cache path: /var/tmp/neuron-compile-cache\n .\n Compiler status PASS\n\nRe-running the above script would fetch the graph from the \nneuron cache and you would see logs as follows:\n\n.. code:: bash\n\n 2023-08-25 21:52:23.000451: INFO ||NCC_WRAPPER||: Compile cache path: /var/tmp/neuron-compile-cache\n 2023-08-25 21:52:23.000453: INFO ||NCC_WRAPPER||: Using a cached neff at /var/tmp/neuron-compile-cache/neuronxcc-2.8.0.25+a3ad0f342/MODULE_198775565831884870+d41d8cd9/model.neff. Exiting with a successfully compiled graph.\n\nAs you can see, the next run picks the compiled graph from\ncache, thereby saving the compilation time.\nThe cache uses hash of the Neuron compiler flags and XLA graph as the\nkey. If the Neuron compiler version or XLA graph changes, you will see\nrecompilation. Examples of changes that would cause XLA graph change\ninclude:\n\n- Model type and size\n- Batch size\n- Optimizer and optimizer hyperparameters\n- Location of xm.mark_step()\n\nTo keep cache size small and to enable weights/parameters updates without recompilation, \nonly the compute graphs are cached when using transformers-neuronx (weights/parameters are inputs to the compute graphs) and \ntraining flow using torch-neuronx's XLA (weights/parameters are inputs and outputs of the compute graphs). \nNote that this caching mechanism doesn't apply to the torch-neuronx trace API where the weights/parameters are frozen and converted to constants, \nthen compiled together with the compute operations (traced graphs with frozen weights/parameters are not cached).\n\nAll compilation results are saved in the cache. To disable the cache, you \ncan pass ``--no_cache`` option via NEURON_CC_FLAGS:\n\n.. code:: python\n\n os.environ['NEURON_CC_FLAGS'] = os.environ.get('NEURON_CC_FLAGS', '') + ' --no_cache'\n\nThe default cache path is the directory ``/var/tmp/neuron-compile-cache``.\nTo change the cache's location, pass ``cache_dir=``\noption via ``NEURON_CC_FLAGS`` or ``NEURON_COMPILE_CACHE_URL=`` environment variables:\n\n.. code:: python\n\n os.environ['NEURON_CC_FLAGS'] = os.environ.get('NEURON_CC_FLAGS', '') + ' --cache_dir='\n\n.. code:: python\n\n os.environ['NEURON_COMPILE_CACHE_URL'] = ''\n\nThe cache URL specified using ``--cache_dir`` is prioritized over that specified using ``NEURON_COMPILE_CACHE_URL`` if both are set.\nIf ```` starts with ``s3://``, it will use the AWS S3 URL as the cache location, provided that the corresponding S3 bucket exists and is both readable and writeable.\n\nYou can change the verbose level of the compiler by adding ``log_level`` to either ``WARNING``, ``INFO``\nor ``ERROR``. This can be done as follows:\n\n.. code:: python\n\n os.environ['NEURON_CC_FLAGS'] = os.environ.get('NEURON_CC_FLAGS', '') + ' --log_level=INFO'\n\nA graph compilation can fail because of a compilation error or an environment issue (for example, compilation is interrupted by ctrl-C). The graph would be marked as failed and subsequent rerun would encounter message like below:\n\n.. code:: bash\n\n INFO ||NCC_WRAPPER||: Got a cached failed neff at /var/tmp/neuron-compile-cache/neuronxcc-2.8.0.25+a3ad0f342/MODULE_12486829708343293975+d41d8cd9/model.neff. Will skip compilation, please set --retry_failed_compilation for recompilation. \n\nTo retry compilation,\nadd ``--retry_failed_compilation`` in ``NEURON_CC_FLAGS`` environment variable. When the script is reran, all the previously failed compilations are recompiled and fresh results are saved in the cache.\n\n.. code:: python\n\n os.environ['NEURON_CC_FLAGS'] = os.environ.get('NEURON_CC_FLAGS', '') + ' --retry_failed_compilation'\n\nNote that all flags demonstrated above will be parsed by a tool called ``neuron_cc_wrapper``, which is a wrapper over Neuron Compiler CLI to provide caching mechanism. All these flags will not be passed into Neuron Compiler CLI. \n\n.. |Image:| image:: ./images/NeuronCaching.png\n" }, { "path": "about-neuron/arch/neuron-features/neuroncore-batching.rst", "content": ".. _neuron-batching:\n\nNeuron Batching\n===============\n\nBatching refers to the process of grouping multiple samples together,\nand processing them as a group (i.e. passing them together through the\nneural network). Batching is typically used as an optimization for\nimproving throughput at the expense of higher latency (and potentially\nhigher memory footprint). Batching considerations are slightly different\nbetween inference and training workloads, and we thus cover them\nseparately below.\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\nBatching in inference workloads\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhat is batched inference?\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe concept of batched inference is conceptually illustrated below, with\na single NeuronCore performing batched computation of a 3 layer neural\nnetwork with a batch-size of 4. The NeuronCore reads the parameters for\na certain layer from the external memory, and then performs the\ncorresponding computations for all 4 inference-requests, before reading\nthe next set of parameters (thus, performing more compute for every\nparameter read from memory). \n\n.. image:: /images/batched-inference.png\n\n\nWhat are the benefits of batched Inference?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor inference, batching is typically used as a trade-off knob between\nthroughput and latency: higher batch-size typically leads to better\nhardware utilization and thus higher throughput, but at the same time\nbatching requires to perform more computation until getting the first\nresults, and hence leads to higher latency. \n\n\n.. image:: /images/tradeoffs.png\n\nTo understand why batching tends to improve throughput (up to a certain max\nvalue), it is useful to consider an intuitive visual performance-model\ncalled ‘the roofline model’, which provides with a theoretical bound on\nthe system’s performance: \n\n\n.. image:: /images/memoryvscompute.png\n\nThe X-axis indicates the\narithmetic intensity (AI) of the workload, which is the ratio between\nthe number of operations and the number of bytes read-from/written-to\nmemory. The Y-axis indicates the theoretical extractable performance.\nFor small(large) AI values, the workload is expected to be\nmemory(compute) bound. For inference workloads, AI is often approximated\nby dividing the model’s number of operations by its memory footprint\n(#params x dtype_size). To a first order approximate, the AI value is\nlinearly dependent on the batch-size, which means that the workloads\nperformance (throughput) is expected to increase with the batch-size. To\nunderstand this more intuitively, for a larger batch size, Neuron can\nbetter amortize the cost of reading parameters from the external memory,\nand thus improve the overall hardware efficiency. It should be noted\nthat while the roofline model can be very useful, it is not perfectly\naccurate (e.g. it doesn’t take into account spill/fills from/to on-chip\nSRAM memories), and thus users are encouraged to use it as a tool for\n**estimating** the optimal batch-size for their workloads.\n\nHow to determine the optimal batch-size for inference workloads?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe optimal batch size is dependent on the application-level\nrequirements: some applications require strict latency guarantees (in\nwhich case, check out the :ref:`neuroncore-pipeline`\ntechnology), while other applications strictly aim to maximize\nthroughput. We thus encourage our users to try out multiple batch-sizes,\nand compare performance between them. A good starting for batch-size\nexploration can be identified using the roofline model: we can choose a\nbatch-size that achieves an Arithmetic Intensity which is at the edge of\nthe compute bound region. By doing that, we aim to achieve max\nthroughput with a minimal batch-size, and thus minimal impact to\nlatency. \n\n.. image:: /images/memoryvscompute2.png\n\n\nThis can be expressed via the following\nequation:\n``batch-size(Inference) = ceiling[0.5 x (/) /``\n``(/(<#model-dense-params> x ))]`` (for\nNeuronDevice PeakFLOPS and MemBW, see the :ref:`trainium-arch`, :ref:`inferentia-arch` and :ref:`inferentia2-arch` pages.\n\nFor example, a BF16 BERT-Large model, with a sequence length of 128,\nwill have the following approximated batch sizes:\n\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1 \n :align: left\n \n\n * - Model\n - NeuronDevice\n - Peak TFLOPS (BF16)\n - MemBW (GB/sec)\n - Model GFLOPs\n - Model Dense Params (Millions)\n - Data-type size (BF16)\n - Approximated optimal batch-size\n\n * - BERT-Large (SeqLen=128)\n - Inferentia\n - 64\n - 50\n - 77.3\n - 302\n - 2\n - 6\n\n * - BERT-Large (SeqLen=128)\n - Trainium\n - 210\n - 820\n - 77.3\n - 302\n - 2\n - 2\n\n * - ResNet-50\n - Inferentia\n - 64\n - 50\n - 7.8\n - 25\n - 2\n - 5\n\n * - ResNet-50\n - Trainium\n - 210\n - 820\n - 7.8\n - 25\n - 2\n - 1\n\nWe recommend to evaluate multiple batch sizes and compare the\nperformance between them, in order to determine the optimal\nlatency/throughput deployment-point.\n\nHow to set the batch-size?\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe Neuron compiler takes a model and its sample input, as inputs for\nthe compilation process. For example, the code snippet below will\ncompile a model with a batch-size of 4:\n\n.. code::\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input of batch size 4\n image = torch.rand([4, 3, 224, 224])\n\n model_neuron = torch.neuron.trace(model, image, dynamic_batch_size=True)\n\n # Execute with a batch of 12 images\n batch = torch.rand([12, 3, 224, 224])\n results = model_neuron(batch)\n\nFor ahead-of-time compiled inference graphs (i.e. Inf1), dynamic\nbatching can be used (as shown in the above code snippet) to process a\nlarger client-side inference batch-size, and allow the framework to\nautomatically break up the user-batch (12 in our case) into smaller\nbatch sizes, to match the compiled batch-size (4 in our case). This\ntechnique increases the achievable throughput by hiding the\nframework-to-neuron overhead, and amortizing it over a larger batch\nsize.\n\n.. seealso::\n\n - :ref:`torch-neuronx-dynamic-batching` in ``torch-neuronx``\n - :ref:`tensorflow-neuronx-special-flags` in ``tensorflow-neuronx``.\n\n\nBatching in training workloads\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUnlike inference workloads, training is inherently an offline process,\nand thus doesn’t have latency requirements. This means that training is\nalmost always batched to some degree.\n\nBatch-size naming\n^^^^^^^^^^^^^^^^^\n\nFor distributed processing, defining the batch size depends on the observation\nlevel. There are multiple terms you should be aware of when running\na distributed training job, especially global batch size (GBS) and micro-batch.\nKnowing the batch size in advance is crucial for precompiling the computational\ngraph and for setting the hyperparameters.\n\n micro-batch size\n Smallest unit of the number of samples getting processed in a single step\n in the accelerator. For very large models, it is frequently chosen to be 1.\n\n gradient accumulation\n Process of iterating over a micro-batch multiple times and summing up the gradients \n before an optimizer update. This can happen in a dedicated loop for gradient \n accumulation or as part of multiple iterations of samples in pipeline parallelism.\n See :ref:`pp_developer_guide` for more details on pipeline parallelism.\n\n data-parallel size (or DP degree)\n Number of model replicas that process different portions of data in parallel.\n Each replica maintains a complete copy of the model while processing unique\n data chunks, after which their gradients are synchronized for the optimizer update. \n See :ref:`neuron_hw_glossary` for more details.\n\n global batch-size\n Number of total samples used for an update of the optimizer.\n This includes all the respective gradients that get added up from\n data-parallel processing or gradient accumulation.\n :literal:`global batch size = micro_batch_size * data_parallel_size * gradient_accumulation_steps`\n\n mini-batch or replica-batch size\n Number of samples that contribute to a gradient within one data-parallel rank.\n A mini-batch gradient is obtained by aggregating multiple\n micro-batch gradients within or without a pipeline (aka. gradient accumulation).\n :literal:`mini_batch_size = micro_batch_size * gradient_accumulation_steps`\n\n worker batch\n The portion of mini-batch samples processed by a worker.\n The idea behind a worker batch is that one worker (node) might have a subset of the dp-degrees \n and we care about how much data gets tackled by this worker.\n\nHow to determine the optimal batch-size for training workloads?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDetermining the optimal batch-size for training workloads can be a\nnon-trivial task. In most cases, we’d want to choose the largest\nbatch-size that we can get away with.\n\nThe most dominant factor for determining the optimal batch-size in\ntraining workloads is memory footprint: training workloads have higher\nmemory footprint compared to inference, as they require saving more\ntensors aside from the model parameters, such as gradients, intermediate\nactivations (passed between forward-pass and backward-pass), and\noptimizer-state. If the batch-size is increased beyond a certain point,\none can run out of device memory (indicated by an ‘Out of device memory’\nerror, typically abbreviated as OOM).\n\nTo estimate the memory footprint of a model, we look at the different\ncontributors:\n\n1. Weights and gradients:\n\n 1. typically 2B each, thus 4B per parameter\n\n2. Optimizer state:\n\n 1. typically 4B - 12B per parameter\n\n3. Intermediate activations:\n\n 1. sum of all tensor sizes for forward pass\n 2. for example, for a transformer neural network, this is roughly 16\n x x x x x = 100MB x\n\n\nFor training workloads, determining the optimal batch size can be a\nlittle more tricky, due to two reasons:\n\n1. *Higher memory footprint:* Training workloads have higher memory\n footprint compared to inference, as they require saving more tensors\n aside from the model parameters, such as gradients,\n intermediate-state and optimizer-state. If the batch-size is\n increased too much, one can run out of device memory (indicated by an\n ‘Out of memory’ error, typically abbreviated as OOM).\n2. *Arithmetic intensity estimation:* Arithmetic intensity is harder to\n estimate in training workloads, compared to inference workloads, as\n the majority of the external memory access are due to reads/writes of\n intermediate activation state (rather than parameters), which\n requires lower level familiarity with the model to estimate\n correctly.\n\nA good first order approximate for the optimal batch-size in a training\nworkload, is the largest one that can fit in the device’s memory (i.e.\nwon’t lead to OOM error).\n:literal:`batch-size(Training) = 0.6 x ( x x ``)`\n:literal:`/ ``(<#model-dense-params> x ``)`\n\nNote TP-rank stands for Tensor-Parallelism rank, i.e. how many\nNeuronCores participate in a single Tensor-Parallelism group. Similarly,\nPP-rank stands for Pipeline-Parallelism rank, i.e. how many NeuronCores\nparticipate in a single Pipeline-Parallelism group.\n\nFor example, for BERT-Large Ph1 training, with a model-state of 4B per\nparameter (2B weights, 2B parameters), and TP-rank = PP-rank = 1, the\napproximated optimal per-NeuronCore training batch-size would be:\n:literal:`batch-size(Training/Trainium) = 0.6 x (1 x 1 x 16e+9``) / ``(300e+6 x 4``) = 8`\n" }, { "path": "about-neuron/arch/neuron-features/neuroncore-pipeline.rst", "content": ".. _neuroncore-pipeline:\n\nNeuronCore Pipeline\n===================\n\nThe Neuron software feature referred to as a NeuronCore Pipeline refers\nto the process of sharding a compute-graph across multiple NeuronCores,\ncaching the model parameters in each core’s on-chip memory (cache), and\nthen streaming inference requests across the cores in a pipelined\nmanner. Based on the number of NeuronCores selected, the model might get\nseamlessly sharded across up-to 16 Inferentia devices (i.e. 64\nNeuronCores). This enables users to optimize for both throughput and\nlatency, as it enables the NeuronCores to process neural-networks with\nlocally cached data and avoid the cost of accessing external memory.\n|Image:|\n\nOne benefit to this approach is that NeuronCore Pipeline can typically\nhit maximal hardware efficiency without the need for batching (e.g.\nBERT, ResNet50).\n\nFor maximal performance, users should choose an instance-size that can\ncache the entire model by using sufficient NeuronCores. Inf1 instance\ntypes have different number of Inferentia devices, each of which has 4\nNeuronCores, as shown here\nhttps://aws.amazon.com/ec2/instance-types/inf1/\n\nTo enable the NeuronCore Pipeline optimization, the compiler should be\ninvoked with the following flags: ``--neuroncore-pipeline-cores N``. The\nnumber of NeuronCores is typically chosen to be the minimal number that\ncan fit the entire model, which is currently done through a\ntrial-and-error process (compiling to different number of cores and\nlooking for compilation success/failure message). This process will be\nautomated in the future. A simple formula to help define the number of\nNeuronCores that may be an appropriate choice is\n\n::\n\n neuroncore-pipeline-cores = 4 * round( number-of-weights-in-model/(2 * 10^7) ) \n\nThis allocates a set of NeuronCores based on the size of the given\nmodel's weights and normalizes to multiples of 4 so it uses full\nInferentias.\n\nThe code snippet below shows how to compile a model with NeuronCore\nPipeline for 16 NeuronCores (instance size inf1.6xlarge).\n\n::\n\n import numpy as np\n import tensorflow.neuron as tfn\n\n example_input = np.zeros([1,224,224,3], dtype='float16')\n tfn.saved_model.compile(\"rn50_fp16\",\n \"rn50_fp16_compiled/1\",\n model_feed_dict={'input_1:0' : example_input },\n compiler_args = ['--neuroncore-pipeline-cores', '16'])\n\n.. |Image:| image:: ./images/NeuronCorePipelining.png\n" }, { "path": "about-neuron/arch/neuron-features/rounding-modes.rst", "content": ".. _neuron-rounding-modes:\n\nNeuron Rounding Modes\n=====================\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 1\n\n\n.. _neuron-rounding-mode-rne:\n\nRound Nearest, ties to Even (RNE)\n---------------------------------\n\nWhen the exact result of a floating point operation cannot be exactly\nrepresented as a floating point value, it must be rounded. The IEEE\n754-2008 standard defines the default rounding mode to be ‘Round\nNearest, ties to Even’ (RNE for short). Under this scheme, numbers are\nrounded to the nearest representable value, and in case of a ‘tie’ (i.e.\nthe number is exactly between the two nearest representable values)\nnumbers will be rounded to the nearest even number.\n\nAll NeuronCore generations support the RNE rounding scheme, which is the\nmost commonly used rounding scheme for Machine Learning workloads. Below\nis an illustration of the RNE rounding scheme: \n\n.. image:: /images/rne1.png\n :width: 700\n\n.. image:: /images/rne2.png\n :width: 700\n\n.. image:: /images/rne3.png\n :width: 700\n\n.. _neuron-rounding-mode-sr:\n\n\nStochastic Rounding (SR)\n------------------------\n\nOne downside of the RNE rounding scheme (and other rounding schemes\ndescribed in the IEEE 754-2008 standard), is that when adding floating\npoint values of significantly different magnitudes, rounding can squash\nsmall values and prevent them from accumulating over time. \n\nTo improve this, starting from the second generation of the NeuronCore\n(NeuronCore-v2), customers can choose between the RNE rounding scheme\ndescribed above, and a second rounding scheme called ‘Stochastic\nRounding’ (SR for short). Stochastic rounding prevents the computation\nprecision-loss described above, by performing the rounding operations in\na probabilistic manner, according to the relative distance from the two\nnearest representable values, as illustrated below: \n\n.. image:: /images/sr.png\n :width: 700\n\n\nBy performing the rounding in a probabilistic manner, this scheme allows\nfor small increments to accumulate over time, even when added to numbers\nof significantly higher magnitude, which leads to more precise results\nwhen performing large floating point computations (as done for machine\nlearning).\n\n\nQuick Tests \n-----------\n\nAs an example, we examine the code-snippet below:\n\n::\n\n import torch\n import torch_xla\n import torch_xla.core.xla_model as xm\n device = xm.xla_device()\n \n a = torch.tensor(1024.0).half().to(device)\n \n for i in range(2048) :\n a = (a + 0.5)\n xm.mark_step()\n \n print(a)\n\n\nThis code shows that rounding can significantly impact the calculation’s precision over time.\nTo use standard RNE rounding, use the environment variable ``NEURON_RT_STOCHASTIC_ROUNDING_EN=0``.\nTo enable stochastic rounding, use the environment variable ``NEURON_RT_STOCHASTIC_ROUNDING_EN=1``.\n\nNOTE: Stochastic rounding mode is enabled by default in PyTorch-Neuron when XLA_USE_BF16=1.\n\nThe first test continues to show 1024 due to RNE rounding after each addition, and the second test shows result that is mostly in line with expectation.\n\n::\n\n $ NEURON_RT_STOCHASTIC_ROUNDING_EN=0 python3 rounding_mode_test.py\n \n tensor(1024., device='xla:1', dtype=torch.float16)\n \n $ NEURON_RT_STOCHASTIC_ROUNDING_EN=1 python3 rounding_mode_test.py\n \n tensor(2056., device='xla:1', dtype=torch.float16)\n\n" }, { "path": "about-neuron/arch/neuron-hardware/inf1-arch.rst", "content": ".. _aws-inf1-arch:\n\nAmazon EC2 Inf1 Architecture\n==============================\n\nOn this page, we provide an architectural overview of the Amazon EC2 Inf1\ninstance and the corresponding :ref:`Inferentia ` NeuronChips that power\nthem (:ref:`Inferentia ` chips from here on).\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n.. _inf1-arch:\n\nInf1 Architecture\n-----------------\n\nThe EC2 Inf1 instance is powered by 16 :ref:`Inferentia ` chips, allowing\ncustomers to choose between four instance sizes:\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1 \n :align: left\n \n\n * - Instance size\n - # of Inferentia chips\n - vCPUs\n - Host Memory (GiB)\n - FP16/BF16 TFLOPS\n - INT8 TOPS\n - Device Memory (GiB)\n - Device Memory bandwidth (GiB/sec)\n - NeuronLink-v1 chip-to-chip bandwidth (GiB/sec/chip)\n - EFA bandwidth (Gbps)\n\n * - Inf1.xlarge\n - 1\n - 4\n - 8\n - 64\n - 128\n - 8\n - 50\n - N/A\n - up-to 25\n\n\n * - Inf1.2xlarge\n - 1\n - 8\n - 16\n - 64\n - 128\n - 8\n - 50\n - N/A\n - up-to 25\n\n * - Inf1.6xlarge\n - 4\n - 24\n - 48\n - 256\n - 512\n - 32\n - 200\n - 32\n - 25\n\n * - Inf1.24xlarge\n - 16\n - 96\n - 192\n - 1024\n - 2048\n - 128\n - 800\n - 32\n - 100\n\n\n\nInf1 offers a direct chip-to-chip interconnect called NeuronLink-v1,\nwhich enables co-optimizing latency and throughput via the :ref:`Neuron Core Pipeline ` technology. \n\n.. image:: /images/inf1-server-arch.png\n\n" }, { "path": "about-neuron/arch/neuron-hardware/inf2-arch.rst", "content": ".. _aws-inf2-arch:\n\nAmazon EC2 Inf2 Architecture\n=============================\n\nOn this page we provide an architectural overview of the Amazon EC2 Inf2\ninstances and the corresponding Inferentia2 NeuronChips that power\nthem (Inferentia2 chips from here on).\n\nInf2 Architecture\n-----------------\n\nThe EC2 Inf2 instance is powered by up to 12 :ref:`Inferentia2 chips `, and allows\ncustomers to choose between four instance sizes:\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1 \n :align: left\n\n * - Instance size\n - # of Inferentia2 chips\n - vCPUs\n - Host Memory (GiB)\n - FP8/FP16/BF16/TF32 TFLOPS\n - FP32 TFLOPS\n - Device Memory (GiB)\n - Instance Memory Bandwidth (GiB/sec)\n - NeuronLink-v2 chip-to-chip (GiB/sec/chip)\n\n * - Inf2.xlarge\n - 1\n - 4\n - 16\n - 190\n - 47.5\n - 32\n - 820\n - N/A\n\n * - Inf2.8xlarge\n - 1\n - 32\n - 128\n - 190\n - 47.5\n - 32\n - 820\n - N/A\n\n * - Inf2.24xlarge\n - 6\n - 96\n - 384\n - 1140\n - 285\n - 192\n - 4920\n - 192\n\n * - Inf2.48xlarge\n - 12\n - 192\n - 768\n - 2280\n - 570\n - 384\n - 9840\n - 192\n\n\nInf2 offers a low-latency, high-bandwidth chip-to-chip interconnect\ncalled NeuronLink-v2, which enables high-performance collective communication operations (e.g., AllReduce and AllGather).\n\nThis allows sharding large models across Inferentia2 chips (e.g., via\nTensor Parallelism), thus optimizing latency and throughput. This\ncapability is especially useful when deploying Large Generative Models.\n\n.. image:: /images/inf2-topology.png\n\n\n" }, { "path": "about-neuron/arch/neuron-hardware/inferentia.rst", "content": ".. _inferentia-arch:\n\n\nInferentia Architecture\n-----------------------\n\nAt the heart of each Inf1 instance are sixteen Inferentia chips, each with four :ref:`NeuronCore-v1 `, as depicted\nbelow:\n\n.. image:: /images/inferentia-neurondevice.png\n\n\n\nEach Inferentia chip consists of:\n\n+---------------+-------------------------------------------+\n| Compute | Four | \n| | :ref:`NeuronCore-v1 `| \n| | cores, delivering 128 INT8 TOPS and 64 | \n| | FP16/BF16 TFLOPS | \n+---------------+-------------------------------------------+\n| Device Memory | 8GiB of device DRAM memory (for storing | \n| | parameters and intermediate state), with | \n| | 50 GiB/sec of bandwidth | \n+---------------+-------------------------------------------+\n| NeuronLink | Enables co-optimization of latency and | \n| | throughput via the :ref:`Neuron Core |\n| | Pipeline ` | \n| | technology | \n+---------------+-------------------------------------------+\n\n" }, { "path": "about-neuron/arch/neuron-hardware/inferentia2.rst", "content": ".. _inferentia2-arch:\n\nInferentia2 Architecture\n------------------------\n\nAt the heart of each Inf2 instance are up to twelve Inferentia2 chips (each with two :ref:`NeuronCore-v2 ` cores). Inferentia2 is the second\ngeneration AWS purpose-built Machine Learning inference accelerator. The Inferentia2 chip architecture is depicted below: \n\n.. image:: /images/inferentia2.png\n\n\nEach Inferentia2 chip consists of:\n\n+----------------------------------+----------------------------------+\n| Compute | Two :ref:`NeuronCore-v2 |\n| | ` |\n| | cores, delivering 380 INT8 TOPS, |\n| | 190 FP16/BF16/cFP8/TF32 TFLOPS, |\n| | and 47.5 FP32 TFLOPS. |\n+----------------------------------+----------------------------------+\n| Device Memory | 32GiB of high-bandwidth device | \n| | memor (HBM) (for storing model | \n| | state), with 820 GiB/sec of | \n| | bandwidth. |\n+----------------------------------+----------------------------------+\n| Data Movement | 1 TB/sec of DMA bandwidth, with |\n| | inline memory |\n| | compression/decompression. |\n+----------------------------------+----------------------------------+\n| NeuronLink | NeuronLink-v2 for | \n| | chip-to-chip interconnect | \n| | enables high-performance | \n| | collective compute for | \n| | co-optimization of latency and | \n| | throughput. |\n+----------------------------------+----------------------------------+\n| Programmability | Inferentia2 supports dynamic |\n| | shapes and control flow, via ISA |\n| | extensions of NeuronCore-v2 and |\n| | :ref:`custom-operators |\n| | ` |\n| | via the deeply embedded GPSIMD |\n| | engines. |\n+----------------------------------+----------------------------------+\n\nFor a more detailed description of all the hardware engines, see :ref:`NeuronCore-v2 `." }, { "path": "about-neuron/arch/neuron-hardware/neuron-core-v1.rst", "content": ".. _neuroncores-v1-arch:\n\n\nNeuronCore-v1 Architecture\n--------------------------\n\nNeuronCore-v1 is the first generation NeuronCore engine, powering\nthe Inferentia chips. Each NeuronCore-v1 is a fully-independent\nheterogenous compute-unit, with three main engines (Tensor/Vector/Scalar\nEngines), and on-chip software-managed SRAM memory, for\nmaximizing data locality (compiler managed, for maximum data locality\nand optimized data prefetch).\n\n.. image:: /images/nc-v1.png\n\n\nThe ScalarEngine is optimized for scalar computations, in which every\nelement of the output is dependent on one element of the input, e.g.,\nnon-linearities such as GELU, SIGMOID, or EXP. The ScalarEngine is highly\nparallelized, and can process 512 floating point operations per cycle.\nIt can handle various data types, including FP16, BF16, FP32, INT8,\nINT16, and INT32. \n\nThe VectorEngine is optimized for vector computations,\nin which every element of the output is dependent on multiple input\nelements. Examples include ‘axpy’ operations (Z=aX+Y), Layer\nNormalization, Pooling operations, and many more. The VectorEngine is\nalso highly parallelized, and can perform 256 floating point operations\nper cycle. It can handle various data-types, including FP16, BF16, FP32,\nINT8, INT16, and INT32.\n\nThe TensorEngine is based on a power-optimized systolic array, which is\nhighly optimized for tensor computations (e.g., GEMM, CONV, Reshape,\nTranspose), and supports mixed-precision computations (FP16/BF16/INT8\ninputs, FP32/INT32 outputs). Each NeuronCore-v1 TensorEngine delivers 16\nTFLOPS of FP16/BF16 tensor computations." }, { "path": "about-neuron/arch/neuron-hardware/neuron-core-v2.rst", "content": ".. _neuroncores-v2-arch:\n\nNeuronCore-v2 Architecture\n--------------------------\n\nNeuronCore-v2 is the second generation of the NeuronCore engine,\npowering the Trainium chips. Each NeuronCore-v2 is a\nfully-independent heterogenous compute-unit, with 4 main engines\n(Tensor/Vector/Scalar/GPSIMD Engines), and on-chip\nsoftware-managed SRAM memory, for maximizing data locality (compiler\nmanaged, for maximum data locality and optimized data prefetch).\n\n\n.. image:: /images/nc-v2.png\n\nJust like in NeuronCore-v1, The ScalarEngine is optimized for\nscalar-computations, in which every element of the output is dependent\non one element of the input. The ScalarEngine is highly parallelized,\nand delivers 2.9 TFLOPS of FP32 computations (3x speedup\nrelative to NeuronCore-v1). The NeuronCore-v2 ScalarEngine can handle\nvarious data types, including cFP8, FP16, BF16, TF32, FP32, INT8, INT16,\nand INT32. \n\nThe VectorEngine is optimized for vector computations, in\nwhich every element of the output is dependent on multiple input\nelements. Examples include ‘axpy’ operations (Z=aX+Y), Layer\nNormalization, Pooling operations, and many more. The VectorEngine is\nalso highly parallelized, and delivers 2.3 TFLOPS of FP32 computations \n(10x speedup vs. NeuronCore-v1). The NeuronCore-v2\nVectorEngine can handle various data-types, including cFP8, FP16, BF16,\nTF32, FP32, INT8, INT16 and INT32.\n\nThe TensorEngine is based on a power-optimized systolic-array, which is\nhighly optimized for tensor computations (e.g., GEMM, CONV, \nTranspose), and supports mixed-precision computations (cFP8 / FP16 /\nBF16 / TF32 / FP32 / INT8 inputs, FP32 / INT32 outputs). Each\nNeuronCore-v2 TensorEngine delivers over 90 TFLOPS of FP16/BF16 tensor\ncomputations (6x speedup from NeuronCore-v1). \n\nNeuronCore-v2 also introduces a new engine called the\nGPSIMD-Engine, which consists of eight fully-programmable 512-bit wide \nvector processors, which can execute general purpose C-code and access the \nembedded on-chip SRAM memory. With these cores, customers can implement \ncustom operators and execute them directly on the NeuronCores.\n\nNeuronCore-v2 also adds support for control flow, dynamic shapes, and\nprogrammable :ref:`rounding mode ` (RNE & Stochastic-rounding).\n" }, { "path": "about-neuron/arch/neuron-hardware/neuron-core-v3.rst", "content": ".. _neuroncores-v3-arch:\n\nNeuronCore-v3 Architecture\n--------------------------\n\nNeuronCore-v3 is the third-generation NeuronCore that powers Trainium2 chips. It is a fully-independent heterogenous compute \nunit consisting of 4 main engines: Tensor, Vector, Scalar, and GPSIMD, with on-chip software-managed SRAM memory to maximize data \nlocality and optimize data prefetch. The following diagram shows a high-level overview of the NeuronCore-V3 architecture.\n\n.. image:: /images/architecture/NeuronCore/nc-v3.png\n :align: center\n :width: 250\n|\nNeuronCore-v3 is made up of the following components:\n\nOn-chip SRAM \n\"\"\"\"\"\"\"\"\"\"\"\"\nEach NeuronCore-v3 has a total of 28MB of on-chip SRAM. NeuronCore-v3 on-chip SRAM is software-managed to maximize data locality \nand optimize data prefetch. \n\nTensor Engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nTensor engines are based on a power-optimized systolic array. They are highly optimized for tensor computations such as GEMM, CONV, and \nTranspose. Tensor Engines support mixed-precision computations, including cFP8, FP16, BF16, TF32, and FP32 inputs and outputs. \nA NeuronCore-v3 Tensor Engine delivers 158 cFP8 TFLOPS, and 79 BF16/FP16/TF32 TFLOPS of tensor computations. \n\nLike NeuronCore-v2, NeuronCore-v3 supports control flow, dynamic shapes, and programmable rounding mode (RNE & Stochastic-rounding). \nNeuronCore-v3 also supports adjustable exponent biasing for the cFP8 data type.\n \nThe NeuronCore-v3 Tensor Engine also supports Structured Sparsity, delivering up to 316 TFLOPS of cFP8/FP16/BF16/TF32 \ncompute. This is useful when one of the input tensors to matrix multiplication exhibits a M:N sparsity pattern, where only M elements \nout of every N contiguous elements are non-zero. NeuronCore-v3 supports several sparsity patterns, including 4:16, 4:12, 4:8, 2:8, \n2:4, 1:4, and 1:2. \n\nVector Engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nOptimized for vector computations, in which every element of the output is dependent on multiple input elements. Examples include \naxpi operations (Z=aX+Y), Layer Normalization, and Pooling operations. \n\nVector Engines are highly parallelized, and deliver a total of 1 TFLOPS of FP32 computations. NeuronCore-v3 Vector Engines can handle \nvarious data-types, including cFP8, FP16, BF16, TF32, FP32, INT8, INT16, and INT32. \n\nScalar engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nOptimized for scalar computations in which every element of the output is dependent on one element of the input. Scalar Engines are \nhighly parallelized, and deliver a total of 1.2 TFLOPS of FP32 computations. NeuronCore-v3 Scalar Engines support multiple data \ntypes, including cFP8, FP16, BF16, TF32, FP32, INT8, INT16, and INT32.\n\nGPSIMD engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nEach GPSIMD engine consists of eight fully-programmable 512-bit wide vector processors. They can execute general purpose C-code and \naccess the embedded on-chip SRAM, allowing you to implement custom operators and execute them directly on the NeuronCores.\n\n" }, { "path": "about-neuron/arch/neuron-hardware/neuron-core-v4.rst", "content": ".. meta::\n :description: \"NeuronCore-v4 architecture overview and components.\"\n :date-modified: 12/02/2025\n\n\n.. _neuroncores-v4-arch:\n\nNeuronCore-v4 Architecture\n===========================\n\nNeuronCore-v4 is the fourth-generation NeuronCore that powers Trainium3 chips. It is a fully-independent heterogenous compute unit consisting of 4 main engines: Tensor, Vector, Scalar, and GPSIMD, with on-chip software-managed SRAM memory to maximize data locality and optimize data prefetch. \n\nThe following diagram shows a high-level overview of the NeuronCore-v4 architecture.\n\n.. image:: /images/architecture/trn3/neuroncore-v4.png\n :align: center\n\nLike previous generations of NeuronCore, NeuronCore-v4 supports control flow, dynamic shapes, and programmable rounding mode (RNE & Stochastic-rounding). NeuronCore-v4 is made up of the following components:\n\nOn-chip SRAM\n-------------\n\nEach NeuronCore-v4 has a total of 32MiB of on-chip SRAM. The on-chip SRAM is software-managed to maximize data locality and optimize data prefetch. NeuronCore-v4 SRAM also introduces a new near-memory accumulation feature, which allows DMA engines to perform a read-add-write operation into existing SRAM data via a single transfer. \n\nTensor Engine\n--------------\n\nTensor engines are based on a power-optimized systolic array. They are highly optimized for tensor computations such as GEMM, CONV, and Transpose. Tensor Engines support mixed-precision computations, including MXFP8/MXFP4, FP16, BF16, TF32, and FP32 inputs. The output data type can either be FP32 or BF16. A NeuronCore-v4 Tensor Engine delivers 315 MXFP8/MXFP4 TFLOPS, where MXFP8/MXFP4 are OCP (Open Compute Project) compliant data type formats. MXFP4 data types are converted to MXFP8 before Tensor Engine computation logic, using any arbitrary programmer-defined mapping. Besides quantized data types, a NeuronCore-v4 Tensor Engine also delivers 79 BF16/FP16/TF32 and 20 FP32 TFLOPS of tensor computations. \n\nThe NeuronCore-v4 Tensor Engine also supports Structured Sparsity, delivering up to 315 TFLOPS of FP16/BF16/TF32 compute. This is useful when one of the input tensors to matrix multiplication exhibits a M:N sparsity pattern, where only M elements out of every N contiguous elements are non-zero. NeuronCore-v4 supports several sparsity patterns, including 4:16, 4:12, 4:8, 2:8, 2:4, 1:4, and 1:2.\n\n\nVector Engine\n----------------\n\nOptimized for vector computations, in which every element of the output is dependent on multiple input elements. Examples include axpi operations (Z=aX+Y), Layer Normalization, and Pooling operations.\n\nVector Engines are highly parallelized, and deliver a total of 1.2 TFLOPS of FP32 computations. NeuronCore-v3 Vector Engines can handle various data-types, including FP8, FP16, BF16, TF32, FP32, INT8, INT16, and INT32. \n\nIn addition, NeuronCore-v4 Vector Engine supports two new features:\n\n1. Data quantization into MXFP8 data type formats from BF16/FP16, which is particularly useful for online data quantization in between MLP (multi-layer perceptron) layers. \n2. Fast exponential functional evaluation, at 4x higher throughput than exponential on Scalar Engine, which is particularly useful in self attention acceleration.\n\n\n\nScalar Engine\n---------------\n\nOptimized for scalar computations in which every element of the output is dependent on one element of the input. Scalar Engines are highly parallelized, and deliver a total of 1.2 TFLOPS of FP32 computations. NeuronCore-v3 Scalar Engines support multiple data types, including FP8, FP16, BF16, TF32, FP32, INT8, INT16, and INT32.\n\nGPSIMD Engine\n---------------\n\nEach GPSIMD engine consists of eight fully-programmable 512-bit wide vector processors. They can execute general purpose C/C++ code and access the embedded on-chip SRAM, allowing you to implement custom operators and execute them directly on the NeuronCores.\n" }, { "path": "about-neuron/arch/neuron-hardware/trainium.rst", "content": ".. _trainium-arch:\n\n\nTrainium Architecture\n----------------------\n\nAt the heart of the Trn1 instance are 16 x Trainium chips (each Trainium include 2 x :ref:`NeuronCore-v2 `). Trainium is the second\ngeneration purpose-built Machine Learning accelerator from AWS. The\nTrainium chip architecture is depicted below:\n\n.. image:: /images/trainium-neurondevice.png\n\n\nEach Trainium chip consists of:\n\n+----------------------------------+----------------------------------+\n| Compute | Two :ref:`NeuronCore-v2 |\n| | ` |\n| | delivering 380 INT8 TOPS, |\n| | 190 FP16/BF16/cFP8/TF32 TFLOPS, |\n| | and 47.5 FP32 TFLOP. |\n+----------------------------------+----------------------------------+\n| Device Memory | 32 GiB of device memory (for | \n| | storing model state), with 820 | \n| | GiB/sec of bandwidth. | \n+----------------------------------+----------------------------------+\n| Data Movement | 1 TB/sec of DMA bandwidth, with |\n| | inline memory |\n| | compression/decompression. |\n+----------------------------------+----------------------------------+\n| NeuronLink | NeuronLink-v2 for |\n| | chip-to-chip interconnect |\n| | enables efficient scale-out |\n| | training, as well as memory |\n| | pooling between the different |\n| | Trainium chips. |\n+----------------------------------+----------------------------------+\n| Programmability | Trainium supports dynamic shapes |\n| | and control flow, via ISA |\n| | extensions of NeuronCore-v2. In |\n| | addition, Trainium also allows |\n| | for user-programmable |\n| | :ref:`rounding mode |\n| | ` |\n| | (Round Nearest Even Stochastic |\n| | Rounding), and custom operators |\n| | via the deeply embedded GPSIMD |\n| | engines. |\n+----------------------------------+----------------------------------+\n\n\nFor a detailed description of all the hardware engines, see :ref:`NeuronCore-v2 `\n" }, { "path": "about-neuron/arch/neuron-hardware/trainium2.rst", "content": ".. _trainium2-arch:\n\n######################\nTrainium2 Architecture\n######################\n\nTrainium2 is the third generation, purpose-built Machine Learning chip from AWS. Every Trainium2 chip contains eight NeuronCore-V3 cores. Beginning with Trainium2, AWS Neuron adds support for Logical \nNeuronCore Configuration (LNC), which lets you combine the compute and memory resources of multiple physical NeuronCores into a \nsingle logical NeuronCore. The following diagram shows the architecture overview of a Trainium2 chip.\n\n.. image:: /images/architecture/Trainium2/trainium2.png\n :align: center\n :width: 400\n \n===========================\nTrainium2 chip components\n===========================\n\nEach Trainium2 chip consists of the following components:\n\n+----------------------------------+-----------------------------------------------------+\n| Compute | Eight NeuronCore-v3 that collectively deliver: |\n| | |\n| | * 1,299 FP8 TFLOPS | \n| | * 667 BF16/FP16/TF32 TFLOPS |\n| | * 2,563 FP8/FP16/BF16/TF32 sparse TFLOPS |\n| | * 181 FP32 TFLOPS |\n| | |\n+----------------------------------+-----------------------------------------------------+\n| Device Memory | 96 GiB of device memory with 2.9 TB/sec of |\n| | bandwidth. | \n+----------------------------------+-----------------------------------------------------+\n| Data Movement | 3.5 TB/sec of DMA bandwidth, with inline |\n| | memory compression and decompression. |\n+----------------------------------+-----------------------------------------------------+\n| NeuronLink | NeuronLink-v3 for chip-to-chip interconnect |\n| | provides 1.28 TB/sec bandwidth per chip. It allows |\n| | for efficient scale-out training and inference, as |\n| | well as memory pooling between Trainium2 chips. |\n+----------------------------------+-----------------------------------------------------+\n| Programmability | Trainium2 supports dynamic shapes and control flow |\n| | via NeuronCore-v3 ISA extensions. Trainium2 also |\n| | allows for user-programmable |\n| | :ref:`rounding mode ` |\n| | (Round Nearest Even or Stochastic Rounding), and |\n| | custom operators via deeply embedded GPSIMD engines.|\n+----------------------------------+-----------------------------------------------------+\n| Collective communication | 16 CC-Cores orchestrate collective communication |\n| | among Trainium2 chips within and across instances. |\n+----------------------------------+-----------------------------------------------------+ \n\n==================================\nTrainium2 performance improvements\n==================================\n\nThe following set of tables offer a comparison between Trainium and Trainium2 chips. \n \nCompute\n\"\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1 \n :stub-columns: 1 \n :align: left\n \n * - \n - Trainium\n - Trainium2\n - Improvement factor\n \n * - FP8 (TFLOPS)\n - 191\n - 1299\n - 6.7x\n * - BF16/FP16/TF32 (TFLOPS)\n - 191\n - 667\n - 3.4x\n * - FP32 (TFLOPS)\n - 48\n - 181\n - 3.7x\n * - FP8/FP16/BF16/TF32 Sparse (TFLOPS)\n - Not applicable\n - 2563 \n - Not applicable\n\nMemory\n\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1 \n :stub-columns: 1 \n :align: left\n \n * - \n - Trainium\n - Trainium2\n - Improvement factor\n \n * - HBM Capacity (GiB)\n - 32\n - 96\n - 3x\n * - HBM Bandwidth (TB/sec)\n - 0.8\n - 2.9\n - 3.6x\n * - SBUF Capacity (MiB)\n - 48\n - 224\n - 4.7x\n * - Memory Pool Size\n - Up to 16 chips\n - Up to 64 chips\n - 4x\n\nInterconnect\n\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1 \n :stub-columns: 1 \n :align: left\n \n * - \n - Trainium\n - Trainium2\n - Improvement factor\n \n * - Inter-chip Interconnect (GB/sec/chip)\n - 384\n - 1280\n - 3.3x\n\nData movement\n\"\"\"\"\"\"\"\"\"\"\"\"\"\n.. list-table::\n :widths: auto\n :header-rows: 1 \n :stub-columns: 1 \n :align: left\n \n * - \n - Trainium\n - Trainium2\n - Improvement factor\n \n * - CC Cores\n - 6\n - 16\n - 3.3x\n * - DMA barriers\n - Write-after-write\n - Strong-order-write\n - \\>1x (Benefit DMA-size dependent)\n * - SBUF memory layout\n - Row-major\n - Row-major, Col-major-2B, Col-major-4B\n - Not applicable\n\n====================\nAdditional resources\n====================\n\nFor a detailed description of NeuronCore-v3 hardware engines, instances powered by AWS Trainium2, and Logical NeuronCore configuration, see the following resources:\n\n* :ref:`NeuronCore-v3 architecture `\n* :ref:`Amazon EC2 Trn2 architecture `\n* :ref:`Logical NeuronCore configuration `\n" }, { "path": "about-neuron/arch/neuron-hardware/trainium3.rst", "content": ".. meta::\n :description: \"Neuron Trainium3 (Trn3) architecture overview.\"\n :date-modified: 12/02/2025\n\n.. _trainium3-arch:\n\nTrainium3 Architecture\n=======================\n\nTrainium3 is the fourth-generation purpose-built Machine Learning chip from AWS. A Trainium3 device contains eight NeuronCore-v4 cores. Similar to Trainium2, AWS Neuron adds support for Logical NeuronCore Configuration (LNC), which lets you combine the compute and memory resources of multiple physical NeuronCores into a single logical NeuronCore. The following diagram shows the architecture overview of a Trainium3 chip.\n\n.. image:: /images/architecture/trn3/neuroncore-v4-overview.png\n :align: center\n\nNeuronCore-v4\n--------------\n\nEach Trainium3 chip consists of the following components:\n\n.. list-table::\n :widths: auto\n :header-rows: 0\n :stub-columns: 1\n :align: left\n\n * - Compute\n - Eight NeuronCore-v4 cores that collectively deliver:\n\n * 2,517 MXFP8/MXFP4 TFLOPS\n * 671 BF16/FP16/TF32 TFLOPS\n * 2,517 FP16/BF16/TF32 sparse TFLOPS\n * 183 FP32 TFLOPS\n\n * - Device memory\n - 144 GiB of device memory, with 4.9 TB/sec of bandwidth.\n\n * - Data movement\n - 4.9 TB/sec of DMA bandwidth, with inline computation.\n\n * - NeuronLink\n - NeuronLink-v4 for device-to-device interconnect provides 2.56 TB/sec bandwidth per device. It enables efficient scale-out training, as well as memory pooling between the different Trainium3 devices.\n\n * - Programmability\n - Trainium3 supports dynamic shapes and control flow, via ISA extensions of NeuronCore-v4. Trainium3 also allows for user-programmable rounding mode (Round Nearest Even or Stochastic Rounding), and custom operators via the deeply embedded GPSIMD engines.\n\n * - Collective communication\n - 16 CC-Cores orchestrate collective communication among Trainium3 devices, both within a server and across servers.\n\nTrainium3 performance improvements\n-----------------------------------\n\nThe following set of tables offer a comparison between Trainium2 and Trainium3 chips.\n\nCompute\n\"\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1\n :align: left\n\n * -\n - Trainium2\n - Trainium3\n - Improvement factor\n\n * - MXFP4 (TFLOPS)\n - Not applicable\n - 2517\n - -\n * - FP8 (TFLOPS)\n - 1299\n - 2517\n - 2x\n * - BF16/FP16/TF32 (TFLOPS)\n - 667\n - 671\n - 1x\n * - FP32 (TFLOPS)\n - 181\n - 183\n - 1x\n\nMemory\n\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1\n :align: left\n\n * -\n - Trainium2\n - Trainium3\n - Improvement factor\n\n * - HBM Capacity (GiB)\n - 96\n - 144\n - 1.5x\n * - HBM Bandwidth (TB/sec)\n - 2.9\n - 4.9\n - 1.7x\n * - SBUF Capacity (MiB)\n - 224\n - 256\n - 1.14x\n\nInterconnect\n\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1\n :align: left\n\n * -\n - Trainium2\n - Trainium3\n - Improvement factor\n\n * - Inter-chip Interconnect (GB/sec/chip)\n - 1280\n - 2560\n - 2x\n\nData movement\n\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1\n :align: left\n\n * -\n - Trainium2\n - Trainium3\n - Improvement factor\n\n * - DMA Bandwidth (TB/sec)\n - 3.5\n - 4.9\n - 1.4x\n\nAdditional resources\n----------------------\n\nFor a detailed description of NeuronCore-v4 hardware engines, instances powered by AWS Trainium3, and Logical NeuronCore configuration, see the following resources:\n\n* :ref:`NeuronCore-v4 architecture `\n\n" }, { "path": "about-neuron/arch/neuron-hardware/trn1-arch.rst", "content": ".. _aws-trn1-arch:\n\nAmazon EC2 Trn1/Trn1n Architecture\n===================================\n\nOn this page, we provide an architectural overview of the AWS Trn1/Trn1n\ninstances, and the corresponding :ref:`Trainium ` NeuronChips that power them\n(Trainium chips from here on).\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n.. _trn1-arch:\n\nTrn1/Trn1n Architecture\n-----------------------\n\nAn EC2 Trn1/Trn1n instance is powered by up to 16 :ref:`Trainium ` chips.\n\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1 \n :align: left\n \n\n * - Instance size\n - # of Trainium chips\n - vCPUs\n - Host Memory (GiB)\n - FP8/FP16/BF16/TF32 TFLOPS\n - FP32 TFLOPS\n - Device Memory (GiB)\n - Device Memory Bandwidth (GiB/sec)\n - EFA bandwidth (Gbps)\n\n * - Trn1.2xlarge\n - 1\n - 8\n - 32\n - 190\n - 47.5\n - 32\n - 820\n - N/A\n - up-to 25 \n\n * - Trn1.32xlarge\n - 16\n - 128\n - 512\n - 3,040\n - 760\n - 512\n - 13,120\n - 384\n - 800\n\n * - Trn1n.32xlarge\n - 16\n - 128\n - 512\n - 3,040\n - 760\n - 512\n - 13,120\n - 768\n - 1,600\n\n\nThe Trn1.2xlarge instance size allows customers to train their models on\na single Trainium chip, which is useful for small model training, as\nwell as for model experimentation. The Trn1.32xlarge and Trn1n.32xlarge instance size come\nwith a high-bandwidth and low-latency NeuronLink-v2 chip-to-chip\ninterconnect, which utilizes a 2D Torus topology. This is useful for\ncollective communication between the Trainium chips during scale-out\ntraining, as well as for pooling the memory capacity of all Trainium\nchips, making it directly addressable from each of the chips.\n\nIn a Trn1/Trn1n server, the Trainium chips are connected in a 2D Torus topology, as depicted below:\n\n.. image:: /images/trn1-topology.png\n\nThe Trn1/Trn1n instances are also available in an EC2 UltraCluster, which\nenables customers to scale Trn1/Trn1n instances to over 100,000 Trainium\nchips, and leverage the AWS-designed non-blocking petabit-scale EFA\nnetworking infrastructure.\n\n.. image:: /images/ultracluster-1.png\n\n\n\n" }, { "path": "about-neuron/arch/neuron-hardware/trn2-arch.rst", "content": ".. _aws-trn2-arch:\n\n############################\nAmazon EC2 Trn2 Architecture\n############################\n\nTrn2 is an Amazon EC2 accelerated computing instance, purpose built for high-performance deep learning training and inference. This page provides \nan architecture overview of the trn2.48xlarge and trn2u.48xlarge instances, and Trn2 UltraServer.\n\n.. contents:: Topics\n :local:\n :depth: 2\n\n.. _trn2-arch:\n\nTrn2 instance sizes\n===================\n\nTrn2 instances and UltraServers are available in the following sizes and configurations:\n\n* trn2.48xlarge\n* trn2u.48xlarge\n* Trn2 UltraServer\n\n.. _trn2-instance:\n\ntrn2.48xlarge / trn2u.48xlarge\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\nTrn2 instances are powered by 16 Trainium2 chips connected using a high-bandwidth, low-latency NeuronLink-v3 \nchip-to-chip interconnect. The NeuronLink-v3 chip-to-chip interconnect enables collective communication between Trainium2 \nchips during distributed training and inference. It also allows for the pooling of memory resources from all 16 Trainium2 chips. \n\nIn a trn2.48xlarge or trn2u.48xlarge instance, 16 Trainium2 chips are connected using a 4x4, 2D Torus topology. The following diagram shows the \nintra-instance connections of a trn2.48xlarge or trn2u.48xlarge instance\n\n.. image:: /images/architecture/Trn2/trn2.48xlarge.png\n :align: center\n :width: 650\n|\n\n.. _trn2-ultraserver: \n\nTrn2 UltraServer\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nA Trn2 UltraServer comprises four trn2u.48xlarge instances connected together via the NeuronLink-v3 chip-to-chip interconnect. \nThis allows for a total of 64 Trainium2 chips to be interconnected within a Trn2 UltraServer. Trainium2 chips with the same \ncoordinates in each Trn2 instance are connected in a ring topology. The following figure shows the inter-instance ring connection \nbetween Trainium2 chips.\n\n.. image:: /images/architecture/Trn2/u-trn2x64.png\n :align: center\n :width: 650\n|\nTrn2 instance specifications \n============================\n\nThe following table shows the performance metrics for Trainium2 based instances.\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :stub-columns: 1 \n :align: left\n \n\n * - Perfomance specification\n - trn2.48xlarge / trn2u.48xlarge\n - Trn2 UltraServer\n * - # of Trainium2 chips\n - 16\n - 64\n * - vCPUs\n - 192\n - 768\n * - Host Memory (GiB)\n - 2,048\n - 8,192\n * - FP8 PFLOPS\n - 20.8\n - 83.2\n * - FP16/BF16/TF32 PFLOPS\n - 10.7\n - 42.8\n * - FP8/FP16/BF16/TF32 Sparse PFLOPS\n - 41\n - 164\n * - FP32 PFLOPS\n - 2.9\n - 11.6\n * - Device Memory (GiB)\n - 1,536\n - 6,144\n * - Device Memory Bandwidth (TB/sec)\n - 46.4\n - 185.6\n * - Intra-instance NeuronLink-v3 bandwidth (GB/sec/chip)\n - 1,024\n - 1,024\n * - Inter-instance NeuronLink-v3 bandwidth (GB/sec/chip)\n - Not applicable\n - 256\n * - EFAv3 bandwidth (Gbps)\n - 3,200\n - 3,200\n\n" }, { "path": "about-neuron/arch/neuron-hardware/trn3-arch.rst", "content": ".. _aws-trn3-arch:\n\n###############################\nAmazon EC2 Trn3 Architecture\n###############################\n\nAmazon EC2 **Trn3** instances are accelerated computing instances powered by Trainium3 AI chips, purpose-built for high-performance deep learning training and inference. Trn3 is available in two UltraServer scale-up configurations: Gen1 with 64 Trainium3 chips per UltraServer, and Gen2 with 144 chips per UltraServer. Both configurations use NeuronSwitch-v1 interconnect technology to enable all-to-all connectivity between chips, especially optimized for workloads that leverage all-to-all communication patterns, such as Mixture of Experts models and autoregressive inference serving.\n\n=====================\nTrn3 Gen1 UltraServer\n=====================\n\nThe EC2 Trn3 Gen1 UltraServers deliver 161 PetaFLOPS of dense MXFP8 compute, 314 TB/s of HBM bandwidth, and 9TB of HBM capacity. Each UltraServer consists of four servers with 16 Trainium3 devices per server. Therefore, the UltraServer integrates a total of 64 Trainium3 devices into a single scale-up domain, interconnected via our latest-generation NeuronLink-v4 and the newly introduced NeuronSwitch-v1. The chip-to-chip topology features an all-to-all connectivity design, replacing the previous 2D-torus architecture. This all-to-all topology is optimized for workloads that require efficient all-to-all communication patterns or ultra-low latency collectives, including Mixture of Experts models and autoregressive inference serving. The following diagram illustrates the Trn3 Gen1 UltraServer connectivity.\n\n.. image:: /images/architecture/trn3/trn3-ultraserver-gen1.png\n :align: center\n\n\n=====================\nTrn3 Gen2 UltraServer\n=====================\n\nThe EC2 Trn3 Gen2 UltraServers deliver 362 PetaFLOPS of dense MXFP8 compute, 706 TB/s of HBM bandwidth, and 20TB of HBM capacity. Each UltraServer consists of 36 servers with 4 Trainium3 devices per server. Trainium3 devices within the same server are connected via a first-level NeuronSwitch-v1, while devices across servers are connected via two second-level NeuronSwitch-v1 and NeuronLink-v4. Therefore, the UltraServer integrates 144 Trainium3 devices into a single scale-up domain. Like Gen1, the chip-to-chip topology features an all-to-all connectivity design optimized for Mixture of Experts models and autoregressive inference serving. The following diagram illustrates the Trn3 Gen2 UltraServer connectivity.\n\n.. image:: /images/architecture/trn3/trn3-ultraserver-gen2.png\n :align: center\n\n==========================================\nTrn3 Gen1/Gen2 UltraServer specifications\n==========================================\n\nThe following table shows the performance metrics for Tranium3 based instances.\n\n.. list-table::\n :header-rows: 2\n :stub-columns: 1\n :widths: 30 20 20\n\n * - \n - Trn3 Gen1 UltraServer\n - Trn3 Gen2 UltraServer\n * - Configuration\n - \n - \n * - # of Trainium3 devices\n - 64\n - 144\n * - Host vCPUs\n - 768\n - 2304\n * - Host Memory (GiB)\n - 8,192\n - 27,648\n * - **Compute**\n - \n - \n * - MXFP8/MXFP4 TFLOPS\n - 161,088\n - 362,448\n * - FP16/BF16/TF32 TFLOPS\n - 42,944\n - 96,624\n * - FP32 TFLOPS\n - 11,712\n - 26,352\n * - **Memory**\n - \n - \n * - Device Memory (GiB)\n - 9,216\n - 20,736\n * - Device Memory Bandwidth (TB/sec)\n - 313.6\n - 705.6\n * - **Interconnect**\n - \n - \n * - NeuronLink-v4 bandwidth (GiB/sec/device)\n - 2,048\n - 2,048\n * - EFA bandwidth (Gbps)\n - 12,800\n - 28,800\n \n============================================\nTrn3 UltraServer Connectivity and Networking\n============================================\n\nTrn3 UltraServers use a PCIe switch-based interconnect architecture for all chip-to-chip communication, both within and across servers. This replaces the point-to-point NeuronLink topology used in previous generations (Trn1, Trn2) with a switched fabric that enables flexible, all-to-all connectivity across the entire UltraServer domain.\n\nIntra-server connectivity\n-------------------------\n\nEach server (sled) contains 4 Trainium3 chips connected through an intra-server PCIe switch. Each chip provides four PCIe Gen6 x8 links to this switch, delivering a total of 256 GB/s of bidirectional bandwidth between chips within the same server. This local switch enables low-latency communication for operations like tensor parallelism and data-parallel gradient synchronization within a server.\n\nInter-server connectivity\n-------------------------\n\nAll servers within a rack are connected through inter-server PCIe switches. Each Trainium3 chip provides five PCIe Gen6 x8 links to the inter-server switch, delivering 320 GB/s of bidirectional bandwidth per chip for cross-server communication. This enables collective operations such as all-reduce and all-gather to span all servers in a rack without requiring host CPU involvement.\n\nInter-rack connectivity\n-----------------------\n\nFor multi-rack configurations, Trainium3 chips in corresponding positions across racks are connected via dedicated direct PCIe links. Each chip provides two PCIe Gen6 x8 links for inter-rack communication, delivering 128 GB/s of bidirectional bandwidth per chip between racks. This direct-link design avoids additional switch hops for cross-rack traffic.\n\nBandwidth summary\n-----------------\n\n.. list-table::\n :header-rows: 1\n :widths: 30 30 40\n\n * - Connectivity level\n - Bandwidth per chip\n - Link configuration\n * - Intra-server (within sled)\n - 256 GB/s\n - 4 × PCIe Gen6 x8 via intra-server switch\n * - Inter-server (within rack)\n - 320 GB/s\n - 5 × PCIe Gen6 x8 via inter-server switch\n * - Inter-rack\n - 128 GB/s\n - 2 × PCIe Gen6 x8 direct links\n\nRouting and address-based switching\n------------------------------------\n\nUnlike Trn1 and Trn2, where NeuronLink connections are point-to-point and require no intermediate routing, Trn3's PCIe switch fabric uses address-based routing to direct transactions to the correct destination chip. Each Trainium3 chip in the system is identified by a tuple of (rack, server, chip), and this identity is encoded in the upper bits of the PCIe address used for outbound transactions. The PCIe switches use BAR (Base Address Register) address matching to determine the correct output port for each transaction.\n\nThis routing is transparent to ML workloads. The Neuron Runtime and compiler handle all address encoding and switch configuration automatically. From the developer's perspective, collective operations and direct memory access between chips work the same way as on previous Trainium generations.\n\nSemaphore-based synchronization\n-------------------------------\n\nTrn3 uses hardware semaphores to synchronize data transfers across the switched fabric. When a chip writes data to a remote chip's HBM, a follow-up semaphore write signals completion to the receiving chip. The system guarantees that data and its associated semaphore always traverse the same physical path through the switch fabric, ensuring correct ordering without additional software synchronization overhead.\n" }, { "path": "about-neuron/benchmarks/index.rst", "content": ".. _benchmark:\n\n.. meta::\n :description: Explore AWS Neuron performance benchmarks for Inf1, Inf2, and Trn1 instances. Find detailed inference and training performance data across NLP, CV, and recommender models to optimize your machine learning workloads.\n :date-modified: 2025-10-03\n\nNeuron performance\n==================\n\nThe Neuron performance pages provide comprehensive benchmarks and performance data for AWS Neuron SDK across different Trainium and Inferentia instance types. These benchmarks cover various open-source models for Natural Language Processing (NLP), Computer Vision (CV), and Recommender systems. Each benchmark includes detailed setup instructions and reproducible test configurations to help you evaluate performance for your specific use cases.\n\nInference performance\n---------------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: appnote-performance-benchmark\n :link-type: ref\n\n **Inf1 Inference Performance**\n ^^^\n Comprehensive inference benchmarks for ``Inf1`` instances across NLP, CV, and recommender models\n\n .. grid-item-card::\n :link: inf2-performance\n :link-type: ref\n\n **Inf2 Inference Performance**\n ^^^\n Latest inference performance data for ``Inf2`` instances with improved throughput and latency metrics\n\n .. grid-item-card::\n :link: trn1-inference-performance\n :link-type: ref\n\n **Trn1 Inference Performance**\n ^^^\n Inference benchmarks for ``Trn1`` instances showcasing versatile training and inference capabilities\n\nTraining performance\n--------------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: trn1-training-performance\n :link-type: ref\n\n **Trn1 Training Performance**\n ^^^\n Training performance benchmarks for ``Trn1`` instances with distributed training metrics and scalability data\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n inf1/index\n inf2/inf2-performance\n trn1/trn1-inference-performance\n trn1/trn1-training-performance\n\n\n" }, { "path": "about-neuron/benchmarks/inf1/data.csv", "content": "Name,Model,Model details,Framework,Application Type,Run Mode,Inst. Type,Num. Cores,Batch Size,Avg Throughput (/sec),Max Throughput,Threads,Ops in Inferentia,Latency P50 (ms),Latency P90 (ms),Latency P95 (ms),Latency P99 (ms),Latency P100 (ms),Neuron Version,Application,Tutorial\r\n\"YOLOv4-PT(fp32,b1,c4)\",YOLO v4,fp32,PyTorch 1.13,Real Time,Data Parallel,inf1.2xlarge,4,1,180.2,,8,,40.1,,,52,,2.15.0,CV,:ref:`Evaluate YOLO v4 on Inferentia `\r\n\"Resnet50-PT(fp32,b5,c4)\",Resnet-50,fp32,PyTorch 1.13,Batch,Data Parallel,inf1.xlarge,4,5,923,,4,,22,,,23,,2.15.0,CV,:ref:`Resnet50 model for Inferentia `\r\n\"Resnet50-TF(fp16,b5,c4)\",Resnet-50,fp16,Tensorflow 1.15,Batch,Data Parallel,inf1.xlarge,4,10,2207,,8,,17.8,,,22.7,,2.12.0,CV,:ref:`ResNet-50 optimization example `\r\n\"OpenPose-TF(fp16,b1,c4)\",OpenPose,fp16,Tensorflow 1.15,Real Time,Data Parallel,inf1.xlarge,4,1,57.5,,4,,60.3,,,67.4,,2.12.0,CV,:ref:`Running OpenPose on Inferentia `\r\n\"BERT-base-PT(fp32,b6,c4)\",BERT base,\"fp32, bert-base-cased-finetuned-mrpc, sequence-length=128\",PyTorch 1.13,Batch,Data Parallel,inf1.xlarge,4,6,966,,4,,21,,,22,,2.15.0,NLP,:ref:`HuggingFace Pretrained BERT `\r\n\"BERT-base-PT(fp32,b1,c16)\",BERT base,\"fp32, bert-base-uncased, sequence-length=128\",PyTorch 1.13,Real Time,Model Pipeline,inf1.6xlarge,16,1,1988.8,,12,,6,,,6.3,,2.15.0,NLP,:ref:`Using NeuronCore Pipeline `\r\n\"BERT-base-TF(fp32,b128,c16)\",BERT base,\"fp32, distilbert-base-uncased-finetuned-sst-2-english, sequence-length=128\",Tensorflow 2.8,Batch,Data Parallel,inf1.6xlarge,16,16,2114.8,,,,30.1,,,33,,2.15.0,NLP,:ref:`HuggingFace distilBERT with Tensorflow2 `" }, { "path": "about-neuron/benchmarks/inf1/index.rst", "content": ".. _appnote-performance-benchmark:\n\nInf1 Inference Performance\n===========================\n\n.. important::\n\n The benchmark scripts linked on this page are provided for historical reference only and are not tested with recent versions of the Neuron SDK. They have been moved to the `archive folder `_.\n\n.. contents:: Table of contents\n :local:\n\nThe following tables contain the reference inference performance for models in the tutorials. Follow the links on each row to replicate similar results in your own environment. Refer to :ref:`ec2-then-ec2-setenv` documentation to create a new environment based on the latest Neuron release.\n\n*Last update: September 16th, 2024*\n\n\n.. _NLP:\n\nEncoder Models\n--------------\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_encoder.csv')\n df_prices = pd.read_csv('instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Avg Throughput (/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model', 'Scripts', 'Framework', 'Inst. Type', 'Avg Throughput (/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model details' ]\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n\n int_cols = ['Avg Throughput (/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(0).astype('int',copy=True)\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_encoder.csv')\n df_prices = pd.read_csv('instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Avg Throughput (/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model', 'Scripts', 'Framework', 'Inst. Type', 'Avg Throughput (/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model details' ]\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n\n int_cols = ['Avg Throughput (/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(0).astype('int',copy=True)\n\n\n.. note::\n Throughput and latency numbers in this table were computed using* NeuronPerf_. To reproduce these results, install NeuronPerf and run the provided scripts.*\n\n.. _NeuronPerf: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuronperf/index.html\n\nConvolutional Neural Networks (CNN) Models\n------------------------------------------\n\n.. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_cnn.csv')\n df_prices = pd.read_csv('instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type').query('`Application`==\"CV\"')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Avg Throughput (/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model', 'Tutorial', 'Framework', 'Inst. Type', 'Avg Throughput (/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model details' ]\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences']).groupby('Model').head(2)\n\n int_cols = ['Avg Throughput (/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(0).astype('int',copy=True)\n\n.. note::\n Throughput and latency numbers in this table were generated using Neuron Tutorials.\n\n.. note::\n **Cost per 1M inferences** is calculated using US East (N. Virginia) RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n" }, { "path": "about-neuron/benchmarks/inf1/instance_prices.csv", "content": "Inst. Type,RI-Effective hourly rate\ninf1.xlarge,0.110\ninf1.2xlarge,0.174\ninf1.6xlarge,0.567\ninf1.24xlarge,2.269\n" }, { "path": "about-neuron/benchmarks/inf1/latency_data_encoder.csv", "content": "Model,Scripts,Source,Framework,Inst. Type,Num Cores,Seq. Length,Avg Throughput (/sec),Max Throughput,Threads,Latency P50 (ms),Latency P90 (ms),Latency P95 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,N Models,Workers per Model,Model details\r\nBERT base (bert-base-cased),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,125.7,,8,7.9,,,8.0,Real Time,2.20.0,Data Parallel,1,1,1,\"fp32, sequence-length=128\"\r\nBERT base (bert-base-uncased),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,284.7,,8,10.5,,,10.7,Real Time,2.20.0,Data Parallel,3,1,1,\"fp32, sequence-length=128\"\r\nDistilBERT base (distilbert-base-uncased-finetuned-sst-2-english),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,593.4,,8,10.0,,,10.7,Real Time,2.20.0,Data Parallel,5,1,1,\"fp32, sequence-length=128\"\r\nDistilBERT base (distilbert-base-uncased),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,538.2,,8,11.1,,,11.5,Real Time,2.20.0,Data Parallel,6,1,1,\"fp32, sequence-length=128\"\r\nDistilRoBERTa base (distilroberta-base),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,417.0,,8,7.0,,,7.8,Real Time,2.20.0,Data Parallel,3,1,1,\"fp32, sequence-length=128\"\r\n" }, { "path": "about-neuron/benchmarks/inf1/throughput_data_cnn.csv", "content": "Name,Model,Model details,Framework,Application Type,Run Mode,Inst. Type,Num. Cores,Batch Size,Avg Throughput (/sec),Max Throughput,Threads,Ops in Inferentia,Latency P50 (ms),Latency P90 (ms),Latency P95 (ms),Latency P99 (ms),Latency P100 (ms),Neuron Version,Application,Tutorial\r\n\"YOLOv4-PT(fp32,b1,c4)\",YOLO v4,fp32,PyTorch 1.13,Real Time,Data Parallel,inf1.2xlarge,4,1,180.3,,8,,40.0,,,50.8,,2.20.0,CV,:ref:`Evaluate YOLO v4 on Inferentia `\r\n\"Resnet50-PT(fp32,b5,c4)\",Resnet-50,fp32,PyTorch 1.13,Batch,Data Parallel,inf1.xlarge,4,5,921.5,,4,,21.6,,,22.9,,2.20.0,CV,:ref:`Resnet50 model for Inferentia `\r\n\"Resnet50-TF(fp16,b5,c4)\",Resnet-50,fp16,Tensorflow 1.15,Batch,Data Parallel,inf1.xlarge,4,10,2207,,8,,17.8,,,22.7,,2.12.0,CV,:ref:`ResNet-50 optimization example `\r\n\"OpenPose-TF(fp16,b1,c4)\",OpenPose,fp16,Tensorflow 1.15,Real Time,Data Parallel,inf1.xlarge,4,1,57.5,,4,,60.3,,,67.4,,2.12.0,CV,:ref:`Running OpenPose on Inferentia `\r\n" }, { "path": "about-neuron/benchmarks/inf1/throughput_data_encoder.csv", "content": "Model,Scripts,Source,Framework,Inst. Type,Num Cores,Seq. Length,Avg Throughput (/sec),Max Throughput,Threads,Latency P50 (ms),Latency P90 (ms),Latency P95 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,N Models,Workers per Model,Model details\r\nBERT base (bert-base-cased),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,1095.4,,8,58.3,,,65.0,Batch,2.20.0,Data Parallel,8,4,2,\"fp32, sequence-length=128\"\r\nBERT base (bert-base-uncased),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,1180.7,,8,40.6,,,45.0,Batch,2.20.0,Data Parallel,6,4,2,\"fp32, sequence-length=128\"\r\nDistilBERT base (distilbert-base-uncased-finetuned-sst-2-english),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,1875.3,,8,33.7,,,54.1,Batch,2.20.0,Data Parallel,8,4,2,\"fp32, sequence-length=128\"\r\nDistilBERT base (distilbert-base-uncased),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,1876.7,,8,33.7,,,53.2,Batch,2.20.0,Data Parallel,8,4,2,\"fp32, sequence-length=128\"\r\nDistilRoBERTa base (distilroberta-base),:compile-pt:`Compile ` + :benchmark-pt:`Benchmark `,HuggingFace,PyTorch 1.13.1,inf1.xlarge,4,128,1512.9,,8,15.0,,,25.9,Batch,2.20.0,Data Parallel,6,4,1,\"fp32, sequence-length=128\"\r\nBERT base,:ref:`HuggingFace Pretrained BERT `,,PyTorch 1.13,inf1.xlarge,,,1056,,,20,,,21,Batch,2.20.0,Data Parallel,4,,,\"fp32, bert-base-cased-finetuned-mrpc, sequence-length=128\"\r\nBERT base,:ref:`Using NeuronCore Pipeline `,,PyTorch 1.13,inf1.6xlarge,,,2009.1,,,5.9,,,6.3,Real Time,2.20.0,Model Pipeline,1,,,\"fp32, bert-base-uncased, sequence-length=128\"\r\nBERT base,:ref:`HuggingFace distilBERT with Tensorflow2 `,,Tensorflow 2.10,inf1.6xlarge,,,2123.4,,,30.0,,,32.2,Batch,2.20.0,Data Parallel,16,,,\"fp32, distilbert-base-uncased-finetuned-sst-2-english, sequence-length=128\"\r\n" }, { "path": "about-neuron/benchmarks/inf2/inf2-performance.rst", "content": ".. _inf2-performance:\n\nInf2 Inference Performance\n==========================\n\n.. important::\n\n The benchmark scripts linked on this page are provided for historical reference only and are not tested with recent versions of the Neuron SDK. They have been moved to the `archive folder `_.\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n*Last update: Feb 26th, 2026*\n\n.. _inf2_inference_perf:\n\nEncoder Models\n--------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_encoder.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (inference/second)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/second)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Sequence Length', 'Model Data Type','Compilation Autocast Data Type', 'OS Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n df['Throughput (inference/second)'] = df['Throughput (inference/second)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_encoder.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (inference/second)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/second)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Sequence Length', 'Model Data Type','Compilation Autocast Data Type', 'OS Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n df['Throughput (inference/second)'] = df['Throughput (inference/second)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\nEncoder-Decoder Models\n----------------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_encoder_decoder.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (tokens/second)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (tokens/second)', 'Latency per Token P50 (ms)', 'Latency per Token P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'TP Degree',\t'DP Degree', 'Batch Size', 'Sequence Length', 'Input Length', 'Output Length', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n\n df['Throughput (tokens/second)'] = df['Throughput (tokens/second)'].round(2).astype('float',copy=True)\n int_cols = ['Latency per Token P50 (ms)', 'Latency per Token P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n Only for Encoder-Decoder\n\n **Throughput (tokens/second)** counts both input and output tokens\n\n **Latency per Token** counts both input and output tokens\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_encoder_decoder.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (tokens/second)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (tokens/second)', 'Latency per Token P50 (ms)', 'Latency per Token P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'TP Degree',\t'DP Degree', 'Batch Size', 'Sequence Length', 'Input Length', 'Output Length', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n\n df['Throughput (tokens/second)'] = df['Throughput (tokens/second)'].round(2).astype('float',copy=True)\n int_cols = ['Latency per Token P50 (ms)', 'Latency per Token P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n **Throughput (tokens/second)** counts both input and output tokens\n\n **Latency per Token** counts both input and output tokens\n \n\nVision Transformers Models\n--------------------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_vision_transformers.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M images', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_vision_transformers.csv')\n\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework','Inst. Type','Task', 'Throughput (inference/sec)','Latency P50 (ms)','Latency P99 (ms)','Cost per 1M images','Application Type','Neuron Version','Run Mode','Batch Size','Model Data Type', 'Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\nConvolutional Neural Networks (CNN) Models\n------------------------------------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_vision_cnn.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M images', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_vision_cnn.csv')\n\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework','Inst. Type','Task', 'Throughput (inference/sec)','Latency P50 (ms)','Latency P99 (ms)','Cost per 1M images','Application Type','Neuron Version','Run Mode','Batch Size','Model Data Type', 'Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\nStable Diffusion Models\n-----------------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_vision_sd.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M images', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n **Cost per 1M images** is calculated using RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_vision_sd.csv')\n\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework','Inst. Type','Task', 'Throughput (inference/sec)','Latency P50 (ms)','Latency P99 (ms)','Cost per 1M images','Application Type','Neuron Version','Run Mode','Batch Size','Model Data Type', 'Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n **Cost per 1M images** is calculated using RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n\nDiffusion Transformer Models\n----------------------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_vision_dit.csv')\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M images', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n **Cost per 1M images** is calculated using RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_vision_dit.csv')\n\n df_prices = pd.read_csv('inf2_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M images'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Image Size','Scripts','Framework','Inst. Type','Task', 'Throughput (inference/sec)','Latency P50 (ms)','Latency P99 (ms)','Cost per 1M images','Application Type','Neuron Version','Run Mode','Batch Size','Model Data Type', 'Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Image Size', 'Cost per 1M images'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n **Cost per 1M images** is calculated using RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n\n\n\n\n.. note::\n\n See :ref:`neuron_hw_glossary` for abbreviations and terms\n" }, { "path": "about-neuron/benchmarks/inf2/inf2_instance_prices.csv", "content": "Inst. Type,RI-Effective hourly rate\nInf2.xlarge,0.328\nInf2.48xlarge,5.608\nInf2.24xlarge,2.804\nInf2.8xlarge,0.850\n" }, { "path": "about-neuron/benchmarks/inf2/latency_data_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Output Token Throughput (tokens/sec),TTFT Latency P50 (ms),TTFT Latency P99 (ms),TPOT Latency P50 (ms),TPOT Latency P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type,Weight Storage Data Type\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,144.98,29.47,42.05,7.41,7.68,Real Time,2.18.1,Tensor Parallel,24,1,8192,128,8064,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,133.63,209.37,232.02,7.47,7.57,Real Time,2.18.1,Tensor Parallel,24,1,8192,4096,4096,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,161.67,25,25.88,6.42,6.58,Real Time,2.18.1,Tensor Parallel,24,1,4096,128,3968,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,153.58,101.81,110.6,6.5,6.6,Real Time,2.18.1,Tensor Parallel,24,1,4096,2048,2048,FP16,Matmult-BF16,int8\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,28.84,745.49,749.48,34.67,35.06,Real Time,2.18.1,Tensor Parallel,24,1,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,29.81,312.86,322.56,33.81,34.13,Real Time,2.18.1,Tensor Parallel,24,1,3072,1024,2048,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.16,310.18,315.23,33.14,34.29,Real Time,2.18.1,Tensor Parallel,24,1,2048,1024,1024,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.82,80,100.47,32.47,33.03,Real Time,2.18.1,Tensor Parallel,24,1,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.9,99.37,142.62,32.48,32.86,Real Time,2.18.1,Tensor Parallel,24,1,512,256,256,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,31.28,77.81,78.52,32.2,33.02,Real Time,2.18.1,Tensor Parallel,24,1,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-7b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,156.1281689,27.63772011,33.7741375,6.46972656,7.07960129,Real Time,2.18.0,Tensor Parallel,24,1,4096,128,3968,FP16,Matmult-BF16,bf16\r\nLlama-2-7b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,145.1665497,29.20985222,33.39338303,7.34019279,7.80153275,Real Time,2.18.0,Tensor Parallel,24,1,8192,128,8064,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,112.520024,25.85077286,26.89838409,9.16552544,9.33074951,Real Time,2.18.0,Tensor Parallel,24,1,4096,128,3968,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,97.41527724,333.7800503,340.9907818,10.17355919,10.37788391,Real Time,2.18.0,Tensor Parallel,24,1,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,73.16747525,994.1797257,999.7954369,13.49759102,13.97609711,Real Time,2.18.0,Tensor Parallel,24,1,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.06356,76.59531,77.12364,32.89557,33.42032,Real Time,2.18.0,Tensor Parallel,24,1,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,29.92419,96.4396,98.47379,33.13422,33.45966,Real Time,2.18.0,Tensor Parallel,24,1,512,256,256,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.07017,76.33042,86.52544,33.15115,34.0786,Real Time,2.18.0,Tensor Parallel,24,1,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,29.426,277.01592,280.12586,33.73241,34.01256,Real Time,2.18.0,Tensor Parallel,24,1,2048,1024,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,28.91353,275.96617,284.77097,34.81936,35.43973,Real Time,2.18.0,Tensor Parallel,24,1,3072,1024,2048,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,28.32725,810.43696,814.87799,34.90329,35.14242,Real Time,2.18.0,Tensor Parallel,24,1,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,149.7363908,27.34160423,29.20722961,6.86240196,7.07960129,Real Time,2.18.0,Tensor Parallel,24,1,4096,128,3968,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,81.7034129,557.9631329,562.8581047,7.86566734,11.64746284,Real Time,2.18.0,Tensor Parallel,24,1,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,95.99325977,539.5913124,557.1010113,10.32972336,10.61367989,Real Time,2.18.0,Tensor Parallel,24,1,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,112.7050057,27.0178318,33.24627876,9.12380219,9.38177109,Real Time,2.18.0,Tensor Parallel,24,1,4096,128,3968,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,97.52121418,338.6683464,340.4603005,10.15138626,10.55026054,Real Time,2.18.0,Tensor Parallel,24,1,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,73.67826681,989.4962311,1000.655413,13.43631744,13.85569572,Real Time,2.18.0,Tensor Parallel,24,1,16384,8192,8192,FP16,Matmult-BF16,bf16\r\n" }, { "path": "about-neuron/benchmarks/inf2/latency_data_encoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (inference/second),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Sequence Length,Model Data Type,Compilation Autocast Data Type,OS Type\r\nalbert-base-v2,:benchmark-pt:`Benchmark `,PyTorch 2.8,Inf2.xlarge,Raw Output (AutoModel),2119.78480993,0.93722343,1.00183487,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.8,Inf2.xlarge,Raw Output (AutoModel),1998.20950133,0.99897385,1.04045868,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nbert-large-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.7,Inf2.xlarge,Raw Output (AutoModel),738.64502335,2.69365311,2.77733803,Real Time,2.25.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\ndistilbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.8,Inf2.xlarge,Raw Output (AutoModel),3401.96550351,0.57864189,0.67734718,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\ngoogle/electra-base-discriminator,:benchmark-pt:`Benchmark `,PyTorch 2.8,Inf2.xlarge,Raw Output (AutoModel),2020.45540243,0.9958744,1.04618073,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nroberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.8,Inf2.xlarge,Raw Output (AutoModel),1989.26102482,0.99945068,1.09100342,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nroberta-large,:benchmark-pt:`Benchmark `,PyTorch 2.8,Inf2.xlarge,Raw Output (AutoModel),738.88441011,2.69317627,2.77304649,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nxlm-roberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.48xlarge,Raw Output (AutoModelForMaskedLM),48.80198341,40.66610336,51.05760336,Real Time,2.22.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\n\r\n" }, { "path": "about-neuron/benchmarks/inf2/latency_data_encoder_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (tokens/second),Latency per Token P50 (ms),Latency per Token P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,DP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type\r\nt5-3b,`Tutorial `_,NeuronX Distributed,Inf2.24xlarge,Text Generation,108.18,9.25,9.26,Real Time,2.18.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\ngoogle/flan-t5-xl,`Tutorial `_,NeuronX Distributed,Inf2.24xlarge,Text Generation,117.6,8.5,8.53,Real Time,2.18.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\n" }, { "path": "about-neuron/benchmarks/inf2/latency_data_vision.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\ndeepmind/multimodal-perceiver,16x224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Multimodal Autoencoding,0.83,1250,1271,Real Time,2.18.0,Data Parallel,1,FP32,None\ndeepmind/vision-perceiver-learned,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,99.6,18.6,18.7,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\ndeepmind/vision-perceiver-fourier,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,67.9,29.5,29.68,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\ndeepmind/vision-perceiver-conv,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,126.5,14.14,14.2,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\ngoogle/vit-base-patch16-224,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,709.468,1.406,1.431,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nopenai/clip-vit-base-patch32,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,163.444,6.113,6.143,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nopenai/clip-vit-large-patch14,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,61.812,16.172,16.216,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nresnet18,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,1385.04,0.72,0.75,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nresnet34,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,1187.64,0.83,0.88,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nresnet50,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,1044.93,0.95,0.98,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nresnet101,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,882.61,1.13,1.15,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nresnet152,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,736.91,1.35,1.39,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion 1.5,512x512,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.421,2369.6,2406.8,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion 2.1,512x512,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.549,1794.5,2103.7,Real Time,2.17.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\nStable Diffusion 2.1,768x768,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.188,5306.7,5368.6,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion 2 Inpainting,936x624,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.15,6701.4,6737.4,Real Time,2.17.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\nStable Diffusion XL Base,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.073,13431.7,15739.0,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion XL Base & Refiner,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.8xlarge,Image Generation,0.078,12651.9,15053.9,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nUNet,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Segmentation,420.16,2.37,2.41,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nvgg11,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,524.10,1.90,1.96,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\nvgg16,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,435.54,2.29,2.33,Real Time,2.14.0,Data Parallel,1,FP32,Matmult-BF16\n" }, { "path": "about-neuron/benchmarks/inf2/latency_data_vision_cnn.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\r\nresnet18,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,1669.796,0.596,0.613,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nresnet34,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,1394.211,0.718,0.726,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nresnet50,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,1218.875,0.83,0.846,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nresnet101,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,994.691,1.007,1.024,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nresnet152,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,837.784,1.185,1.219,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nUNet,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Segmentation,447.094,2.232,2.253,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nvgg11,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,629.189,1.59,1.605,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nvgg16,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,508.665,1.956,1.995,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16" }, { "path": "about-neuron/benchmarks/inf2/latency_data_vision_dit.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\nPixArt Alpha,256x256,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,1.975,502.587,537.258,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\nPixArt Alpha,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,0.565,1769.756,1775.697,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\nPixArt Sigma,256x256,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,1.86,540.832,548.41,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\nPixArt Sigma,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,0.543,1841.882,1850.683,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\n" }, { "path": "about-neuron/benchmarks/inf2/latency_data_vision_sd.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\r\nStable Diffusion 1.5,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.494,2023.741,2031.705,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nStable Diffusion 2.1,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.596,1679.805,1685.442,Real Time,2.21.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\r\nStable Diffusion 2.1,768x768,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.187,5337.509,5357.361,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nStable Diffusion 2 Inpainting,936x624,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.133,7546.004,7550.984,Real Time,2.21.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\r\nStable Diffusion XL Base,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.083,12048.659,12102.431,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nStable Diffusion XL Base & Refiner,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.8xlarge,Image Generation,0.095,10546.45,10704.566,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16" }, { "path": "about-neuron/benchmarks/inf2/latency_data_vision_transformers.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\r\ndeepmind/multimodal-perceiver,16x224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Multimodal Autoencoding,0.853,1170.045,1232.056,Real Time,2.21.0,Data Parallel,1,FP32,None\r\ndeepmind/vision-perceiver-learned,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,99.6,18.6,18.7,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\r\ndeepmind/vision-perceiver-fourier,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,67.9,29.5,29.68,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\r\ndeepmind/vision-perceiver-conv,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,126.5,14.14,14.2,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\r\ngoogle/vit-base-patch16-224,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,746.139,1.322,1.378,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nopenai/clip-vit-base-patch32,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,161.047,6.213,6.246,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nopenai/clip-vit-large-patch14,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,73.261,13.643,13.685,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Output Token Throughput (tokens/sec),TTFT Latency P50 (ms),TTFT Latency P99 (ms),TPOT Latency P50 (ms),TPOT Latency P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type,Weight Storage Data Type\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,649.17,68.95,99.28,15.22,15.48,Batch,2.18.1,Tensor Parallel,24,8,8192,128,8064,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,521.96,1992.59,2016.73,15.31,15.64,Batch,2.18.1,Tensor Parallel,24,8,8192,4096,4096,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,859.09,66.02,75.73,10.45,10.76,Batch,2.18.1,Tensor Parallel,24,8,4096,128,3968,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,759.15,823.53,832.84,10.5,11.02,Batch,2.18.1,Tensor Parallel,24,8,4096,2048,2048,FP16,Matmult-BF16,int8\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,28.84,745.49,749.48,34.67,35.06,Batch,2.18.1,Tensor Parallel,24,1,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,29.81,312.86,322.56,33.81,34.13,Batch,2.18.1,Tensor Parallel,24,1,3072,1024,2048,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.16,310.18,315.23,33.14,34.29,Batch,2.18.1,Tensor Parallel,24,1,2048,1024,1024,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.82,80,100.47,32.47,33.03,Batch,2.18.1,Tensor Parallel,24,1,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.9,99.37,142.62,32.48,32.86,Batch,2.18.1,Tensor Parallel,24,1,512,256,256,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,31.28,77.81,78.52,32.2,33.02,Batch,2.18.1,Tensor Parallel,24,1,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-7b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,725.82805,77.36206,87.27574,12.10523,13.05699,Batch,2.18.0,Tensor Parallel,24,8,4096,128,3968,FP16,Matmult-BF16,bf16\r\nLlama-2-7b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,577.97078,80.11794,89.68878,16.39295,17.81178,Batch,2.18.0,Tensor Parallel,24,8,8192,128,8064,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,589.88712,108.80947,113.89017,14.89663,15.79142,Batch,2.18.0,Tensor Parallel,24,8,4096,128,3968,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,351.75817,7083.72855,7158.32424,20.9856,21.80099,Batch,2.18.0,Tensor Parallel,24,8,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,178.56973,5141.32094,5160.92515,21.70897,22.74466,Batch,2.18.0,Tensor Parallel,24,4,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.06356,76.59531,77.12364,32.89557,33.42032,Batch,2.18.0,Tensor Parallel,24,1,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,29.92419,96.4396,98.47379,33.13422,33.45966,Batch,2.18.0,Tensor Parallel,24,1,512,256,256,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,30.07017,76.33042,86.52544,33.15115,34.0786,Batch,2.18.0,Tensor Parallel,24,1,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,29.426,277.01592,280.12586,33.73241,34.01256,Batch,2.18.0,Tensor Parallel,24,1,2048,1024,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,28.91353,275.96617,284.77097,34.81936,35.43973,Batch,2.18.0,Tensor Parallel,24,1,3072,1024,2048,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,28.32725,810.43696,814.87799,34.90329,35.14242,Batch,2.18.0,Tensor Parallel,24,1,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,761.88605,77.62027,86.62724,11.63864,12.49599,Batch,2.18.0,Tensor Parallel,24,8,4096,128,3968,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,450.37555,4740.11564,4783.75316,16.54649,17.52925,Batch,2.18.0,Tensor Parallel,24,8,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,411.04655,11085.12306,11125.86117,18.01157,19.9585,Batch,2.18.0,Tensor Parallel,24,8,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,546.51472,115.81421,121.49906,15.87224,17.21263,Batch,2.18.0,Tensor Parallel,24,8,4096,128,3968,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,333.24073,7115.97776,7231.01234,22.26758,23.81206,Batch,2.18.0,Tensor Parallel,24,8,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,Inf2.48xlarge,Text Generation,178.79017,5136.61623,5192.58666,21.6732,22.73154,Batch,2.18.0,Tensor Parallel,24,4,16384,8192,8192,FP16,Matmult-BF16,bf16\r\n" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_encoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (inference/second),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Sequence Length,Model Data Type,Compilation Autocast Data Type,OS Type\r\nalbert-base-v2,:benchmark-pt:`Benchmark `,PyTorch 2.7,Inf2.xlarge,Raw Output (AutoModel),3147.09984049,5.0675869,5.27883291,Batch,2.25.0,Data Parallel,8,128,FP32,Matmult-BF16,U22\r\nbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,Inf2.xlarge,Raw Output (AutoModel),2674.18956433,5.97381591,6.17100715,Batch,2.27.0,Data Parallel,8,128,FP32,Matmult-BF16,U22\r\nbert-large-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Raw Output (AutoModel),950.0496231,8.41140747,8.84652853,Batch,2.21.0,Data Parallel,4,128,FP32,Matmult-BF16,U22\r\ndistilbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,Inf2.xlarge,Raw Output (AutoModel),5307.87660777,6.01053237,6.23083114,Batch,2.27.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\ngoogle/electra-base-discriminator,:benchmark-pt:`Benchmark `,PyTorch 2.7,Inf2.xlarge,Raw Output (AutoModel),2889.75325068,11.02411747,11.97555304,Batch,2.25.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\nroberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.7,Inf2.xlarge,Raw Output (AutoModel),2920.37954741,5.42390347,5.82957506,Batch,2.25.0,Data Parallel,8,128,FP32,Matmult-BF16,U22\r\nroberta-large,:benchmark-pt:`Benchmark `,PyTorch 2.7,Inf2.xlarge,Raw Output (AutoModel),962.70185508,8.31007957,8.60977411,Batch,2.25.0,Data Parallel,4,128,FP32,Matmult-BF16,U22\r\nxlm-roberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.48xlarge,Raw Output (AutoModelForMaskedLM),51.13695938,625.66077709,694.93403673,Batch,2.22.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\n" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_encoder_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (tokens/second),Latency per Token P50 (ms),Latency per Token P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,DP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type\r\nt5-3b,`Tutorial `_,NeuronX Distributed,Inf2.24xlarge,Text Generation,111.92,8.97,8.98,Batch,2.17.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\ngoogle/flan-t5-xl,`Tutorial `_,NeuronX Distributed,Inf2.24xlarge,Text Generation,117.61,8.51,8.53,Batch,2.17.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\n" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_vision.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\ndeepmind/multimodal-perceiver,16x224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Multimodal Autoencoding,0.83,1250,1271,Real Time,2.18.0,Data Parallel,1,FP32,None\ndeepmind/vision-perceiver-learned,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,99.6,18.6,18.7,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\ndeepmind/vision-perceiver-fourier,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,67.9,29.5,29.68,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\ndeepmind/vision-perceiver-conv,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,126.5,14.14,14.2,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\ngoogle/vit-base-patch16-224,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,1632.359,4.716,5.902,Batch,2.14.0,Data Parallel,2,FP32,Matmult-BF16\nopenai/clip-vit-base-patch32,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,5178.833,48.973,57.002,Batch,2.14.0,Data Parallel,64,FP32,Matmult-BF16\nopenai/clip-vit-large-patch14,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,200.997,78.331,92.452,Batch,2.14.0,Data Parallel,4,FP32,Matmult-BF16\nresnet18,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,6635.04,4.80,4.88,Batch,2.14.0,Data Parallel,8,FP32,Matmult-BF16\nresnet34,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,4848.72,6.56,6.66,Batch,2.14.0,Data Parallel,8,FP32,Matmult-BF16\nresnet50,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,4269.12,7.49,7.55,Batch,2.14.0,Data Parallel,8,FP32,Matmult-BF16\nresnet101,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,3066.24,83.38,83.56,Batch,2.14.0,Data Parallel,64,FP32,Matmult-BF16\nresnet152,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,2323.20,110.06,110.21,Batch,2.14.0,Data Parallel,64,FP32,Matmult-BF16\nStable Diffusion 1.5,512x512,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.421,2369.6,2406.8,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion 2.1,512x512,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.549,1794.5,2103.7,Real Time,2.17.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\nStable Diffusion 2.1,768x768,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.188,5306.7,5368.6,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion 2 Inpainting,936x624,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.15,6701.4,6737.4,Real Time,2.17.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\nStable Diffusion XL Base,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Generation,0.073,13431.7,15739.0,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nStable Diffusion XL Base & Refiner,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.8xlarge,Image Generation,0.078,12651.9,15053.9,Real Time,2.17.0,Data Parallel,1,FP32,Matmult-BF16\nUNet,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Segmentation,866.96,18.37,18.86,Batch,2.14.0,Data Parallel,4,FP32,Matmult-BF16\nvgg11,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,3955.20,64.15,64.24,Batch,2.14.0,Data Parallel,64,FP32,Matmult-BF16\nvgg16,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,1964.16,16.27,16.35,Batch,2.14.0,Data Parallel,8,FP32,Matmult-BF16\n" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_vision_cnn.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\r\nresnet18,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,6949.174,4.587,4.659,Batch,2.21.0,Data Parallel,8,FP32,Matmult-BF16\r\nresnet34,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,5158.607,6.18,6.251,Batch,2.21.0,Data Parallel,8,FP32,Matmult-BF16\r\nresnet50,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,4393.304,7.283,7.331,Batch,2.21.0,Data Parallel,8,FP32,Matmult-BF16\r\nresnet101,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,3164.991,80.818,80.938,Batch,2.21.0,Data Parallel,64,FP32,Matmult-BF16\r\nresnet152,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,2449.875,104.406,104.531,Batch,2.21.0,Data Parallel,64,FP32,Matmult-BF16\r\nUNet,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Segmentation,1010.803,15.818,15.875,Batch,2.21.0,Data Parallel,4,FP32,Matmult-BF16\r\nvgg11,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,4734.402,54.044,54.09,Batch,2.21.0,Data Parallel,64,FP32,Matmult-BF16\r\nvgg16,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,2161.392,14.77,14.832,Batch,2.21.0,Data Parallel,8,FP32,Matmult-BF16" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_vision_dit.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\nPixArt Alpha,256x256,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,1.975,502.587,537.258,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\nPixArt Alpha,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,0.565,1769.756,1775.697,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\nPixArt Sigma,256x256,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,1.86,540.832,548.41,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\nPixArt Sigma,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.1,Inf2.xlarge,Image Generation,0.543,1841.882,1850.683,Real Time,2.20,Data Parallel,1,\"\"\"FP32, BF16\"\"\",Matmult-BF16\n" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_vision_sd.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\r\nStable Diffusion 1.5,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.494,2023.741,2031.705,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nStable Diffusion 2.1,512x512,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.596,1679.805,1685.442,Real Time,2.21.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\r\nStable Diffusion 2.1,768x768,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.187,5337.509,5357.361,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nStable Diffusion 2 Inpainting,936x624,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.133,7546.004,7550.984,Real Time,2.21.0,Data Parallel,1,\"FP32, BF16\",Matmult-BF16\r\nStable Diffusion XL Base,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Generation,0.083,12048.659,12102.431,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16\r\nStable Diffusion XL Base & Refiner,1024x1024,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.8xlarge,Image Generation,0.095,10546.45,10704.566,Real Time,2.21.0,Data Parallel,1,FP32,Matmult-BF16" }, { "path": "about-neuron/benchmarks/inf2/throughput_data_vision_transformers.csv", "content": "Model,Image Size,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Model Data Type,Compilation Autocast Data Type\r\ndeepmind/multimodal-perceiver,16x224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Multimodal Autoencoding,0.853,1170.045,1232.056,Real Time,2.21.0,Data Parallel,1,FP32,None\r\ndeepmind/vision-perceiver-learned,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,99.6,18.6,18.7,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\r\ndeepmind/vision-perceiver-fourier,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,67.9,29.5,29.68,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\r\ndeepmind/vision-perceiver-conv,224x224,:benchmark-pt:`Benchmark `,PyTorch 1.13.1,Inf2.xlarge,Image Classification,126.5,14.14,14.2,Real Time,2.18.0,Data Parallel,1,FP32,Matmult-BF16\r\ngoogle/vit-base-patch16-224,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,1955.406,4.087,4.125,Batch,2.21.0,Data Parallel,2,FP32,Matmult-BF16\r\nopenai/clip-vit-base-patch32,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,6509.83,135.806,136.003,Batch,2.21.0,Data Parallel,64,FP32,Matmult-BF16\r\nopenai/clip-vit-large-patch14,224x224,:benchmark-pt:`Benchmark `,PyTorch 2.5,Inf2.xlarge,Image Classification,285.938,113.117,115.940,Batch,2.21.0,Data Parallel,8,FP32,Matmult-BF16" }, { "path": "about-neuron/benchmarks/trn1/latency_data_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Output Token Throughput (tokens/sec),TTFT Latency P50 (ms),TTFT Latency P99 (ms),TPOT Latency P50 (ms),TPOT Latency P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type,Weight Storage Data Type\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,157.25202,17.09,21.62,7.03,7.16,Real Time,2.18.1,Tensor Parallel,32,1,8192,128,8064,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,140.50031,153.02,159.13,7.04,7.13,Real Time,2.18.1,Tensor Parallel,32,1,8192,4096,4096,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,178.18923,14.75,22.94,5.86,6,Real Time,2.18.1,Tensor Parallel,32,1,4096,128,3968,FP16,Matmult-BF16,int8\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,37.70379,547,553.89,26.2,26.79,Real Time,2.18.1,Tensor Parallel,32,1,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,40.63808,53.2,59.5,24.48,26.17,Real Time,2.18.1,Tensor Parallel,32,1,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,40.80995,52.53,52.79,26.48,24.22,Real Time,2.18.1,Tensor Parallel,32,1,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-7b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,161.7081305,13.32402229,14.1210556,6.69956207,6.84595108,Real Time,2.18.0,Tensor Parallel,32,1,8192,128,8064,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,60.43330245,864.1381264,865.9124374,9.84406471,10.14947891,Real Time,2.18.0,Tensor Parallel,32,1,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,31.3990051,2367.928505,2369.139671,13.40842247,15.76948166,Real Time,2.18.0,Tensor Parallel,32,1,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,39.28574,53.91026,54.9469,25.18129,26.58272,Real Time,2.18.0,Tensor Parallel,32,1,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,39.17668,81.882,98.77896,25.26712,25.7585,Real Time,2.18.0,Tensor Parallel,32,1,512,256,256,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,39.16379,57.75213,64.75568,25.44856,26.1333,Real Time,2.18.0,Tensor Parallel,32,1,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,38.09518,232.47981,239.02893,26.03793,26.17574,Real Time,2.18.0,Tensor Parallel,32,1,2048,1024,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,37.70947,236.78207,241.14895,26.62468,27.02999,Real Time,2.18.0,Tensor Parallel,32,1,3072,1024,2048,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,36.78021,690.95588,695.91761,26.85046,27.04263,Real Time,2.18.0,Tensor Parallel,32,1,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,49.55890938,1322.874308,1325.857162,9.89246368,10.18333435,Real Time,2.18.0,Tensor Parallel,32,1,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,60.21552741,868.635416,870.9816933,9.86456871,10.24436951,Real Time,2.18.0,Tensor Parallel,32,1,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,31.37781421,2372.928381,2375.921965,13.3998394,13.79013062,Real Time,2.18.0,Tensor Parallel,32,1,16384,8192,8192,FP16,Matmult-BF16,bf16\r\n" }, { "path": "about-neuron/benchmarks/trn1/latency_data_encoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Sequence Length,Model Data Type,Compilation Autocast Data Type,OS Type\r\nalbert-base-v2,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),2321.97758889,0.85997581,0.9086132,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.8,trn1.2xlarge,Raw Output (AutoModel),2085.45272427,0.94294548,1.02853775,Real Time,2.26.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nbert-large-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),747.48212826,2.66885757,2.73442268,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\ndistilbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),3672.38478861,0.54264069,0.58531761,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\ngoogle/electra-base-discriminator,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),2127.07474023,0.93317032,0.9958744,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nroberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),2094.37288172,0.95796585,1.00588799,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nroberta-large,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),747.58300171,2.66981125,2.73323059,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\nxlm-roberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.32xlarge,Raw Output (AutoModelForMaskedLM),46.89836990,42.62268543,44.11746978,Real Time,2.27.0,Data Parallel,1,128,FP32,Matmult-BF16,U22\r\n" }, { "path": "about-neuron/benchmarks/trn1/latency_data_encoder_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (tokens/second),Latency per Token P50 (ms),Latency per Token P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,DP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type\r\nt5-3b,`Tutorial `_,NeuronX Distributed,trn1.32xlarge,Text Generation,110.23,9.07,9.12,Real Time,2.18.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\ngoogle/flan-t5-xl,`Tutorial `_,NeuronX Distributed,trn1.32xlarge,Text Generation,120.29,8.31,8.34,Real Time,2.18.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\n" }, { "path": "about-neuron/benchmarks/trn1/throughput_data_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Output Token Throughput (tokens/sec),TTFT Latency P50 (ms),TTFT Latency P99 (ms),TPOT Latency P50 (ms),TPOT Latency P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type,Weight Storage Data Type\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,933.50053,55.16,61.47,9.95,10.1,Batch,2.18.1,Tensor Parallel,32,8,8192,128,8064,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,770.16291,1265.95,1292.94,10.04,10.33,Batch,2.18.1,Tensor Parallel,32,8,8192,4096,4096,FP16,Matmult-BF16,int8\r\nLlama-3-8B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,1142.69582,49.05,52.79,7.65,7.94,Batch,2.18.1,Tensor Parallel,32,8,4096,128,3968,FP16,Matmult-BF16,int8\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,120.3614,1661.12,1672.71,32.33,33.27,Batch,2.18.1,Tensor Parallel,32,4,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,140.51039,129.86,132.03,28.38,29.11,Batch,2.18.1,Tensor Parallel,32,4,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-3-70B,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,138.01357,130.37,130.48,28.08,28.53,Batch,2.18.1,Tensor Parallel,32,4,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-7b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,917.2452652,66.4024353,70.63961029,10.09511948,10.46204567,Batch,2.18.0,Tensor Parallel,32,8,8192,128,8064,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,371.7031,6668.70475,6689.8005,19.85741,21.0557,Batch,2.18.0,Tensor Parallel,32,8,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nLlama-2-13b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,184.28337,4628.44729,4635.24675,21.09194,22.3856,Batch,2.18.0,Tensor Parallel,32,4,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,141.45357,156.84581,158.41317,26.72362,30.16973,Batch,2.18.0,Tensor Parallel,32,4,256,128,128,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,143.42503,270.15853,270.55573,26.9084,27.90999,Batch,2.18.0,Tensor Parallel,32,4,512,256,256,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,145.12799,156.68869,161.41367,27.21453,30.60174,Batch,2.18.0,Tensor Parallel,32,4,1152,128,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,133.25056,1478.64008,1479.77638,28.55039,29.49882,Batch,2.18.0,Tensor Parallel,32,4,2048,1024,1024,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,129.27628,1478.84846,1482.93161,31.67439,32.01842,Batch,2.18.0,Tensor Parallel,32,4,3072,1024,2048,FP16,Matmult-BF16,bf16\r\nLlama-2-70b,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,120.62953,2722.03422,2730.95036,31.78978,33.2315,Batch,2.18.0,Tensor Parallel,32,4,4096,2048,2048,FP16,Matmult-BF16,bf16\r\nMistral-7B-Instruct-v0.2,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,484.5773,8614.85291,8630.24068,15.43713,15.9421,Batch,2.18.0,Tensor Parallel,32,8,16384,8192,8192,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,370.97736,6625.1595,6628.26467,19.91653,20.94936,Batch,2.18.0,Tensor Parallel,32,8,8192,4096,4096,FP16,Matmult-BF16,bf16\r\nCodeLlama-13b-hf,:llama-sample:`Sample `,Transformers NeuronX,trn1.32xlarge,Text Generation,184.17898,4626.17469,4630.66864,21.09528,22.16578,Batch,2.18.0,Tensor Parallel,32,4,16384,8192,8192,FP16,Matmult-BF16,bf16\r\n" }, { "path": "about-neuron/benchmarks/trn1/throughput_data_encoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (inference/sec),Latency P50 (ms),Latency P99 (ms),Application Type,Neuron Version,Run Mode,Batch Size,Sequence Length,Model Data Type,Compilation Autocast Data Type,OS Type\r\nalbert-base-v2,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),3442.53392946,9.28854942,9.35173273,Batch,2.27.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\nbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),3421.56625089,9.34481621,9.41992044,Batch,2.27.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\nbert-large-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),1104.43610458,7.24101067,7.29799271,Batch,2.27.0,Data Parallel,4,128,FP32,Matmult-BF16,U22\r\ndistilbert-base-uncased,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),6369.44180331,5.00988960,5.09214401,Batch,2.28.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\ngoogle/electra-base-discriminator,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),3425.55803570,9.32765007,9.45640087,Batch,2.28.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\nroberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),3378.10764201,9.46044921,9.53317165,Batch,2.28.0,Data Parallel,16,128,FP32,Matmult-BF16,U22\r\nroberta-large,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.2xlarge,Raw Output (AutoModel),1123.90475943,14.23048973,14.30106163,Batch,2.27.0,Data Parallel,8,128,FP32,Matmult-BF16,U22\r\nxlm-roberta-base,:benchmark-pt:`Benchmark `,PyTorch 2.9,trn1.32xlarge,Raw Output (AutoModelForMaskedLM),46.68898543,342.50581264,350.86465597,Batch,2.27.0,Data Parallel,8,128,FP32,Matmult-BF16,U22\r\n" }, { "path": "about-neuron/benchmarks/trn1/throughput_data_encoder_decoder.csv", "content": "Model,Scripts,Framework,Inst. Type,Task,Throughput (tokens/second),Latency per Token P50 (ms),Latency per Token P99 (ms),Application Type,Neuron Version,Run Mode,TP Degree,DP Degree,Batch Size,Sequence Length,Input Length,Output Length,Model Data Type,Compilation Autocast Data Type\r\nt5-3b,`Tutorial `_,NeuronX Distributed,trn1.32xlarge,Text Generation,116.29,8.58,8.66,Batch,2.17.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\ngoogle/flan-t5-xl,`Tutorial `_,NeuronX Distributed,trn1.32xlarge,Text Generation,122.52,8.16,8.19,Batch,2.17.0,Tensor Parallel,8,1,1,128,128,84,FP32,Matmult-BF16\r\n" }, { "path": "about-neuron/benchmarks/trn1/training_data_decoder.csv", "content": "Model,Instance-Type,Training Data-Type,Nodes,Topology,Microbatch,Globalbatch, Optimizer, Sequence Length, Performance [seq/sec],Strong/Weak Scaling,Neuron Version,Neuron Tutorial/Example,Pytorch Neuron(torch-neuronx) Version, OS Type.\r\nLlama-3.1-8B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+FP32Optimizer,32,TP=32 DP=32 PP=1 ZeRO-1,1,1024,AdamW,8192,47.95,strong scaling,2.24.0,`NeuronX Distributed `_,2.7.0.2.8.6896,U22\r\nLlama-3.1-70B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+FP32Optimizer,32,TP=32 DP=4 PP=8,1,1024,AdamW,8192,7.94,strong scaling,2.24.0,`NeuronX Distributed `_,2.7.0.2.8.6896,U22\r\n" }, { "path": "about-neuron/benchmarks/trn1/training_data_encoder.csv", "content": "Model,Instance-Type,Training Data-Type,Nodes,Topology,Microbatch,Globalbatch, Optimizer, Sequence Length, Performance [seq/sec],Strong/Weak Scaling,Neuron Version,Neuron Tutorial/Example,Pytorch Neuron(torch-neuronx) Version, OS Type.\r\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,16,[32xNC(DP)] x 16Nodes(DP),16,1048576,Lamb,128,57407.9207,weak scaling,2.28.0,:ref:`hf-bert-pretraining-tutorial`,2.9.0.2.12.21983, U22\r\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,FP32,16,[32xNC(DP)] x 16Nodes(DP),8,1048576,Lamb,128,32362.6714,weak scaling,2.28.0,:ref:`hf-bert-pretraining-tutorial`,2.9.0.2.12.21983, U22\r\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],16,16384,AdamW,128,3826.6103,strong scaling,2.28.0,:ref:`hf-bert-pretraining-tutorial`,2.9.0.2.12.21983, U22\r\n\r\n" }, { "path": "about-neuron/benchmarks/trn1/training_data_vision_transformers.csv", "content": "Model,Instance-Type,Training Data-Type,Nodes,Topology,Microbatch,Globalbatch, Optimizer, Performance [seq/sec],Strong/Weak Scaling,Neuron Version,Neuron Tutorial/Example,Pytorch Neuron(torch-neuronx) Version, OS Type.\r\nHuggingFace ViT-Base fine-tuning,trn1.32xlarge/trn1n.32xlarge,BF16,1,[32xNC(DP)],64,2048,AdamW,6587.25,weak scaling,2.25.0,`ViT-Base Fine-tuning Example `_,2.7.0.2.9.0, U22\r\n" }, { "path": "about-neuron/benchmarks/trn1/trn1-inference-performance.rst", "content": ".. _trn1-inference-performance:\n\nTrn1/Trn1n Inference Performance\n================================\n\n.. important::\n\n The benchmark scripts linked on this page are provided for historical reference only and are not tested with recent versions of the Neuron SDK. They have been moved to the `archive folder `_.\n\n.. contents:: Table of contents\n :local:\n\n\n*Last update: Feb 26th, 2026*\n\n\n.. _NLP:\n\nEncoder Models\n--------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_encoder.csv')\n df_prices = pd.read_csv('trn1_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size','Sequence Length', 'Model Data Type','Compilation Autocast Data Type','OS Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_encoder.csv')\n df_prices = pd.read_csv('trn1_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (inference/sec)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (inference/sec)', 'Latency P50 (ms)', 'Latency P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'Batch Size','Sequence Length', 'Model Data Type','Compilation Autocast Data Type','OS Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n\n df['Throughput (inference/sec)'] = df['Throughput (inference/sec)'].round(2).astype('float',copy=True)\n int_cols = ['Latency P50 (ms)', 'Latency P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\nEncoder-Decoder Models\n----------------------\n\n.. tab-set::\n\n .. tab-item:: Throughput optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('throughput_data_encoder_decoder.csv')\n df_prices = pd.read_csv('trn1_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (tokens/second)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (tokens/second)', 'Latency per Token P50 (ms)', 'Latency per Token P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'TP Degree', 'DP Degree', 'Batch Size', 'Sequence Length', 'Input Length', 'Output Length', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n df['Throughput (tokens/second)'] = df['Throughput (tokens/second)'].round(2).astype('float',copy=True)\n int_cols = ['Latency per Token P50 (ms)', 'Latency per Token P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n Only for Encoder-Decoder\n\n **Throughput (tokens/second)** counts both input and output tokens\n\n **Latency per Token** counts both input and output tokens\n\n Applicable to all models\n\n **Cost per 1M inferences** is calculated using RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n\n\n .. tab-item:: Latency optimized\n\n .. df-table::\n :header-rows: 1\n\n df = pd.read_csv('latency_data_encoder_decoder.csv')\n df_prices = pd.read_csv('trn1_instance_prices.csv')\n df = pd.merge(df,df_prices,on='Inst. Type')\n df['Cost per 1M inferences'] = ((1.0e6 / df['Throughput (tokens/second)']) * (df['RI-Effective hourly rate'] / 3.6e3 )).map('${:,.3f}'.format)\n cols_to_show = ['Model','Scripts','Framework', 'Inst. Type', 'Task', 'Throughput (tokens/second)', 'Latency per Token P50 (ms)', 'Latency per Token P99 (ms)', 'Cost per 1M inferences', 'Application Type', 'Neuron Version', 'Run Mode', 'TP Degree', 'DP Degree', 'Batch Size', 'Sequence Length', 'Input Length', 'Output Length', 'Model Data Type','Compilation Autocast Data Type']\n df = df[cols_to_show].sort_values(['Model', 'Cost per 1M inferences'])\n df['Throughput (tokens/second)'] = df['Throughput (tokens/second)'].round(2).astype('float',copy=True)\n int_cols = ['Latency per Token P50 (ms)', 'Latency per Token P99 (ms)']\n df[int_cols] = df[int_cols].round(2).astype('float',copy=True)\n\n .. note::\n\n Only for Encoder-Decoder\n\n **Throughput (tokens/second)** counts both input and output tokens\n\n **Latency per Token** counts both input and output tokens\n\n\n .. note::\n\n **Cost per 1M inferences** is calculated using RI-Effective hourly rate.\n\n **Real Time** application refers to batch size 1 inference for minimal latency. **Batch** application refers to maximum throughput with minimum cost-per-inference.\n" }, { "path": "about-neuron/benchmarks/trn1/trn1-training-performance.rst", "content": ".. _trn1-training-performance:\n\nTrn1/Trn1n Training Performance\n===============================\n\nThis section provides benchmark results for training various deep learning models on AWS Trn1 and Trn1n instances powered by AWS Trainium chips. The benchmarks cover a range of model architectures, including encoder models, decoder models, and vision transformers, demonstrating the performance capabilities of Trn1/Trn1n instances for different training workloads.\n\n**Last update: February 19th, 2026**\n\n.. contents:: Table of contents\n :local:\n\n.. _NLP:\n\nEncoder Models\n--------------\n\n.. csv-table::\n :file: training_data_encoder.csv\n :header-rows: 1\n\n\nDecoder Models\n--------------\n\n.. csv-table::\n :file: training_data_decoder.csv\n :header-rows: 1\n\n.. note::\n **TP (Tensor Parallel), PP (Pipeline Parallel) and DP (Data Parallel) Topology** configuration refers to the degrees of 3D Parallelism (How the model and data is sharded across NeuronCores).\n\n TP and PP are specified in the run script and DP is calculated by dividing **world size** (Number of nodes/instances * Number of neuron cores per instance) by TP * PP degrees.\n\n For example, ``TP = 4, PP = 4`` and`` Number of instances is 32 (trn1.32xlarge)``. The world size will be: ``32(num instances) * 32(Neuron Cores per instance) = 1024``. Now, ``DP degree = 1024 (World size)/ 4 (TP) * 4 (PP) = 64``\n\n For more information on batch sizes please refer to :ref:`neuron-batching`\n\n\nVision Transformer Models\n--------------------------\n\n.. csv-table::\n :file: training_data_vision_transformers.csv\n :header-rows: 1\n\n.. note::\n Read more about strong vs weak scaling here :ref:`neuron-training-faq`\n" }, { "path": "about-neuron/benchmarks/trn1/trn1_instance_prices.csv", "content": "Inst. Type,RI-Effective hourly rate\ntrn1.2xlarge,0.512\ntrn1.32xlarge,8.197\n" }, { "path": "about-neuron/benchmarks/trn1/trn1_trn1n_nlp_data.csv", "content": "Model,Instance-Type,Training Data-Type,Nodes,Topology,Microbatch,Globalbatch, Optimizer, Performance [seq/sec],MFU[%],ComputeCostPerToken(Tflops),Strong/Weak Scaling,Neuron Version,Neuron Tutorial/Example,Pytorch Neuron(torch-neuronx) Version, OS Type.\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,16,[32xNC(DP)] x 16Nodes(DP),16,1048576,Lamb,53069,25.83,,weak scaling,2.15.0,:ref:`hf-bert-pretraining-tutorial`,1.13.1.1.12.0, U20\nHuggingFace BERT-Large Ph2 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,16,[32xNC(DP)] x 16Nodes(DP),2,524288,Lamb,7507,15.5,,weak scaling,2.15.0,:ref:`hf-bert-pretraining-tutorial`,1.13.1.1.12.0, U20\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16/AMP,16,[32xNC(DP)] x 16Nodes(DP),16,16384,AdamW,24518.47,,,strong scaling,2.14.0,:ref:`hf-bert-pretraining-tutorial`,1.13.1.1.11.0, U20\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,FP32,16,[32xNC(DP)] x 16Nodes(DP),8,1048576,Lamb,28432,13.83,,weak scaling,2.14.0,:ref:`hf-bert-pretraining-tutorial`,1.13.1.1.12.0, U20\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],16,16384,AdamW,3530,27.49,,strong scaling,2.15.0,:ref:`hf-bert-pretraining-tutorial`,1.13.1.1.12.0, U20\nHuggingFace BERT-Large Ph1 pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],16,65536,Lamb,3733,29.07,,strong scaling,2.15.0,:ref:`hf-bert-pretraining-tutorial`,1.13.1.1.12.0, U20\nGPT3-23B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,32,TP=8 DP=32 PP=4,1,1024,AdamW,100,29.65,289,strong scaling,2.15.0,`nemo-megatron `_,1.13.1.1.12.0, U20\nGPT3-46B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,32,TP=8 DP=16 PP=8,1,1024,AdamW,47.2,27.7,578,strong scaling,2.15.0,`nemo-megatron `_,1.13.1.1.12.0, U20\nGPT3-175B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,32,TP=32 DP=4 PP=8,1,1024,AdamW,12.7,33.14,2197,strong scaling,2.13.0,`nemo-megatron `_,1.13.1.1.10.0, U20\nLlama2-7B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,32,TP=8 DP=4 PP=4,1,1024,AdamW,82,14.8,336,strong scaling,2.15.0,`nemo-megatron `_,1.13.1.1.12.0, U20\nLlama2-13B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,32,TP=8 DP=4 PP=4,1,1024,AdamW,60,20.7,336,strong scaling,2.15.0,`nemo-megatron `_,1.13.1.1.12.0, U20\nLlama2-7B pre-training,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+FP32Optimizer,16,TP=8 DP=64,1,1024,AdamW,81,30.8,,strong scaling,2.15.0,`neuronx-distributed `_,1.13.1.1.12.0, U20\nHuggingFace ViT-Base fine-tuning,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],64,2048,AdamW,5232.78,,,weak scaling,2.17.0,`ViT-Base Fine-tuning Example `_,1.13.1.1.13.0, U20\nHuggingFace CLIP-Base fine-tuning,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],80,2560,AdamW,5152.76,,,weak scaling,2.17.0,`CLIP-Base Fine-tuning `_,1.13.1.1.13.0, U20\nHuggingFace Vision-Perveriver-Conv fine-tuning,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],4,128,AdamW,423.32,,,weak scaling,2.17.0,`Vision Perceiver Conv Fine-tuning `_,1.13.1.1.13.1, U20\nHuggingFace Language-Perveriver fine-tuning,trn1.32xlarge/trn1n.32xlarge,Autocast:BF16+SR,1,[32xNC(DP)],20,640,AdamW,1407.02,,,weak scaling,2.17.0,`Language Perceiver Fine-tuning `_,1.13.1.1.13.1, U20\n" }, { "path": "about-neuron/beta-participation.rst", "content": ".. meta::\n :description: Information about participating in the AWS Neuron SDK beta program.\n :date-modified: 12/19/2025\n\nParticipate in the AWS Neuron SDK Beta Program\n===============================================\n\nAWS Neuron SDK users can participate in our beta program to get early access to new features and improvements. By joining the beta program, you can provide valuable feedback that helps us enhance the AWS Neuron SDK for everyone.\n\nCurrently, we are taking requests to join our Beta program for the new Neuron Kernel Interface and its associated features. If you are interested in participating, `fill out this online form `__ and we'll get back to you! Read more about the new NKI features `here `__.\n\n.. admonition:: Disclaimer\n\n Beta features are not recommended for production workloads. They may contain bugs or incomplete functionality. Use them at your own risk and provide feedback to help us improve." }, { "path": "about-neuron/calculator/neuron-calculator.rst", "content": ".. _neuron_calculator:\n\nNeuron Calculator\n=================\n\n.. raw:: html\n\n \n\n \n\n\n \n\n \n\n\n\n

\n\n
\n \n
\n\n \n\n
\n
\n\n\n
\n\n

Number of NeuronCores needed for LLM Inference

\n Please enter model configurations (You can enter multiple values of each hyperparameter. Press enter after adding each value in the text field) \n\n \n
\n \n
\n\n \n
\n \n
\n\n\n
\n \n
\n \n
\n
\n\n\n
\n \n
\n \n
\n
\n\n\n
\n \n
\n \n
\n
\n\n \n\n
\n \n
\n \n
\n
\n\n\n
\n \n
\n \n
\n
\n\n
\n \n
\n \n
\n
\n\n\n
\n \n
\n \n
\n
\n \n \n
\n
\n \n \n
\n
\n\n\n\n
Invalid model configurations entered. Each text field accepts multiple values. Please press Enter after adding a new value to the text field.
\n\n\n
\n
\n
\n \n
\n
\n
\n\n\n\n
\n \n \n\n \n
\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Batch Size Max Seq Length Embedding Dimension Num Attention Heads Num Layers Memory Footprint (GB) TP Degree(NeuronCores) Instances Recommended
\n\n\n \n\n
\n \n
\n
\n \n \n\n \n\n
\n
\n \n
\n\n\n \n\n.. raw:: html\n\n \n \n\n" }, { "path": "about-neuron/faq/contributing-faq.rst", "content": ".. _contribute-faq:\n\nContributing Guidelines FAQs\n============================\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nWhether it's\na bug report, new feature, correction, or additional documentation, we\ngreatly value feedback and contributions from our community.\n\nPlease read through this document before submitting any issues or pull\nrequests to ensure we have all the necessary information to effectively\nrespond to your bug report or contribution.\n\nHow to reporting Bugs/Feature Requests\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe welcome you to use the GitHub issue tracker to report bugs or suggest\nfeatures.\n\nWhen filing an issue, please check existing open, or recently closed,\nissues to make sure somebody else hasn't already reported the issue.\nPlease try to include as much information as you can. Details like these\nare incredibly useful:\n\n- A reproducible test case or series of steps\n- The version of our code being used\n- Any modifications you've made relevant to the bug\n- Anything unusual about your environment or deployment\n\nContributing via Pull Requests\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nContributions via pull requests are much appreciated. Before sending us\na pull request, please ensure that:\n\n1. You are working against the latest source on the *master* branch.\n2. You check existing open, and recently merged, pull requests to make\n sure someone else hasn't addressed the problem already.\n3. You open an issue to discuss any significant work - we would hate for\n your time to be wasted.\n\nTo send us a pull request, please:\n\n1. Fork the repository.\n2. Modify the source; please focus on the specific change you are\n contributing. If you also reformat all the code, it will be hard for\n us to focus on your change.\n3. Ensure local tests pass.\n4. Commit to your fork using clear commit messages.\n5. Send us a pull request, answering any default questions in the pull\n request interface.\n6. Pay attention to any automated CI failures reported in the pull\n request, and stay involved in the conversation.\n\nGitHub provides additional document on `forking a\nrepository `__ and\n`creating a pull\nrequest `__.\n\nHow to find contributions to work on\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLooking at the existing issues is a great way to find something to\ncontribute on. As our projects, by default, use the default GitHub issue\nlabels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix),\nlooking at any 'help wanted' issues is a great place to start.\n\nWhat is the code of conduct\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis project has adopted the `Amazon Open Source Code of\nConduct `__. For more information\nsee the `Code of Conduct\nFAQ `__ or contact\nopensource-codeofconduct@amazon.com with any additional questions or\ncomments.\n\nHow to notify for a security issue\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf you discover a potential security issue in this project we ask that\nyou notify AWS/Amazon Security via our `vulnerability reporting\npage `__.\nPlease do **not** create a public github issue.\n\nWhat is the licensing\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nSee the `link `_ \nand `link `_ files\nfor our project's licensing. We will ask you to confirm the licensing of\nyour contribution.\n\nWe may ask you to sign a `Contributor License Agreement\n(CLA) `__\nfor larger changes.\n" }, { "path": "about-neuron/faq/index.rst", "content": ".. _neuron_faq:\n\nOther Neuron FAQs\n=================\n\nFrequently asked questions about AWS Neuron SDK, covering general topics, inference, training, ONNX support, and contributing guidelines. \n\n.. note::\n This content may not be be up to date as of 2026, and often pertains to older or now-unsupported platforms and components.\n\nGeneral FAQs\n-------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: neuron2-intro-faq\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron 2.x Introduction FAQ**\n ^^^\n Common questions about Neuron 2.x and Trn1 general availability\n\n .. grid-item-card::\n :link: onnx-faq\n :link-type: doc\n :class-card: sd-border-1\n\n **ONNX FAQ**\n ^^^\n Using ONNX models with AWS Neuron\n\n .. grid-item-card::\n :link: contributing-faq\n :link-type: doc\n :class-card: sd-border-1\n\n **Contributing Guidelines FAQ**\n ^^^\n How to report bugs, request features, and contribute to Neuron\n\nInference FAQs\n---------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: inference/neuron-faq\n :link-type: doc\n :class-card: sd-border-1\n\n **Inference with Neuron FAQ**\n ^^^\n Common questions about running inference workloads on AWS Neuron\n\n .. grid-item-card::\n :link: inference/trouble-shooting-faq\n :link-type: doc\n :class-card: sd-border-1\n\n **Troubleshooting for Inf1 FAQ**\n ^^^\n Debugging and troubleshooting inference issues on Inf1 instances\n\nTraining FAQs\n-------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: training/neuron-training\n :link-type: doc\n :class-card: sd-border-1\n\n **Training with Neuron FAQ**\n ^^^\n Common questions about training models on Trainium instances\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Neuron 2.x Introduction FAQ \n ONNX FAQ \n Contributing Guidelines FAQ \n Inference with Neuron FAQ \n Troubleshooting for Inf1 FAQ \n Training with Neuron FAQ \n" }, { "path": "about-neuron/faq/inference/neuron-faq.rst", "content": ".. _neuron-f1-faq:\n\nInference with Neuron - FAQ\n---------------------------\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nWhat ML model types and operators are supported by AWS Neuron?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAWS Neuron includes a compiler that converts your trained machine\nlearning models to a binary object for execution. The Neuron\ncompiler supports many commonly used machine learning operators used in computer vision, natural language processing, recommender engines and more. A list of supported ML operators and supported inputs are in :ref:`neuron-supported-operators` .\n\nIt's important to mention that to get good performance doesn't require all of the model operators to run on the chip. In many cases, some of the operators will continue to run on the instance CPUs, like the case of embeddings or image pre-processing, and will still provide a compelling end to end performance. We call this approach auto-partitioning, where the Neuron compiler optimizes the model execution based on operators that are most suitable to run on the CPU or the chip.\n\nFor the latest model architecture support, please refer to the model architecture fit and performance pages.\n\nWhy is a compiler needed, and how do I use it?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe Neuron compiler converts a model from a framework level Neural Network\ngraph, with operators like convolution and pooling, into a\nNeuron Device-specific instruction set, builds the schedule for\nexecution of these instructions, and converts the model parameters into\nformat that the neuron device can consume. The supported input formats include\nTensorFlow, PyTorch, and MXNet. The output from the\ncompiler is a Neuron Executable File Format (NEFF) artifact. The NEFF\ncontains a combination of binary code, the model parameters, and\nadditional meta-data needed by the Neuron runtime and profiler.\n\nI am using a ML framework today – what will change for me to use this?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo use Inferentia within the Inf1 instances, the developer needs to perform one-time compilation\nof the pre-trained model to generate a NEFF, and use this as the inference\nmodel in fleet of Inf1 instances.\n\n- :doc:`TensorFlow Neuron `\n- :ref:`neuron-pytorch`\n- :ref:`neuron-mxnet`\n\nWhat is a NeuronCore Pipeline? How do I take advantage of it?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nA NeuronCore Pipeline is a unique technique to shard a specific Neural\nNetwork across multiple NeuronCores, to take advantage of the large\non-chip cache instead of moving data in and out of external memory. The result is an increased throughput and reduce latency\ntypically important for real-time inference applications. All Inf1 instances support it, and the Inf1\ninstances with multiple Inferentia accelerators, such as inf1.6xlarge or\ninf1.24xlarge support it thanks to the fast chip-to-chip interconnect.\n\nDevelopers can choose to use NeuronCore Pipeline mode during compile\nstage, with an opt-in flag. :ref:`neuron-cc` provides further details.\n\nNeuronCores, NeuronCore Groups and NeuronCore Pipelines: What do they do?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEach Inferentia chip has four compute engines called NeuronCores. A\nNeuronCore Group is a way to aggregate NeuronCores to increase hardware\nutilization and assign models with the right compute sizing for a\nspecific application. If you want to run multiple models in parallel,\nyou can assign different models to separate NeuronCore Groups. A model\ncompiled to use multiple NeuronCores in a NeuronCore Pipeline can be\nassigned to a NeuronCore Group with enough NeuronCores to load into.\nFinally- it is also possible for sets of Inferentia devices to be mapped\nto separate Neuron Runtimes. :ref:`neuron-features-index` section has more\ninformation and examples.\n\nCan I use TensorFlow networks from tfhub.dev as-is ? if not, what should I do?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nYes. Models format can be imported into TensorFlow, either as a standard\nmodel-server, in which case it appears as a simple command line utility,\nor via the Python based TensorFlow environment. The primary additional\nstep needed is to compile the model into Inferentia NEFF format.\n" }, { "path": "about-neuron/faq/inference/trouble-shooting-faq.rst", "content": ".. _trouble-shooting-inf1-faq:\n\nTroubleshooting for Inf1 - FAQ\n==============================\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\nPerformance is not what I expect it to be, what's the next step?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPlease check our performance optimization section on performance\ntuning and other notes on how to use pipelining and batching to improve\nperformance.\n\nDo I need to worry about size of model and size of inferentia memory? what problems can I expect to have?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nErrors like this will be logged and can be found as shown\n:ref:`neuron_gatherinfo`.\n\nHow can I debug / profile my inference request?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSee :ref:`neuron-plugin-tensorboard`\n\n\nHow to report Bug/Feature Requests\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe welcome you to use the Neuron GitHub issue tracker to report bugs or suggest\nfeatures.\n\nWhen filing an issue, please check existing open, or recently closed,\nissues to make sure somebody else hasn't already reported the issue.\nPlease try to include as much information as you can. Details like these\nare incredibly useful:\n\n- A reproducible test case or series of steps\n- The version of our code being used\n- Any modifications you've made relevant to the bug\n- Anything unusual about your environment or deployment\n" }, { "path": "about-neuron/faq/neuron2-intro-faq.rst", "content": ".. _neuron2-intro-faq:\n\nNeuron 2.x Introduction at Trn1 GA - FAQ\n----------------------------------------\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n.. include:: /release-notes/templates/n2.x-trn1-ga-faq.txt\n" }, { "path": "about-neuron/faq/onnx-faq.rst", "content": ".. _onnx-faq:\n\nONNX FAQ\n---------\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\nCan I use ONNX models with Neuron ? If not, what should I do?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAWS Neuron does not directly support compilation of models in the ONNX file format. The recommended way to compile a model that is in the ONNX file format is to first convert the model to PyTorch using a publicly available tool\nlike `onnx2pytorch `_ . Once the ONNX model is converted to PyTorch, it can then be compiled with the :func:`torch_neuron.trace` function to produce a model that can run on Neuron.\n\n\n" }, { "path": "about-neuron/faq/roadmap-faq.rst", "content": ".. _neuron_roadmap_faq:\n\nRoadmap FAQ\n===========\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\nWhy did you build this?\n~~~~~~~~~~~~~~~~~~~~~~~\n\nA: We know that our customers are making decisions and plans based on\nwhat we are developing, and we want to provide them with the right\nvisibility to what we are working on, as well as the opportunity to\nprovide direct feedback.\n\nWhat do the roadmap categories mean?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- **Roadmap Requests** - Requests we received and we are considering to add to the roadmap, \n this is a great phase to give us feedback and let us know if you need this feature as well.\n- **Working on it** - In progress, we might still be\n working through the implementation details, or scoping stuff out.\n This is a great phase to give us feedback as to how you want to see\n something implemented. We’ll benefit from your specific use cases\n here.\n- **Completed** - Feature complete and supported by Neuron.\n\n\nWhy are there no dates on your roadmap?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA: We are not providing exact target dates for releases because we\nprioritize operational excellence, security and quality over hitting a\nspecific date. If you have an urgent need for a feature, please contact\nus directly at aws-neuron-support@amazon.com.\n\nIs everything on the roadmap?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA: We are focusing on upgrades for existing features, as well as\nbuilding new features. We will keep adding features and capabilities to\nthis roadmap as time progresses.\n\nHow can I provide feedback or ask for more information?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA: When in doubt, please create an issue or post a question on the `AWS\nNeuron support\nforum `__.\n\nHow can I request a feature be added to the roadmap?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA: We encourage you to open an issue. All community-submitted issues\nwill be reviewed by the roadmap maintainers.\n\nCan I \"+1\" existing issues?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA:We strongly encourage you to do so, as it helps us understand which\nissues will have the widest impact. You can navigate to the issue\ndetails page and add a reaction (thumbs up). There are six types of\nreactions supported (thumbs down “-1”, confused, heart, watching, laugh,\nhooray, and thumbs up +1)." }, { "path": "about-neuron/faq/training/neuron-training.rst", "content": ".. _neuron-training-faq:\n\nTraining with Neuron - FAQ\n==========================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nCompute\n-------\n\nHow do I get started with training my model on Trn1?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOnce you select your machine learning framework, you can get started here: :ref:`docs-quick-links`\n\n\nHow do I setup EFA for multi-node training?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFor setting up EFA that is needed for multi-node training, please see :ref:`setup-trn1-multi-node-execution`\n\n\nHow do I know if I can train my models with Trainium?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWe aim to support a broad set of models and distribution libraries. We continuously add more capabilities and enable new features via Neuron SDK releases and suggest you will follow our public roadmap and join our slack and email lists.\n\nHow should I size Trainium NeuronCores vs GPUs?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor simplicity, you should consider each NeuronCore within your instances as an independent deep learning compute engine, the equivalent of a GPU. As point of comparison, a trn1.32xlarge has 32 NeuronCores, and their max performance is 40% higher than of P4d for BF16/FP16/FP8, 2.5X faster for TF32, and 5X faster for FP32. Each NeuronCore is independent and connected to the rest of the NeuronCores within the instance via NeuronLink, and across instances with EFA. Each NeuronCore has also full access to the accelerator memory in the instance, which helps scale large models across NeuronCores using various collective compute ops techniques. \n\nWhat are the time to train advantages of Trn1?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhile the answer is largely model dependent, training performance on Trn1 is fast due thanks for multiple system wide optimizations working in concert. Dependent on the data type, you should expect between 1.4-5X higher throughput on Trn1 as compared to the latest GPUs instances (P4d). For distributed workloads, 800Gbps EFA gives customers lower latency, and 2x the throughput as compared to P4d. (a Trn1n 1.6Tb option is coming soon). Each Trainium also has a dedicated collective compute (CC) engine, which enables running the CC ops in parallel to the NeuronCores compute. This enables another 10-15% acceleration of the overall workload. Finally, stochastic rounding enables running at half precision speeds (BF16) while maintaining accuracy at near full precision, this is not only simplifying model development (no need for mixed precision) it also helps the loss function converge faster and reduce memory footprint.\n\nWhat are some of the training performance results for Trn1?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThey are great! please refer to the :ref:`benchmark` page for open-source model performance results. We encourage you to try it for your own models/application.\n\nCan I use CUDA libraries with AWS Trainium?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAWS Trainium and Neuron are plugged into popular frameworks, and is automatically optimizing model deployment on Neuron devices like Inferentia and Trainium. The Neuron SDK automatically optimizes for Trainium without using closed source dependencies like Nvidia CUDA, not requiring any application level code changes to accelerate models. We believe this intentional approach allows developers freedom of choice with their code and models. If you have applications dependencies on CUDA (or other 3rd party closed source artifacts) you will need to strip them out, and from that point the Neuron compiler will take the model as is and optimize it at the hardware level. \n\n\nNetworking\n----------\n\nWhat’s important to know about the networking in Trn1?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTrn1 have the fastest EFA in AWS, clocked at 800Gbps they enable more collective communication as compared to other training instances, which is important if your training job spans across multiple servers. You should also expect lower latency as we streamline the communication path between the dedicated collective communication engine on Trainium, and the AWS Nitro EFA NICs.\n\nHow does Trainium accelerates collective communication operations?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTrainium introduces a dedicated collective compute engine, that runs in parallel to the compute cores (aka NeuronCores). This improves convergence time of intermediate steps as the communication happens in parallel to the compute. This capability, in addition to the faster and optimized EFA, results in better scalability and faster time to train, as compared to other training instances in AWS.\n\nWhat does Strong/Weak Scaling mean?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo enable strong scaling, we optimized Trainium to be efficient at small batch sizes. Compared to GPUs, Trn1 maintains high efficiency even for small batch sizes. This allows you to scale-out to thousands of devices without increasing the global mini-batch size at the same rate, which in turn leads to faster end-to-end training convergence.\n\nIn weak scaling setup, we show the optimal throughput with sufficiently large batch size per Trainium. The large batch size is set to leverage the high core utilization so that the overall end-to-end training will be fast. This setup also enables a large global batch size as it scales with the total number of nodes in the cluster.\n\nUsability\n---------\n\nWhat have AWS done to improve usability of Trainium?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nStochastic rounding enables running at half precision speeds (BF16) while maintaining accuracy at near full precision. This of course helps the loss function converge faster and reduce memory footprint, but equally important, it is simplifying model development as you can write your model in FP32, and Neuron/Trainium will auto-cast the model to BF16, and execute it with SR enabled. There is no need to loss accuracy with pure BF16 runs, and more importantly no need for experimenting with mixed precision strategies to find the optimal settings.\n\nEager debug mode provides a convenient utility to step through the code and evaluate operator correctness as part of your model creation/debug. For more details, please refer to the Neuron documentation\n\nWhat other AWS services work with Trn1?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTrn1 via its Neuron SDK supports Amazon ECS, EKS, ParallelCluster, Batch, and Amazon SageMaker. Customers can also choose to run in a Neuron container within their self-managed containers orchestration service (e.g., Kubernetes and Ray).\n\nWhat tools are available to develop models with Trn1?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen running training, evaluation or inference workloads you can use Neuron 2.x CLI tools such as neuron-ls and neuron-top to get insights into the NeuronCores and NeuronDevices performance and memory utilization, topology and host vCPU performance and memory utilization. In addition, the Neuron Plugin for TensorBoard provides a standard GUI that enables profile and debug of models. TensorBoard views include:\n\n- Model overview: provide a summary of the model and the utilization on the Host and NeuronDevice\n- Operators’ view: provide a breakdown of ML framework and HLO operators on both Host and NeuronDevice\n- Code trace view: show a timeline of the model execution at the framework and HLO operators level \n- Hardware trace view: show a timeline of the model execution at the level of hardware (Host, NeuronDevice, Data Transfer)\n- Topology view: show the NeuronDevices topology within an instance\n\n\nHow will compile time impact my work flow?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWe understand compilation is a new step with Trainium, but as long as the overall time to train and cost to train is optimized, the compilation impact on these two metrics is minimized. To further help reduce compilation time impact on usability, Neuron supports a persistent cache, where artifacts that have not changed since the last run can be reused, skipping compilation all together. For developing and experimenting with new models, you can use the eager debug mode, that compiles (and caches) op-by-op, enabling quick evaluation without compiling large models. We are also working on Neuron model analyzer (see Neuron roadmap) that will recommend optimized hyper parameters, skipping full compilation per experiment.\n" }, { "path": "about-neuron/faq.rst", "content": ".. _neuron_faq:\n\n.. meta::\n :description: Frequently Asked Questions (FAQ) about the AWS Neuron SDK, including topics on Neuron 2.x, training, inference, runtime, compiler, containers, and ONNX support.\n :date-modified: 2025-10-03\n\nNeuron FAQ\n==========\n\nThis topic provides links to frequently asked questions (FAQs) about the AWS Neuron SDK, organized by Neuron component.\n\nNeuron 2.x FAQ\n--------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: neuron2-intro-faq\n :link-type: ref\n\n **Neuron 2.x Introduction FAQ**\n ^^^\n Common questions about Neuron 2.x features and migration\n\nTraining-specific FAQ\n---------------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: neuron-training-faq\n :link-type: ref\n\n **Neuron Training FAQ**\n ^^^\n Frequently asked questions about training models on Neuron\n\nInference-specific FAQ\n----------------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: neuron-f1-faq\n :link-type: ref\n\n **Neuron F1 FAQ**\n ^^^\n Questions about F1 instance inference capabilities\n\n .. grid-item-card::\n :link: trouble-shooting-inf1-faq\n :link-type: ref\n\n **Inf1 Troubleshooting FAQ**\n ^^^\n Common ``Inf1`` instance issues and solutions\n\n .. grid-item-card::\n :link: neuronperf_faq\n :link-type: ref\n\n **NeuronPerf FAQ**\n ^^^\n Performance benchmarking tool questions\n\nNeuron Runtime FAQ\n------------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: neuron-runtime-faq\n :link-type: ref\n\n **Neuron Runtime FAQ**\n ^^^\n Runtime configuration and execution questions\n\nNeuron Compiler FAQ\n-------------------\n\n.. grid:: 1 1 2 2\n :gutter: 2\n\n .. grid-item-card::\n :link: neuronx_compiler_faq\n :link-type: ref\n\n **NeuronX Compiler FAQ**\n ^^^\n Questions about the NeuronX compiler for Trn1/Inf2\n\n .. grid-item-card::\n :link: neuron_compiler_faq\n :link-type: ref\n\n **Neuron Compiler FAQ**\n ^^^\n Questions about the Neuron compiler for Inf1\n\nNeuron DLCs FAQ\n---------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: container-faq\n :link-type: ref\n\n **Neuron Containers FAQ**\n ^^^\n Container deployment and configuration questions\n\nSupport\n-------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: contribute-faq\n :link-type: ref\n\n **Contribute FAQ**\n ^^^\n Questions about contributing to the Neuron project\n" }, { "path": "about-neuron/index.rst", "content": ".. _about-neuron:\n\nAbout the AWS Neuron SDK\n========================\n\nAWS Neuron is a software development kit (SDK) enabling high-performance deep learning acceleration using AWS Inferentia and Trainium, AWS's custom designed machine learning accelerators. It enables you to develop, profile, and deploy high-performance machine learning workloads on AWS Inferentia and Trainium instances. \n\nThe AWS Neuron SDK includes:\n\n* **Neuron Compiler** - Compiles high-level, framework-based models for optimal performance on Neuron devices\n* **Neuron Kernel Interface (NKI)** - Provides direct compiler access to Neuron device capabilities\n* **Neuron Runtime** - Executes compiled models on Neuron devices\n* **ML Framework integration** - Deep support for PyTorch and JAX\n* **Training and inference libraries** - Distributable training and inference libraries for large-scale models\n* **Deployment support** - Integration with AWS services like SageMaker, EC2, EKS, and ECS\n* **Developer tools** - Profiling, monitoring, and debugging utilities\n\nFor a full list of AWS Neuron features, see :ref:`what-is-neuron`.\n\n.. admonition:: Join our Beta program\n\n Get early access to new Neuron features and tools! `Fill out this form and apply to join our Beta program `__.\n\nWhat is \"NeuronX\"?\n------------------\n\n\"NeuronX\" refers to the next-generation AWS Neuron SDK, which provides enhanced capabilities for both inference and training on AWS Inferentia and Trainium instances. NeuronX includes:\n\n* Support for the latest versions of PyTorch and JAX\n* Advanced compiler optimizations for improved performance\n* Enhanced distributed training libraries for large-scale models\n* Improved profiling and debugging tools\n* Ongoing feature development and support for new instance types\n\nCatch up on the latest Neuron news\n-----------------------------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: /about-neuron/whats-new\n :link-type: doc\n :class-card: sd-border-1\n\n **What's New in Neuron**\n ^^^\n Read about the latest releases and features of the Neuron SDK\n\n\n\n\nLearn about AWS Neuron\n----------------------\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card::\n :link: /about-neuron/what-is-neuron\n :link-type: doc\n :class-card: sd-border-1\n\n **What is AWS Neuron?**\n ^^^\n Short overview of the AWS Neuron SDK and its components\n\n.. grid:: 1 1 2 2\n :gutter: 2\n \n .. grid-item-card::\n :link: /about-neuron/arch/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron architecture**\n ^^^\n Understand the Neuron hardware and software architecture\n\n .. grid-item-card::\n :link: /about-neuron/arch/neuron-features/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron features**\n ^^^\n Overviews of model development features provided by Neuron\n\n .. grid-item-card::\n :link: /frameworks/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Supported ML frameworks**\n ^^^\n Neuron support for popular ML frameworks including PyTorch and JAX\n\n .. grid-item-card::\n :link: /libraries/index\n :link-type: doc\n :class-card: sd-border-1\n\n **NeuronX distributed (NxD) libraries**\n ^^^\n NeuronX distributed libraries for training and inference\n\n .. grid-item-card::\n :link: /nki/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron Kernel Interface (NKI)**\n ^^^\n NKI is a low-level interface for custom, bare-metal kernel development\n\n .. grid-item-card::\n :link: /compiler/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron Compiler**\n ^^^\n The Neuron compiler optimizes models for Neuron hardware\n\n .. grid-item-card::\n :link: /neuron-runtime/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron Runtime**\n ^^^\n Runtime for executing compiled models on Neuron devices\n\n .. grid-item-card::\n :link: /tools/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron developer tools**\n ^^^\n Tools for profiling, debugging, and monitoring Neuron applications\n\n .. grid-item-card::\n :link: /dlami/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron AWS Neuron Deep Learning AMIs**\n ^^^\n Deploy the Neuron SDK on EC2 instances with pre-installed Amazon Machine Images (AMIs)\n\n .. grid-item-card::\n :link: /containers/index\n :link-type: doc\n :class-card: sd-border-1\n\n **Neuron AWS Neuron Deep Learning Containers**\n ^^^\n Deploy the Neuron SDK using pre-built Docker deep learning containers (DLCs)\n\nResources\n---------\n\n* :ref:`Setup Guide `\n* :ref:`Release Notes `\n* :ref:`Neuron FAQ `\n* :doc:`Older Neuron FAQs `\n\nSupport\n-------\n\n* :doc:`Neuron Open Source GitHub Repos `\n* :ref:`AWS Neuron SDK maintenance policy `\n\n.. _contact-us:\n\nContact us\n----------\n\nFor support, submit a request with AWS Neuron `Github issues `_ or visit the `Neuron AWS forums `_ for an answer. \n\nIf you want to request a feature or report a critical issue, you can contact us directly at ``aws-neuron-support@amazon.com``.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n App Notes \n Ask Amazon AI helper tools \n Benchmarks \n Beta Participation \n Model Samples \n Neuron FAQ \n Neuron Features \n Open Source \n SDK Maintenance Policy \n Security \n Term Glossary \n Troubleshooting \n What is AWS Neuron? \n Older Neuron FAQS \n" }, { "path": "about-neuron/models/index.rst", "content": ".. _model_samples_tutorials:\n\nModel samples and tutorials\n===========================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Training on Trn1 \n Inference on Inf2/Trn1/Trn2 \n Inference on Inf1 \n \nThis section gives you the consolidated list of code samples and tutorials published by AWS Neuron across documentation \nand various GitHub repositories.\n\n.. card:: Training on Trn1\n :link: model_samples_training_trn1\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. card:: Inference on Inf2, Trn1 and Trn2\n :link: model_samples_inference_inf2_trn1\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. card:: Inference on Inf1\n :link: model_samples_inference_inf1\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\nFor links to individual GitHub sample repositories, see :ref:`neuron-github-samples`\n" }, { "path": "about-neuron/models/inference-inf1-samples.rst", "content": ".. _model_samples_inference_inf1:\n\nInference Samples/Tutorials (Inf1)\n==================================\n\n.. important::\n\n The samples linked on this page have been archived and are provided for historical reference only. They are not tested with recent versions of the Neuron SDK.\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n \n.. _encoder_model_samples_inference_inf1:\n \nEncoders \n--------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - bert-base-cased-finetuned-mrpc\n - torch-neuron\n - * HuggingFace pretrained BERT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * `BertBaseCased Inference on Inf1 instances `_\n * Bert TorchServe tutorial :ref:`[html] `\n * Bring your own HuggingFace pretrained BERT container to Sagemaker Tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - bert-base-uncased\n - torch-neuron\n - * NeuronCore Pipeline tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - bert-large-uncased\n - torch-neuron\n - * `BertLargeUncased Inference on Inf1 instances `_\n \n * - roberta-base\n - torch-neuron\n - * `Roberta-Base inference on Inf1 instances `_\n\n * - distilbert-base-uncased-finetuned-sst-2-english\n - tensorflow-neuron \n - * Tensorflow 2.x - HuggingFace Pipelines distilBERT with Tensorflow2 Neuron :ref:`[html] ` :github:`[notebook] `\n \n * - gluon bert\n - mxnet-neuron \n - * MXNet 1.8: Using data parallel mode tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n\n\n\n.. _vision_transformer_model_samples_inference_inf1:\n\nVision Transformers \n-------------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n \n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - ssd\n - torch-neuron\n - * `Inference of SSD model on inf1 instances `_\n \n\n * - TrOCR\n - torch-neuron\n - * `TrOCR inference on Inf1 instances `_\n\n \n * - vgg\n - torch-neuron\n - * `VGG inference on Inf1 instances `_\n\n\n * - google/vit-base-patch16-224\n - torch-neuron\n - * `ViT model inference on Inf1 `_\n\n\n\n.. _cnn_model_samples_inference_inf1:\n\nConvolutional Neural Networks(CNN)\n----------------------------------\n\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - EfficientNet\n - torch-neuron\n - * `EfficientNet model inference on Inf1 instances `_\n\n * - GFL (MMDetection)\n - torch-neuron\n - * `GFL (MMDetection) inference on Inf1 instances `_\n\n * - HRNet\n - torch-neuron\n - * `HRNET - Pose Estimation `_\n\n * - MarianMT\n - torch-neuron\n - * HuggingFace MarianMT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * `Inference of Pre-trained MarianMT model on Inf1 `_\n\n * - Detectron2 R-CNN \n - torch-neuron\n - * `R-CNN inference on Inf1 `_\n\n * - resnet\n - torch-neuron\n - * `Inference of Pre-trained Resnet model (18,34,50,101,152) on Inf1 `_\n * ResNet-50 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - resnet\n - tensorflow-neuron\n - * Tensorflow 2.x - Using NEURON_RT_VISIBLE_CORES with TensorFlow Serving :ref:`[html] `\n \n * - resnet\n - mxnet-neuron\n - * ResNet-50 tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n * Getting started with Gluon tutorial :ref:`[html] ` :github:`[notebook] `\n * NeuronCore Groups tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n \n\n * - Resnext\n - torch-neuron\n - * `Inference of Resnext model on Inf1 `_\n\n\n * - Yolov4\n - torch-neuron \n - * PyTorch YOLOv4 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - Yolov5\n - torch-neuron\n - * `Inference of Yolov5 on Inf1 `_\n\n\n * - Yolov6\n - torch-neuron \n - * `Inference of Yolov6 on Inf1 instances `_\n\n\n * - Yolov7\n - torch-neuron\n - * `Inference of Yolov7 model on Inf1 `_\n\n * - Yolof\n - torch-neuron\n - * `Inference of Yolof model on Inf1 `_\n\n * - fairseq\n - torch-neuron\n - * `Inference of fairseq model on Inf1 `_\n\n * - unet\n - tensorflow-neuron\n - * `Unet - Tensorflow 2.x tutorial `_\n\n\n\n.. _vision_model_samples_inference_inf1:\n\nVision\n------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - craft-pytorch\n - torch-neuron\n - * `CRAFT model inference on Inf1 `_\n\n \n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "about-neuron/models/inference-inf2-trn1-samples.rst", "content": ".. _model_samples_inference_inf2_trn1:\n\nInference Samples/Tutorials (Inf2/Trn1/Trn2)\n============================================\n\n.. important::\n\n Some samples linked on this page have been archived and are provided for historical reference only. They are not tested with recent versions of the Neuron SDK. For the latest inference tutorials, refer to :ref:`NxD Inference Tutorials `.\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\n.. _encoder_model_samples_inference_inf2_trn1:\n \nEncoders \n--------\n\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - bert-base-cased-finetuned-mrpc\n - torch-neuronx\n - * :ref:`BERT TorchServe tutorial `\n * HuggingFace pretrained BERT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * `LibTorch C++ Tutorial for HuggingFace Pretrained BERT `_\n * `Compiling and Deploying HuggingFace Pretrained BERT on Inf2 on Amazon SageMaker `_\n\n\n * - bert-base-cased-finetuned-mrpc\n - neuronx-distributed\n - * :ref:`tp_inference_tutorial`\n\n\n * - bert-base-uncased\n - torch-neuronx\n - * `HuggingFace Pretrained BERT Inference on Trn1 `_\n\n * - distilbert-base-uncased\n - torch-neuronx\n - * `HuggingFace Pretrained DistilBERT Inference on Trn1 `_\n\n\n * - roberta-base\n - tensorflow-neuronx\n - * HuggingFace Roberta-Base :ref:`[html]` :github:`[notebook] `\n\n\n * - roberta-large\n - torch-neuronx\n - * `HuggingFace Pretrained RoBERTa Inference on Trn1 `_\n\n\n\n.. _decoder_model_samples_inference_inf2_trn1:\n\nDecoders\n--------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - gpt2\n - torch-neuronx\n - * `HuggingFace Pretrained GPT2 Feature Extraction on Trn1 `_\n \n * - meta-llama/Llama-3.3-70B\n - neuronx-distributed-inference\n - * :ref:`nxdi-trn2-llama3.3-70b-tutorial`\n * :ref:`/libraries/nxd-inference/tutorials/trn2-llama3.3-70b-dp-tutorial.ipynb`\n * :ref:`nxdi-sd-inference-tutorial`\n\n * - meta-llama/Llama-3.1-8b\n - transformers-neuronx\n - * `Run Hugging Face Llama 3.1 8B autoregressive sampling on Inf2 & Trn1 with 32k sequence length `_\n * `Run Hugging Face Llama 3.1 8B autoregressive sampling on Inf2 & Trn1 with 128k sequence length `_\n * `Run meta-llama/Meta-Llama-3.1-8B autoregressive sampling on Inf2 & Trn1 `_\n \n * - meta-llama/Llama-3.1-70b\n - transformers-neuronx\n - * `Run Hugging Face Llama 3.1 70B autoregressive sampling on Trn1 with 64k sequence length `_\n * `Run Hugging Face meta-llama/Meta-Llama-3.1-70B autoregressive sampling on Inf2 & Trn1 `_\n\n * - meta-llama/Llama-3.1-70b-Instruct\n - transformers-neuronx\n - * `Run Hugging Face Llama-3.1-70B-Instruct + Llama-3.2-1B-Instruct Speculative Decoding on Trn1 with transformers-neuronx and vLLM `_\n * `Run Hugging Face Llama-3.1-70B-Instruct EAGLE Speculative Decoding on Trn1 with transformers-neuronx and vLLM `_\n\n * - meta-llama/Llama-3.1-405b\n - neuronx-distributed-inference\n - * :ref:`Tutorial for deploying Llama-3.1-405B on Trn2 `\n * :ref:`nxdi-trn2-llama3.1-405b-speculative-tutorial`\n \n * - meta-llama/Llama-3.1-405b\n - transformers-neuronx\n - * `Run Hugging Face Llama 3.1 405B autoregressive sampling on Trn1/Trn1n with 16k sequence length `_\n\n * - meta-llama/Llama-3-8b\n - transformers-neuronx\n - * `Run Hugging Face meta-llama/Llama-3-8b autoregressive sampling on Inf2 & Trn1 `_\n\n * - meta-llama/Llama-3-70b\n - transformers-neuronx\n - * `Run Hugging Face meta-llama/Llama-3-70b autoregressive sampling on Inf2 & Trn1 `_\n\n * - meta-llama/Llama-2-13b\n - transformers-neuronx\n - * `Run Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1 `_\n\n * - meta-llama/Llama-2-70b\n - transformers-neuronx\n - * `Run Hugging Face meta-llama/Llama-2-70b autoregressive sampling on Inf2 & Trn1 `_\n * `Run speculative sampling on Meta Llama models [Beta] `_\n\n * - meta-llama/Llama-3.2-1B-Instruct\n - neuronx-distributed\n - * `Run meta-llama/Llama-3.2-1B-Instruct on Inf2 and Trn1 `_\n\n * - meta-llama/codellama-13b\n - neuronx-distributed\n - * `Run meta-llama/codellama-13b-16k-sampling `_\n\n * - mistralai/Mistral-7B-Instruct-v0.1\n - transformers-neuronx\n - * :ref:`Run Mistral-7B-Instruct-v0.1 autoregressive sampling on Inf2 & Trn1 `\n\n * - mistralai/Mistral-7B-Instruct-v0.2\n - transformers-neuronx\n - * `Run Hugging Face mistralai/Mistral-7B-Instruct-v0.2 autoregressive sampling on Inf2 & Trn1 [Beta] `_\n\n * - Mixtral-8x7B-v0.1\n - transformers-neuronx\n - * `Run Hugging Face mistralai/Mixtral-8x7B-v0.1 autoregressive sampling on Inf2 & Trn1 `_\n\n * - Mixtral-8x7B\n - neuronx-distributed\n - * `Mixtral inference with NeuronX Distributed on Inf2 & Trn1 `_\n\n\n * - DBRX\n - neuronx-distributed\n - * `DBRX inference with NeuronX Distributed on Inf2 & Trn1 `_ \n\n * - codellama/CodeLlama-13b-hf\n - transformers-neuronx\n - * `Run Hugging Face codellama/CodeLlama-13b-hf autoregressive sampling on Inf2 & Trn1 `_\n\n.. _encoder_decoder_model_samples_inference_inf2_trn1:\n\nEncoder-Decoders \n----------------\n\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - t5-large\n - * torch-neuronx\n * optimum-neuron\n - * T5 inference tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - t5-3b\n - neuronx-distributed\n - * T5 inference tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - google/flan-t5-xl\n - neuronx-distributed\n - * flan-t5-xl inference tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n\n.. _vision_transformer_model_samples_inference_inf2_trn1:\n\nVision Transformers \n-------------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n \n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - google/vit-base-patch16-224\n - torch-neuronx\n - * `HuggingFace Pretrained ViT Inference on Trn1 `_\n\n * - clip-vit-base-patch32\n - torch-neuronx\n - * `HuggingFace Pretrained CLIP Base Inference on Inf2 `_\n\n\n * - clip-vit-large-patch14\n - torch-neuronx\n - * `HuggingFace Pretrained CLIP Large Inference on Inf2 `_\n\n\n\n.. _cnn_model_samples_inference_inf2_trn1:\n\nConvolutional Neural Networks(CNN)\n----------------------------------\n\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - resnet50\n - torch-neuronx\n - * `Torchvision Pretrained ResNet50 Inference on Trn1 / Inf2 `_\n * Torchvision ResNet50 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n * - resnet50\n - tensorflow-neuronx\n - * :ref:`tensorflow-servingx-neuronrt-visible-cores`\n\n * - unet\n - torch-neuronx\n - * `Pretrained UNet Inference on Trn1 / Inf2 `_\n\n * - vgg\n - torch-neuronx\n - * `Torchvision Pretrained VGG Inference on Trn1 / Inf2 `_\n\n\n.. _sd_model_samples_inference_inf2_trn1:\n\nStable Diffusion\n----------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - stable-diffusion-v1-5\n - torch-neuronx\n - * `HuggingFace Stable Diffusion 1.5 (512x512) Inference on Trn1 / Inf2 `_\n\n * - stable-diffusion-2-1-base\n - torch-neuronx\n - * `HuggingFace Stable Diffusion 2.1 (512x512) Inference on Trn1 / Inf2 `_\n\n * - stable-diffusion-2-1\n - torch-neuronx\n - * `HuggingFace Stable Diffusion 2.1 (768x768) Inference on Trn1 / Inf2 `_\n * `Deploy & Run Stable Diffusion on SageMaker and Inferentia2 `_\n\n * - stable-diffusion-xl-base-1.0\n - torch-neuronx\n - * `HuggingFace Stable Diffusion XL 1.0 (1024x1024) Inference on Inf2 `_\n * `HuggingFace Stable Diffusion XL 1.0 Base and Refiner (1024x1024) Inference on Inf2 `_\n\n * - stable-diffusion-2-inpainting\n - torch-neuronx\n - * `stable-diffusion-2-inpainting model Inference on Trn1 / Inf2 `_\n\n\n.. _diffusion_transformers_samples_inference_inf2_trn1:\n\nDiffusion Transformers\n----------------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - pixart-alpha\n - torch-neuronx\n - * `HuggingFace PixArt Alpha (256x256, 512x512 square resolution) Inference on Trn1 / Inf2 `_\n\n * - pixart-sigma\n - torch-neuronx\n - * `HuggingFace PixArt Sigma (256x256, 512x512 square resolution) Inference on Trn1 / Inf2 `_\n\n \n\n.. _audio_model_samples_inference_inf2_trn1:\n\nAudio\n-----\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n \n * - wav2vec2-conformer\n - torch-neuronx\n - * `Run HuggingFace Pretrained Wav2Vec2-Conformer with Rotary Position Embeddings Inference on Inf2 `_\n * `Run HuggingFace Pretrained Wav2Vec2-Conformer with Relative Position Embeddings Inference on Inf2 & Trn1 `_\n\n\n\n.. _multi_modal_model_samples_inference_inf2_trn1:\n\nMulti Modal\n-----------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n \n\n * - multimodal-perceiver\n - torch-neuronx\n - * `HuggingFace Multimodal Perceiver Inference on Trn1 / Inf2 `_\n\n\n * - language-perceiver\n - torch-neuronx\n - * `HF Pretrained Perceiver Language Inference on Trn1 / Inf2 `_\n\n\n * - vision-perceiver-conv\n - torch-neuronx\n - * `HF Pretrained Perceiver Image Classification Inference on Trn1 / Inf2 `_\n\n\n\n \n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "about-neuron/models/training-trn1-samples.rst", "content": ".. _model_samples_training_trn1:\n\nTraining Samples/Tutorials (Trn1/Trn1n)\n=======================================\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\n.. _encoder_model_samples_training_trn1:\n \nEncoders \n--------\n\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - bert-base-cased\n - torch-neuronx\n - * `Fine-tune a \"bert-base-cased\" PyTorch model for Text Classification `_\n * `How to fine-tune a \"bert base cased\" PyTorch model with AWS Trainium (Trn1 instances) for Sentiment Analysis `_\n \n * - bert-base-uncased\n - torch-neuronx\n - * `Fine-tune a \"bert-base-uncased\" PyTorch model `_\n * `Fine tuning BERT base model from HuggingFace on Amazon SageMaker `_\n \n * - bert-large-cased\n - torch-neuronx\n - * `Fine-tune a \"bert-large-cased\" PyTorch model `_\n \n * - bert-large-uncased\n - torch-neuronx\n - * :ref:`hf-bert-pretraining-tutorial`\n * `Launch Bert Large Phase 1 pretraining job on Parallel Cluster `_\n * `Launch a Multi-Node PyTorch Neuron Training Job on Trainium Using TorchX and EKS `_\n * :ref:`torch-hf-bert-finetune`\n * `Fine-tune a \"bert-large-uncased\" PyTorch model `_\n \n\n * - roberta-base\n - tensorflow-neuronx\n - * `Fine-tune a \"roberta-base\" PyTorch model `_\n\n\n * - roberta-large\n - torch-neuronx\n - * `Fine-tune a \"roberta-large\" PyTorch model `_\n\n \n * - xlm-roberta-base\n - torch-neuronx\n - * `Fine-tune a \"xlm-roberta-base\" PyTorch model `_\n\n\n * - alberta-base-v2\n - torch-neuronx\n - * `Fine-tune a \"alberta-base-v2\" PyTorch model `_\n\n\n * - distilbert-base-uncased\n - torch-neuronx\n - * `Fine-tune a \"distilbert-base-uncased\" PyTorch model `_\n\n\n * - camembert-base\n - torch-neuronx\n - * `Fine-tune a \"camembert-base PyTorch model `_\n\n * - cl-tohoku/bert-base-japanese-whole-word-masking\n - torch-neuronx\n - * `Fine-tuning & Deployment Hugging Face BERT Japanese model\t`_\n\n\n\n\n.. _decoder_model_samples_training_trn1:\n\n\nDecoders\n--------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n \n * - gpt-2\n - torch-neuronx\n - * `How to run training jobs for \"gpt2\" PyTorch model with AWS Trainium `_\n * :ref:`zero1-gpt2-pretraining-tutorial`\n \n \n * - gpt-3\n - neuronx-nemo-megatron\n - * `Launch a GPT-3 23B pretraining job using neuronx-nemo-megatron `_\n * `Launch a GPT-3 46B pretraining job using neuronx-nemo-megatron `_\n * `Launch a GPT-3 175B pretraining job using neuronx-nemo-megatron `_\n\n\n * - GPT-NEOX-20B\n - neuronx-distributed\n - * :ref:`gpt_neox_20b_tp_zero1_tutorial`\n * `Training GPT-NEOX 20B model using neuronx-distributed\t `_\n * `Pre-train GPT Neox 20b on Wikicorpus dataset using Neuronx Distributed library `_\n\n \n * - GPT-NEOX-6.9B\n - neuronx-distributed\n - * :ref:`gpt_neox_tp_zero1_tutorial`\n * `Training GPT-NEOX 6.9B model using neuronx-distributed\t\t `_\n * `Pre-train GPT Neox 6.9b on Wikicorpus dataset using Neuronx Distributed library `_\n\n * - meta-llama/Llama-3.1-70b\n - neuronx-distributed\n - * :ref:`llama2_tp_pp_tutorial`\n\n * - meta-llama/Llama-3.1-8b\n - neuronx-distributed\n - * :ref:`llama2_7b_tp_zero1_tutorial`\n\n * - meta-llama/Llama-3-70b\n - neuronx-distributed\n - * :ref:`llama2_tp_pp_tutorial`\n\n * - meta-llama/Llama-3-8b\n - nxd-training\n - * :ref:`hf_llama3_8B_pretraining`\n * :ref:`hf_llama3_8B_SFT`\n\n * - meta-llama/Llama-3-8b\n - neuronx-distributed\n - * :ref:`Training Llama3 8B Model with Tensor Parallelism and ZeRO-1 Optimizer `\n * :ref:`Tutorial for Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning with NeuronX Distributed `\n \n * - meta-llama/Llama-2-7b\n - neuronx-distributed\n - * :ref:`llama2_7b_tp_zero1_tutorial`\n * `Training Llama2 7B Model with AWS Batch and Trainium `_\n * :ref:`llama2_7b_tp_zero1_ptl_finetune_tutorial`\n * `Pre-train Llama2-7B on Wikicorpus dataset using Neuronx Distributed library `_\n\n * - meta-llama/Llama-2-13b\n - neuronx-distributed\n - * :ref:`llama2_tp_pp_tutorial`\n\n * - meta-llama/Llama-2-70b\n - neuronx-distributed\n - * :ref:`llama2_tp_pp_tutorial`\n\n * - codegen25-7b-mono\n - neuronx-distributed\n - * :ref:`codegen25_7b_tp_zero1_tutorial`\n\n * - meta-llama/Llama-2\n - neuronx-nemo-megatron\n - * `Launch a Llama-2-7B pretraining job using neuronx-nemo-megatron `_\n * `Launch a Llama-2-13B pretraining job using neuronx-nemo-megatron `_\n * `Launch a Llama-2-70B pretraining job using neuronx-nemo-megatron `_\n\n * - Mistral-7B\n - neuronx-nemo-megatron\n - * `Training Mistral-7B `_\n\n.. _encoder_decoder_model_samples_training_trn1:\n\nEncoder-Decoders \n----------------\n\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - t5-small\n - * torch-neuronx\n * optimum-neuron\n - * :ref:`torch-hf-t5-finetune`\n\n * - facebook/bart-large\n - * torch-neuronx\n - * `How to fine-tune a \"Bart-Large\" PyTorch model with AWS Trainium (trn1 instances) `_\n\n\n.. _vision_transformer_model_samples_training_trn1:\n\nVision Transformers \n-------------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n \n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - google/vit-base-patch16-224-in21k\n - torch-neuronx\n - * `Fine-tune a pretrained HuggingFace vision transformer PyTorch model `_\n\n \n * - openai/clip-vit-base-patch32\n - torch-neuronx\n - * `Fine-tune a pretrained HuggingFace CLIP-base PyTorch model with AWS Trainium `_\n\n\n * - openai/clip-vit-large-patch14\n - torch-neuronx\n - * `Fine-tune a pretrained HuggingFace CLIP-large PyTorch model with AWS Trainium `_\n\n\n\n.. _sd_model_samples_training_trn1:\n\nStable Diffusion\n----------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n \n\n * - stabilityai/stable-diffusion-2-1-base\n - torch-neuronx\n - * [Beta] `Train stabilityai/stable-diffusion-2-1-base with AWS Trainium (trn1 instances) `_\n\n\n * - runwayml/stable-diffusion-v1-5\n - torch-neuronx\n - * [Beta] `Train runwayml/stable-diffusion-v1-5 with AWS Trainium (trn1 instances) `_\n \n\n\n.. _multi_modal_model_samples_training_trn1:\n\nMulti Modal\n-----------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n\n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n \n\n * - language-perceiver\n - torch-neuronx\n - * `How to fine-tune a \"language perceiver\" PyTorch model with AWS Trainium (trn1 instances) `_\n\n\n * - vision-perceiver-conv\n - torch-neuronx\n - * `How to fine-tune a pretrained HuggingFace Vision Perceiver Conv `_\n\n\n\n.. _cnn_model_samples_training_trn1:\n\nConvolutional Neural Networks(CNN)\n----------------------------------\n\n.. list-table::\n :widths: 20 15 45 \n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n \n * - Model\n - Frameworks/Libraries\n - Samples and Tutorials\n\n * - resnet50\n - torch-neuronx\n - `How to fine-tune a pretrained ResNet50 Pytorch model with AWS Trainium (trn1 instances) using NeuronSDK `_\n" }, { "path": "about-neuron/monitoring-tools.rst", "content": ".. _monitoring_tools:\n\nMonitoring Tools\n=================\n\n.. toctree:: \n :maxdepth: 1\n\n Neuron-Monitor User Guide \n Neuron-Top User Guide \n Neuron-LS User Guide \n Neuron-Sysfs User Guide \n NCCOM-TEST User Guide \n What's New \n" }, { "path": "about-neuron/news-and-blogs/CONTRIBUTING.md", "content": "# Contributing to AWS Neuron News and Blogs\n\nThank you for your interest in sharing content about AWS Neuron, Trainium, and Inferentia! This page collects external articles, blog posts, tutorials, and news to help the community discover valuable content.\n\n## How to Add Your Article\n\n### Quick Steps\n\n1. **Fork the repository** on GitHub\n2. **Edit the data file**: `about-neuron/news-and-blogs/news-and-blogs.yaml`\n3. **Add your article** following the format below\n4. **Submit a pull request** with your changes\n\n### Article Entry Format\n\nAdd your article to the appropriate section in `news-and-blogs.yaml`:\n\n```yaml\n- title: \"Your Article Title\"\n url: \"https://example.com/your-article\"\n description: \"A brief 1-2 sentence description of your article content.\"\n author: \"Your Name or Organization\"\n author_url: \"https://your-website.com\" # Optional for featured articles\n date: \"YYYY-MM-DD\" # Publication date\n category: \"blog\" # Options: blog, news, tutorial, case-study, benchmark\n locale: \"en-US\" # Language/region code (e.g., en-US, ja-JP, zh-CN, de-DE, fr-FR)\n featured: false # Set to true only if approved by AWS Neuron team\n icon: \"📝\" # Optional emoji icon for featured articles\n```\n\n### Sections\n\n- **`featured_articles`**: Highlighted content (requires AWS Neuron team approval)\n- **`all_articles`**: All community and official content\n\n### Categories\n\nChoose the most appropriate category for your content:\n\n- **`blog`**: Technical blog posts and articles\n- **`news`**: News announcements and press releases\n- **`tutorial`**: Step-by-step guides and how-tos\n- **`case-study`**: Customer success stories and use cases\n- **`benchmark`**: Performance benchmarks and comparisons\n\n### Locale Codes\n\nSpecify the language and region of your article using standard locale codes:\n\n**Common Locales:**\n- `en-US` - English (United States) 🇺🇸\n- `en-GB` - English (United Kingdom) 🇬🇧\n- `ja-JP` - Japanese 🇯🇵\n- `zh-CN` - Chinese (Simplified) 🇨🇳\n- `zh-TW` - Chinese (Traditional) 🇹🇼\n- `ko-KR` - Korean 🇰🇷\n- `de-DE` - German 🇩🇪\n- `fr-FR` - French 🇫🇷\n- `es-ES` - Spanish (Spain) 🇪🇸\n- `es-MX` - Spanish (Mexico) 🇲🇽\n- `pt-BR` - Portuguese (Brazil) 🇧🇷\n- `it-IT` - Italian 🇮🇹\n- `nl-NL` - Dutch 🇳🇱\n- `ru-RU` - Russian 🇷🇺\n- `ar-SA` - Arabic 🇸🇦\n- `hi-IN` - Hindi 🇮🇳\n\nA flag emoji will be automatically displayed next to your article based on the locale. If your locale isn't in the list, a 🌐 globe icon will be shown.\n\n### Example Entry\n\n```yaml\nall_articles:\n - title: \"Building Large Language Models on AWS Trainium\"\n url: \"https://example.com/llm-trainium-guide\"\n description: \"A comprehensive guide to training and deploying LLMs using AWS Trainium instances with practical code examples.\"\n author: \"Jane Developer\"\n date: \"2026-01-15\"\n category: \"tutorial\"\n locale: \"en-US\"\n featured: false\n```\n\n### Guidelines\n\n1. **Content must be relevant** to AWS Neuron, Trainium, or Inferentia\n2. **Provide accurate information** - ensure URLs work and descriptions are clear\n3. **Use proper formatting** - follow YAML syntax exactly\n4. **One article per pull request** - makes review easier\n5. **Include context** in your PR description about why this content is valuable\n\n### Featured Articles\n\nTo request your article be featured:\n\n1. Add it to `all_articles` first with `featured: false`\n2. In your pull request, explain why it should be featured\n3. AWS Neuron team will review and may promote it to `featured_articles`\n\nFeatured articles should be:\n- High-quality, in-depth content\n- Particularly valuable to the community\n- Recent (typically within the last 6 months)\n\n### Review Process\n\n1. Submit your pull request\n2. AWS Neuron team will review within 5-7 business days\n3. May request changes or clarifications\n4. Once approved, your article will appear on the next documentation build\n\n### Questions?\n\n- Open an issue in the repository\n- Contact your AWS Neuron support representative\n- Email: aws-neuron-support@amazon.com\n\n## Content Guidelines\n\n### What to Include\n\n✅ Technical tutorials and guides \n✅ Performance benchmarks and analysis \n✅ Customer success stories \n✅ Integration guides with other tools \n✅ Best practices and optimization tips \n✅ Conference talks and presentations \n✅ Research papers using Neuron/Trainium/Inferentia \n\n### What Not to Include\n\n❌ Marketing content without technical substance \n❌ Broken or paywalled links \n❌ Content unrelated to AWS Neuron ecosystem \n❌ Duplicate submissions \n❌ Self-promotional content without value to community \n\n## Technical Details\n\nThis page uses:\n- **Sphinx** with `sphinxcontrib.datatemplates` extension\n- **YAML** for data storage\n- **Jinja2** templates for rendering\n- **sphinx-design** for grid layouts\n\nThe system is fully static - no backend required. All content is rendered at build time.\n\n## License\n\nBy contributing, you agree that your contributions will be licensed under the same license as this project. See the repository LICENSE files for details.\n" }, { "path": "about-neuron/news-and-blogs/JIRA-INTEGRATION-DESIGN.md", "content": "# Jira Integration Design for News & Blogs\n\n## Overview\n\nThis document describes a design for populating the `news-and-blogs.yaml` file from Jira tickets, allowing contributors to submit article links via Jira instead of direct pull requests.\n\n## Design Goals\n\n1. **Simple for contributors**: Submit a Jira ticket with article metadata\n2. **Automated**: Minimal manual intervention to add articles to YAML\n3. **Quality control**: Review process before articles appear on the site\n4. **Compatible**: Works with existing Sphinx build process\n5. **No backend required**: Leverages existing CI/CD infrastructure\n\n## Architecture\n\n### Option 1: GitHub Actions + Jira API (Recommended)\n\n```\nJira Ticket Created → GitHub Action Triggered → Parse Ticket → Update YAML → Create PR\n```\n\n**Components:**\n\n1. **Jira Ticket Template**: Custom issue type \"News Article Submission\"\n2. **GitHub Action**: Runs on schedule (e.g., hourly) or webhook\n3. **Python Script**: Fetches approved tickets, generates YAML entries\n4. **Automated PR**: Creates pull request with new articles\n\n**Workflow:**\n\n```yaml\n# .github/workflows/sync-jira-articles.yml\nname: Sync Jira Articles to YAML\n\non:\n schedule:\n - cron: '0 */6 * * *' # Every 6 hours\n workflow_dispatch: # Manual trigger\n\njobs:\n sync-articles:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n \n - name: Set up Python\n uses: actions/setup-python@v4\n with:\n python-version: '3.9'\n \n - name: Install dependencies\n run: pip install jira pyyaml\n \n - name: Fetch and process Jira tickets\n env:\n JIRA_URL: ${{ secrets.JIRA_URL }}\n JIRA_USER: ${{ secrets.JIRA_USER }}\n JIRA_TOKEN: ${{ secrets.JIRA_TOKEN }}\n run: python scripts/sync_jira_articles.py\n \n - name: Create Pull Request\n uses: peter-evans/create-pull-request@v5\n with:\n commit-message: 'Add articles from Jira'\n title: 'Add news articles from Jira submissions'\n body: 'Automated PR from Jira article submissions'\n branch: jira-articles-sync\n```\n\n### Option 2: Jira Automation + Webhook\n\n```\nJira Ticket Approved → Webhook to GitHub → GitHub Action → Update YAML → Create PR\n```\n\n**Advantages:**\n- Real-time updates when tickets are approved\n- No polling required\n- More efficient\n\n**Setup:**\n1. Configure Jira Automation rule\n2. Trigger on status change to \"Approved\"\n3. Send webhook to GitHub repository dispatch endpoint\n\n### Option 3: Manual Script (Simplest)\n\n```\nDeveloper runs script → Fetches approved tickets → Updates YAML → Commits changes\n```\n\n**Use case:** Lower volume, manual review preferred\n\n## Jira Ticket Structure\n\n### Custom Fields Required\n\n```\nIssue Type: News Article Submission\n\nFields:\n- Article Title (text, required)\n- Article URL (URL, required)\n- Description (text area, required)\n- Author Name (text, required)\n- Author URL (URL, optional)\n- Publication Date (date, required)\n- Category (dropdown: blog|news|tutorial|case-study|benchmark)\n- Locale (dropdown: en-US|ja-JP|zh-CN|ko-KR|de-DE|fr-FR|es-ES|pt-BR)\n- Keywords (labels or multi-select)\n- Featured (checkbox)\n- Icon (text, optional, for featured articles)\n\nStatus Workflow:\n- Submitted → Under Review → Approved → Published → Rejected\n```\n\n### Example Jira Ticket\n\n```\nTitle: Add Karakuri AWS Trainium Tutorial\n\nFields:\n- Article Title: AWS Trainium: 50 Exercises\n- Article URL: https://zenn.dev/karakuri_blog/articles/5ccedeee1beb08\n- Description: Learn how to build LLMs for Trainium accelerators...\n- Author Name: Karakuri\n- Author URL: https://about.karakuri.ai/\n- Publication Date: 2026-02-19\n- Category: tutorial\n- Locale: ja-JP\n- Keywords: trainium, llm, training, tutorial\n- Featured: Yes\n- Icon: 🚀\n```\n\n## Implementation Script\n\n### `scripts/sync_jira_articles.py`\n\n```python\n#!/usr/bin/env python3\n\"\"\"\nSync approved Jira article submissions to news-and-blogs.yaml\n\"\"\"\n\nimport os\nimport yaml\nfrom jira import JIRA\nfrom datetime import datetime\n\n# Configuration\nJIRA_URL = os.environ.get('JIRA_URL')\nJIRA_USER = os.environ.get('JIRA_USER')\nJIRA_TOKEN = os.environ.get('JIRA_TOKEN')\nYAML_FILE = 'about-neuron/news-and-blogs/news-and-blogs.yaml'\n\n# JQL to find approved, unpublished articles\nJQL_QUERY = 'project = NEURON AND issuetype = \"News Article Submission\" AND status = \"Approved\" AND labels != \"published\"'\n\ndef connect_jira():\n \"\"\"Connect to Jira instance\"\"\"\n return JIRA(server=JIRA_URL, basic_auth=(JIRA_USER, JIRA_TOKEN))\n\ndef fetch_approved_articles(jira):\n \"\"\"Fetch approved article submissions from Jira\"\"\"\n issues = jira.search_issues(JQL_QUERY, maxResults=100)\n articles = []\n \n for issue in issues:\n article = {\n 'title': issue.fields.customfield_10001, # Article Title\n 'url': issue.fields.customfield_10002, # Article URL\n 'description': issue.fields.customfield_10003, # Description\n 'author': issue.fields.customfield_10004, # Author Name\n 'date': issue.fields.customfield_10006, # Publication Date\n 'category': issue.fields.customfield_10007.value, # Category\n 'locale': issue.fields.customfield_10008.value, # Locale\n 'keywords': [label.name for label in issue.fields.labels if label.name != 'published'],\n 'featured': bool(issue.fields.customfield_10009), # Featured checkbox\n }\n \n # Optional fields\n if issue.fields.customfield_10005: # Author URL\n article['author_url'] = issue.fields.customfield_10005\n \n if issue.fields.customfield_10010: # Icon (for featured)\n article['icon'] = issue.fields.customfield_10010\n \n articles.append({\n 'article': article,\n 'issue_key': issue.key\n })\n \n return articles\n\ndef load_yaml():\n \"\"\"Load existing YAML file\"\"\"\n with open(YAML_FILE, 'r', encoding='utf-8') as f:\n return yaml.safe_load(f)\n\ndef article_exists(data, url):\n \"\"\"Check if article URL already exists in YAML\"\"\"\n all_urls = [a['url'] for a in data.get('featured_articles', [])]\n all_urls.extend([a['url'] for a in data.get('all_articles', [])])\n return url in all_urls\n\ndef add_articles_to_yaml(data, new_articles):\n \"\"\"Add new articles to appropriate sections\"\"\"\n added_keys = []\n \n for item in new_articles:\n article = item['article']\n \n # Skip if already exists\n if article_exists(data, article['url']):\n print(f\"Skipping duplicate: {article['title']}\")\n continue\n \n # Add to appropriate section\n if article.get('featured', False):\n data['featured_articles'].append(article)\n else:\n data['all_articles'].append(article)\n \n added_keys.append(item['issue_key'])\n print(f\"Added: {article['title']} ({item['issue_key']})\")\n \n return added_keys\n\ndef save_yaml(data):\n \"\"\"Save updated YAML file\"\"\"\n with open(YAML_FILE, 'w', encoding='utf-8') as f:\n yaml.dump(data, f, allow_unicode=True, sort_keys=False, default_flow_style=False)\n\ndef mark_as_published(jira, issue_keys):\n \"\"\"Add 'published' label to Jira tickets and transition to Published status\"\"\"\n for key in issue_keys:\n issue = jira.issue(key)\n \n # Add published label\n labels = issue.fields.labels\n if 'published' not in labels:\n labels.append('published')\n issue.update(fields={'labels': labels})\n \n # Transition to Published status (adjust transition ID as needed)\n try:\n jira.transition_issue(issue, 'Published')\n except Exception as e:\n print(f\"Could not transition {key}: {e}\")\n\ndef main():\n print(\"Connecting to Jira...\")\n jira = connect_jira()\n \n print(\"Fetching approved articles...\")\n new_articles = fetch_approved_articles(jira)\n \n if not new_articles:\n print(\"No new articles to add.\")\n return\n \n print(f\"Found {len(new_articles)} approved articles\")\n \n print(\"Loading existing YAML...\")\n data = load_yaml()\n \n print(\"Adding articles to YAML...\")\n added_keys = add_articles_to_yaml(data, new_articles)\n \n if added_keys:\n print(\"Saving YAML...\")\n save_yaml(data)\n \n print(\"Marking Jira tickets as published...\")\n mark_as_published(jira, added_keys)\n \n print(f\"Successfully added {len(added_keys)} articles!\")\n else:\n print(\"No new articles added (all were duplicates)\")\n\nif __name__ == '__main__':\n main()\n```\n\n## Setup Instructions\n\n### 1. Configure Jira\n\n1. Create custom issue type \"News Article Submission\"\n2. Add custom fields (see structure above)\n3. Configure workflow: Submitted → Under Review → Approved → Published\n4. Create Jira API token for automation user\n\n### 2. Configure GitHub Secrets\n\nAdd these secrets to your GitHub repository:\n\n```\nJIRA_URL: https://your-company.atlassian.net\nJIRA_USER: automation@your-company.com\nJIRA_TOKEN: \n```\n\n### 3. Add GitHub Action\n\nCreate `.github/workflows/sync-jira-articles.yml` with the workflow above.\n\n### 4. Install Dependencies\n\nAdd to `requirements.txt`:\n```\njira==3.5.0\nPyYAML==6.0\n```\n\n### 5. Test\n\n1. Create a test Jira ticket\n2. Approve it\n3. Run workflow manually: Actions → Sync Jira Articles → Run workflow\n4. Verify PR is created with new article\n\n## Alternative: Simpler Webhook Approach\n\nIf you want something lighter without Jira API polling:\n\n### Jira Automation Rule\n\n```\nTrigger: Issue transitioned to \"Approved\"\nCondition: Issue type = \"News Article Submission\"\nAction: Send web request\n\nURL: https://api.github.com/repos/aws-neuron/aws-neuron-sdk/dispatches\nMethod: POST\nHeaders:\n Authorization: Bearer ${GITHUB_TOKEN}\n Accept: application/vnd.github.v3+json\nBody:\n{\n \"event_type\": \"jira-article-approved\",\n \"client_payload\": {\n \"issue_key\": \"{{issue.key}}\",\n \"title\": \"{{issue.customfield_10001}}\",\n \"url\": \"{{issue.customfield_10002}}\",\n \"description\": \"{{issue.customfield_10003}}\",\n \"author\": \"{{issue.customfield_10004}}\",\n \"date\": \"{{issue.customfield_10006}}\",\n \"category\": \"{{issue.customfield_10007}}\",\n \"locale\": \"{{issue.customfield_10008}}\"\n }\n}\n```\n\nThen GitHub Action receives webhook and processes directly without Jira API calls.\n\n## Maintenance\n\n### Regular Tasks\n\n1. **Monitor failed syncs**: Check GitHub Action logs\n2. **Review PRs**: Automated PRs should still be reviewed before merge\n3. **Clean up Jira**: Archive old Published tickets\n4. **Update mappings**: If custom field IDs change, update script\n\n### Troubleshooting\n\n**Articles not syncing:**\n- Check Jira API credentials\n- Verify custom field IDs match\n- Check JQL query returns expected tickets\n\n**Duplicate articles:**\n- Script checks URL before adding\n- Manually remove duplicates from YAML if needed\n\n**Formatting issues:**\n- Validate YAML after sync: `python -m yaml about-neuron/news-and-blogs/news-and-blogs.yaml`\n- Check for special characters in descriptions\n\n## Security Considerations\n\n1. **API Tokens**: Store in GitHub Secrets, never commit\n2. **Permissions**: Use dedicated Jira service account with minimal permissions\n3. **Validation**: Sanitize all input from Jira before adding to YAML\n4. **Review**: Always review automated PRs before merging\n\n## Cost & Complexity\n\n| Approach | Setup Time | Maintenance | Cost |\n|----------|-----------|-------------|------|\n| GitHub Actions + Jira API | 4-6 hours | Low | Free (GitHub Actions) |\n| Webhook + GitHub Actions | 2-3 hours | Very Low | Free |\n| Manual Script | 1-2 hours | Medium | Free |\n\n## Recommendation\n\n**For production use**: Start with **Option 3 (Manual Script)** to validate the workflow, then upgrade to **Option 1 (GitHub Actions)** once the process is proven and volume increases.\n\n**For high volume**: Use **Option 2 (Webhook)** for real-time updates.\n\n## Future Enhancements\n\n1. **Validation**: Add URL validation, duplicate detection in Jira\n2. **Preview**: Generate preview of how article will appear\n3. **Scheduling**: Support future publication dates\n4. **Analytics**: Track article submissions and approval rates\n5. **Notifications**: Notify submitters when articles are published\n6. **Bulk import**: Support CSV upload for multiple articles\n" }, { "path": "about-neuron/news-and-blogs/README.md", "content": "# AWS Neuron News and Blogs System\n\nThis directory contains a dynamic, community-driven news and blogs page for AWS Neuron, Trainium, and Inferentia content.\n\n## Overview\n\nThe system allows external contributors to add links to relevant articles, blog posts, and news through a simple YAML data file, without requiring any backend infrastructure.\n\n## Architecture\n\n```\nabout-neuron/news-and-blogs/\n├── index.rst # Main page (uses datatemplate directives)\n├── news-and-blogs.yaml # Data file with all article metadata\n├── featured-articles.tmpl # Jinja2 template for featured section\n├── all-articles.tmpl # Jinja2 template for all articles section\n├── CONTRIBUTING.md # Contribution guidelines\n└── README.md # This file\n```\n\n## How It Works\n\n1. **Data Storage**: Article metadata is stored in `news-and-blogs.yaml`\n2. **Templating**: Jinja2 templates (`*.tmpl`) define how articles are rendered\n3. **Rendering**: Sphinx's `datatemplates` extension processes the YAML and templates at build time\n4. **Output**: Static HTML with grid cards using `sphinx-design`\n\n## Key Features\n\n- ✅ **No backend required** - fully static site generation\n- ✅ **Easy contributions** - edit a YAML file and submit a PR\n- ✅ **Version controlled** - all changes tracked in Git\n- ✅ **Automated rendering** - Sphinx handles everything at build time\n- ✅ **Responsive design** - uses sphinx-design grid system\n- ✅ **Maintainable** - clear separation of data, templates, and content\n\n## Adding New Articles\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for detailed instructions.\n\nQuick example:\n\n```yaml\nall_articles:\n - title: \"My Article Title\"\n url: \"https://example.com/article\"\n description: \"Brief description\"\n author: \"Author Name\"\n date: \"2026-01-15\"\n category: \"blog\"\n featured: false\n```\n\n## Modifying Templates\n\nTemplates use Jinja2 syntax and have access to the YAML data structure.\n\n### Featured Articles Template (`featured-articles.tmpl`)\n\nRenders articles from the `featured_articles` section with:\n- Large cards with borders\n- Icons and bold titles\n- Author attribution with links\n- Publication dates\n\n### All Articles Template (`all-articles.tmpl`)\n\nRenders articles from the `all_articles` section with:\n- 2-column grid on desktop, 1-column on mobile\n- Simple card layout\n- Title and description\n\n## Customization\n\n### Adding New Fields\n\n1. Add field to YAML entries:\n ```yaml\n - title: \"Article\"\n new_field: \"value\"\n ```\n\n2. Update template to use it:\n ```jinja\n {{ article.new_field }}\n ```\n\n### Changing Layout\n\nEdit the grid directive in templates:\n```rst\n.. grid:: 1 1 2 3 # 1 col mobile, 1 tablet, 2 desktop, 3 wide\n :gutter: 2\n```\n\n### Adding Filters/Sorting\n\nYou can add Jinja2 filters in templates:\n\n```jinja\n{% for article in all_articles | sort(attribute='date', reverse=True) %}\n {# Sorted by date, newest first #}\n{% endfor %}\n```\n\n## Dependencies\n\nRequired Sphinx extensions (already in `conf.py`):\n- `sphinxcontrib.datatemplates` - YAML data processing\n- `sphinx_design` - Grid card layouts\n\n## Testing Locally\n\n1. Install dependencies:\n ```bash\n pip install -r requirements.txt\n ```\n\n2. Build documentation:\n ```bash\n sphinx-build -b html . _build/html\n ```\n\n3. View the page:\n ```bash\n open _build/html/about-neuron/news-and-blogs/index.html\n ```\n\n## Troubleshooting\n\n### Template Not Found Error\n\nEnsure templates are in the same directory as `index.rst` or add the directory to `templates_path` in `conf.py`.\n\n### YAML Parse Error\n\nValidate your YAML:\n```bash\npython -c \"import yaml; yaml.safe_load(open('news-and-blogs.yaml'))\"\n```\n\n### Articles Not Rendering\n\nCheck that:\n1. YAML file is in the same directory as `index.rst`\n2. Template files exist and have correct names\n3. YAML structure matches template expectations\n\n## Future Enhancements\n\nPossible improvements:\n- Add category filtering/grouping\n- Add search functionality\n- Add RSS feed generation\n- Add automatic link checking\n- Add article metadata validation\n- Sort by date automatically\n- Add pagination for large lists\n\n## Support\n\nFor questions or issues:\n- Open a GitHub issue\n- Contact AWS Neuron support team\n- See main repository CONTRIBUTING.md\n" }, { "path": "about-neuron/news-and-blogs/article-template.yaml", "content": "# Article Entry Template\n# \n# Copy this template and fill in your article details.\n# Then add it to the appropriate section in news-and-blogs.yaml\n#\n# For featured articles (requires AWS Neuron team approval):\n# Add to the 'featured_articles' section with featured: true and an icon\n#\n# For regular articles:\n# Add to the 'all_articles' section with featured: false\n\n# TEMPLATE - Copy everything below this line\n# ============================================\n\n- title: \"Your Article Title Here\"\n url: \"https://your-website.com/path-to-article\"\n description: \"A clear, concise description of your article in 1-2 sentences. Explain what readers will learn or discover.\"\n author: \"Your Name or Organization Name\"\n author_url: \"https://your-website.com\" # Optional: Your website or profile URL (for featured articles)\n date: \"YYYY-MM-DD\" # Publication date in YYYY-MM-DD format (e.g., 2026-01-27)\n category: \"blog\" # Choose ONE: blog, news, tutorial, case-study, benchmark\n locale: \"en-US\" # Language/region code (e.g., en-US, ja-JP, zh-CN, de-DE, fr-FR, es-ES, pt-BR, ko-KR)\n keywords: [\"keyword1\", \"keyword2\", \"keyword3\"] # List of 3-10 relevant keywords for filtering/search\n featured: false # Set to false unless approved by AWS Neuron team\n icon: \"📝\" # Optional: Single emoji for featured articles (e.g., 🚀 📊 🎯 💡 ⚡)\n\n# ============================================\n# FIELD DESCRIPTIONS\n# ============================================\n#\n# title (required):\n# - Clear, descriptive title of your article\n# - Keep under 100 characters\n# - Use title case\n#\n# url (required):\n# - Full HTTPS URL to your article\n# - Must be publicly accessible\n# - Should not require login or paywall\n#\n# description (required):\n# - Brief summary of article content\n# - 20-500 characters recommended\n# - Focus on what readers will learn\n# - Avoid marketing language\n#\n# author (required):\n# - Your name or organization\n# - Will be displayed as attribution\n#\n# author_url (optional):\n# - Link to your website or profile\n# - Only used in featured articles\n# - Must be valid HTTPS URL\n#\n# date (required):\n# - Publication date in YYYY-MM-DD format\n# - Use the date the article was published\n# - Not the date you're adding it here\n#\n# category (required):\n# - blog: Technical blog posts and articles\n# - news: News announcements and press releases\n# - tutorial: Step-by-step guides and how-tos\n# - case-study: Customer success stories and use cases\n# - benchmark: Performance benchmarks and comparisons\n#\n# locale (required):\n# - Language and region code in format: language-REGION\n# - Examples: en-US (English-US), ja-JP (Japanese), zh-CN (Chinese-Simplified)\n# - Common codes: en-US, en-GB, ja-JP, zh-CN, zh-TW, ko-KR, de-DE, fr-FR, \n# es-ES, es-MX, pt-BR, pt-PT, it-IT, nl-NL, ru-RU, ar-SA, hi-IN\n# - A flag emoji will be displayed based on the locale\n# - Unknown locales will display 🌐 globe icon\n#\n# keywords (required):\n# - List of 3-10 relevant keywords for filtering and search\n# - Use lowercase, hyphenated format (e.g., \"machine-learning\", \"pytorch\")\n# - Include technology names, topics, and key concepts\n# - Examples: [\"trainium\", \"inference\", \"pytorch\", \"llm\", \"optimization\"]\n# - Keywords help users find your article through filtering and search\n#\n# featured (required):\n# - true: Article appears in featured section (requires approval)\n# - false: Article appears in all articles section\n# - Most submissions should use false\n#\n# icon (optional):\n# - Single emoji character\n# - Only used for featured articles\n# - Examples: 🚀 📊 🎯 💡 ⚡ 🔥 ✨ 🌟 📈 🛠️\n#\n# ============================================\n# EXAMPLES\n# ============================================\n\n# Example 1: Tutorial Article\n- title: \"Getting Started with PyTorch on AWS Trainium\"\n url: \"https://example.com/pytorch-trainium-tutorial\"\n description: \"A comprehensive guide to training PyTorch models on AWS Trainium instances, including setup, optimization tips, and common pitfalls to avoid.\"\n author: \"Jane Developer\"\n date: \"2026-01-15\"\n category: \"tutorial\"\n locale: \"en-US\"\n keywords: [\"pytorch\", \"trainium\", \"training\", \"tutorial\", \"getting-started\"]\n featured: false\n\n# Example 2: Benchmark Article\n- title: \"BERT Inference Performance: Inferentia2 vs GPU Comparison\"\n url: \"https://example.com/bert-benchmark\"\n description: \"Detailed performance comparison of BERT inference on AWS Inferentia2 vs leading GPU instances, including cost analysis and throughput metrics.\"\n author: \"ML Performance Lab\"\n date: \"2026-01-20\"\n category: \"benchmark\"\n locale: \"en-US\"\n keywords: [\"inferentia\", \"bert\", \"benchmark\", \"performance\", \"gpu-comparison\"]\n featured: false\n\n# Example 3: Case Study\n- title: \"How Acme Corp Reduced ML Training Costs by 60% with Trainium\"\n url: \"https://example.com/acme-case-study\"\n description: \"Learn how Acme Corp migrated their large language model training to AWS Trainium and achieved significant cost savings while maintaining performance.\"\n author: \"Acme Corp Engineering Team\"\n date: \"2026-01-25\"\n category: \"case-study\"\n locale: \"en-US\"\n keywords: [\"trainium\", \"cost-optimization\", \"llm\", \"case-study\", \"migration\"]\n featured: false\n\n# Example 4: Featured Article (requires approval)\n- title: \"Advanced Optimization Techniques for Neuron Compiler\"\n url: \"https://example.com/neuron-optimization\"\n description: \"Deep dive into advanced compiler optimization techniques for AWS Neuron, with practical examples and performance improvements.\"\n author: \"AWS Neuron Team\"\n author_url: \"https://aws.amazon.com/machine-learning/neuron/\"\n date: \"2026-01-27\"\n category: \"blog\"\n locale: \"en-US\"\n keywords: [\"neuron\", \"compiler\", \"optimization\", \"performance\", \"advanced\"]\n featured: true\n icon: \"⚡\"\n\n# Example 5: Japanese Article\n- title: \"AWS Trainiumで大規模言語モデルを訓練する\"\n url: \"https://example.jp/trainium-llm-training\"\n description: \"AWS Trainiumを使用して大規模言語モデルを効率的に訓練する方法を詳しく解説します。\"\n author: \"日本のMLエンジニア\"\n date: \"2026-01-20\"\n category: \"tutorial\"\n locale: \"ja-JP\"\n keywords: [\"trainium\", \"llm\", \"training\", \"japanese\", \"tutorial\"]\n featured: false\n" }, { "path": "about-neuron/news-and-blogs/index.rst", "content": ".. meta::\n :description: Links to external news and blog articles about AWS Neuron and Trainium/Inferentia ML accelerators.\n :date-modified: 02/26/2026\n\n.. _neuron-news:\n\nAWS Neuron News and Blogs\n=========================\n\nStay up to date with the latest news, announcements, and technical blog posts about AWS Neuron, AWS Trainium, and AWS Inferentia. Discover customer success stories, performance benchmarks, best practices, and deep dives into machine learning acceleration on AWS.\n\n----\n\nFeatured Articles\n-----------------\n\nRead recent blogs and technical content about Neuron, Trainium, and Inferentia from AWS subject matter experts and our highly experienced customers.\n\n.. datatemplate:yaml:: news-and-blogs.yaml\n\n .. grid:: 1\n :gutter: 2\n {% for article in data.featured_articles %}\n {% if article.locale == 'en-US' %}{% set flag = '🇺🇸' %}{% set locale_name = 'English' %}{% elif article.locale == 'ja-JP' %}{% set flag = '🇯🇵' %}{% set locale_name = 'Japanese' %}{% elif article.locale == 'zh-CN' %}{% set flag = '🇨🇳' %}{% set locale_name = 'Chinese' %}{% elif article.locale == 'ko-KR' %}{% set flag = '🇰🇷' %}{% set locale_name = 'Korean' %}{% else %}{% set flag = '🌐' %}{% set locale_name = 'Unknown' %}{% endif %}\n\n .. grid-item-card::\n :class-card: sd-border-2\n :link: {{ article.url }}\n\n {{ article.icon }} **{{ article.title }}**\n ^^^\n {{ article.description }}\n +++\n **Published on**: {{ article.date }} | {{ flag }} ({{ locale_name }}) | Content by `{{ article.author }} <{{ article.author_url }}>`__\n {% endfor %}\n\n.. note::\n \n This page is regularly updated with new content. Bookmark it to stay informed about the latest developments in AWS Neuron, Trainium, and Inferentia.\n \n**For the full list of featured articles and posts, go to the :ref:`News & Blogs ` section of this page.**\n\n.. _all-articles:\n\nNews & Blogs \n-------------\n\nExplore the latest news, press releases, and industry coverage about AWS Neuron, Trainium, and Inferentia.\n\n.. raw:: html\n\n
\n \n \n
\n\n.. datatemplate:yaml:: news-and-blogs.yaml\n\n .. grid:: 1 1 2 2\n :gutter: 2\n :class-container: articles-grid news-blogs-grid\n {% for article in data.all_articles|sort(attribute='date', reverse=True) %}\n {% if article.locale == 'en-US' %}{% set flag = '🇺🇸' %}{% set locale_name = 'English' %}{% elif article.locale == 'ja-JP' %}{% set flag = '🇯🇵' %}{% set locale_name = 'Japanese' %}{% elif article.locale == 'zh-CN' %}{% set flag = '🇨🇳' %}{% set locale_name = 'Chinese' %}{% elif article.locale == 'ko-KR' %}{% set flag = '🇰🇷' %}{% set locale_name = 'Korean' %}{% else %}{% set flag = '🌐' %}{% set locale_name = 'Unknown' %}{% endif %}\n\n .. grid-item-card::\n :link: {{ article.url }}\n :class-card: sd-border-1 article-card\n :class-body: article-locale-{{ article.locale }}\n\n **{{ article.title }}**\n ^^^\n {{ article.description }}\n +++\n **Published on**: {{ article.date }} | {{ flag }} ({{ locale_name }})\n {% endfor %}\n\n.. raw:: html\n\n \n\n.. important::\n\n AWS and Neuron provide links to external articles and posts to help you discover them, but do not commission or own any content not created by AWS employees. This list is curated based on internal and customer recommendations. \n\n**Want to add your article?** Go to `https://github.com/aws-neuron/aws-neuron-sdk `_, edit ``about-neuron/news-and-blogs/news-and-blogs.yaml`` to add your submission, and submit a pull request. \n\n\n\n" }, { "path": "about-neuron/news-and-blogs/news-and-blogs.yaml", "content": "# AWS Neuron News and Blogs Data File\n# \n# This file contains metadata for external articles, blog posts, and news about\n# AWS Neuron, Trainium, and Inferentia.\n#\n# To contribute a new article:\n# 1. Add a new entry to the appropriate section below\n# 2. Follow the existing format exactly\n# 3. Submit a pull request with your changes\n#\n# Entry format:\n# - title: \"Article Title\"\n# url: \"https://example.com/article\"\n# description: \"Brief description of the article content\"\n# author: \"Author Name or Organization\"\n# date: \"YYYY-MM-DD\"\n# category: \"blog|news|tutorial|case-study|benchmark\"\n# locale: \"en-US\" # Language/region code (e.g., en-US, ja-JP, zh-CN, de-DE, fr-FR, es-ES, pt-BR, ko-KR)\n# keywords: [\"keyword1\", \"keyword2\", \"keyword3\"] # List of relevant keywords for filtering\n# featured: true|false # Set to true for featured articles section\n\nfeatured_articles:\n - title: \"AWS Trainium: 50 Exercises\"\n url: \"https://zenn.dev/karakuri_blog/articles/5ccedeee1beb08\"\n description: \"Learn how to build LLMs for Trainium accelerators with this rich 50-lesson guide from customer Karakuri.\"\n author: \"Karakuri\"\n author_url: \"https://about.karakuri.ai/\"\n date: \"2026-02-19\"\n category: \"tutorial\"\n locale: \"en-US\"\n keywords: [\"trainium\", \"llm\", \"training\", \"tutorial\", \"japanese\"]\n featured: true\n icon: \"🚀\"\n\n - title: \"Cost-effective AI image generation with PixArt-Sigma inference on AWS Trainium and AWS Inferentia\"\n url: \"https://aws.amazon.com/blogs/machine-learning/cost-effective-ai-image-generation-with-pixart-sigma-inference-on-aws-trainium-and-aws-inferentia/\"\n description: \"Learn how to use AWS Trainium and Inferentia to deploy a PixArt-Sigma diffusion transformer model.\"\n author: \"AWS Neuron Team\"\n author_url: \"https://aws.amazon.com/machine-learning/neuron/\"\n date: \"2026-02-19\"\n category: \"blog\"\n locale: \"en-US\"\n keywords: [\"inferentia\", \"trainium\", \"inference\", \"diffusion\", \"image-generation\"]\n featured: true\n icon: \"📊\"\n\nall_articles:\n # Japanese Articles\n - title: \"AWS Neuron 関連記事まとめ\"\n url: \"https://zenn.dev/tosshi/articles/36f3615e26c323\"\n description: \"AWS Neuron エコシステムに関する自身が作成した一連の技術記事のインデックス\"\n author: \"littlemex\"\n date: \"2026-02-20\"\n category: \"blog\"\n locale: \"ja-JP\"\n keywords: [\"trainium\", \"neuron\", \"collective-communication\", \"architecture\", \"japanese\"]\n featured: false\n\n - title: \"【AWS re:Invent 2025 速報】AWS 自社設計 AIチップ AWS Trainium3 の全貌\"\n url: \"https://zenn.dev/aws_japan/articles/06808526d5c75f\"\n description: \"AWS re:Invent 2025で発表されたAWS Trainium3カスタムAIチップの完全な概要をお届けします。\"\n author: \"AWS Japan\"\n date: \"2025-12-06\"\n category: \"news\"\n locale: \"ja-JP\"\n keywords: [\"trainium3\", \"reinvent\", \"announcement\", \"ai-chip\"]\n featured: false\n\n - title: \"【AWS Trainium 50本ノック #0】はじめに\"\n url: \"https://zenn.dev/karakuri_blog/articles/77d93c40b27b60\"\n description: \"AWS Trainium 50本ノックシリーズの紹介 - 入門ガイド。\"\n author: \"Karakuri\"\n date: \"2025-11-18\"\n category: \"tutorial\"\n locale: \"ja-JP\"\n keywords: [\"trainium\", \"tutorial\", \"getting-started\", \"series\"]\n featured: false\n\n - title: \"「Syn Pro」開発レポート：AWS TrainiumとRFTによる高性能日本語LLMの実現\"\n url: \"https://zenn.dev/karakuri_blog/articles/b923acfc86083b\"\n description: \"AWS TrainiumとRFTを使用した高性能日本語LLMの構築に関する開発レポート。\"\n author: \"Karakuri\"\n date: \"2025-10-24\"\n category: \"case-study\"\n locale: \"ja-JP\"\n keywords: [\"trainium\", \"llm\", \"japanese\", \"rft\", \"case-study\"]\n featured: false\n\n - title: \"AWS Inferentia2 + Llama 3.2 にできること\"\n url: \"https://zenn.dev/exwzd/articles/20250930-inferentia-llama\"\n description: \"AWS Inferentia2とLlama 3.2モデルでできることを紹介します。\"\n author: \"exwzd\"\n date: \"2025-09-30\"\n category: \"blog\"\n locale: \"ja-JP\"\n keywords: [\"inferentia2\", \"llama\", \"capabilities\", \"inference\"]\n featured: false\n\n - title: \"AWS Inferentia2とvLLMでLlama 3.2の推論サーバーを構築する手順\"\n url: \"https://zenn.dev/exwzd/articles/20250827_inferentia_compile\"\n description: \"AWS Inferentia2とvLLMを使用してLlama 3.2推論サーバーを構築するステップバイステップガイド。\"\n author: \"exwzd\"\n date: \"2025-08-28\"\n category: \"tutorial\"\n locale: \"ja-JP\"\n keywords: [\"inferentia2\", \"vllm\", \"llama\", \"inference\", \"tutorial\"]\n featured: false\n\n - title: \"【開催報告】Neuron Community – Vol.2\"\n url: \"https://aws.amazon.com/jp/blogs/news/neuron-community-vol-2/\"\n description: \"Neuron Community Vol.2の開催報告。\"\n author: \"AWS Japan\"\n date: \"2025-07-24\"\n category: \"news\"\n locale: \"ja-JP\"\n keywords: [\"community\", \"event\", \"neuron\", \"japan\"]\n featured: false\n\n - title: \"KARAKURI VL - 日本語コンピュータユースに特化した視覚言語モデル\"\n url: \"https://zenn.dev/karakuri_blog/articles/28c73f2ada797a\"\n description: \"日本語コンピュータユースに特化したビジョン言語モデルKARAKURI VLの紹介。\"\n author: \"Karakuri\"\n date: \"2025-07-11\"\n category: \"blog\"\n locale: \"ja-JP\"\n keywords: [\"vision-language\", \"japanese\", \"multimodal\", \"karakuri\"]\n featured: false\n\n - title: \"LLM-jp Chatbot Arenaを試験運用しました\"\n url: \"https://llm-jp.nii.ac.jp/ja/blog/blog-836/\"\n description: \"LLM-jp Chatbot Arenaの試験運用に関するレポート。\"\n author: \"LLM-jp\"\n date: \"2025-05-12\"\n category: \"blog\"\n locale: \"ja-JP\"\n keywords: [\"llm\", \"chatbot\", \"arena\", \"japanese\"]\n featured: false\n\n - title: \"【開催報告】Neuron Community – Day One\"\n url: \"https://aws.amazon.com/jp/blogs/news/neuron-community-day-one/\"\n description: \"初回Neuron Community Dayの開催報告。\"\n author: \"AWS Japan\"\n date: \"2025-04-14\"\n category: \"news\"\n locale: \"ja-JP\"\n keywords: [\"community\", \"event\", \"neuron\", \"japan\"]\n featured: false\n\n - title: \"EKS Auto Mode でサクッと機械学習用インスタンスを利用してみる。 AWS 独自設計チップ搭載の Trainium と Inferentia を使ってみた！\"\n url: \"https://dev.classmethod.jp/articles/eks-auto-mode-gpu-aws-trainium-inferentia/\"\n description: \"EKS Auto Modeを使用してMLインスタンスを簡単に利用する方法。AWS TrainiumとInferentiaチップの活用ガイド。\"\n author: \"Classmethod\"\n date: \"2025-01-02\"\n category: \"tutorial\"\n locale: \"ja-JP\"\n keywords: [\"eks\", \"trainium\", \"inferentia\", \"kubernetes\", \"tutorial\"]\n featured: false\n\n # Korean Articles\n - title: \"Nota AI가 제안하는 AWS Inferentia에서 다양한 LLM 모델 양자화 최적화기법 사용하기\"\n url: \"https://aws.amazon.com/ko/blogs/tech/llm-model-quantization-techniques-for-aws-inferentia-by-nota-ai/\"\n description: \"Nota AI가 제안하는 AWS Inferentia에서 LLM 모델 양자화 최적화 기법.\"\n author: \"Nota AI / AWS Korea\"\n date: \"2026-01-20\"\n category: \"blog\"\n locale: \"ko-KR\"\n keywords: [\"inferentia\", \"quantization\", \"llm\", \"optimization\", \"nota-ai\"]\n featured: false\n\n - title: \"Nota AI가 제안하는 Transformer 모델을 AWS Inferentia/Trainium에 손쉽게 배포하는 방법\"\n url: \"https://aws.amazon.com/ko/blogs/tech/tips-for-using-transformer-models-on-aws-inf-and-trn/\"\n description: \"Nota AI가 제안하는 AWS Inferentia/Trainium에서 Transformer 모델을 쉽게 배포하는 방법.\"\n author: \"Nota AI / AWS Korea\"\n date: \"2025-04-09\"\n category: \"blog\"\n locale: \"ko-KR\"\n keywords: [\"transformer\", \"deployment\", \"inferentia\", \"trainium\", \"nota-ai\"]\n featured: false\n\n - title: \"콜드스타트 추천 문제를 AWS Trainium과 vLLM으로 해결하는 자동화 전략\"\n url: \"https://blog.a-cloud.co.kr/2025/07/25/%EC%BD%9C%EB%93%9C%EC%8A%A4%ED%83%80%ED%8A%B8-%EC%B6%94%EC%B2%9C-%EB%AC%B8%EC%A0%9C%EB%A5%BC-aws-trainium%EA%B3%BC-vllm%EC%9C%BC%EB%A1%9C-%ED%95%B4%EA%B2%B0%ED%95%98%EB%8A%94-%EC%9E%90%EB%8F%99/\"\n description: \"AWS Trainium과 vLLM을 사용하여 콜드 스타트 추천 문제를 해결하는 자동화 전략.\"\n author: \"A-Cloud\"\n date: \"2025-07-25\"\n category: \"blog\"\n locale: \"ko-KR\"\n keywords: [\"trainium\", \"vllm\", \"cold-start\", \"recommendations\", \"automation\"]\n featured: false\n\n - title: \"DeepSeek-R1 모델 AWS 출시\"\n url: \"https://aws.amazon.com/ko/blogs/korea/deepseek-r1-models-now-available-on-aws/\"\n description: \"AWS에서 DeepSeek-R1 모델을 사용할 수 있게 되었습니다.\"\n author: \"AWS Korea\"\n date: \"2025-02-05\"\n category: \"news\"\n locale: \"ko-KR\"\n keywords: [\"deepseek\", \"r1\", \"model\", \"launch\", \"aws\"]\n featured: false\n\n # Chinese Articles\n - title: \"使用亚马逊云科技自研芯片 Inferentia2 部署 DeepSeek R1 Distillation 模型（一）\"\n url: \"https://aws.amazon.com/cn/blogs/china/deploying-the-deepseek-r1-distillation-model-using-amazon-inferentia2/\"\n description: \"使用亚马逊云科技自研芯片 Inferentia2 部署 DeepSeek R1 Distillation 模型（第一部分）。\"\n author: \"AWS China\"\n date: \"2025-02-12\"\n category: \"tutorial\"\n locale: \"zh-CN\"\n keywords: [\"inferentia2\", \"deepseek\", \"r1\", \"deployment\", \"distillation\"]\n featured: false\n\n - title: \"使用亚马逊云科技自研芯片 Inferentia2 部署 DeepSeek R1 Distillation 模型（二）\"\n url: \"https://aws.amazon.com/cn/blogs/china/deploying-the-deepseek-r1-distillation-model-using-amazon-inferentia2-part-two/\"\n description: \"使用亚马逊云科技自研芯片 Inferentia2 部署 DeepSeek R1 Distillation 模型（第二部分）。\"\n author: \"AWS China\"\n date: \"2025-02-14\"\n category: \"tutorial\"\n locale: \"zh-CN\"\n keywords: [\"inferentia2\", \"deepseek\", \"r1\", \"deployment\", \"distillation\"]\n featured: false\n\n - title: \"Bytedance processes billions of daily videos using their multimodal video understanding models on AWS Inferentia2\"\n url: \"https://aws.amazon.com/blogs/machine-learning/bytedance-processes-billions-of-daily-videos-using-their-multimodal-video-understanding-models-on-aws-inferentia2/\"\n description: \"How Bytedance processes billions of daily videos using multimodal models on AWS Inferentia2.\"\n author: \"AWS\"\n date: \"2025-02-26\"\n category: \"case-study\"\n locale: \"en-US\"\n keywords: [\"inferentia2\", \"bytedance\", \"video\", \"multimodal\", \"case-study\"]\n featured: false\n\n - title: \"基于 HAMi 实现亚马逊云科技 Trainium 与 Inferentia 核心级共享与策略性拓扑调度\"\n url: \"https://aws.amazon.com/cn/blogs/china/achieve-trainium-and-inferentia-core-level-sharing-and-strategic-topology-scheduling/\"\n description: \"基于 HAMi 实现亚马逊云科技 Trainium 与 Inferentia 核心级共享与策略性拓扑调度。\"\n author: \"AWS China\"\n date: \"2025-11-06\"\n category: \"blog\"\n locale: \"zh-CN\"\n keywords: [\"trainium\", \"inferentia\", \"hami\", \"scheduling\", \"topology\"]\n featured: false\n\n # Red Hat / AWS Neuron Collaboration\n - title: \"Red Hat to Deliver Enhanced AI Inference Across AWS\"\n url: \"https://www.redhat.com/en/about/press-releases/red-hat-deliver-enhanced-ai-inference-across-aws\"\n description: \"Red Hat and AWS expand collaboration to power enterprise-grade generative AI using Red Hat AI Inference Server on AWS Inferentia2 and Trainium3.\"\n author: \"Red Hat\"\n date: \"2025-12-02\"\n category: \"news\"\n locale: \"en-US\"\n keywords: [\"red-hat\", \"inferentia2\", \"trainium3\", \"vllm\", \"openshift\", \"inference\", \"collaboration\"]\n featured: false\n\n - title: \"Run cost-effective AI workloads on OpenShift with AWS Neuron Operator\"\n url: \"https://developers.redhat.com/articles/2025/12/02/cost-effective-ai-workloads-openshift-aws-neuron-operator\"\n description: \"How to use the AWS Neuron Operator to run LLM inference with vLLM on AWS AI chips in Red Hat OpenShift.\"\n author: \"Red Hat\"\n date: \"2025-12-02\"\n category: \"tutorial\"\n locale: \"en-US\"\n keywords: [\"red-hat\", \"openshift\", \"neuron-operator\", \"vllm\", \"inferentia\", \"trainium\", \"kubernetes\"]\n featured: false\n\n - title: \"AWS Neuron Operator for AI Chips on AWS — GitHub Releases\"\n url: \"https://github.com/awslabs/operator-for-ai-chips-on-aws/releases\"\n description: \"Open-source AWS Neuron Operator for Kubernetes and Red Hat OpenShift, enabling native support for AWS Inferentia and Trainium accelerators.\"\n author: \"AWS\"\n date: \"2025-12-02\"\n category: \"news\"\n locale: \"en-US\"\n keywords: [\"neuron-operator\", \"kubernetes\", \"openshift\", \"open-source\", \"inferentia\", \"trainium\"]\n featured: false\n\n - title: \"Red Hat AI Inference Server — vLLM Neuron Container Image (RHEL 9)\"\n url: \"https://catalog.redhat.com/en/software/containers/rhaiis/vllm-neuron-rhel9/698c42b20b626d81c97abd7f\"\n description: \"Certified container image for the Red Hat AI Inference Server with vLLM optimized for AWS Inferentia and Trainium accelerators via the AWS Neuron SDK. Provides enterprise-grade, high-performance LLM inference serving on RHEL 9, enabling production deployment of generative AI models on AWS AI chips through Red Hat OpenShift or Podman.\"\n author: \"Red Hat\"\n date: \"2025-12-02\"\n category: \"news\"\n locale: \"en-US\"\n keywords: [\"red-hat\", \"vllm\", \"neuron\", \"inferentia\", \"trainium\", \"container\", \"rhel9\", \"inference\", \"openshift\"]\n featured: true\n\n" }, { "path": "about-neuron/news-and-blogs/validate_articles.py", "content": "#!/usr/bin/env python3\n\"\"\"\nValidation script for news-and-blogs.yaml\n\nThis script validates the structure and content of article entries\nto ensure they meet the required format before submission.\n\nUsage:\n python validate_articles.py\n\"\"\"\n\nimport sys\nfrom pathlib import Path\nfrom datetime import datetime\nimport re\n\ntry:\n import yaml\nexcept ImportError:\n print(\"Error: PyYAML is required. Install with: pip install pyyaml\")\n sys.exit(1)\n\n\nVALID_CATEGORIES = {'blog', 'news', 'tutorial', 'case-study', 'benchmark'}\nREQUIRED_FIELDS = {'title', 'url', 'description', 'author', 'date', 'category', 'locale', 'keywords'}\nOPTIONAL_FIELDS = {'featured', 'author_url', 'icon'}\nALL_FIELDS = REQUIRED_FIELDS | OPTIONAL_FIELDS\n\n# Valid locale codes\nVALID_LOCALES = {\n 'en-US', 'en-GB', 'en-CA', 'en-AU', 'en-NZ', 'en-IE', 'en-IN', 'en-SG', 'en-ZA',\n 'ja-JP', 'zh-CN', 'zh-TW', 'zh-HK', 'ko-KR', 'th-TH', 'vi-VN', 'id-ID', 'ms-MY', 'fil-PH',\n 'de-DE', 'fr-FR', 'es-ES', 'es-MX', 'es-AR', 'pt-BR', 'pt-PT', 'it-IT', 'nl-NL', 'pl-PL',\n 'ru-RU', 'tr-TR', 'sv-SE', 'da-DK', 'no-NO', 'fi-FI', 'cs-CZ', 'hu-HU', 'ro-RO', 'el-GR',\n 'uk-UA', 'ar-SA', 'ar-AE', 'ar-EG', 'he-IL', 'fa-IR', 'hi-IN', 'bn-BD', 'ur-PK', 'sw-KE'\n}\n\n\ndef validate_url(url):\n \"\"\"Validate URL format\"\"\"\n url_pattern = re.compile(\n r'^https?://' # http:// or https://\n r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\\.)+[A-Z]{2,6}\\.?|' # domain\n r'localhost|' # localhost\n r'\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})' # or IP\n r'(?::\\d+)?' # optional port\n r'(?:/?|[/?]\\S+)$', re.IGNORECASE)\n return url_pattern.match(url) is not None\n\n\ndef validate_date(date_str):\n \"\"\"Validate date format (YYYY-MM-DD)\"\"\"\n try:\n datetime.strptime(date_str, '%Y-%m-%d')\n return True\n except ValueError:\n return False\n\n\ndef validate_article(article, index, section):\n \"\"\"Validate a single article entry\"\"\"\n errors = []\n warnings = []\n \n # Check for required fields\n missing_fields = REQUIRED_FIELDS - set(article.keys())\n if missing_fields:\n errors.append(f\"Missing required fields: {', '.join(missing_fields)}\")\n \n # Check for unknown fields\n unknown_fields = set(article.keys()) - ALL_FIELDS\n if unknown_fields:\n warnings.append(f\"Unknown fields (will be ignored): {', '.join(unknown_fields)}\")\n \n # Validate title\n if 'title' in article:\n if not article['title'] or not isinstance(article['title'], str):\n errors.append(\"Title must be a non-empty string\")\n elif len(article['title']) > 200:\n warnings.append(f\"Title is very long ({len(article['title'])} chars). Consider shortening.\")\n \n # Validate URL\n if 'url' in article:\n if not validate_url(article['url']):\n errors.append(f\"Invalid URL format: {article['url']}\")\n \n # Validate description\n if 'description' in article:\n if not article['description'] or not isinstance(article['description'], str):\n errors.append(\"Description must be a non-empty string\")\n elif len(article['description']) < 20:\n warnings.append(\"Description is very short. Consider adding more detail.\")\n elif len(article['description']) > 500:\n warnings.append(f\"Description is very long ({len(article['description'])} chars). Consider shortening.\")\n \n # Validate author\n if 'author' in article:\n if not article['author'] or not isinstance(article['author'], str):\n errors.append(\"Author must be a non-empty string\")\n \n # Validate author_url (optional)\n if 'author_url' in article:\n if article['author_url'] and not validate_url(article['author_url']):\n errors.append(f\"Invalid author_url format: {article['author_url']}\")\n \n # Validate date\n if 'date' in article:\n if not validate_date(str(article['date'])):\n errors.append(f\"Invalid date format: {article['date']}. Use YYYY-MM-DD\")\n else:\n article_date = datetime.strptime(str(article['date']), '%Y-%m-%d')\n if article_date > datetime.now():\n warnings.append(f\"Date is in the future: {article['date']}\")\n \n # Validate category\n if 'category' in article:\n if article['category'] not in VALID_CATEGORIES:\n errors.append(f\"Invalid category: {article['category']}. Must be one of: {', '.join(VALID_CATEGORIES)}\")\n \n # Validate locale\n if 'locale' in article:\n if not isinstance(article['locale'], str):\n errors.append(\"Locale must be a string\")\n elif article['locale'] not in VALID_LOCALES:\n warnings.append(f\"Locale '{article['locale']}' not in standard list. Will display with 🌐 globe icon. Common locales: en-US, ja-JP, zh-CN, de-DE, fr-FR, es-ES, pt-BR, ko-KR\")\n \n # Validate keywords\n if 'keywords' in article:\n if not isinstance(article['keywords'], list):\n errors.append(\"Keywords must be a list\")\n elif len(article['keywords']) == 0:\n warnings.append(\"Keywords list is empty. Consider adding relevant keywords for better filtering\")\n else:\n for i, keyword in enumerate(article['keywords']):\n if not isinstance(keyword, str):\n errors.append(f\"Keyword at index {i} must be a string\")\n elif len(keyword.strip()) == 0:\n warnings.append(f\"Keyword at index {i} is empty or whitespace\")\n if len(article['keywords']) > 10:\n warnings.append(f\"Article has {len(article['keywords'])} keywords. Consider limiting to 5-10 most relevant keywords\")\n \n # Validate featured\n if 'featured' in article:\n if not isinstance(article['featured'], bool):\n errors.append(\"Featured must be true or false (boolean)\")\n if section == 'all_articles' and article['featured']:\n warnings.append(\"Article marked as featured but in all_articles section\")\n \n # Validate icon (optional)\n if 'icon' in article:\n if not isinstance(article['icon'], str) or len(article['icon']) > 10:\n warnings.append(\"Icon should be a short string (emoji recommended)\")\n \n return errors, warnings\n\n\ndef main():\n \"\"\"Main validation function\"\"\"\n yaml_file = Path(__file__).parent / 'news-and-blogs.yaml'\n \n if not yaml_file.exists():\n print(f\"❌ Error: {yaml_file} not found\")\n return 1\n \n print(f\"Validating {yaml_file}...\\n\")\n \n try:\n with open(yaml_file, 'r', encoding='utf-8') as f:\n data = yaml.safe_load(f)\n except yaml.YAMLError as e:\n print(f\"❌ YAML Parse Error: {e}\")\n return 1\n \n if not isinstance(data, dict):\n print(\"❌ Error: YAML file must contain a dictionary\")\n return 1\n \n total_errors = 0\n total_warnings = 0\n \n # Validate featured_articles section\n if 'featured_articles' in data:\n print(\"📌 Validating featured_articles section...\")\n if not isinstance(data['featured_articles'], list):\n print(\"❌ Error: featured_articles must be a list\")\n total_errors += 1\n else:\n for i, article in enumerate(data['featured_articles'], 1):\n errors, warnings = validate_article(article, i, 'featured_articles')\n if errors or warnings:\n print(f\"\\n Article #{i}: {article.get('title', 'NO TITLE')}\")\n for error in errors:\n print(f\" ❌ Error: {error}\")\n total_errors += 1\n for warning in warnings:\n print(f\" ⚠️ Warning: {warning}\")\n total_warnings += 1\n print()\n \n # Validate all_articles section\n if 'all_articles' in data:\n print(\"📚 Validating all_articles section...\")\n if not isinstance(data['all_articles'], list):\n print(\"❌ Error: all_articles must be a list\")\n total_errors += 1\n else:\n for i, article in enumerate(data['all_articles'], 1):\n errors, warnings = validate_article(article, i, 'all_articles')\n if errors or warnings:\n print(f\"\\n Article #{i}: {article.get('title', 'NO TITLE')}\")\n for error in errors:\n print(f\" ❌ Error: {error}\")\n total_errors += 1\n for warning in warnings:\n print(f\" ⚠️ Warning: {warning}\")\n total_warnings += 1\n print()\n \n # Summary\n print(\"=\" * 60)\n if total_errors == 0 and total_warnings == 0:\n print(\"✅ Validation passed! No errors or warnings found.\")\n return 0\n else:\n print(f\"Validation complete:\")\n if total_errors > 0:\n print(f\" ❌ {total_errors} error(s) found - must be fixed\")\n if total_warnings > 0:\n print(f\" ⚠️ {total_warnings} warning(s) found - should be reviewed\")\n \n if total_errors > 0:\n print(\"\\n❌ Validation FAILED - please fix errors before submitting\")\n return 1\n else:\n print(\"\\n✅ Validation PASSED - warnings are optional to fix\")\n return 0\n\n\nif __name__ == '__main__':\n sys.exit(main())\n" }, { "path": "about-neuron/oss/index.rst", "content": ".. meta::\n :description: GitHub repositories for AWS Neuron open source components, libraries, and tools.\n :date-modified: 12/02/2025\n\nNeuron Open Source Repositories and Contribution\n===================================================\n\nAWS Neuron provides open source code and samples for some of its components, libraries, and tools under the Apache 2.0 license. The current public repositories open to contribution at this time are listed below.\n\nNeuron Open Source GitHub Repositories\n---------------------------------------\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: \n :class-body: sphinx-design-class-title-small\n \n **TorchNeuron PyTorch Extension Open Source**\n ^^^\n Source code for the Neuron Native PyTorch extension and the TorchNeuron library that implements it for AWS Trainium.\n\n * Neuron GitHub source repository: https://github.com/aws-neuron/torch-neuronx\n\n .. grid-item-card:: \n :class-body: sphinx-design-class-title-small\n \n **Neuron Kernel Library Open Source**\n ^^^\n Source code and specifications for the pre-built kernels that ship with the NKI Library .\n\n * Neuron GitHub source repository: https://github.com/aws-neuron/nki-library\n \n .. grid-item-card:: \n :class-body: sphinx-design-class-title-small\n \n **vLLM for Neuron Open Source**\n ^^^\n Source code for the vLLM integrations with Neuron, supporting AWS Trainium and Inferentia.\n\n * Neuron GitHub source repository: https://github.com/vllm-project/vllm-neuron\n * **Note**: Released under vLLM project license (`LICENSE `__). \n \n .. grid-item-card:: \n :class-body: sphinx-design-class-title-small\n \n **NKI Samples**\n ^^^\n Full code examples that support NKI kernel development.\n\n * Neuron GitHub source repository: https://github.com/aws-neuron/nki-samples\n\nHow to Contribute to Neuron Open Source\n----------------------------------------\n\nContributions via pull requests are appreciated! Before sending us a pull request, please ensure that:\n\n1. You are working against the latest source on the `main`` branch.\n2. You check existing open and recently merged pull requests and GitHub Issues to make sure someone else hasn't addressed the problem already.\n3. You open a GitHub Issue for the repo to discuss any significant work.\n\nTo send us a pull request:\n\n1. Fork the repository.\n2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.\n3. Ensure local tests pass.\n4. Commit to your fork using clear commit messages.\n5. Send us a pull request, answering any default questions in the pull request interface.\n6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.\n\nGitHub provides documentation on `forking a repository `_ and `creating a pull request `_.\n\nFor the specific details on licenses and contributing to each OSS repo, review the ``CONTRIBUTING.md`` pages linked below:\n\n* Contribute to TorchNeuron: https://github.com/aws-neuron/torch-neuronx/blob/main/CONTRIBUTING.md\n* Contribute to the NKI Library: https://github.com/aws-neuron/nki-library/blob/main/CONTRIBUTING.md\n* Contribute the the NKI samples: https://github.com/aws-neuron/nki-samples/blob/main/CONTRIBUTING.md\n \n.. Re-add this when available: * Contribute to vLLM Neuron: https://github.com/vllm-project/vllm-neuron/blob/main/CONTRIBUTING.md\n" }, { "path": "about-neuron/profiling-tools.rst", "content": ".. _profiling-tools:\n\nProfiling Tools\n================\n\n.. toctree:: \n :maxdepth: 1\n\n Neuron Profiler User Guide \n Neuron Profiler 2.0 (Beta) User Guide \n What's New \n\n\n" }, { "path": "about-neuron/quick-start/_specs/REFACTORING_NOTES.md", "content": "# Quick-Start Refactoring Notes\n\n## Summary\n\nThe quick-start documentation has been restructured with a modern, task-based information architecture. The new structure eliminates the need for .txt includes in the primary quickstart paths.\n\n## New Structure (No .txt includes)\n\n### Primary Quickstarts (Self-contained)\n- `index.rst` - Main landing page with decision tree\n- `training-quickstart.rst` - Complete training workflow (no includes)\n- `inference-quickstart.rst` - Complete inference workflow (no includes)\n\nThese files follow the procedural-quickstart template and contain all content inline. No external includes required.\n\n### Supporting Pages\n- `docs-quicklinks.rst` - Quick navigation links\n- `github-samples.rst` - GitHub repository links\n\n## Legacy Structure (Uses .txt includes)\n\n### Legacy Quick-Start Pages (Inf1 only)\n- `torch-neuron.rst` - Uses tab-inference-torch-neuronx.txt and tab-inference-torch-neuron.txt\n- `tensorflow-neuron.rst` - Uses tab-inference-tensorflow-neuronx.txt and tab-inference-tensorflow-neuron.rst\n- `mxnet-neuron.rst` - Uses tab-inference-mxnet-neuron.txt\n\nThese legacy pages:\n- Target Inf1 instances (NeuronCore v1)\n- Use .txt includes that reference `/src/helperscripts/installationScripts/python_instructions.txt`\n- Are de-emphasized in the new navigation (under \"Legacy\" section)\n- Are preserved for backward compatibility and existing links\n\n### .txt Include Files (Legacy only)\nAll .txt files in this directory are used exclusively by the legacy quick-start pages:\n- `tab-inference-torch-neuronx*.txt` (various OS versions)\n- `tab-inference-torch-neuron*.txt` (various OS versions)\n- `tab-inference-tensorflow-neuronx*.txt` (various OS versions)\n- `tab-inference-tensorflow-neuron*.txt` (various OS versions)\n- `tab-inference-mxnet-neuron*.txt` (various OS versions)\n- `select-framework-note.txt`\n\n## Design Decision\n\n**Why not refactor legacy files?**\n1. They target deprecated Inf1 hardware\n2. They're not prominently featured in new navigation\n3. Refactoring would require updating installation script references\n4. Risk of breaking existing external links\n5. New users are directed to the new self-contained quickstarts\n\n**Why are new quickstarts self-contained?**\n1. Easier to maintain (all content in one place)\n2. Better for AI/LLM context retrieval\n3. Follows modern docs-as-code best practices\n4. Clearer for human readers (no jumping between files)\n5. Follows the procedural-quickstart template structure\n\n## Migration Path\n\nFor users currently using legacy quick-starts:\n- Inf1 users: Continue using legacy pages (torch-neuron.rst, etc.)\n- New projects: Use new quickstarts (training-quickstart.rst, inference-quickstart.rst)\n- Inf2/Trn1/Trn2/Trn3 users: Use new quickstarts\n\n## Future Cleanup\n\nWhen Inf1 support is fully deprecated:\n1. Archive legacy quick-start pages to `/archive/quick-start/`\n2. Remove .txt include files\n3. Update any remaining cross-references\n4. Update neuron_tag.py to remove special handling\n" }, { "path": "about-neuron/quick-start/docs-quicklinks.rst", "content": ".. _docs-quick-links:\n\nNeuron Quick Links\n==================\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: Overview\n \n * :ref:`neuron-quickstart`\n * :ref:`amazon-q-dev`\n * :ref:`model_samples_tutorials`\n * :ref:`benchmark`\n * :ref:`neuron_release_notes`\n * :ref:`announcements-main`\n\n .. grid-item-card:: ML frameworks\n \n * :ref:`pytorch-neuronx-main`\n * :ref:`jax-neuron-main`\n * :ref:`tensorflow-neuron-main`\n * :doc:`MXNet Neuron (archived) `\n\n .. grid-item-card:: ML libraries\n\n * :ref:`nxdt`\n * :ref:`NxD Inference `\n * :ref:`neuronx-distributed-index`\n * :ref:`transformers_neuronx_readme`\n * :ref:`nemo-megatron-index`\n\n .. grid-item-card:: User Guides\n \n * :ref:`neuron_runtime`\n * :ref:`neuron_cc`\n * :ref:`Neuron Kernel Interface (NKI) (beta) `\n * :ref:`Neuron Custom C++ Operators (beta) `\n * :ref:`monitoring_tools`\n * :ref:`profiling-tools`\n * :ref:`setup-guide-index`\n * :ref:`neuron-dlami-overview`\n * :ref:`neuron_containers`\n * :ref:`neuron-devflows`\n\n .. grid-item-card:: Learn AWS Neuron\n\n * :ref:`neuron-architecture-index`\n * :ref:`neuron-features-index`\n * :ref:`neuron-appnotes-index`\n * :ref:`neuron_faq`\n * :ref:`general-troubleshooting`\n\n .. grid-item-card:: About AWS Neuron\n\n * :ref:`neuron_release_notes`\n\n" }, { "path": "about-neuron/quick-start/github-samples.rst", "content": ".. _neuron-github-samples:\n\nNeuron GitHub Samples\n=====================\n\n.. grid:: 2\n\n .. dropdown:: Training Samples for ``Trn1``\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n :open:\n\n * `PyTorch Neuron (torch-neuronx) samples for Trn1 `_\n * `Nemo Megatron for Neuron for Trn1 `_\n * `AWS Neuron samples for ParallelCluster `_\n * `AWS Neuron samples for EKS `_\n * `AWS Neuron samples for SageMaker `_\n * `AWS Neuron samples for Batch `_\n\n\n .. dropdown:: Inference Samples for ``Inf2 & Trn1``\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n :open:\n\n * `PyTorch Neuron (torch-neuronx) samples for Inf2 & Trn1 `_\n * `Transformers Neuron (transformers-neuronx) samples `_\n * `AWS Neuron samples for SageMaker `_\n\n .. dropdown:: Inference Samples for ``Inf1``\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n :open:\n\n * `PyTorch Neuron (torch-neuron) samples for Inf1 `_\n * `TensorFlow Neuron (tensorflow-neuron) samples for Inf1 `_\n\n \n" }, { "path": "about-neuron/quick-start/index.rst", "content": ".. meta::\n :description: Get started quickly with AWS Neuron SDK for PyTorch, JAX, and TensorFlow on Inferentia and Trainium\n :keywords: neuron, quickstart, getting started, pytorch, jax, tensorflow, inferentia, trainium, training, inference\n :instance-types: inf2, trn1, trn2, trn3\n :content-type: navigation-hub\n :date-modified: 2026-03-03\n\n.. _neuron-quickstart:\n\nGet Started with AWS Neuron\n============================\n\nGet up and running with AWS Neuron SDK in minutes. These quickstarts guide you through your first training or inference workload on Inferentia and Trainium instances.\n\n.. note::\n \n **First time using AWS Neuron?** These quickstarts assume you have:\n \n - An active AWS account with EC2 access\n - Basic familiarity with your chosen ML framework (PyTorch, JAX, or TensorFlow)\n - SSH access to launch and connect to EC2 instances\n \n For detailed installation instructions, see the :doc:`Setup Guide `.\n\nChoose Your Path\n----------------\n\nSelect the quickstart that matches your use case:\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: 🚀 Training Quickstart\n :link: training-quickstart\n :link-type: ref\n :class-card: sd-border-2\n \n Train your first model on Trainium\n \n - Launch a Trn1 instance\n - Run a PyTorch training script\n - Monitor training progress\n \n **Time**: ~15 minutes\n \n :bdg-primary:`Trn1` :bdg-primary:`Trn2` :bdg-primary:`Trn3`\n\n .. grid-item-card:: 🎯 Inference Quickstart\n :link: inference-quickstart\n :link-type: ref\n :class-card: sd-border-2\n \n Run your first inference on Inferentia\n \n - Launch an Inf2 instance\n - Load a pre-compiled model\n - Run predictions\n \n **Time**: ~10 minutes\n \n :bdg-success:`Inf2` :bdg-success:`Trn1`\n\nSpecialized Quickstarts\n-----------------------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: 💬 LLM Serving with vLLM\n :class-card: sd-border-1\n \n Deploy large language models for production inference\n \n - :doc:`Online serving ` (OpenAI-compatible API)\n - :doc:`Offline batch inference `\n \n **Time**: ~20 minutes\n \n :bdg-info:`Inf2` :bdg-info:`Trn1`\n\n .. grid-item-card:: 🤖 Amazon AI helper tools\n :link: amazon-q-dev\n :link-type: ref\n :class-card: sd-border-1\n \n Use AI-powered code assistance for Neuron development\n \n - Get code suggestions\n - Debug Neuron applications\n - Optimize performance\n \n **Time**: ~5 minutes\n\nFramework-Specific Guides\n-------------------------\n\nNeed framework-specific setup instructions?\n\n.. grid:: 1 1 3 3\n :gutter: 2\n\n .. grid-item-card:: PyTorch\n :link: /setup/pytorch/index\n :link-type: doc\n :class-card: sd-border-1\n :class-body: sphinx-design-class-title-small\n \n PyTorch 2.9+ setup\n\n .. grid-item-card:: JAX\n :link: /setup/jax/index\n :link-type: doc\n :class-card: sd-border-1\n :class-body: sphinx-design-class-title-small\n \n JAX 0.7+ setup\n\n .. grid-item-card:: TensorFlow\n :link: /archive/tensorflow/index\n :link-type: doc\n :class-card: sd-border-1\n :class-body: sphinx-design-class-title-small\n \n TensorFlow 2.x setup\n\nAdditional Resources\n--------------------\n\n- :doc:`/about-neuron/models/index` - Pre-tested model samples and tutorials\n- :doc:`/devflows/ec2-flows` - Detailed EC2 deployment workflows\n- :doc:`/containers/index` - Use Deep Learning Containers\n- :doc:`docs-quicklinks` - Quick links to all Neuron documentation\n- :doc:`github-samples` - GitHub sample repositories\n\nLegacy Quick-Start Pages (Inf1)\n--------------------------------\n\n.. warning::\n \n The following pages are for legacy Inf1 instances only. For new projects, use the quickstarts above for Inf2, Trn1, Trn2, or Trn3.\n\n- :doc:`torch-neuron` - PyTorch on Inf1\n- :doc:`tensorflow-neuron` - TensorFlow on Inf1\n- :doc:`mxnet-neuron` - MXNet on Inf1\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n \n training-quickstart\n inference-quickstart\n /libraries/nxd-inference/vllm/quickstart-vllm-online-serving\n /libraries/nxd-inference/vllm/quickstart-vllm-offline-serving\n /about-neuron/amazonq-getstarted\n docs-quicklinks\n github-samples\n torch-neuron\n tensorflow-neuron\n mxnet-neuron\n" }, { "path": "about-neuron/quick-start/inference-quickstart.rst", "content": ".. meta::\n :description: Run your first inference workload on AWS Inferentia with PyTorch and Neuron SDK\n :keywords: neuron, inference, quickstart, pytorch, inferentia, inf2, getting started\n :instance-types: inf2, trn1\n :content-type: quickstart\n :date-modified: 2026-03-03\n\n.. _inference-quickstart:\n\nQuickstart: Run Inference on Inferentia\n========================================\n\nThis quickstart guides you through running your first PyTorch inference workload on AWS Inferentia. You'll launch an Inf2 instance, compile a model for Neuron, and run predictions. When you complete this quickstart, you'll understand the basic workflow for deploying models on Inferentia.\n\n**This quickstart is for**: ML engineers and developers deploying inference workloads\n\n**Time to complete**: ~10 minutes\n\nPrerequisites\n-------------\n\nBefore you begin, ensure you have:\n\n- An AWS account with EC2 launch permissions\n- AWS CLI configured with your credentials\n- SSH key pair for EC2 access\n- Basic familiarity with PyTorch\n- Terminal access (Linux, macOS, or WSL on Windows)\n\nStep 1: Launch an Inferentia instance\n--------------------------------------\n\nIn this step, you will launch an Inf2 instance using the AWS Deep Learning AMI.\n\nLaunch an Inf2.xlarge instance with the latest Deep Learning AMI:\n\n.. code-block:: bash\n\n aws ec2 run-instances \\\n --image-id resolve:ssm:/aws/service/deep-learning-base-neuron/ubuntu-22-04/latest \\\n --instance-type inf2.xlarge \\\n --key-name YOUR_KEY_NAME \\\n --security-group-ids YOUR_SECURITY_GROUP \\\n --subnet-id YOUR_SUBNET_ID\n\n.. note::\n \n Replace ``YOUR_KEY_NAME``, ``YOUR_SECURITY_GROUP``, and ``YOUR_SUBNET_ID`` with your values.\n \n Alternatively, launch the instance through the `EC2 Console `_.\n\nConnect to your instance via SSH:\n\n.. code-block:: bash\n\n ssh -i YOUR_KEY.pem ubuntu@YOUR_INSTANCE_IP\n\nVerify Neuron devices are available:\n\n.. code-block:: bash\n\n neuron-ls\n\nYou should see output showing available NeuronCores:\n\n.. code-block:: text\n\n +--------+--------+--------+---------+\n | NEURON | NEURON | NEURON | PCI |\n | DEVICE | CORES | MEMORY | BDF |\n +--------+--------+--------+---------+\n | 0 | 2 | 32 GB | 00:1e.0 |\n +--------+--------+--------+---------+\n\nStep 2: Set up your environment\n--------------------------------\n\nIn this step, you will create a Python virtual environment and install PyTorch with Neuron support.\n\nCreate and activate a virtual environment:\n\n.. code-block:: bash\n\n python3 -m venv neuron_env\n source neuron_env/bin/activate\n\nInstall PyTorch Neuron and dependencies:\n\n.. code-block:: bash\n\n pip install torch-neuronx neuronx-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\nVerify the installation:\n\n.. code-block:: bash\n\n python -c \"import torch; import torch_neuronx; print(f'PyTorch: {torch.__version__}')\"\n\nYou should see output confirming PyTorch is installed:\n\n.. code-block:: text\n\n PyTorch: 2.9.0+cpu\n\nStep 3: Compile a model for Neuron\n-----------------------------------\n\nIn this step, you will create a simple model and compile it for Neuron inference.\n\nCreate a file named ``compile_model.py``:\n\n.. code-block:: python\n\n import torch\n import torch.nn as nn\n import torch_neuronx\n \n # Simple neural network\n class SimpleNet(nn.Module):\n def __init__(self):\n super().__init__()\n self.fc1 = nn.Linear(784, 128)\n self.fc2 = nn.Linear(128, 10)\n self.relu = nn.ReLU()\n \n def forward(self, x):\n x = self.relu(self.fc1(x))\n return self.fc2(x)\n \n # Create model and set to eval mode\n model = SimpleNet()\n model.eval()\n \n # Create example input\n example_input = torch.randn(1, 784)\n \n # Trace and compile for Neuron\n print(\"Compiling model for Neuron...\")\n neuron_model = torch_neuronx.trace(model, example_input)\n \n # Save compiled model\n neuron_model.save('simple_net_neuron.pt')\n print(\"Model compiled and saved to simple_net_neuron.pt\")\n\nRun the compilation script:\n\n.. code-block:: bash\n\n python compile_model.py\n\nYou should see compilation progress and success message:\n\n.. code-block:: text\n\n Compiling model for Neuron...\n INFO:Neuron:Compiling function _NeuronGraph$1 with neuronx-cc\n INFO:Neuron:Compilation successful\n Model compiled and saved to simple_net_neuron.pt\n\n.. note::\n \n Model compilation happens once. The compiled model (``simple_net_neuron.pt``) can be reused for inference without recompiling.\n\nStep 4: Run inference\n----------------------\n\nIn the final step, you will load the compiled model and run predictions.\n\nCreate a file named ``run_inference.py``:\n\n.. code-block:: python\n\n import torch\n import torch_neuronx\n \n # Load compiled model\n print(\"Loading compiled model...\")\n neuron_model = torch.jit.load('simple_net_neuron.pt')\n \n # Create sample input\n sample_input = torch.randn(1, 784)\n \n # Run inference\n print(\"Running inference...\")\n with torch.no_grad():\n output = neuron_model(sample_input)\n \n # Get prediction\n predicted_class = output.argmax(dim=1).item()\n print(f\"Predicted class: {predicted_class}\")\n print(f\"Output logits: {output[0][:5].tolist()}\") # Show first 5 logits\n \n # Run multiple inferences to measure throughput\n print(\"\\nRunning 100 inferences...\")\n import time\n start = time.time()\n \n with torch.no_grad():\n for _ in range(100):\n output = neuron_model(sample_input)\n \n elapsed = time.time() - start\n throughput = 100 / elapsed\n print(f\"Throughput: {throughput:.2f} inferences/second\")\n print(f\"Latency: {elapsed/100*1000:.2f} ms per inference\")\n\nRun the inference script:\n\n.. code-block:: bash\n\n python run_inference.py\n\nYou should see inference results:\n\n.. code-block:: text\n\n Loading compiled model...\n Running inference...\n Predicted class: 7\n Output logits: [0.123, -0.456, 0.789, -0.234, 0.567]\n \n Running 100 inferences...\n Throughput: 245.67 inferences/second\n Latency: 4.07 ms per inference\n\nMonitor Neuron device utilization in another terminal:\n\n.. code-block:: bash\n\n neuron-top\n\nThis shows real-time NeuronCore utilization and inference metrics.\n\nConfirmation\n------------\n\nCongratulations! You've successfully run inference on AWS Inferentia. You should have:\n\n- ✅ Launched an Inf2 instance with Neuron SDK\n- ✅ Installed PyTorch with Neuron support\n- ✅ Compiled a model for Neuron inference\n- ✅ Ran predictions and measured throughput\n- ✅ Monitored inference with Neuron tools\n\nIf you encountered any issues, see the **Common issues** section below.\n\nCommon issues\n-------------\n\n**Issue**: ``ModuleNotFoundError: No module named 'torch_neuronx'``\n\n**Solution**: Ensure you activated the virtual environment and installed packages:\n\n.. code-block:: bash\n\n source neuron_env/bin/activate\n pip install torch-neuronx neuronx-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n**Issue**: ``RuntimeError: No Neuron devices found``\n\n**Solution**: Verify you're on an Inferentia instance and devices are visible:\n\n.. code-block:: bash\n\n neuron-ls\n\nIf no devices appear, check instance type and driver installation.\n\n**Issue**: Compilation takes a long time\n\n**Solution**: Model compilation is a one-time cost. For this simple model, compilation should take 1-2 minutes. Larger models take longer but only need to be compiled once. The compiled model can be saved and reused.\n\n**Issue**: Lower throughput than expected\n\n**Solution**: This quickstart uses a small model and batch size for demonstration. For production workloads:\n\n- Use larger batch sizes (e.g., 4, 8, 16)\n- Enable dynamic batching\n- Use multiple NeuronCores in parallel\n- See :doc:`/frameworks/torch/torch-neuronx/programming-guide/inference/index` for optimization techniques\n\nClean up\n--------\n\nTo avoid ongoing charges, terminate your instance when finished:\n\n.. code-block:: bash\n\n # From your local machine\n aws ec2 terminate-instances --instance-ids YOUR_INSTANCE_ID\n\nOr use the EC2 Console to terminate the instance.\n\nNext steps\n----------\n\nNow that you've completed this quickstart, explore more advanced inference topics:\n\n- :doc:`/frameworks/torch/torch-neuronx/programming-guide/inference/index` - Comprehensive inference guide\n- :doc:`/libraries/nxd-inference/index` - Production inference with NeuronX Distributed\n- :doc:`/libraries/nxd-inference/vllm/quickstart-vllm-online-serving` - Deploy LLMs with vLLM\n- :doc:`/about-neuron/models/index` - Pre-tested model samples\n- :doc:`/tools/neuron-explorer/index` - Profile and optimize inference performance\n\nFurther reading\n---------------\n\n- :doc:`/setup/pytorch/index` - Detailed PyTorch installation options\n- :doc:`/devflows/ec2-flows` - EC2 deployment workflows\n- :doc:`/frameworks/torch/index` - Complete PyTorch Neuron documentation\n- :doc:`/compiler/index` - Understanding Neuron compilation\n" }, { "path": "about-neuron/quick-start/mxnet-neuron.rst", "content": ".. _mxnet_quick_start:\n\n\nGet Started with Apache MXNet Neuron\n=====================================\n\nThis page provide links that will assist you to quickly start with :doc:`MXNet Neuron ` (supporting inference only).\n\n.. note::\n Below instructions are for Ubuntu20, if you looking for complete setup instructions for different platforms, please :ref:`Check Here. `\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /setup/install-templates/launch-instance.txt\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 5\n :end-line: 6\n\t\t\n.. include:: /includes/setup/tab-inference-mxnet-neuron.txt" }, { "path": "about-neuron/quick-start/tab-inference-tensorflow-neuron.rst", "content": ".. dropdown:: Install TensorFlow Neuron (``tensorflow-neuron``)\n :class-title: drop-down-class-title-small\n :class-body: drop-down-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=compiler_framework\n\n.. dropdown:: Get Started with Inference (``Inf1``)\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n :ref:`ResNet-50 `\n\n.. card:: Visit TensorFlow Neuron section for more\n :class-body: sphinx-design-class-body-small\n :link: tensorflow-neuron-main\n :link-type: ref" }, { "path": "about-neuron/quick-start/tensorflow-neuron.rst", "content": ".. _tensorflow_quick_start:\n\nGet Started with TensorFlow Neuron\n==================================\n\nThis page provide links that will assist you to quickly start with :ref:`tensorflow-neuron-main`.\n\n\n.. note::\n Below instructions are for Ubuntu20, if you looking for complete setup instructions for different platforms, please :ref:`Check Here. `\n\n.. _tensorflow_quick_start_inference:\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /setup/install-templates/launch-instance.txt\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 5\n :end-line: 6\n\n\n.. tab-set::\n\n .. tab-item:: tensorflow-neuronx (``Trn1, Inf2``)\n\n .. include:: /includes/setup/tab-inference-tensorflow-neuronx.txt\n\n .. tab-item:: tensorflow-neuron (``Inf1``)\n\n .. include:: /includes/setup/tab-inference-tensorflow-neuron.rst" }, { "path": "about-neuron/quick-start/torch-neuron-tab-training.rst", "content": "\n.. dropdown:: Launch Trn1 Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /setup/install-templates/launch-instance.txt\n\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. code:: bash\n\n # Configure Linux for Neuron repository updates\n\n sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <`\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /setup/install-templates/launch-instance.txt\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 5\n :end-line: 6\n\n.. tab-set::\n\n .. tab-item:: torch-neuronx (``Trn1, Inf2``)\n\n .. include:: /includes/setup/tab-inference-torch-neuronx.txt\n\n\n .. tab-item:: torch-neuron (``Inf1``)\n\n .. include:: /includes/setup/tab-inference-torch-neuron.txt" }, { "path": "about-neuron/quick-start/training-quickstart.rst", "content": ".. meta::\n :description: Train your first model on AWS Trainium with PyTorch and Neuron SDK\n :keywords: neuron, training, quickstart, pytorch, trainium, trn1, getting started\n :instance-types: trn1, trn2, trn3\n :content-type: quickstart\n :date-modified: 2026-03-03\n\n.. _training-quickstart:\n\nQuickstart: Train a Model on Trainium\n======================================\n\nThis quickstart guides you through training your first PyTorch model on AWS Trainium. You'll launch a Trn1 instance, install Neuron SDK, and run a simple training script. When you complete this quickstart, you'll understand the basic workflow for training models with Neuron.\n\n**This quickstart is for**: ML engineers and data scientists new to AWS Trainium\n\n**Time to complete**: ~15 minutes\n\nPrerequisites\n-------------\n\nBefore you begin, ensure you have:\n\n- An AWS account with EC2 launch permissions\n- AWS CLI configured with your credentials\n- SSH key pair for EC2 access\n- Basic familiarity with PyTorch\n- Terminal access (Linux, macOS, or WSL on Windows)\n\nStep 1: Launch a Trainium instance\n-----------------------------------\n\nIn this step, you will launch a Trn1 instance using the AWS Deep Learning AMI.\n\nFirst, launch a Trn1.2xlarge instance with the latest Deep Learning AMI:\n\n.. code-block:: bash\n\n aws ec2 run-instances \\\n --image-id resolve:ssm:/aws/service/deep-learning-base-neuron/ubuntu-22-04/latest \\\n --instance-type trn1.2xlarge \\\n --key-name YOUR_KEY_NAME \\\n --security-group-ids YOUR_SECURITY_GROUP \\\n --subnet-id YOUR_SUBNET_ID\n\n.. note::\n \n Replace ``YOUR_KEY_NAME``, ``YOUR_SECURITY_GROUP``, and ``YOUR_SUBNET_ID`` with your values.\n \n Alternatively, launch the instance through the `EC2 Console `_.\n\nOnce the instance is running, connect via SSH:\n\n.. code-block:: bash\n\n ssh -i YOUR_KEY.pem ubuntu@YOUR_INSTANCE_IP\n\nVerify Neuron devices are available:\n\n.. code-block:: bash\n\n neuron-ls\n\nYou should see output showing available NeuronCores:\n\n.. code-block:: text\n\n +--------+--------+--------+---------+\n | NEURON | NEURON | NEURON | PCI |\n | DEVICE | CORES | MEMORY | BDF |\n +--------+--------+--------+---------+\n | 0 | 2 | 32 GB | 00:1e.0 |\n | 1 | 2 | 32 GB | 00:1f.0 |\n +--------+--------+--------+---------+\n\nStep 2: Set up your environment\n--------------------------------\n\nIn this step, you will create a Python virtual environment and install PyTorch with Neuron support.\n\nCreate and activate a virtual environment:\n\n.. code-block:: bash\n\n python3 -m venv neuron_env\n source neuron_env/bin/activate\n\nInstall PyTorch Neuron and dependencies:\n\n.. code-block:: bash\n\n pip install torch-neuronx neuronx-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\nVerify the installation:\n\n.. code-block:: bash\n\n python -c \"import torch; import torch_neuronx; print(f'PyTorch: {torch.__version__}')\"\n\nYou should see output confirming PyTorch is installed:\n\n.. code-block:: text\n\n PyTorch: 2.9.0+cpu\n\nStep 3: Create a training script\n---------------------------------\n\nIn this step, you will create a simple PyTorch training script that uses Neuron acceleration.\n\nCreate a file named ``train_simple.py``:\n\n.. code-block:: python\n\n import torch\n import torch.nn as nn\n import torch.optim as optim\n import torch_neuronx\n \n # Simple neural network\n class SimpleNet(nn.Module):\n def __init__(self):\n super().__init__()\n self.fc1 = nn.Linear(784, 128)\n self.fc2 = nn.Linear(128, 10)\n self.relu = nn.ReLU()\n \n def forward(self, x):\n x = self.relu(self.fc1(x))\n return self.fc2(x)\n \n # Create model and move to Neuron device\n model = SimpleNet().to('neuron')\n criterion = nn.CrossEntropyLoss()\n optimizer = optim.SGD(model.parameters(), lr=0.01)\n \n # Generate dummy training data\n batch_size = 32\n num_batches = 100\n \n print(\"Starting training...\")\n model.train()\n \n for batch_idx in range(num_batches):\n # Create dummy batch\n inputs = torch.randn(batch_size, 784).to('neuron')\n targets = torch.randint(0, 10, (batch_size,)).to('neuron')\n \n # Training step\n optimizer.zero_grad()\n outputs = model(inputs)\n loss = criterion(outputs, targets)\n loss.backward()\n optimizer.step()\n \n if batch_idx % 10 == 0:\n print(f\"Batch {batch_idx}/{num_batches}, Loss: {loss.item():.4f}\")\n \n print(\"Training complete!\")\n\nThis script creates a simple neural network, moves it to the Neuron device, and trains it on synthetic data.\n\nStep 4: Run training\n---------------------\n\nIn the final step, you will run the training script and monitor its progress.\n\nExecute the training script:\n\n.. code-block:: bash\n\n python train_simple.py\n\nYou should see training progress output:\n\n.. code-block:: text\n\n Starting training...\n Batch 0/100, Loss: 2.3156\n Batch 10/100, Loss: 2.2845\n Batch 20/100, Loss: 2.2534\n ...\n Training complete!\n\nMonitor Neuron device utilization in another terminal:\n\n.. code-block:: bash\n\n neuron-top\n\nThis shows real-time NeuronCore utilization, memory usage, and other metrics.\n\nConfirmation\n------------\n\nCongratulations! You've successfully trained your first model on AWS Trainium. You should have:\n\n- ✅ Launched a Trn1 instance with Neuron SDK\n- ✅ Installed PyTorch with Neuron support\n- ✅ Created and ran a training script on Neuron devices\n- ✅ Monitored training with Neuron tools\n\nIf you encountered any issues, see the **Common issues** section below.\n\nCommon issues\n-------------\n\n**Issue**: ``ModuleNotFoundError: No module named 'torch_neuronx'``\n\n**Solution**: Ensure you activated the virtual environment and installed packages:\n\n.. code-block:: bash\n\n source neuron_env/bin/activate\n pip install torch-neuronx neuronx-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n**Issue**: ``RuntimeError: No Neuron devices found``\n\n**Solution**: Verify you're on a Trainium instance and devices are visible:\n\n.. code-block:: bash\n\n neuron-ls\n\nIf no devices appear, check instance type and driver installation.\n\n**Issue**: Training is slower than expected\n\n**Solution**: This quickstart uses a small model for demonstration. For production workloads:\n\n- Use larger batch sizes\n- Enable XLA compilation with ``torch.compile()``\n- See :doc:`/frameworks/torch/torch-neuronx/programming-guide/training/pytorch-neuron-programming-guide` for optimization techniques\n\nClean up\n--------\n\nTo avoid ongoing charges, terminate your instance when finished:\n\n.. code-block:: bash\n\n # From your local machine\n aws ec2 terminate-instances --instance-ids YOUR_INSTANCE_ID\n\nOr use the EC2 Console to terminate the instance.\n\nNext steps\n----------\n\nNow that you've completed this quickstart, explore more advanced training topics:\n\n- :doc:`/frameworks/torch/torch-neuronx/programming-guide/training/pytorch-neuron-programming-guide` - Comprehensive training guide\n- :doc:`/libraries/nxd-training/index` - Distributed training with NeuronX Distributed\n- :doc:`/about-neuron/models/index` - Pre-tested model samples\n- :doc:`/tools/neuron-explorer/index` - Profile and optimize training performance\n\nFurther reading\n---------------\n\n- :doc:`/setup/pytorch/index` - Detailed PyTorch installation options\n- :doc:`/devflows/ec2-flows` - EC2 deployment workflows\n- :doc:`/frameworks/torch/index` - Complete PyTorch Neuron documentation\n" }, { "path": "about-neuron/quick-start/user-guide-quickstart.rst", "content": ".. _userguide-quickstart:\n\nUser Guide Quick Start\n======================\n\n* :ref:`setup-guide-index`\n* :ref:`Neuron Containers `\n* :ref:`neuron-devflows`\n\n" }, { "path": "about-neuron/sdk-policy.rst", "content": ".. _sdk-maintenance-policy:\n.. _neuron-maintenance-policy:\n\nNeuron Software Maintenance policy\n==================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\nOverview\n--------\n\nThis document outlines software maintenance policy for AWS Neuron\nSoftware Development Kit (SDK), Neuron Components, both extension and\nstandalone components, supported model classes, features, APIs, DLAMIs\nand DLCs, and dependency software. AWS Neuron is the SDK for Amazon EC2\n`Inferentia `__ and\nAmazon EC2\n`Trainium `__ based\ninstances purpose-built for deep learning. Neuron integrates with\npopular Machine Learning (ML) frameworks like PyTorch, JAX, and\nTensorFlow and includes a compiler, runtime, driver, profiling tools,\nand libraries to support high performance training of generative AI\nmodels on Trainium and Inferentia powered instances.\n\nThis document addresses Neuron Software life-cycle and the Neuron SDK\nrelease versioning.\n\n.. _neuron-software-definitions:\n\nNeuron Software Definitions\n---------------------------\n\nNeuron Software refers to the complete set of software elements\nprovided by AWS Neuron, including:\n\nNeuron SDK\n~~~~~~~~~~\n\nThe core software development kit that enables users to build, train,\nand deploy machine learning models on Inferentia and Trainium based\ninstances. The Neuron SDK encompasses the entire set of components,\nfeatures, APIs, and other elements that are bundled together and made\navailable in a particular version of the Neuron SDK release.\n\nNeuron components\n~~~~~~~~~~~~~~~~~\n\nNeuron components refer to any packages or libraries within the Neuron\nSDK that offer specific functionality. These components are typically\naccessible through PIP, RPM, or Debian packages for easy installation\nand usage. There are two main categories of Neuron components: Neuron\nextension components and Neuron standalone components.\n\nNeuron extension components\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNeuron extension components are components that integrate Neuron support\ninto open source machine learning frameworks, libraries or tools\nenhancing their functionality and extending their capabilities as\nnecessary. When referring to Neuron extension components, we are also\nreferring to the parts of the open source machine learning framework or\nlibrary that are supported by Neuron. The software life-cycle of the\nopen source machine learning frameworks, libraries or tools that are\nextended by Neuron is managed and maintained by their respective\ncommunities or the vendors responsible for those specific components.\nExamples for Neuron extension components are:\n\n- **Third party ML Library**: Examples include Neuron Nemo Megatron.\n- **Third party ML Framework**: Examples include PyTorch NeuronX and\n TensorFlow Neuron.\n\nNeuron standalone components\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNeuron standalone components are self-contained components within the\nNeuron SDK. Examples of such components are Neuron Compiler, Neuron\nTools and Neuron Runtime.\n\nNeuron Model Classes\n~~~~~~~~~~~~~~~~~~~~\n\nA Neuron supported model class is tightly coupled with a specific Neuron\nextension component (e.g. PyTorch NeuronX) or Neuron library (e.g.\nNeuronX Distributed) and the workload type (e.g. Training or Inference).\nFor example a model can be supported at Beta level in PyTorch NeuronX\nfor training and Stable level in PyTorch NeuronX for inference.\n\nNeuron features\n~~~~~~~~~~~~~~~\n\nA Neuron feature refers to any functionality or attribute that is part\nof the Neuron SDK, whether it belongs to the entire Neuron SDK or to one\nof its specific components.\n\nNeuron APIs\n~~~~~~~~~~~\n\nA Neuron API refers to any API, CLI, environment variables, or flag that\nbelong to to the entire Neuron SDK or to one the Neuron components. A\nNeuron API allows developers to interact with and leverage the\ncapabilities of the Neuron SDK and its components.\n\nExamples include :ref:`Neuron Trace API ` and :ref:`Neuron Compiler flags `\n\nDependency software components\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nExternal software components or frameworks that the Neuron\nSDK and its components rely on for proper functioning and compatibility,\nsuch as language runtimes or operating systems.\n\nThe software life-cycle of the dependency software components, is\nmanaged and maintained by their respective communities or the vendors\nresponsible for those specific dependency software components. The\nfollowing terms are examples of underlying dependency software\ncomponents:\n\n- **Operating System (OS)**: Examples include Ubuntu 22 and Amazon\n Linux 2023\n- **Language Runtime**: Examples include Python 3.10\n\nNeuron Deep Learning AMIs and Deep Learning Containers\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:ref:`Neuron Deep Learning AMIs\n(DLAMIs) `\nand :ref:`Neuron Deep Learning Containers\n(DLCs) ` are pre-configured Amazon Machine Images and Docket container that\ncome with the Neuron SDK and necessary dependencies pre-installed,\nproviding a ready-to-use environment for machine learning development.\n\n.. _neuron-software-lifecycle:\n\nNeuron Software Life-cycle\n--------------------------\n\nThe typical life-cycle for Neuron software consists of several phases, though not all phases are applicable to every type of Neuron software. The phases are as follows:\n\n- **Developer Preview or Beta** (these terms are used interchangeably in\n Neuron collaterals)\n- **Release Candidate (RC)**\n- **General Availability (GA) or Stable** (these terms are used\n interchangeably in Neuron collaterals)\n- **Maintenance**\n- **End-of-Support (EOS)**\n\nThe following table outlines the details for each phase for Neuron software:\n\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n| | Description | Comments |\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n| Developer Preview (Beta) | In this phase, Neuron Software is not supported, should not be used in production environments, | |\n| | and is meant for early access and feedback purposes only. It is possible for future releases | |\n| | to introduce breaking changes. | |\n| | See :ref:`Neuron Software Classification ` for more information | |\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n| Release Candidate (RC) | Once AWS identifies a release to be a stable product, it may be marked as a Release Candidate (RC). | This phase applies only to Neuron SDK |\n| | This phase is usually short and during it AWS will provide for Neuron Software on an as-needed basis. | and Neuron components |\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n| General Availability (Stable) | During this phase, AWS releases :ref:`regular ` updates for the Neuron Software based | |\n| | on a predefined release cadence of the Neuron SDK or provides :ref:`maintenance updates `| |\n| | for Neuron Software on an as-needed basis. | |\n| | See :ref:`Neuron Software Classification ` for more information | |\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n| Maintenance | During the maintenance phase, AWS will provide :ref:`maintenance updates ` | This phase does not apply to Dependency Software |\n| | for Neuron Software on an as-needed basis. Any new PIP, RPM, and Debian packages for the Neuron | Components, Neuron DLCs, |\n| | Software, as well as updated versions of the Neuron DLAMIs and Neuron DLCs, will be released | Neuron DLAMIs, Neuron Features and APIs |\n| | only when deemed necessary by the AWS Neuron team. | |\n| | Users can expect updates to be less frequent compared to :ref:`regular ` | |\n| | as the focus will be on addressing critical issues and ensuring the stability of the software. | |\n| | | |\n| | Maintenance Announcement: AWS will make a public :ref:`announcement ` at least one month | |\n| | before the Neuron Software enters Maintenance phase. | |\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n| End of Support (EOS) | When Neuron Software reaches the end of its support lifecycle, it will no longer receive | |\n| | :ref:`regular ` updates and :ref:`maintenance updates ` | |\n| | (including security updates). While AWS will continue to provide access to all previously released | |\n| | PIP, RPM, and Debian packages for the Neuron Software, as well as earlier versions of the Neuron DLAMIs | |\n| | and Neuron DLCs, it's important to note that these older versions will not receive any updates or support. | |\n| | Customers can still use these resources at their own discretion, but it is highly recommended to upgrade | |\n| | to the latest available versions | |\n| | | |\n| | End of Support Announcement: AWS will make a public :ref:`announcement ` at least one month | |\n| | before a Neuron Software enters End of Support. | |\n+-------------------------------+----------------------------------------------------------------------------------------------------------------------+--------------------------------------------------+\n\n.. _neuron-regular-updates:\n\nNeuron Software Regular Updates\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nRegular updates for Neuron Software address the following areas: new\nfeatures, feature improvements, performance enhancements, bug\nresolution, security vulnerability fixes, upgrades to Neuron dependency\nsoftware components and upgrades to Neuron extension components. To\nhandle these regular updates, AWS will release a new version of the\nNeuron SDK, incrementing the minor version (the second digit in the\nversion number) for a minor release or incrementing the major version\n(the first digit in the version number) for a major release when\nsignificant changes that break compatibility are introduced. It's\nimportant to note that any bug-fixes or security issues in regular\nupdates are not applied retroactively to previous versions of the Neuron\nSDK. To benefit from these updates, users must adopt the latest release.\n\nFor more information see:\n\n- :ref:`Neuron DLAMIs and DLCs Updates `\n- :ref:`Neuron Extension Components Updates `\n- :ref:`Neuron Software Versioning `\n\n**Neuron SDK Installation and Update instructions**\nTo install and update to the latest Neuron packages, customers need to pin the major\nversion of the Neuron package. For example, to install latest Neuron\ntools package, call ``sudo apt-get install aws-neuronx-tools=2.*`` and\nto install latest PyTorch Neuron package for Trn1, call\n``pip install torch-neuronx==2.1.0.1.*``. This is done to future-proof\ninstructions for new, backwards-incompatible major version releases.\n\n.. _neuron-maintenance-updates:\n\nNeuron Software Maintenance Updates\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nMaintenance updates for Neuron Software address three key areas:\nresolving bugs, fixing security vulnerabilities, and upgrading\ndependency software components. At AWS discretion, additional critical\nfeatures or performance enhancement may also be included. To handle\nthese maintenance updates, AWS will release a new version of the Neuron\nSDK, incrementing the patch number (the last digit in the version\nnumber) to indicate a patch release. Major or minor releases may also\ncontain maintenance updates. It's important to note that these\nmaintenance updates are not applied retroactively to previous versions\nof the Neuron SDK. To take advantage of these updates, users must adopt\nthe latest patch release.\n\nFor more information see:\n\n- :ref:`Neuron DLAMIs and DLCs Updates `\n- :ref:`Neuron Extension Components Updates `\n- :ref:`Neuron Software Versioning `\n\n.. _neuron-dlami-dlc-updates:\n\nNeuron DLAMIs and DLCs Updates\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAWS will address :ref:`regular ` updates, life-cycle changes, maintenance\nupdates, and security issues related to any third-party software\nincluded in the Neuron DLAMI or DLCs by releasing new versions of the\nNeuron DLAMI or DLCs. However, updates won't be applied retroactively to\nolder versions of the Neuron DLAMI or DLCs. Instead, users will need to\nuse the new versions to get the latest updates. Generally, Neuron DLAMIs and Deep Learning Containers (DLCs) will support one latest LTS Linux Distribution version (Ubuntu, Amazon Linux, and Rocky9), with exceptions. Neuron Base DLAMIs (which come pre-installed with Neuron driver, EFA, and Neuron tools) will support the two latest versions of LTS Linux Distributions.\n\n\nFor more information see:\n\n- :ref:`Neuron Extension Components Updates `\n- :ref:`Neuron Software Versioning `\n\n.. _neuron-extension-components-updates:\n\nNeuron Extension Components Updates\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen a new version of an open source ML framework (e.g. PyTorch) is\nsupported by a Neuron extension component (e.g., PyTorch NeuronX), the\nNeuron extension component for the latest supported ML framework version\nwill become the default for installation. If users wish to use a Neuron\nextension component for an earlier supported ML framework version, they\nwill need to explicitly specify the desired version during installation.\nAfter upgrading a Neuron extension component to support a newer version\nof an ML framework, AWS will continue to provide :ref:`regular updates `\nfor the Neuron extension component that supports the earlier ML\nframework version for a minimum of 6 months. After the 6 months period,\nthe Neuron extension component for the earlier supported ML framework\nversion may transition into a maintenance mode. In the maintenance mode,\nupdates for the older Neuron extension component versions will be\nprovided on an as-needed basis, focusing on critical bug fixes and\nsecurity patches. For more information see: :ref:`Neuron extension component versioning `\n\n.. _neuron-communication:\n\nCommunication methods\n~~~~~~~~~~~~~~~~~~~~~\n\nNeuron software classification and lifecycle announcements are\ncommunicated as follows:\n\n- Neuron SDK documentation under\n `Announcements `__\n\nTo see the list of available Neuron SDK versions and supported\ndependency software components versions:\n\n- Neuron SDK documentation under `Release\n Content `__\n- Neuron SDK documentation under `What’s\n New `__\n\n.. _neuron-software-versioning:\n\nNeuron Software Versioning\n--------------------------\n\nNeuron SDK Documentation Versioning\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron SDK documentation is versioned and maps to the corresponding\nNeuron SDK version. Users can switch to earlier versions of the Neuron\nSDK documentation by selecting the version from the dropdown in bottom\nleft portion of the side bar.\n\nNeuron SDK Versioning\n~~~~~~~~~~~~~~~~~~~~~\n\nThe AWS SDK release versions are in the form of ``[A.B.C]`` where\n``(A)`` represents the major version, ``(B)`` represents\nthe minor version, and ``(C)`` represents the patch version.\n\n.. _neuron-extension-components-versioning:\n\nNeuron extension components Versioning\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron extension components versioning (like PyTorch NeuronX) is in the\nform ``[X.Y.Z].[A.B.C]``, where ``[X.Y.Z]`` represents the\nthird party component’s major (``X``), minor (``Y``), and patch\n(``Z``) versions and ``[A.B.C]`` represents the Neuron extension\ncomponents (``A``), minor (``B``), and patch (``C``)\nversions.\n\nNeuron Standalone Component Versioning\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron Component versioning (except of Neuron extension components like\nPyTorch NeuronX) is in the form ``[A.B.C.D]``, where ``A``\nrepresents the major version, ``B`` represents the minor version,\nand ``C.D`` represents the patch version.\n\n.. _neuron-releases-types:\n\nNeuron Software Release Types\n-----------------------------\n\nMajor release\n~~~~~~~~~~~~~~~~~\n\nIncreasing the major version indicates that the Neuron software\nunderwent significant and substantial changes in an incompatible manner.\nApplications need to be updated in order for them to work with the\nnewest SDK version. It is important to update major versions carefully\nand in accordance with the upgrade guidelines provided by AWS. After\nincreasing the major version, the Neuron software may not maintain\ncompatibility with previous supported versions of :ref:`Neuron\nRuntime `, :ref:`Neuron Compiler `, and\n:ref:`NEFF `.\n\nMinor release\n~~~~~~~~~~~~~~~~~\n\nIncreasing the minor version indicates that the Neuron software added\nfunctionality in a backwards compatible manner.\n\nPatch release\n~~~~~~~~~~~~~~~~~\n\nIncreasing the patch version indicates that the Neuron software\nadded backward compatible bug or security fixes. A bug fix is defined as\nan internal change that fixes incorrect behavior.\n\nPre-releases\n~~~~~~~~~~~~~~~~\n\n- **Developer Preview (Beta)**: During this phase, the Neuron software\n is not supported, should not be used in production environments, and\n is meant for early access and feedback purposes only. It is possible\n for future releases to introduce breaking changes. In the case of a\n Developer Preview (Beta) release, the minor version will include a\n lower case ``b`` along with a (Beta) tag.\n- **Release Candidate (RC)**: Once Neuron identifies a release to be a\n stable product, it may mark it as a Release Candidate. Release\n Candidates are ready for GA release unless significant bugs emerge,\n and will receive full AWS Neuron support. In the case of a RC\n release, the minor version will include a lower case ``rc``\n along with a (RC) tag.\n\n.. _sdk-classification:\n\nNeuron Software Classification\n------------------------------\n\nThis section explains the Neuron software classification for APIs,\nlibraries, packages, features, and Neuron supported model classes\nmentioned in the Neuron documentation.\n\nNeuron SDK and Neuron components\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n+-----------------+-----------------+-----------------+-------------+\n| | Testing | Features | Performance |\n+=================+=================+=================+=============+\n| Developer | Basic | Minimal Viable | |\n| Preview (Beta) | | Product (MVP) \\*| |\n+-----------------+-----------------+-----------------+-------------+\n| Release | Basic | Minimal Viable | Tested |\n| Candidate (RC) | | Product (MVP)\\* | |\n+-----------------+-----------------+-----------------+-------------+\n| GA (Stable) | Standard | Incremental | Tested |\n| | Product Testing | additions or | |\n| | | changes | |\n| | | in new releases | |\n+-----------------+-----------------+-----------------+-------------+\n\n\\* A minimum viable product (MVP) for a Neuron Component contains just\nenough features to be usable by early customers who can then provide\nfeedback for future development. MVP can be different per use case\nand depends on the specific package/library of interest. Please note\nthat in many cases, an MVP can also represent an advanced level of\nfeatures.\n\n.. _neuron-apis-classification:\n\nNeuron APIs\n~~~~~~~~~~~\n\n+----------------------+----------------------+----------------------+\n| | API Contract | API Backward |\n| | | Compatibility |\n+======================+======================+======================+\n| Alpha | Unstable and | No |\n| | undocumented | |\n+----------------------+----------------------+----------------------+\n| Developer Preview | Major changes may | No |\n| (Beta) | happen | |\n+----------------------+----------------------+----------------------+\n| GA (Stable) | Incremental changes | Yes \\* |\n| | in new releases | |\n| | (without breaking | |\n| | the API contract) | |\n+----------------------+----------------------+----------------------+\n\n\\* In certain cases, when necessary, AWS may introduce API changes that may break compatibility, with notice provided ahead of time.\n\n.. _neuron-features-classification:\n\nNeuron Features\n~~~~~~~~~~~~~~~\n\n+-----------------+-----------------+------------------------+-------------+\n| | Testing | Functionality | Performance |\n+=================+=================+========================+=============+\n| | No formal | Partial funcitonality | Not tested |\n| Alpha | testing done | with limited set of | or |\n| | | core capabilities, | evaluated |\n| | | far from Minium Viable | |\n| | | Product (MVP) \\* | |\n+-----------------+-----------------+------------------------+-------------+\n| Developer | Basic | Minimum Viable | |\n| Preview (Beta) | | Product (MVP) \\* | |\n+-----------------+-----------------+------------------------+-------------+\n| GA (Stable) | Standard | Incremental | Tested |\n| | Product Testing | additions or changes | |\n| | | in new releases | |\n+-----------------+-----------------+------------------------+-------------+\n\n\\* A minimum viable product (MVP) for a Neuron Feature contains just\nenough functionality to be usable by early customers who can then\nprovide feedback for future development. MVP can be different per use\ncase and depends on the specific feature of interest. Please note\nthat in many cases, an MVP can also represent an advanced level of\nfunctionality.\n\n.. _neuron-models-classification:\n\nNeuron Supported Model Classes\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n+----------------------+----------------------+----------------------+\n| | Accuracy / | Throughput / Latency |\n| | Convergence | |\n+======================+======================+======================+\n| Developer Preview | Validated | Tested |\n| (Beta) | | |\n+----------------------+----------------------+----------------------+\n| GA (Stable) | Validated | Tested |\n+----------------------+----------------------+----------------------+\n" }, { "path": "about-neuron/security.rst", "content": ".. meta::\n :description: Security disclosures and notification for the AWS Neuron SDK.\n :date-modified: 01/27/2026\n\n.. _security:\n\nNeuron Security Disclosures\n===========================\n\nIf you think you've found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here\n(https://aws.amazon.com/security/vulnerability-reporting/) or email AWS\nsecurity directly (`mailto:aws-security@amazon.com `__).\n\nImportant Security Information for Trainium Hardware\n-----------------------------------------------------\n\nTrainium hardware is designed to optimize performance for machine learning workloads. To deliver high performance, applications with access to Trainium devices have unrestricted access to instance physical memory.\n\nWhat this means for your deployment:\n\n* Instance-level isolation is maintained: AWS EC2 ensures Trainium devices cannot access physical memory of other EC2 instances.\n* As a best practice to prevent unrestricted access to host physical memory by any user/application, we recommend implementing a permission model where:\n\n * A dedicated system group owns the device nodes\n * Only explicitly authorized users are added to this group\n * Device permissions prevent access by users outside the group\n \nCustomer responsibility: Ensure that only trusted applications have access to Tranium devices on Trainium instances. For more information, see `the AWS Shared Responsibility Model `__.\n\nExample Implementation Steps\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe steps below are an example you can follow to implement a security group using udev rules:\n\n1. Create a dedicated security group (in this example, ``neuron``): ``sudo groupadd -r neuron``\n\n2. Add authorized users to that security group: ``sudo usermod -aG neuron {username-to-add-here}``, repeat for each user\n\n3. Configure udev rules. Create a udev rule to automatically set correct ownership and permissions when Trainium (neuron) devices are detected.\n\n Create the file ``/etc/udev/rules.d/neuron-udev.rules`` with the following content:\n \n .. code-block:: shell\n\n # Neuron device access control\n # Only members of the 'neuron' group can access 'neuron' devices.\n\n SUBSYSTEM==\"neuron*\", KERNEL==\"neuron*\", GROUP=\"neuron\", MODE=\"0660\"\n\n4. Apply the configuration:\n\n ``sudo udevadm control —-reload``\n ``sudo udevadm trigger —-subsystem-match=neuron``\n\n5. Verify the configuration:\n\n ``ls -l /dev/neuron*``\n\n Expected output:\n\n ``crw-rw---- 1 root neuron 239, 0 Jan 9 15:58 /dev/neuron0``\n\n" }, { "path": "about-neuron/troubleshooting.rst", "content": ".. _general-troubleshooting:\n\nTroubleshooting Guide\n=====================\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\nTraining Only Troubleshooting\n-----------------------------\n\n* :ref:`PyTorch Neuron for Training `\n\n\nInference Only Troubleshooting\n------------------------------\n\n* :ref:`PyTorch Neuron for Inference `\n* :ref:`NeuronPerf `\n* :ref:`MXNet Neuron `\n\n\nRuntime Troubleshooting\n------------------------------\n\n* :ref:`Neuron Runtime Troubleshooting on Inf1 and Trn1 `\n\n\nContainers Troubleshooting\n--------------------------\n\n* :ref:`Containers `\n\n\nSetup Troubleshooting\n---------------------\n\n* :ref:`neuron-setup-troubleshooting`\n" }, { "path": "about-neuron/what-is-neuron.rst", "content": ".. _what-is-neuron:\n\n.. meta::\n :description: AWS Neuron is a software development kit for high-performance machine learning on AWS Inferentia and Trainium, enabling developers to compile, optimize, and deploy deep learning models at scale.\n\nWhat is AWS Neuron?\n===================\n\nAWS Neuron is the software stack for running deep learning and generative AI workloads on AWS Trainium and AWS Inferentia. Built on an open source foundation, Neuron enables developers to build, deploy and explore natively with PyTorch and JAX frameworks and with ML libraries such as Hugging Face, vLLM, PyTorch Lightning, and others without modifying your code. It includes a compiler, runtime, training and inference libraries, and developer tools for monitoring, profiling, and debugging. Neuron supports your end-to-end machine learning (ML) development lifecycle from building and deploying deep learning and AI models, optimizing to achieve highest performance and lowest cost, and getting deeper insights into model behavior.\n\nNeuron enables rapid experimentation, production scale training of frontier models, low level performance optimization through the Neuron Kernel Interface (NKI) for custom kernels, cost optimized inference deployment for agentic AI and reinforcement learning workloads, and comprehensive profiling and debugging with Neuron Explorer.\n\nFor more details, see the detailed documentation under :ref:`About the AWS Neuron SDK `.\n\nWho is AWS Neuron for?\n-----------------------\n\n* **ML engineers** can use Neuron's vLLM integration to migrate their models to Trainium for improved performance and without code modifications. They can\n* **Performance engineers** can use NKI and our Developer Tools to create new ML kernels and optimize existing ones.\n* **ML researchers** can use their existing PyTorch experience and ecosystem tools to experiment freely on Trainium using our native PyTorch implementatio, without having to learn new frameworks or APIs\n\nWhat is AWS Neuron used for?\n-----------------------------\n\n**Research and Development**: Neuron provides native PyTorch execution on Trainium with full Eager mode compatibility. The stack supports standard distributed training patterns including FSDP, DDP, and DTensor for model sharding across devices and nodes. torch.compile integration enables graph optimization, while existing frameworks like TorchTitan and HuggingFace Transformers run without code modifications. JAX support includes XLA compilation targeting Inferentia and Trainium hardware. \n\n**Production Inference**: Neuron implements vLLM V1 API compatibility on Trainium and Inferentia with optimizations for large-scale inference workloads. The runtime supports Expert Parallelism for MoE models, disaggregated inference architectures, and speculative decoding. Optimized kernels from the NKI Library provide hardware-specific implementations. Training workflows integrate with HuggingFace Optimum Neuron, PyTorch Lightning, and TorchTitan, with seamless deployment through standard vLLM interfaces. \n\n**Performance Engineering**: Neuron Kernel Interface (NKI) provides direct access to Trainium instruction set architecture with APIs for memory management, execution scheduling, and low-level kernel development. The NKI Compiler, built on MLIR, offers full visibility into the compilation pipeline from high-level operations to hardware instructions. The NKI Library contains optimized kernel implementations with source code and performance benchmarks. Neuron Explorer enables comprehensive profiling from application code to hardware execution, supporting both single-node and distributed workload analysis with detailed performance metrics and optimization recommendations.\n\nAWS Neuron Core Components\n----------------------------\n\n**vLLM**\n Neuron enables production inference deployment with standard frameworks and APIs on Trainium and Inferentia. Use Neuron's vLLM integration with standard APIs to deliver high-performance model serving with optimized kernels from the NKI Library. \n\n It provides:\n\n * **Standard vLLM APIs**: Full compatibility with vLLM V1 APIs, enabling customers to use familiar vLLM interfaces on Neuron hardware without code changes\n * **Advanced Inference Features**: Support for Expert Parallelism for MoE models, disaggregated inference for flexible deployment architectures, and speculative decoding for improved latency\n * **Optimized Performance**: Pre-optimized kernels from the NKI Library for peak performance across dense, MoE, and multimodal models\n * **Open Source**: Source code released under the vLLM project organization with source code on GitHub, enabling community contributions\n\n**Native PyTorch**\n Neuron provides native integration with PyTorch, enabling researchers and ML developers to run existing code unchanged on Trainium. Train models with familiar workflows and tools, from pre-training to post-training with reinforcement learning, while leveraging Trainium's performance and cost advantages for both experimentation and production scale training.\n\n It provides:\n\n * **Native Device Support**: Neuron registers as a native device type in PyTorch with standard device APIs like ``torch.tensor([1,2,3], device='neuron')`` and ``.to('neuron')``\n * **Standard Distributed Training APIs**: Support for FSDP, DTensor, DDP, tensor parallelism, context parallelism, and distributed checkpointing\n * **Eager Mode Execution**: Immediate operation execution for interactive development and debugging in notebook environments\n * **torch.compile Integration**: Support for ``torch.compile`` for optimized performance\n * **Open Source**: Released as an open source package on GitHub under Apache 2.0, enabling community contributions. \n\n**Neuron Kernel Interface (NKI)**\n For performance engineers seeking maximum hardware efficiency, Neuron provides complete control through the Neuron Kernel Interface (NKI), with direct access to the NeuronISA (NISA) instruction set, memory allocation, and execution scheduling. Developers can create new operations not available in standard frameworks and optimize performance critical code with custom kernels. \n\n It includes:\n\n * The NKI Compiler, built on MLIR, which provides greater transparency into the kernel compilation process\n * The NKI Library , which provides pre-built kernels you can use to optimize the performance of your models\n\n**Neuron Tools**\n Debug and profiling utilities including:\n \n * Neuron Monitor for real-time performance monitoring\n * Neuron Explorer, built on the Neuron Profiler (``neuron-profile``), for detailed performance analysis\n\n Neuron Explorer provides:\n\n * **Hierarchical Profiling**: Top-down visualization from framework layers through HLO operators to hardware instructions, enabling developers to understand execution at any level of the stack\n * **Code Linking**: Direct navigation between PyTorch, JAX, and NKI source code and performance timeline with automatic annotations showing metrics for specific code lines\n * **IDE Integration**: VSCode extension for profile visualization and analysis directly within the development environment\n * **Device Profiling**: Unified interface for comprehensive view of system-wide metrics and device-specific execution details\n\n**Neuron Compiler**\n Optimizes machine learning models for AWS Inferentia and Trainium chips, converting models from popular frameworks into efficient executable formats.\n\n**Neuron Runtime**\n Manages model execution on Neuron devices, handling memory allocation, scheduling, and inter-chip communication for maximum throughput.\n\n**AWS DLAMIs and DLCs**\n Orchestrate and deploy your models using Deep Learning AWS Machine Images (DLAMIs) and Deep Learning Containers (DLCs).\n\n Neuron DLAMIs come pre-configured with the Neuron SDK, popular frameworks, and helpful libraries, allowing you to quickly begin training and running inference on AWS Inferentia. Or, quickly deploy models using pre-configured AWS Neuron Deep Learning Containers (Neuron DLCs) with optimized frameworks for AWS Trainium and Inferentia. \n\nSupported Hardware\n------------------\n\n**AWS Inferentia**\n Purpose-built for high-performance inference workloads:\n \n * ``Inf1`` instances - First-generation Inferentia chips\n * ``Inf2`` instances - Second-generation with improved performance and efficiency\n\n**AWS Trainium**\n Designed for distributed training of large models:\n \n * ``Trn1`` instances - High-performance training acceleration\n * ``Trn1n`` instances - Enhanced networking for large-scale distributed training\n * ``Trn2`` instances - Next-generation Trainium with superior performance\n * ``Trn2`` UltraServer - High-density Trainium servers for massive training workloads\n * ``Trn3`` UltraServer -- The next generation of Trainium servers for massive training workloads\n\n\nHow do I get more information?\n------------------------------\n\n* Review the comprehensive documentation and follow the tutorials on this site\n* Check the Neuron GitHub repositories for code examples. GitHub repos include:\n\n * `Neuron SDK code samples `_\n * `Neuron NKI ML kernel samples `_\n * `Neuron container confirguations `_\n * `Helm charts for Kubernetes deployment `_\n * `NeuronX Distributed Core library sources `_\n * `NeuronX Distributed Training library sources `_\n * `NeuronX Distributed Inference library sources `_\n * `Linux kernel driver sources `_\n * `Neuron workshop model samples `_\n\n* Visit the `AWS Neuron support forum `_ for community assistance\n" }, { "path": "about-neuron/whats-new.rst", "content": ".. _main_whats-new:\n\n.. meta::\n :description: Blog posts for the latest features and updates for the AWS Neuron SDK\n :date-modified: 03/13/2026\n\nWhat's New in the AWS Neuron SDK\n================================\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n\n Release Notes \n\n*Explore detailed posts about the latest releases, updates, and upcoming changes to the AWS Neuron SDK.*\n\n.. grid:: 1\n :gutter: 2\n\n .. grid-item-card:: Neuron Release Notes\n :link: /release-notes/index\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n **Latest release**: 2.29.0 (04/09/2026)\n\n----\n\n.. _whats-new-2026-04-02-v2_29:\n\nAWS Neuron SDK 2.29.0: NKI Exits Beta, CPU Simulator, and Expanded NKI Library\n-------------------------------------------------------------------------------\n\n**Posted on**: April 09, 2026\n\nToday we are releasing AWS Neuron SDK 2.29.0. This release brings NKI 0.3.0 out of Beta into Stable, featuring the new NKI Standard Library and an experimental CPU Simulator for local kernel development without Trainium hardware. The NKI Library adds 7 new experimental kernels including Conv1D, a Transformer TKG megakernel, and fused communication-compute primitives, along with improvements to existing attention, MLP, and MoE kernels. NxD Inference delivers performance gains for Qwen2 VL, Qwen3 VL, and Flux.1 models. Neuron Runtime introduces new APIs for collective stream management and network proxy tuning. Neuron Explorer is now out of Beta and Stable, with full Device widget support in the System Trace Viewer and availability on the VS Code Extension Marketplace. The Neuron Driver adds support for new Trn3 Gen2 Ultraserver configurations.\n\n\nNeuron Kernel Interface (NKI)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAWS Neuron SDK 2.29.0 introduces NKI 0.3.0, the latest update to the Neuron Kernel Interface. NKI 0.3.0 is now out of Beta and Stable. It features the NKI Standard Library (``nki-stdlib``), which provides developer-visible code for all NKI APIs and native language objects (such as ``NkiTensor``). This release provides new exposed Trainium capabilities and features in the NKI API and introduces ``nki.language`` APIs.\n\n**NKI CPU Simulator (Experimental)**: NKI 0.3.0 includes a CPU Simulator, which executes NKI kernels entirely on CPU and allows for a fast development cycle on inexpensive CPUs and compute instances to validate kernel correctness, using standard Python step-by-step debugging tools and instrumentation to print results for every line of kernel code. Activate it with ``NKI_SIMULATOR=1`` or use ``nki.simulate(kernel)``.\n\n**New Language APIs (Experimental)**: Introduced ``nki.language`` high-level convenience wrappers including ``nl.load``, ``nl.store``, ``nl.copy``, ``nl.matmul``, ``nl.transpose``, and ``nl.softmax``. \n\n**New ISA and Hardware Features**: Added the ability to set DMA priority of DMA operations and collectives operations for Trn3 (NeuronCore-v4). A dedicated ``nki.isa.exponential`` instruction is optimized for vectorising exponents (``exp``) with VectorE. Matmul accumulation control is added via the ``accumulate`` parameter on ``nc_matmul`` and ``nc_matmul_mx``. Variable-length all-to-all collectives are now available via ``nki.collectives.all_to_all_v``.\n\n**Breaking Changes**: NKI 0.3.0 includes several API breaking changes that improve correctness and consistency. All kernels must be updated to NKI 0.3.0; mixing with Beta 2 kernels in the same model is not supported. For the full list of changes and migration examples, see the :doc:`NKI 0.3.0 Update Guide `.\n\nFor more details, see :ref:`nki-2-29-0-rn`.\n\n\nNKI Library\n^^^^^^^^^^^\n\n**New Experimental Kernels (7 added)**: Conv1D provides 1D convolution with stride, padding, dilation, bias, activation fusion, and LNC sharding. Transformer TKG is a multi-layer transformer forward pass megakernel for token generation. Fine-Grained All-Gather and FGCC (All-Gather + Matmul) enable ring-based communication with compute overlap on Trn2. SBUF-to-SBUF All-Gather provides two variants for small and large tensors. Top-K Reduce supports MoE output gathering with LNC sharding. Dynamic Elementwise Add handles runtime-variable M-dimension tiling. The ``find_nonzero_indices`` subkernel is promoted from experimental to core.\n\n**Key Improvements to Existing Kernels**: Attention CTE increases max batch size from 32 to 512 and max sequence length from 36,864 to 131,072 with sequence packing support. Attention Block TKG adds fused QK-norm before RoPE and KVDP attention sharding. MLP adds BufferManager support and MXFP4/MXFP8 quantization paths. MoE TKG introduces a dynamic all-expert algorithm with ``block_size``. QKV adds flexible weight layout support. PyTorch reference implementations are added for 22 kernels.\n\n**Breaking Changes**: Multiple kernel signatures have changed with new parameters inserted mid-signature; callers using positional arguments must switch to keyword arguments. ``SbufManager`` is renamed to ``BufferManager``. MoE TKG replaces boolean sharding flags with ``LNCShardingStrategy`` enum. For the full list of breaking changes, see :ref:`nki-lib-2-29-0-rn`.\n\nFor more details, see :ref:`nki-lib-2-29-0-rn`.\n\n\nInference Updates\n^^^^^^^^^^^^^^^^^\n\n**NxD Inference 0.9.17155**: Qwen2 VL gains vision data parallelism with 7% QPS improvement for image-heavy workloads. Qwen3 VL adds text-model sequence parallelism with 2.2x QPS throughput improvement. Flux.1 adds CFG parallelism with 19% end-to-end latency improvement and 23% instance throughput improvement.\n\n**vLLM Neuron Plugin 0.5.0**: Updated alongside NxD Inference with model performance improvements.\n\n**Hardware Support Change**: NxD Inference no longer supports Trn1/Inf2. Only Trn2 and newer hardware is supported. Pin to Neuron SDK 2.28 for Trn1/Inf2 support.\n\nFor more details, see :ref:`nxd-inference-2-29-0-rn`.\n\n\nRuntime and Driver\n^^^^^^^^^^^^^^^^^^\n\n**Neuron Runtime Library 2.31**: New ``nrt_cc_create_stream`` API creates a collective stream to be used by host-initiated collectives, replacing the previous environment variable approach. New ``nrt_get_attached_efa_bdf`` API returns the BDF string of the EFA device for optimal network interface selection. New environment variables ``NEURON_RT_ONE_THREAD_PER_CORE`` (up to 2x improvement in collective communication latency) and ``NEURON_RT_RANKS_PER_NETWORK_PROXY`` provide fine-grained control over network proxy threading. RDMA support extends to Trn3. Collectives XU gains profiling support, context caching with up to 90% performance improvement, and removal of the 512 queue set instance limit. The async API version is bumped from 2.x to 3.0; applications using the async API must be recompiled.\n\n**Neuron Driver 2.27**: Adds support for new Trn3 Gen2 Ultraserver configurations: US3 (2-node), US4 (4-node), US16 (4-node), and US18 (4-node). Top-level DMA reset support is added during TPB reset on Trn3 and later platforms.\n\n**Neuron Collectives 2.31**: EFA device processing is restructured to per-stream granularity for improved stability. Fixed incorrect interface selection in multi-ultraserver collectives and crash on channel initialization failures.\n\nFor more details, see :ref:`runtime-2-29-0-rn`.\n\n\nNeuron Explorer\n^^^^^^^^^^^^^^^\n\nNeuron Explorer is now out of Beta and Stable. The System Trace Viewer now supports the full suite of Device widgets, enabling multi-device profile analysis across all linked Device Profiles within a single System Profile. The Summary Viewer includes system-level profile data for both system and device profiles. New System Timeline HBM Usage shows device HBM usage with memory allocation breakdown by category. Box Selection Summary enables viewing aggregated device profile information for a selected region in the trace viewer. Neuron Explorer for VS Code is now available on the Visual Studio Code Extension Marketplace and Open VSX, enabling simpler installation and automatic updates.\n\nFor more details, see :ref:`dev-tools-2-29-0-rn`.\n\n\nPyTorch Framework\n^^^^^^^^^^^^^^^^^\n\nPyTorch 2.7 and 2.8 have reached end of support starting with this release. Use PyTorch 2.9 on Ubuntu 24.04. Starting with PyTorch 2.10 support (planned for a future Neuron release), AWS Neuron will transition from PyTorch/XLA to native PyTorch support via TorchNeuron.\n\nFor more details, see :ref:`pytorch-2-29-0-rn`.\n\n\nEnd of Support and Migration Notices\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Effective this release:**\n\n* PyTorch 2.7 and 2.8 have reached end of support. Pin to Neuron SDK 2.28 if required.\n* NeuronX Distributed Training (NxDT) and NxD Core training APIs reach end of support; DLCs and DLAMI virtual environments pinned to SDK 2.28.0.\n* ``neuron-profile analyze`` subcommand is no longer supported. Migrate to Neuron Explorer.\n* Ubuntu 22.04 Multi-Framework DLAMI is no longer published. Use Ubuntu 24.04.\n\n**Hardware support:**\n\n* NxD Inference no longer supports Trn1/Inf2. Pin to Neuron SDK 2.28 for continued support.\n\n**NKI namespace migration:**\n\n* Removal of ``neuronxcc.nki.*`` namespace postponed to a future release. Both ``neuronxcc.nki.*`` and ``nki.*`` namespaces continue to work. Migration to ``nki.*`` is encouraged.\n\n**Effective with PyTorch 2.10 support:**\n\n* PyTorch/XLA will be replaced by TorchNeuron.\n\n* Read the :doc:`Neuron 2.29.0 component release notes ` for specific Neuron component improvements and details.\n\n----\n\n.. _whats-new-2026-03-13-v2_28_1:\n\nAWS Neuron SDK 2.28.1 Patch Available\n--------------------------------------\n\n**Posted on**: March 13, 2026\n\nAWS Neuron provides a patch version, 2.28.1, to address a Neuron Driver compatibility issue with Linux kernel 6.18. \n\n.. _whats-new-2026-02-26-v2_28:\n\nAWS Neuron SDK 2.28.0: Enhanced Profiling, Vision Language Models, and Expanded NKI Capabilities\n--------------------------------------------------------------------------------------------------\n\n**Posted on**: February 26, 2026\n\nToday we are releasing AWS Neuron SDK 2.28.0. This release enhances Neuron Explorer with system profiling, Tensor Viewer, and Database Viewer for comprehensive performance analysis. NxD Inference adds support for Qwen2/Qwen3 VL vision language models, Flux.1 inpainting capabilities, and Eagle3 speculative decoding. The NKI Library expands with 9 new kernels including RoPE, MoE operations, and experimental kernels for attention and cross entropy. NKI (Beta 2) introduces LNC multi-core support with intra-LNC collectives and new APIs. Kubernetes users gain Neuron DRA Driver support for advanced resource allocation.\n\n\nDeveloper Tools and Profiling\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Neuron Explorer Enhancements** - Added system profiling support with drill-down navigation to device profiles. New Tensor Viewer helps identify memory bottlenecks by displaying tensor names, shapes, sizes, and memory usage. Database Viewer provides an interactive interface for querying profiling data using SQL or natural language. Profile Manager now supports tag-based organization and search. A migration guide from Neuron Profiler/Profiler 2.0 is now available.\n\n**nccom-test Improvements** - Enhanced data integrity checks use pseudo-random data patterns for better corruption detection. Added support for ``alltoallv`` collective operation for benchmarking variable-sized all-to-all communication patterns.\n\nFor more details, see :ref:`dev-tools-2-28-0-rn`.\n\nInference Updates\n^^^^^^^^^^^^^^^^^\n\n**NxD Inference 0.8.16251** - Added support for vision language models including Qwen2 VL (Qwen2-VL-7B-Instruct) and Qwen3 VL (Qwen3-VL-8B-Thinking) for processing text and image inputs (Beta). Pixtral model support improved with batch size 32 and sequence length 10240 on Trn2 with vLLM V1. Flux.1 model gains new functionality for in-paint, out-paint, canny edge detection, and depth-based image generation (Beta).\n\n**vLLM Neuron Plugin 0.4.1** - Multi-LoRA serving enhancements enable streaming LoRA adapters via vLLM's ``load_adapter`` API with dynamic runtime loading. Users can now run the base model alone when multi-LoRA serving is enabled. Added Eagle3 speculative decoding support for Llama 3.1 8B. Updated to support vLLM v0.13.0 and PyTorch 2.9.\n\nFor more details, see :ref:`nxd-inference-2-28-0-rn`.\n\nNKI Library\n^^^^^^^^^^^\n\n**9 New Kernels** - The NKI Library expands from 7 to 16 documented kernel APIs. New core kernels include RoPE (Rotary Position Embedding), Router Top-K (expert selection for MoE), MoE CTE (Context Encoding), MoE TKG (Token Generation), and Cumsum. New experimental kernels include Attention Block TKG (fused attention for token generation), Cross Entropy (forward and backward passes), Depthwise Conv1D, and Blockwise MM Backward (for MoE training).\n\n**Enhanced Quantization Support** - Existing kernels receive FP8 and MX quantization support across QKV, MLP, and Output Projection kernels. QKV kernel adds fused FP8 KV cache quantization and block-based KV cache layout. MLP kernel adds gate/up projection clamping and fp16 support for TKG mode. Attention CTE kernel adds strided Q slicing for context parallelism.\n\n**Improved Utilities** - TensorView gains ``rearrange`` method for dimension reordering and ``has_dynamic_access`` for runtime-dependent addressing checks. SbufManager provides hierarchical tree-formatted allocation logging with new query methods for SBUF utilization. New utilities include ``rmsnorm_mx_quantize_tkg``, ``interleave_copy``, ``LncSubscriptable``, and ``TreeLogger``.\n\nFor more details, see :ref:`nki-lib-2-28-0-rn`.\n\nNeuron Kernel Interface (NKI)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**NKI Beta 2 (0.2.0)** - This release includes LNC multi-core support for LNC=2, enabling kernels to leverage multiple NeuronCores within a logical NeuronCore. The compiler now tracks ``shared_hbm`` tensors and canonicalizes LNC kernel outputs. Users can declare tensors private to a single NeuronCore using ``private_hbm`` memory type.\n\n**New nki.collectives Module** - Enables collective communication across multiple NeuronCores with operations including ``all_reduce``, ``all_gather``, ``reduce_scatter``, ``all_to_all``, ``collective_permute`` variants, and ``rank_id``.\n\n**New APIs and Features** - New ``nki.isa`` APIs include ``nonzero_with_count`` for sparse computation and ``exponential`` for element-wise operations. New ``float8_e4m3fn`` dtype supports FP8 workloads. Language features include ``no_reorder`` blocks for instruction ordering control, ``__call__`` special method support, ``tensor.view`` method for reshaping, and shared constants as string arguments.\n\n**API Improvements** - ``dma_transpose`` now supports indirect addressing, ``dma_copy`` adds the ``unique_indices`` parameter, and ``register_alloc`` accepts optional tensor arguments for pre-filling. The compiler no longer truncates diagnostic output.\n\nFor more details, see :ref:`nki-2-28-0-rn`.\n\nKubernetes Support\n^^^^^^^^^^^^^^^^^^\n\n**Neuron DRA Driver** - Introduced Neuron Dynamic Resource Allocation (DRA) Driver enabling advanced resource allocation using the Kubernetes DRA API for flexible and efficient Neuron device management. The DRA API provides topology-aware scheduling, atomic resource allocation, and per-workload configuration. Neuron Helm Charts now include DRA Driver support.\n\nFor more details, see :ref:`containers-2-28-0-rn`.\n\nPyTorch Framework (torch-neuronx)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Transition to Native PyTorch Support** - Starting with PyTorch 2.10 support (planned for a future Neuron release), AWS Neuron will transition from PyTorch/XLA to native PyTorch support via TorchNeuron. PyTorch 2.9 is the last version using PyTorch/XLA. Users will need to update their scripts when upgrading to PyTorch 2.10 or later. See :ref:`native-pytorch-trainium` for migration guidance.\n\nFor more details, see :ref:`pytorch-2-28-0-rn`.\n\n* Read the :doc:`Neuron 2.28.1 component release notes ` for specific Neuron component improvements and details.\n\n.. _whats-new-2025-12-19-v2_27:\n\nAWS Neuron SDK 2.27.0: Trainium3 Support, Enhanced NKI, and Unified Profiling with Neuron Explorer\n---------------------------------------------------------------------------------------------------\n\n**Posted on**: December 19, 2025\n\nToday we are releasing AWS Neuron SDK 2.27.0. This release adds support for Trainium3 (``Trn3``) instances. Enhanced NKI with new NKI Compiler introduces the ``nki.*`` namespace with updated APIs and language constructs. The NKI Library provides pre-optimized kernels for common model operations including attention, MLP, and normalization. Neuron Explorer delivers a unified profiling suite with AI-driven optimization recommendations. vLLM V1 integration is now available through the vLLM-Neuron Plugin. Deep Learning Containers and AMIs are updated with vLLM V1, PyTorch 2.9, JAX 0.7, Ubuntu 24.04, and Python 3.12.\n\nIn addition to this release, we are introducing new capabilities and features in private beta access (see Private Beta Access section). We are also announcing our transition to PyTorch native support starting with PyTorch 2.10 in Neuron 2.28, plans to simplify NxDI in upcoming releases, and other important updates.\n\nNeuron Kernel Interface (NKI)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**NKI Compiler** - The new ``nki.*`` namespace replaces the legacy ``neuronxcc.nki.*`` namespace. Top-level kernel functions now require the ``@nki.jit`` annotation. Neuron 2.27 supports both namespaces side by side; the legacy namespace will be removed in Neuron 2.28. A kernel migration guide is available in the documentation.\n\nFor more details, see :ref:`neuron-2-27-0-nki`.\n\nNKI Library\n^^^^^^^^^^^\n\nThe NKI Library provides pre-optimized kernels: Attention CTE, Attention TKG, MLP, Output Projection CTE, Output Projection TKG, QKV, and RMSNorm-Quant. Kernels are accessible via the ``nkilib.*`` namespace in neuronx-cc or from the GitHub repository.\n\nFor more details, see :ref:`neuron-2-27-0-nkilib`.\n\nDeveloper Tools\n^^^^^^^^^^^^^^^\n\n**Neuron Explorer** - A a suite of tools designed to support ML engineers throughout their development journey on AWS Trainium. This release features improved performance and user expereince for device profiling, with four core viewers to provide insights into model performance:\n\n* **Hierarchy Viewer**: Visualizes model structure and component interactions\n* **AI Recommendation Viewer**: Delivers AI-driven optimization recommendations\n* **Source Code Viewer**: Links profiling data directly to source code\n* **Summary Viewer**: Displays high-level performance metrics\n\nNeuron Explorer is available through UI, CLI, and VSCode IDE integration. Existing NTFF files are compatible but require reprocessing for new features.\n\nNew tutorials cover profiling NKI kernels, multi-node training jobs, and vLLM inference workloads. The ``nccom-test`` tool now includes fine-grained collective communication support.\n\nFor more details, see :ref:`neuron-2-27-0-tools`.\n\nInference Updates\n^^^^^^^^^^^^^^^^^\n\n**vLLM V1** - The vLLM-Neuron Plugin enables vLLM V1 integration for inference workloads. vLLM V0 support ends in Neuron 2.28.\n\n**NxD Inference** - Model support expands with beta releases of Qwen3 MoE (Qwen3-235B-A22B) for multilingual text and Pixtral (Pixtral-Large-Instruct-2411) for image understanding. Both models use HuggingFace checkpoints and are supported on ``Trn2`` and ``Trn3`` instances.\n\nFor more details, see :ref:`neuron-2-27-0-nxd-inference`.\n\nNeuron Graph Compiler\n^^^^^^^^^^^^^^^^^^^^^\n\nDefault accuracy settings are now optimized for precision. The ``--auto-cast`` flag defaults to ``none`` (previously ``matmul``), and ``--enable-mixed-precision-accumulation`` is enabled by default. FP32 models may see performance impacts; restore previous behavior with ``--auto-cast=matmul`` and ``--disable-mixed-precision-accumulation``. Python 3.10 or higher is now required.\n\nFor more details, see :ref:`neuron-2-27-0-compiler`.\n\nRuntime Improvements\n^^^^^^^^^^^^^^^^^^^^\n\n**Neuron Runtime Library 2.29** adds support for Trainium3 (``Trn3``) instances and delivers performance improvements for Collectives Engine overhead, NeuronCore branch overhead, NEFF program startup, and all-gather latency.\n\nFor more details, see :ref:`neuron-2-27-0-runtime`.\n\nDeep Learning AMIs and Containers\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Platform Updates** - All DLCs are updated to Ubuntu 24.04 and Python 3.12. DLAMIs add Ubuntu 24.04 support for base, single framework, and multi-framework configurations.\n\n**Framework Updates**:\n\n* vLLM V1 single framework DLAMI and multi-framework virtual environments\n* PyTorch 2.9 single framework DLAMIs and multi-framework virtual environments (Amazon Linux 2023, Ubuntu 22.04, Ubuntu 24.04)\n* JAX 0.7 single framework DLAMI and multi-framework virtual environments\n\n**New Container** - The ``pytorch-inference-vllm-neuronx`` 0.11.0 DLC provides a complete vLLM inference environment with PyTorch 2.8 and all dependencies.\n\nFor more details, see :ref:`neuron-2-27-0-dlami` and :ref:`neuron-2-27-0-dlc`.\n\n\nEnd of Support and Migration Notices\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Effective this release:**\n\n* :ref:`announcement-python-3-9-eol`\n* :ref:`announcement-end-of-support-pytorch-2-6`\n* :ref:`announce-no-support-tensorflow2-10`\n* :ref:`announce-eos-inf1-virtual-environments`\n* :ref:`announcement-end-of-support-parallel-model-trace`\n* :ref:`announce-eos-tensorboard-tools`\n\n**Effective Neuron 2.28:**\n\n* :ref:`announcement-end-of-support-neuronxcc-nki`\n* :ref:`announcement-nki-library-namespace-changes`\n* :ref:`announcement-nki-library-kernel-migration`\n* :ref:`announcement-end-of-support-vllm-v0`\n\n**Effective with PyTorch 2.10 support:**\n\n* :ref:`announce-transition-pytorch-trainium`\n* :ref:`announcement-end-of-support-nxdt-nxd-core`\n\n**Future Releases:**\n\n* :ref:`announce-nxdi-changes`\n* :ref:`announce-eos-dlami-ubuntu-22-04`\n* :ref:`announce-eos-pytorch-profling-api`\n* :ref:`announce-eos-neuron-profiler`\n\nDetailed Release Notes\n^^^^^^^^^^^^^^^^^^^^^^^\n\n* Read the :doc:`Neuron 2.27.0 component release notes ` for specific Neuron component improvements and details.\n\n----\n\n.. _whats-new-2025-12-02-riv:\n\nAWS Neuron Expands with Trainium3, Native PyTorch, Faster NKI, and Open Source at re:Invent 2025\n------------------------------------------------------------------------------------------------\n\n**Posted on**: 12/02/2025\n\n.. image:: /images/NeuronStandalone_white_small.png\n :alt: AWS Neuron Logo\n :align: right\n :width: 120px\n\nAt re:Invent 2025, AWS Neuron introduces support for `Trainium3 UltraServer `__ with expanded open source components and enhanced developer experience. These updates enable standard frameworks to run unchanged on Trainium, removing barriers for researchers to experiment and innovate. For developers requiring deeper control, the enhanced Neuron Kernel Interface (NKI) provides direct access to hardware-level optimizations, enabling customers to scale AI workloads with improved performance.\n\n**Expanded capabilities and enhancements include**:\n\n* :doc:`Trainium3 UltraServer support `: Enabling customers to scale AI workloads with improved performance\n* :doc:`Native PyTorch support `: Standard PyTorch runs unchanged on Trainium without platform-specific modifications\n* :doc:`Enhanced Neuron Kernel Interface (NKI) ` with open source :doc:`NKI Compiler `: Improved programming capabilities with direct access to Trainium hardware instructions and fine-grained optimization control, compiler built on MLIR\n* :doc:`NKI Library `: Open source collection of optimized, ready-to-use kernels for common ML operations\n* :doc:`Neuron Explorer `: Tools suite to support developers and performance engineers in their performance optimization journey from framework operations to hardware instructions\n* :doc:`Neuron DRA for Kubernetes `: Kubernetes-native resource management eliminating custom scheduler extensions\n* :doc:`Expanded open source components `: Open sourcing more components including NKI Compiler, Native PyTorch, NKI Library, and more released under Apache 2.0\n\n\nAI development requires rapid experimentation, hardware optimization, and production scale workloads. These updates enable researchers to experiment with novel architectures using familiar workflows, ML developers to build AI applications using standard frameworks, and performance engineers to optimize workloads using low-level hardware optimization.\n\n.. admonition:: Looking to try out our Beta features?\n\n Submit your beta access request through `this form `__ and the Neuron Product team will get back to you.\n\nNative PyTorch Support\n^^^^^^^^^^^^^^^^^^^^^^\n\n**Private Preview**\n\nAWS Neuron now natively supports PyTorch through TorchNeuron, an open source native PyTorch backend for Trainium. TorchNeuron integrates with PyTorch through the PrivateUse1 device backend mechanism, registering Trainium as a native device alongside other backends and allowing researchers and ML developers to run their code without modifications.\n\nTorchNeuron provides eager mode execution for interactive development and debugging, native distributed APIs including FSDP and DTensor for distributed training, and torch.compile support for optimization. TorchNeuron enables compatibility with minimal code changes with ecosystem tools like TorchTitan and HuggingFace Transformers.\n\nUse TorchNeuron to run your PyTorch research and training workloads on Trainium without platform-specific code changes.\n\n**Learn more**: :doc:`documentation `, and `TorchNeuron GitHub repository `__.\n\n**Access**: Contact your AWS account team for access.\n\n\nEnhanced NKI\n^^^^^^^^^^^^\n\n**Public Preview**\n\nThe enhanced Neuron Kernel Interface (NKI) provides developers with complete hardware control through advanced APIs for fine-grained scheduling and allocation. The enhanced NKI enables instruction-level programming, memory allocation control, and execution scheduling with direct access to the Trainium ISA. \n\nWe are also releasing the NKI Compiler as open source under Apache 2.0, built on MLIR to enable transparency and collaboration with the broader compiler community. NKI integrates with PyTorch and JAX, enabling developers to use custom kernels within their training workflows.\n\nUse Enhanced NKI to innovate and build optimized kernels on Trainium. Explore the NKI Compiler source code to inspect and contribute to the MLIR-based compilation pipeline. \n\n.. note::\n The NKI Compiler source code is currently in **Private Preview**, while the NKI programming interface is in **Public Preview**.\n\n**Learn more**: :doc:`NKI home page ` and :doc:`NKI Language Guide `.\n\nNKI Library\n^^^^^^^^^^^\n\n**Public Preview**\n\nThe NKI Library provides an open source collection of optimized, ready-to-use kernels for common ML operations. The library includes kernels for dense transformer operations, MoE-specific operations, and attention mechanisms, all with complete source code, documentation, and benchmarks.\n\nUse NKI Library kernels directly in your models to improve performance, or explore the implementations as reference for best practices of performance optimizations on Trainium.\n\n**Learn more**: `GitHub repository `__ and :doc:`API documentation `.\n\n\nNeuron Explorer\n^^^^^^^^^^^^^^^\n\n**Public Preview**\n\nNeuron Explorer is a tools suite that supports developers and performance engineers in their performance optimization journey. It provides capabilities to inspect and optimize code from framework operations down to hardware instructions with hierarchical profiling, source code linking, IDE integration, and AI-powered recommendations for optimization insights.\n\nUse Neuron Explorer to understand and optimize your model performance on Trainium, from high-level framework operations to low-level hardware execution.\n\n**Learn more**: :doc:`Neuron Explorer documentation `.\n\n\nKubernetes-Native Resource Management with Neuron DRA\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Private Preview**\n\nNeuron Dynamic Resource Allocation (DRA) provides Kubernetes-native resource management for Trainium, eliminating custom scheduler extensions. DRA enables topology-aware scheduling using the default Kubernetes scheduler, atomic UltraServer allocation, and flexible per-workload configuration.\n\nNeuron DRA supports EKS, SageMaker HyperPod, and UltraServer configurations. The driver is open source with container images in AWS ECR public gallery.\n\nUse Neuron DRA to simplify Kubernetes resource management for your Trainium workloads with native scheduling and topology-aware allocation.\n\n**Learn more**: :doc:`Neuron DRA documentation `.\n\n**Access**: Contact your AWS account team to participate in the Private Preview.\n\n\nResources and Additional Information\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor more information visit the `AWS Trainium official page `__, the :doc:`AWS Neuron Documentation `, and :doc:`the AWS Neuron GitHub repositories `.\n\n\n\n\n" }, { "path": "archive/helper-tools/index.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nHelper Tools\n============\n\n.. toctree:: \n :maxdepth: 1\n\n \n Check Model \n GatherInfo " }, { "path": "archive/helper-tools/tutorial-neuron-check-model.rst", "content": ".. _neuron_check_model:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuron Check Model\n^^^^^^^^^^^^^^^^^^\n\nOverview\n========\n\nNeuron Check Model tool provides user with basic information about the compiled and uncompiled model's operations\nwithout the use of TensorBoard-Neuron. For additional visibility into the models, please see :ref:`neuron-plugin-tensorboard`.\n\nNeuron Check Model tool scans the user's uncompiled model and provides a table of the operations within the uncompiled\nmodel. By default, the table shows each operation type and number of instances of that type within model, and whether\nthe type is supported in Neuron. If --show_names option is specified, the table shows each operation by name and\nwhether the type of that operation is supported in Neuron.\n\nIf the model is already compiled, the tool also provides the table of operations as for uncompiled model. The table\ninclude the Neuron subgraph type and number of instances of that type, along with operations that have not been\ncompiled to Neuron. Additionally, the tool displays a message showing the minimum number of NeuronCores required to run the\nmodel, followed by another table which shows the list of Neuron subgraphs by name and the number of pipelined\nNeuronCores used by each subgraph. More information about NeuronCore pipeline can be found in\n:ref:`neuroncore-pipeline`. If --expand_subgraph option is specified, the operations within each subgraph are\nprinted below the subgraph information.\n\nNeuron Check Model tool is currently available for TensorFlow and MXNet. To check PT model, please use\ntorch.neuron.analyze_model function as shown in PyTorch-Neuron Getting Started tutorial :ref:`/src/examples/pytorch/resnet50.ipynb`\n\nTensorFlow-Neuron Check Model\n=============================\n\nThe following example shows how to run TensorFlow-Neuron Check Model tool with TensorFlow ResNet50 tutorial.\n\n1. Start with the TensorFlow ResNet50 tutorial at :ref:`/src/examples/tensorflow/tensorflow_resnet50/resnet50.ipynb` and do the first three steps of the\ntutorial. Please stay in the Python environment that you setup during the tutorial.\n\n2. Install needed tensorflow_hub package and download the tool:\n\n::\n\n pip install tensorflow_hub\n wget https://raw.githubusercontent.com/aws/aws-neuron-sdk/master/src/neuron-gatherinfo/tf_neuron_check_model.py\n python tf_neuron_check_model.py -h\n\n::\n\n usage: tf_neuron_check_model.py [-h] [--show_names] [--expand_subgraph]\n model_path\n\n positional arguments:\n model_path a TensorFlow SavedModel directory (currently supporting\n TensorFlow v1 SaveModel only).\n\n optional arguments:\n -h, --help show this help message and exit\n --show_names list operation by name instead of summarizing by type\n (caution: this option will generate many lines of output\n for a large model).\n --expand_subgraph show subgraph operations.\n\n3. After step 3 of the TensorFlow ResNet50 tutorial, you can check the uncompiled model to see Neuron supported operations (currently supporting TensorFlow v1 SaveModel only):\n\n::\n\n $ python tf_neuron_check_model.py ws_resnet50/resnet50/\n\n * The following table shows the supported and unsupported operations within this uncompiled model.\n * Each line shows an operation type, the number of instances of that type within model,\n * and whether the type is supported in Neuron.\n * Some operation types are excluded from table because they are no-operations or training-related operations:\n ['Placeholder', 'PlaceholderWithDefault', 'NoOp', 'Const', 'Identity', 'IdentityN', 'VarHandleOp',\n 'VarIsInitializedOp', 'AssignVariableOp', 'ReadVariableOp', 'StringJoin', 'ShardedFilename', 'SaveV2',\n 'MergeV2Checkpoints', 'RestoreV2']\n\n Op Type Num Instances Neuron Supported ?\n ------- ------------- ------------------\n Pad 2 Yes\n RandomUniform 54 Yes\n Sub 54 Yes\n Mul 54 Yes\n Add 54 Yes\n Conv2D 53 Yes\n BiasAdd 54 Yes\n FusedBatchNormV3 53 Yes\n Relu 49 Yes\n MaxPool 1 Yes\n AddV2 16 Yes\n Fill 56 Yes\n Mean 1 Yes\n MatMul 1 Yes\n Softmax 1 Yes\n Pack 1 Yes\n\n * Total inference operations: 504\n * Total Neuron supported inference operations: 504\n * Percent of total inference operations supported by Neuron: 100.0\n\n4. You can also check the compiled model to see the number of pipeline NeuronCores for each subgraph:\n\n::\n\n $ python tf_neuron_check_model.py ws_resnet50/resnet50_neuron/\n\n * Found 1 Neuron subgraph(s) (NeuronOp(s)) in this compiled model.\n * Use this tool on the original uncompiled model to see Neuron supported operations.\n * The following table shows all operations, including Neuron subgraphs.\n * Each line shows an operation type, the number of instances of that type within model,\n * and whether the type is supported in Neuron.\n * Some operation types are excluded from table because they are no-operations or training-related operations:\n ['Placeholder', 'PlaceholderWithDefault', 'NoOp', 'Const', 'Identity', 'IdentityN', 'VarHandleOp',\n 'VarIsInitializedOp', 'AssignVariableOp', 'ReadVariableOp', 'StringJoin', 'ShardedFilename', 'SaveV2',\n 'MergeV2Checkpoints', 'RestoreV2']\n\n Op Type Num Instances Neuron Supported ?\n ------- ------------- ------------------\n NeuronOp 1 Yes\n\n * Please run this model on Inf1 instance with at least 1 NeuronCore(s).\n * The following list show each Neuron subgraph with number of pipelined NeuronCores used by subgraph\n * (and subgraph operations if --expand_subgraph is used):\n\n Subgraph Name Num Pipelined NeuronCores\n ------------- -------------------------\n conv5_block3_3_bn/FusedBatchNormV3/ReadVariableOp/neuron_op_d6f098c01c780733 1\n\n5. When showing subgraph information, you can use --expand_subgraph to show operation types in each subgraph:\n\n::\n\n $ python tf_neuron_check_model.py ws_resnet50/resnet50_neuron/ --expand_subgraph\n\n (output truncated to show subgraph information only)\n\n Subgraph Name Num Pipelined NeuronCores\n ------------- -------------------------\n conv5_block3_3_bn/FusedBatchNormV3/ReadVariableOp/neuron_op_d6f098c01c780733 1\n Op Type Num Instances\n ------- -------------\n MatMul 1\n Relu 49\n Add 16\n FusedBatchNorm 53\n BiasAdd 54\n Conv2D 53\n Pad 2\n Mean 1\n MaxPool 1\n Softmax 1\n\n6. Use --show_names to see full operation names (caution: this option will generate many lines of output for a large model):\n\n::\n\n $ python tf_neuron_check_model.py ws_resnet50/resnet50_neuron/ --show_names\n\n * Found 1 Neuron subgraph(s) (NeuronOp(s)) in this compiled model.\n * Use this tool on the original uncompiled model to see Neuron supported operations.\n * The following table shows all operations, including Neuron subgraphs.\n * Each line shows an operation name and whether the type of that operation is supported in Neuron.\n * Some operation types are excluded from table because they are no-operations or training-related operations:\n ['Placeholder', 'PlaceholderWithDefault', 'NoOp', 'Const', 'Identity', 'IdentityN', 'VarHandleOp',\n 'VarIsInitializedOp', 'AssignVariableOp', 'ReadVariableOp', 'StringJoin', 'ShardedFilename', 'SaveV2',\n 'MergeV2Checkpoints', 'RestoreV2']\n\n Op Name Op Type Neuron Supported ?\n ------- ------- ------------------\n conv5_block3_3_bn/FusedBatchNormV3/ReadVariableOp/neuron_op_d6f098c01c780733 NeuronOp Yes\n\n * Please run this model on Inf1 instance with at least 1 NeuronCore(s).\n * The following list show each Neuron subgraph with number of pipelined NeuronCores used by subgraph\n * (and subgraph operations if --expand_subgraph is used):\n\n Subgraph Name Num Pipelined NeuronCores\n ------------- -------------------------\n conv5_block3_3_bn/FusedBatchNormV3/ReadVariableOp/neuron_op_d6f098c01c780733 1\n\n\nMXNet-Neuron Check Model\n=========================\n\nThe following example shows how to run MXNet-Neuron Check Model tool with MXNet ResNet50 tutorial.\n\n1. Start with the MXNet ResNet50 tutorial at :ref:`/src/examples/mxnet/resnet50/resnet50.ipynb` and do the first three steps of the tutorial.\nPlease stay in the Python environment that you setup during the tutorial.\n\n2. Download the tool:\n\n::\n\n wget https://raw.githubusercontent.com/aws/aws-neuron-sdk/master/src/neuron-gatherinfo/mx_neuron_check_model.py\n python mx_neuron_check_model.py -h\n\n::\n\n usage: mx_neuron_check_model.py [-h] [--show_names] [--expand_subgraph]\n model_path\n\n positional arguments:\n model_path path prefix to MXNet model (the part before -symbol.json)\n\n optional arguments:\n -h, --help show this help message and exit\n --show_names list operation by name instead of summarizing by type\n (caution: this option will generate many lines of output\n for a large model).\n --expand_subgraph show subgraph operations.\n\n3. After step 3 of MXNet ResNet50 tutorial, you can check the uncompiled model to see Neuron supported operations:\n\n::\n\n $ python mx_neuron_check_model.py resnet-50\n\n * The following table shows the supported and unsupported operations within this uncompiled model.\n * Each line shows an operation type, the number of instances of that type within model,\n * and whether the type is supported in Neuron.\n * Some operation types are excluded from table because they are no-operations or training-related operations:\n ['null']\n\n Op Type Num Instances Neuron Supported ?\n ------- ------------- ------------------\n BatchNorm 51 Yes\n Convolution 53 Yes\n Activation 50 Yes\n Pooling 2 Yes\n elemwise_add 16 Yes\n Flatten 1 Yes\n FullyConnected 1 Yes\n SoftmaxOutput 1 No\n\n * Total inference operations: 175\n * Total Neuron supported inference operations: 174\n * Percent of total inference operations supported by Neuron: 99.4\n\n4. You can also check the compiled model to see the number of pipeline NeuronCores for each subgraph:\n\n::\n\n $ python mx_neuron_check_model.py resnet-50_compiled\n\n * Found 1 Neuron subgraph(s) (_neuron_subgraph_op(s)) in this compiled model.\n * Use this tool on the original uncompiled model to see Neuron supported operations.\n * The following table shows all operations, including Neuron subgraphs.\n * Each line shows an operation type, the number of instances of that type within model,\n * and whether the type is supported in Neuron.\n * Some operation types are excluded from table because they are no-operations or training-related operations:\n ['null']\n\n Op Type Num Instances Neuron Supported ?\n ------- ------------- ------------------\n _neuron_subgraph_op 1 Yes\n SoftmaxOutput 1 No\n\n * Please run this model on Inf1 instance with at least 1 NeuronCore(s).\n * The following list show each Neuron subgraph with number of pipelined NeuronCores used by subgraph\n * (and subgraph operations if --expand_subgraph is used):\n\n Subgraph Name Num Pipelined NeuronCores\n ------------- -------------------------\n _neuron_subgraph_op0 1\n\n5. When showing subgraph information, you can use --expand_subgraph to show operation types in each subgraph:\n\n::\n\n $ python mx_neuron_check_model.py resnet-50_compiled --expand_subgraph\n\n (output truncated to show subgraph information only)\n\n Subgraph Name Num Pipelined NeuronCores\n ------------- -------------------------\n _neuron_subgraph_op0 1\n Op Type Num Instances\n ------- -------------\n BatchNorm 51\n Convolution 53\n Activation 50\n Pooling 2\n elemwise_add 16\n Flatten 1\n FullyConnected 1\n\n6. Use --show_names to see full operation names (caution: this option will generate many lines of output for a large model):\n\n::\n\n $ python mx_neuron_check_model.py resnet-50_compiled --show_names\n\n * Found 1 Neuron subgraph(s) (_neuron_subgraph_op(s)) in this compiled model.\n * Use this tool on the original uncompiled model to see Neuron supported operations.\n * The following table shows all operations, including Neuron subgraphs.\n * Each line shows an operation name and whether the type of that operation is supported in Neuron.\n * Some operation types are excluded from table because they are no-operations or training-related operations:\n ['null']\n\n Op Name Op Type Neuron Supported ?\n ------- ------- ------------------\n _neuron_subgraph_op0 _neuron_subgraph_op Yes\n softmax SoftmaxOutput No\n\n * Please run this model on Inf1 instance with at least 1 NeuronCore(s).\n * The following list show each Neuron subgraph with number of pipelined NeuronCores used by subgraph\n * (and subgraph operations if --expand_subgraph is used):\n\n Subgraph Name Num Pipelined NeuronCores\n ------------- -------------------------\n _neuron_subgraph_op0 1\n" }, { "path": "archive/helper-tools/tutorial-neuron-gatherinfo.rst", "content": ".. _neuron_gatherinfo:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nUsing Neuron GatherInfo Tool to collect debug and support information\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOverview\n========\n\nThe Neuron GatherInfo tool ``neuron-gatherinfo.py`` can assist in\nautomating the collection and packaging of information from Neuron SDK\ntools that is useful to both user and AWS for issue resolution. The tool\ngathers log files and other system information. If being used to supply\nthat info to AWS, the tool will redact proprietary and confidential\ninformation. The GatherInfo tool is supplied in source code form -\navailable here: :github:`Neuron Gatherinfo `\n\nThe tool enables developers to gather compiler and inference/runtime\nlogs. Additionally, the common usage is from within one of the supported\nML frameworks that have been integrated with Neuron, and information can\nbe captured from those compile/runtime environments using the\nframeworks.\n\nSteps Overview:\n~~~~~~~~~~~~~~~\n\n1. Obtain a copy of neuron-gatherinfo.py from\n :github:`Neuron Gatherinfo `\n2. Install into a location in your $PATH or into a location from where\n you can launch the script\n3. Use with compile and/or runtime environments\n\nNeuron-CC information gathering\n-------------------------------\n\nStep 1: Re-run the compile steps for your workload with increased verbosity or debug levels\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- For TensorFlow-Neuron, change the Python code as shown. Note that\n ‘compiler-workdir’ is expected to be an empty directory to prevent\n files from other runs from interfering with the information\n gathering. The call to the compile function has to be augmented with\n the **verbose** and the \\**compiler_workdir \\**arguments. In\n addition, please capture the stdout messages into a file (for\n example, by redirecting the stdout to a file)\n\n::\n\n tfn.saved_model.compile(model_dir, compiled_model_dir, compiler_args=['--verbose', '2', '--pipeline', 'compile', 'SaveTemps'], compiler_workdir='./compiler-workdir')\n\n- For Neuron Apache MXNet, add compiler arguments as shown below and run the\n compilation process from an empty workdir:\n\n::\n\n import mxnet as mx\n import os\n\n from packaging import version\n mxnet_version = version.parse(mx.__version__)\n if mxnet_version >= version.parse(\"1.8\"):\n import mx_neuron as neuron\n else: \n from mxnet.contrib import neuron\n\n ...\n os.environ['SUBGRAPH_INFO'] = '1'\n compile_args = { '--verbose' : 2, '--pipeline' : 'compile', 'flags' : ['SaveTemps'] }\n csym, cargs, cauxs = neuron.compile(sym, args, auxs, inputs=inputs, **compile_args)\n\n.. _step-2-run-neuron-gatherinfopy-to-gather-information-to-share:\n\nStep 2: Run neuron-gatherinfo.py to gather information to share\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe output result will be a tar.gz file.\n\nNeuron Runtime information gathering\n------------------------------------\n\nStep 1: EXECUTE inference steps for your workload with increased verbosity or debug levels\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn the case of runtime information, the tool **neuron-dump.py** is used\nby \\**neuron-gatherinfo.py \\**to gather that information. Make sure that\nyou have the neuron tools package (aws-neuron-tools) installed.\n\n.. _step-2-run-neuron-gatherinfopy-to-gather-information-to-share-1:\n\nStep 2: Run neuron-gatherinfo.py to gather information to share\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe output result will be a tar.gz file.\n\nTool Usage Reference\n====================\n\nRun neuron-gatherinfo.py using the “—help“ option:\n\n::\n\n bash $ ~/bin/neuron-gatherinfo.py --help\n usage: neuron-gatherinfo.py [-h] [--additionalfileordir ADDFLDIR] [-c CCDIR]\n [-i] [-f FILTERFILE] [-m] -o OUTDIR [-r RTDIR] -s\n STDOUT [-v]\n\n Usage: /home/user/bin/neuron-gatherinfo.py [options]\n This program is used to gather information from this system for analysis\n and debugging\n\n\n optional arguments:\n -h, --help show this help message and exit\n --additionalfileordir ADDFLDIR\n Additional file or directory that the user wants to\n provide in the archive. The user can sanitize this\n file or directory before sharing\n -c CCDIR, --compileroutdir CCDIR\n Location of the neuron-cc generated files\n -i, --include By default, only the lines containing (grep) patterns\n like 'nrtd|neuron|kernel:' from the syslog are copied.\n Other lines are excluded. Using this option allows the\n timestamp section of other lines to be included. The\n rest of the contents of the line itself are elided.\n Providing the timestamp section may provide time\n continuity while viewing the copied syslog file\n -f FILTERFILE, --filter FILTERFILE\n -m, --modeldata By using this option, the entire compiler work\n directory's contents will be included (excluding the\n .pb files, unless an additional option is used). This\n would include model information, etc. The files that\n are included, by default, are these: graph_def.neuron-\n cc.log, all_metrics.csv, hh-tr-operand-\n tensortensor.json\n -o OUTDIR, --out OUTDIR\n The output directory where all the files and other\n information will be stored. The output will be stored\n as an archive as well as the actual directory where\n all the contents are copied. This will allow a simple\n audit of the files, if necessary. *** N O T E ***:\n Make sure that this directory has enough space to hold\n the files and resulting archive\n -r RTDIR, --runtimeoutdir RTDIR\n Location of the neuron runtime generated files\n -s STDOUT, --stdout STDOUT\n The file where the stdout of the compiler run was\n saved\n -v, --verbose Verbose mode displays commands executed and any\n additional information which may be useful in\n debugging the tool itself\n\nExamples\n========\n\nExample 1: no ML model information gathered (default behavior)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn this case, the tool will archive just the default information\ngathering:\n\n::\n\n bash $ sudo ~/bin/neuron-gatherinfo.py -o compile-and-run-info-for-debugging-no-model-info -i --verbose -s stdout-from-compile_resnet50.out -c compiler-workdir\n\n Running cmd: lscpu and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-lscpu.txt\n Running cmd: lshw and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-lshw.txt\n Running cmd: lspci | grep -i Amazon and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-lspci.txt\n Running cmd: neuron-cc --version and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-neuron-cc.txt\n Running cmd: neuron-ls and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-neuron-ls.txt\n \n ******\n Archive created at:\n /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo.tar.gz\n From directory:\n /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo\n ******\n\n\n.. _example-2--model-ml-information-gathered-using-the-modeldata-option:\n\nExample 2 : model ML information gathered using the “—modeldata” option\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn this case, the tool will archive the compiler work directory in\naddition to the default information gathering\n\n::\n\n bash $ sudo ~/bin/neuron-gatherinfo.py -o compile-and-run-info-for-debugging -i --verbose -s stdout-from-compile_resnet50.out -c compiler-workdir --modeldata\n\n \n Running cmd: lscpu and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging/neuron-gatherinfo/report-lscpu.txt\n Running cmd: lshw and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging/neuron-gatherinfo/report-lshw.txt\n Running cmd: lspci | grep -i Amazon and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging/neuron-gatherinfo/report-lspci.txt\n Running cmd: neuron-cc --version and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-neuron-cc.txt\n Running cmd: neuron-ls and capturing output in file: /home/user/tutorials-3/compile-and-run-info-for-debugging-no-model-info/neuron-gatherinfo/report-neuron-ls.txt\n \n\n ******\n Archive created at:\n /home/user/tutorials-3/compile-and-run-info-for-debugging/neuron-gatherinfo.tar.gz\n From directory:\n /home/user/tutorials-3/compile-and-run-info-for-debugging/neuron-gatherinfo\n ******\n\n\n **************************\n Based on your command line option, we're also packaging these files:\n\n graph_def.neuron-cc.log\n all_metrics.csv\n hh-tr-operand-tensortensor.json\n\n And this directory: /home/user/tutorials-3/compiler-workdir\n\n **************************\n\n" }, { "path": "archive/index.rst", "content": ".. meta::\n :description: Archived AWS Neuron SDK documentation\n :keywords: AWS Neuron SDK, archived tutorials, legacy documentation\n :date-modified: 12-02-2025\n\n=====================================\nArchived AWS Neuron SDK documentation\n=====================================\n\n.. note::\n\n This page contains archived tutorials and other documentation for older versions of the AWS Neuron SDK.\n These pages are no longer actively maintained and may reference unsupported features or deprecated APIs. They are provided as-is and may not reflect the current state of the AWS Neuron SDK.\n\nOverview\n--------\n\nThe following content has been archived for reference purposes. For the latest documentation and guides, visit the `AWS Neuron SDK documentation `_.\n\nArchived feature docs\n---------------------\n\n.. list-table::\n :header-rows: 1\n\n * - Feature\n - Last release supported\n - Date archived\n * - :doc:`tensorboard/getting-started-tensorboard-neuron-plugin`\n - Neuron 2.27.0\n - Archived on: 12/2/2025\n * - :doc:`neuronperf/index`\n - Neuron 2.27.0\n - Archived on: 12/2/2025\n * - :doc:`helper-tools/index`\n - Neuron 2.27.0\n - Archived on: 12/2/2025\n * - :doc:`transformers-neuronx/index`\n - Neuron 2.25.0\n - Archived on: 9/15/2025\n * - :doc:`MXNet Neuron Setup Guides `\n - Neuron 2.27.0\n - Archived on: 3/30/2026\n * - :doc:`mxnet-neuron/index`\n - Neuron 2.16.0\n - Archived on: 3/11/2026\n * - :doc:`tensorflow/index`\n - Neuron 2.22.0\n - Archived on: 3/11/2026\n * - :doc:`torch-neuron/index`\n - Neuron 2.22.0\n - Archived on: 3/11/2026\n\n\nArchived tutorials\n------------------\n\n.. list-table::\n :header-rows: 1\n\n * - Tutorial\n - Last release supported\n - Date archived\n * - :doc:`tutorials/finetune_t5`\n - Neuron 2.24.0\n - Archived on: 7/31/2025\n * - :doc:`tutorials/ssd300_demo/ssd300_demo`\n - Neuron 2.24.0\n - Archived on: 7/31/2025\n * - :doc:`tutorials/megatron_gpt_pretraining`\n - Neuron 2.25.0\n - Archived on: 7/31/2025\n * - :doc:`tutorials/finetuning_llama2_7b_ptl`\n - Neuron 2.26.0\n - Archived on: 8/25/2025\n * - :doc:`tutorials/training_llama2_tp_pp_ptl`\n - Neuron 2.26.0\n - Archived on: 8/25/2025\n * - :doc:`tutorials/training_codegen25_7b`\n - Neuron 2.26.0\n - Archived on: 8/25/2025\n * - :doc:`tutorials/gpt3_neuronx_nemo_megatron_pretraining`\n - Neuron 2.26.0\n - Archived on: 8/25/2025\n * - :doc:`tutorials/multinode-training-model-profiling`\n - Neuron 2.29.0\n - Archived on: 3/30/2026\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n tutorials/finetune_t5\n tutorials/ssd300_demo/ssd300_demo\n tutorials/megatron_gpt_pretraining\n tutorials/training-gpt-neox-20b\n tutorials/finetuning_llama2_7b_ptl\n tutorials/training_llama2_tp_pp_ptl\n tutorials/training_codegen25_7b\n tutorials/multinode-training-model-profiling\n tutorials/training-gpt-neox\n tensorboard/getting-started-tensorboard-neuron-plugin\n neuronperf/index\n helper-tools/index\n transformers-neuronx/index\n mxnet-neuron/index\n tensorflow/index\n torch-neuron/index\n\nAccessing Archived Content\n--------------------------\n\nEach tutorial listed above corresponds to a specific version or feature set of the Neuron SDK that has since been superseded. Use these resources for historical context or migration guidance.\n\n.. warning::\n\n Archived tutorials may not be compatible with current Neuron SDK releases. Exercise caution when following instructions from these documents.\n" }, { "path": "archive/mxnet-neuron/api-compilation-python-api.rst", "content": ".. _ref-mxnet-neuron-compilation-python-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nNeuron Apache MXNet Compilation Python API\n=======================================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe MXNet-Neuron compilation Python API provides a method to compile\nmodel graph for execution on Inferentia.\n\n\nDescription\n-----------\n\nWithin the graph or subgraph, the compile method selects and sends\nNeuron-supported operations to Neuron-Compiler for compilation and saves\nthe compiled artifacts in the graph. Uncompilable operations are kept as\noriginal operations for framework execution.\n\nThe compiled graph can be saved using the MXNet save_checkpoint and\nserved using MXNet Model Serving. Please see\n:ref:`mxnet-neuron-model-serving` for more information about exporting\nto saved model and serving using MXNet Model Serving.\n\nOptions can be passed to Neuron compiler via the compile function. For\nexample, the “\\ ``--neuroncore-pipeline-cores``\\ ” option directs Neuron compiler\nto compile each subgraph to fit in the specified number of NeuronCores.\nThis number can be less than the total available NeuronCores on an Inf1\ninstance. See :ref:`neuron-compiler-cli-reference` for more information\nabout compiler options.\n\nFor debugging compilation, use SUBGRAPH_INFO=1 environment setting before\ncalling the compilation script. The extract subgraphs are preserved as hidden\nfiles in the run directory. For more information, see :ref:`neuron_gatherinfo`\n\n**MXNet 1.5**\n-------------\n\nMethod\n------\n\n\n.. code:: python\n\n from mxnet.contrib import neuron\n neuron.compile(sym, args, aux, inputs, **compile_args)\n\n\nArguments\n---------\n\n- **sym** - Symbol object loaded from symbol.json file\n- **args** - args/params dictionary loaded from params file\n- **aux** - aux/params dictionary loaded from params file\n- **inputs** - a dictionary with key/value mappings for input name to\n input numpy arrays\n- **kwargs** (optional) - a dictionary with key/value mappings for\n MXNet-Neuron compilation and Neuron Compiler options.\n\n - For example, to limit the number of NeuronCores per subgraph, use\n ``compile_args={'--neuroncore-pipeline-cores' : N}`` where N is an integer\n representing the maximum number of NeuronCores per subgraph.\n - Additional compiler flags can be passed using\n ``'flags' : []`` where is a comma separated list of\n strings. See :ref:`neuron_gatherinfo` for example of passing debug\n flags to compiler.\n - Advanced option to exclude node names:\n ``compile_args={'excl_node_names' : []}`` where is a\n comma separated list of node name strings.\n\nReturns\n-------\n\n- **sym** - new partitioned symbol\n- **args** - modified args/params\n- **auxs** - modified aux/params\n\nExample Usage: Compilation\n--------------------------\n\nThe following is an example usage of the compilation, with default\ncompilation arguments:\n\n\n.. code:: python\n\n from mxnet.contrib import neuron\n ...\n neuron.compile(sym, args, aux, inputs={'data' : img})\n\n\n\n**MXNet 1.8**\n-------------\n\n\nMethod\n------\n\n\n.. code:: python\n\n import mx_neuron as neuron\n neuron.compile(obj, args=None, aux=None, inputs=None, **compile_args)\n\n\nArguments\n---------\n\n- **obj** - Symbol object loaded from symbol.json file or gluon.HybridBlock object\n- **args** (optional) - args/params dictionary loaded from params file. Only needed in case of Symbol object\n- **aux** (optional) - aux/params dictionary loaded from params file. Only needed in case of Symbol object\n- **inputs** - a dictionary with key/value mappings for input name to\n input numpy arrays.\n- **kwargs** (optional) - a dictionary with key/value mappings for\n MXNet-Neuron compilation and Neuron Compiler options.\n\n - For example, to limit the number of NeuronCores per subgraph, use\n ``compile_args={'--neuroncore-pipeline-cores' : N}`` where N is an integer\n representing the maximum number of NeuronCores per subgraph.\n - Additional compiler flags can be passed using\n ``'flags' : []`` where is a comma separated list of\n strings. See :ref:`neuron_gatherinfo` for example of passing debug\n flags to compiler.\n - Advanced option to exclude node names:\n ``compile_args={'excl_node_names' : []}`` where is a\n comma separated list of node name strings.\n - work_dir: relative or absolute path for storing compiler artifacts (including params and jsons) generated \n during compilation when SUBGRAPH_INFO=1.\n\nReturns\n-------\n- **(sym, args, auxs)** - for symbol object as input. sym, args and auxs are new partitioned symbol, modified args/params and modified aux/params repectively.\n- **(obj)** - for gluon.HybridBlock object as input. obj is the parititioned and optimized gluon.Hybrid block object for Neuron backend.\n\n\nExample Usage: Compilation\n--------------------------\n\nThe following is an example usage of the compilation, with default\ncompilation arguments for symbol object:\n\n\n.. code:: python\n\n import mx_neuron as neuron\n ... \n neuron.compile(sym, args, aux, inputs={'data' : img})\n\n\nThe following is an example usage of the compilation, with default\ncompilation arguments for gluon.HybridBlock object (only supported in MXNet-Neuron 1.8):\n\n.. code:: python\n\n import mx_neuron as neuron\n ... \n neuron.compile(obj, inputs={'data' : img})\n\n\nExample Usage: Extract Compilation Statistics\n---------------------------------------------\n\nTo extract operation counts, insert the following code after compile\nstep (assume csym is the compiled MXNet symbol):\n\n.. code:: python\n\n import json\n\n # Return list of nodes from MXNet symbol\n def sym_nodes(sym):\n return json.loads(sym.tojson())['nodes']\n\n # Return number of operations in node list \n def count_ops(graph_nodes):\n return len([x['op'] for x in graph_nodes if x['op'] != 'null'])\n\n # Return triplet of compile statistics\n # - count of operations in symbol database\n # - number of Neuron subgraphs\n # - number of operations compiled to Neuron runtime \n def get_compile_stats(sym):\n cnt = count_ops(sym_nodes(sym))\n neuron_subgraph_cnt = 0\n neuron_compiled_cnt = 0\n for g in sym_nodes(sym):\n if g['op'] == '_neuron_subgraph_op':\n neuron_subgraph_cnt += 1\n for sg in g['subgraphs']:\n neuron_compiled_cnt += count_ops(sg['nodes'])\n return (cnt, neuron_subgraph_cnt, neuron_compiled_cnt)\n\n original_cnt = count_ops(sym_nodes(sym))\n post_compile_cnt, neuron_subgraph_cnt, neuron_compiled_cnt = get_compile_stats(csym)\n print(\"INFO:mxnet: Number of operations in original model: \", original_cnt)\n print(\"INFO:mxnet: Number of operations in compiled model: \", post_compile_cnt)\n print(\"INFO:mxnet: Number of Neuron subgraphs in compiled model: \", neuron_subgraph_cnt)\n print(\"INFO:mxnet: Number of operations placed on Neuron runtime: \", neuron_compiled_cnt)\n\n.. code:: bash\n\n INFO:mxnet: Number of operations in original model: 67\n INFO:mxnet: Number of operations in compiled model: 4\n INFO:mxnet: Number of Neuron subgraphs in compiled model: 2\n INFO:mxnet: Number of operations placed on Neuron runtime: 65\n" }, { "path": "archive/mxnet-neuron/api-reference-guide.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nAPI Reference Guide (mxnet-neuron)\n==================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /archive/mxnet-neuron/api-compilation-python-api\n\n\n.. include:: /archive/mxnet-neuron/api-reference-guide.txt\n" }, { "path": "archive/mxnet-neuron/api-reference-guide.txt", "content": "* :ref:`ref-mxnet-neuron-compilation-python-api`" }, { "path": "archive/mxnet-neuron/developer-guide.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nDeveloper Guide\n===============\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n /about-neuron/appnotes/mxnet-neuron/flex-eg\n\n\n.. include:: /archive/mxnet-neuron/developer-guide.txt\n" }, { "path": "archive/mxnet-neuron/developer-guide.txt", "content": "* :ref:`flexeg`" }, { "path": "archive/mxnet-neuron/ec2-then-ec2-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/ec2-then-ec2-devflow.rst\n" }, { "path": "archive/mxnet-neuron/index.rst", "content": "Neuron Apache MXNet Release Notes\n==============================================\n\n.. toctree::\n :maxdepth: 1\n\n /release-notes/archive/mxnet-neuron\n" }, { "path": "archive/mxnet-neuron/inference-mxnet-neuron.rst", "content": ".. _inference-mxnet-neuron:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInference (mxnet-neuron) (maintenance)\n=======================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Tutorials \n API Reference Guide \n Developer Guide \n Misc \n\n\n.. include:: inference-mxnet-neuron.txt\n" }, { "path": "archive/mxnet-neuron/inference-mxnet-neuron.txt", "content": ".. card:: Setup (``mxnet-neuron``)\n :link: setup-mxnet-neuron\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. dropdown:: Tutorials\n :class-title: sphinx-design-class-title-small\n :animate: fade-in\n\n .. include:: /archive/mxnet-neuron/tutorials/tutorials-mxnet-neuron.txt\n\n\n.. dropdown:: API Reference Guide\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /archive/mxnet-neuron/api-reference-guide.txt\n\n\n.. dropdown:: Developer Guide\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /archive/mxnet-neuron/developer-guide.txt\n\n\n.. dropdown:: Misc\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /archive/mxnet-neuron/misc-mxnet-neuron.txt" }, { "path": "archive/mxnet-neuron/misc-mxnet-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nMisc (mxnet-neuron)\n===================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /archive/mxnet-neuron/troubleshooting-guide\n What's New \n /release-notes/archive/neuron-cc/neuron-cc-ops/neuron-cc-ops-mxnet\n\n\n.. include:: /archive/mxnet-neuron/misc-mxnet-neuron.txt\n" }, { "path": "archive/mxnet-neuron/misc-mxnet-neuron.txt", "content": "* :ref:`mxnet_troubleshooting_guide`\n* :ref:`What's New `\n* :ref:`neuron-cc-ops-mxnet`" }, { "path": "archive/mxnet-neuron/mxnet-neuron-setup.rst", "content": ".. _mxnet-setup:\n\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nMXNet Neuron Setup\n==================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: mxnet-neuron-setup.txt\n" }, { "path": "archive/mxnet-neuron/mxnet-neuron-setup.txt", "content": "\n.. card:: MxNet Neuron (``mxnet-neuron``) Setup for Inf1 Instances\n :link: setup-mxnet-neuron\n :link-type: ref\n :class-body: sphinx-design-class-title-small" }, { "path": "archive/mxnet-neuron/neo-then-hosting-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/neo-then-hosting-devflow.rst\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-install-prev-al2.rst", "content": ".. _mxnet-neuron-install-prev-al2:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous MXNet Neuron Releases for Amazon Linux (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.17.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.17.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.16.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.16.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-install-prev-al2023.rst", "content": ".. _mxnet-neuron-install-prev-al2023:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous MXNet Neuron Releases for Amazon Linux 2023 (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-install-prev-u20.rst", "content": "\n.. Install previous MXNet Neuron releases for Ubuntu 20.04 - archived\n\nUse the tabs below to install a specific previous Neuron SDK release. Select the Neuron version you need.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-install-prev-u22.rst", "content": ".. _mxnet-neuron-install-prev-u22:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous MXNet Neuron Releases for Ubuntu 22 (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-install.rst", "content": ".. _install-neuron-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron\n=====================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-neuron-al2-base-dlami.rst", "content": ".. _setup-mxnet-neuron-al2-base-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nMXNet Neuron (\"mxnet-neuron\") Setup on Amazon Linux 2\n=========================================================\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`install-neuron-mxnet`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instance sizes and pricing see: `Inf1 web page `_\n * Check for the latest version of the `DLAMI Base AMI `_ and copy the AMI name that starts with \"Deep Learning Base Neuron AMI (Amazon Linux 2) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see a matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-mxnet-neuron-al2.txt\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-update-u20.rst\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-install-prev-al2.rst" }, { "path": "archive/mxnet-neuron/setup/mxnet-neuron-al2.rst", "content": ".. _setup-mxnet-neuron-al2:\n\n.. include:: /setup/install-templates/al2-python.rst\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nMXNet Neuron (\"mxnet-neuron\") Setup on Amazon Linux 2\n======================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n.. include:: /setup/install-templates/al2-python.rst\n\nGet Started with Latest Release of MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`install-neuron-mxnet`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Amazon Linux 2 AMI(HVM) - Kernel 5.10\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-mxnet-neuron-al2.txt\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-update-u20.rst\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-install-prev-al2.rst" }, { "path": "archive/mxnet-neuron/setup/mxnet-neuron-al2023.rst", "content": ".. _setup-mxnet-neuron-al2023:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nMXNet Neuron (\"mxnet-neuron\") Setup on Amazon Linux 2023\n=========================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\nGet Started with Latest Release of MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`install-neuron-mxnet`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Amazon Linux 2023 AMI\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n\n.. include:: /includes/setup/tab-inference-mxnet-neuron-al2023.txt\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-install-prev-al2023.rst" }, { "path": "archive/mxnet-neuron/setup/mxnet-neuron-ubuntu20-base-dlami.rst", "content": ".. _setup-mxnet-neuron-u20-base-dlami:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nMXNet Neuron (\"mxnet-neuron\") Setup on Ubuntu 20\n================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`install-neuron-mxnet`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instance sizes and pricing see: `Inf1 web page `_\n * Check for the latest version of the `DLAMI Base AMI `_ and copy the AMI name that starts with \"Deep Learning Base Neuron AMI (Ubuntu 20.04) \" from \"AMI Name:\" section\n * Search for the copied AMI name in the AMI Search , you should see a matching AMI with the AMI name in Community AMIs. Select the AMI and use it to launch the instance.\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance\n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n.. include:: /includes/setup/tab-inference-mxnet-neuron-u20.txt\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-update-u20.rst\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-install-prev-u20.rst" }, { "path": "archive/mxnet-neuron/setup/mxnet-neuron-ubuntu20.rst", "content": ".. _setup-mxnet-neuron-u20:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nMXNet Neuron (\"mxnet-neuron\") Setup on Ubuntu 20\n=================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`install-neuron-mxnet`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Ubuntu Server 20 AMI\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n.. include:: /includes/setup/tab-inference-mxnet-neuron-u20.txt\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-update-u20.rst\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-install-prev-u20.rst" }, { "path": "archive/mxnet-neuron/setup/mxnet-neuron-ubuntu22.rst", "content": ".. _setup-mxnet-neuron-u22:\n\n.. card:: Select a Different Framework or Platform for Setup\n :link: setup-guide-index\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\nMXNet Neuron (\"mxnet-neuron\") Setup on Ubuntu 22\n=================================================\n\n\n.. contents:: Table of contents\n\t:local:\n\t:depth: 2\n\n\nGet Started with Latest Release of MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis section provide links that will assist you to quickly start with a fresh installation of :ref:`install-neuron-mxnet`.\n\n\n.. dropdown:: Launch the Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n * Please follow the instructions at `launch an Amazon EC2 Instance `_ to launch an instance. When choosing the instance type at the EC2 console. please make sure to select the correct instance type.\n * To get more information about instances sizes and pricing see: `Inf1 web page `_\n * Select Ubuntu Server 20 AMI\n * After launching the instance, follow the instructions in `Connect to your instance `_ to connect to the instance \n\n.. dropdown:: Install Drivers and Tools\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami --category=driver_runtime_tools\n\n.. include:: /includes/setup/tab-inference-mxnet-neuron-u22.txt\n\n.. include:: /archive/mxnet-neuron/setup/mxnet-install-prev-u22.rst" }, { "path": "archive/mxnet-neuron/setup/mxnet-update-u20.rst", "content": "\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n.. mxnet-neuron-u20-update:\n\nUpdate to latest MXNet Neuron (``mxnet-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/mxnet-update.rst", "content": ".. _update-neuron-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest MXNet Neuron\n===============================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=mxnet --framework-version=1.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.14.2-mxnet-install.rst", "content": ".. _install-neuron-1.14.2-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron (Neuron 1.14.2)\n======================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.15.0-mxnet-install.rst", "content": ".. _install-neuron-1.15.0-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron (Neuron 1.15.0)\n======================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.15.1-mxnet-install.rst", "content": ".. _install-neuron-1.15.1-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron (Neuron 1.15.1)\n======================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.15.2-mxnet-install.rst", "content": ".. _install-neuron-1.15.2-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron (Neuron 1.15.2)\n======================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.16.3-mxnet-install.rst", "content": ".. _install-neuron-1.16.3-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron\n=====================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.17.2-mxnet-install.rst", "content": ".. _install-neuron-1.17.2-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron\n=====================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.18.0-mxnet-install.rst", "content": ".. _install-neuron-1.18.0-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron\n=====================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/prev-releases/neuron-1.19.0-mxnet-install.rst", "content": ".. _install-neuron-1.19.0-mxnet:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall MXNet Neuron\n=====================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: MXNet 1.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n\n\n .. tab-item:: MXNet 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install mxnet --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=mxnet-1.5.1\n\n" }, { "path": "archive/mxnet-neuron/setup/setup-inference", "content": "Setup Guide for Inf1\n====================\n\n.. toctree::\n :maxdepth: 1\n\n Fresh install \n Update to latest release \n Install previous releases " }, { "path": "archive/mxnet-neuron/troubleshooting-guide.rst", "content": ".. _mxnet_troubleshooting_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTroubleshooting Guide for Neuron Apache MXNet \n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nInference Runtime Error\n=======================\n\nOut-of-memory error when calling Symbol API bind() too many times\n-----------------------------------------------------------------\n\n.. important ::\n\n ``NEURONCORE_GROUP_SIZES`` will no longer be supported starting Neuron 1.19.0 release if your application is using ``NEURONCORE_GROUP_SIZES`` please \n see :ref:`neuron-migrating-apps-neuron-to-libnrt` and :ref:`eol-ncgs-env_2` for more details.\n\nIf you see out-of-memory error when using Symbol API's bind() function, please ensure that the bind() function is\ncalled once for each desired model instance. For example, on inf1.xlarge, use Symbol API to create 4 parallel \ninstances of a model that was compiled to 1 NeuronCore (--neuroncore-pipeline-cores=1), each is bound to an \ndifferent mx.neuron(i) context where i is the NeuronCore Group index ranging from 0 to 3. Then use 4 threads to feed\nthe 4 instances in parallel. For example:\n\n.. code:: python\n\n NUM_PARALLEL = 4\n os.environ['NEURONCORE_GROUP_SIZES'] = ','.join('1' for _ in range(NUM_PARALLEL))\n \n data_iter = []\n for i in range(NUM_PARALLEL):\n data_iter.append(mx.io.ImageRecordIter(\n path_imgrec=recfile_base, data_shape=(3, 224, 224), batch_size=1, \n prefetch_buffer=1,\n num_parts=NUM_PARALLEL, part_index=i))\n\n sym, args, auxs = mx.model.load_checkpoint('resnet-50_compiled', 0)\n\n exec_list = []\n for i in range(NUM_PARALLEL):\n exec = sym.bind(ctx=mx.neuron(i), args=args, aux_states=auxs, grad_req='null')\n exec_list.append(exec)\n\n def single_thread_infer(i):\n for batch in data_iter[i]:\n img = batch.data[0]\n label = batch.label\n feed_dict = {'data': img}\n exe = exec_list[i]\n exe.copy_params_from(feed_dict)\n exe.forward()\n out = exe.outputs[0]\n\n future_list = []\n with futures.ThreadPoolExecutor(max_workers=NUM_PARALLEL) as executor:\n for i in range(NUM_PARALLEL):\n future_list.append(executor.submit(single_thread_infer, i))\n\n\nInference crashed with MXNetError: InferShapeKeyword argument name xyz not found\n--------------------------------------------------------------------------------\n\nIf you see MXNetError:\n\n.. code:: bash\n\n mxnet.base.MXNetError: [11:55:39] src/c_api/c_api_symbolic.cc:508: InferShapeKeyword argument name xyz not found.\"\n\nThis is followed by a list of \"Candidate arguments\". This list shows all the input argument names that the model knows about, and 'xyz' is not in the list. To fix this, remove entry xyz from the feed dictionary.\n\n\nInference crashed at mx.nd.waitall() with MXNetError: Check failed: bin.dtype() == mshadow::kUint8\n--------------------------------------------------------------------------------------------------\n\nWhen executing Symbol API's forward function followed by mx.nd.waitall(), where MXNetError exception occurs with 'Check failed: bin.dtype() == mshadow::kUint8'.\n\n\nInference crashed with NRTD error 1002\n--------------------------------------\n\nDuring inference, the user may encounter an error with details \"[NRTD:infer_wait] error: 1002\":\n\n.. code:: bash\n\n mxnet.base.MXNetError: [11:26:56] src/operator/subgraph/neuron/./neuron_util.h:1175: Check failed: rsp_wait.status().code() == 0 || rsp_wait.status().code() == 1003: Failed\n Infer Wait with Neuron-RTD Error. Neuron-RTD Status Code: 1002, details: \"[NRTD:infer_wait] error: 1002\n \"\n\nRuntime errors are listed in the Neuron Runtime return codes documentation. In particular, 1002 means that some invalid input has been submitted to infer, e.g. missing some of the input tensors, incorrect input tensor sizes. Please examine /var/log/syslog to see imore details on the error. For example, you may see:\n\n.. code::\n\n Oct 30 19:13:39 ip-172-31-93-131 nrtd[1125]: [TDRV:io_queue_prepare_input_nonhugetlb] Unexpected input size, for data00, expected: 2097152, received: 33554432\n\nThis means that the input tensor size is larger than what the model was compiled for (i.e. the example input tensor shapes passed during compilation.\n\n\nMulti-Model Server\n==================\n\n\nFailed to create NEURONCORE Group with GRPC Error. Status Error: 14, Error message: \"Connect Failed\"\n----------------------------------------------------------------------------------------------------\n\nNOTE: This error only applies to MXNet 1.5.\n\nIf the client is unable to start workers and you get a message that MMS is unable to create NeuronCore Group,\nplease check that Neuron RTD is running (neuron-rtd process).\n\n.. code:: json\n\n {\n \"code\": 500,\n \"type\": \"InternalServerException\",\n \"message\": \"Failed to start workers“\n }\n\n.. code:: bash\n\n 2019-10-23 19:56:23,187 [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [19:56:23] src/operator/subgraph/inferentia/./inferentia_util.h:218: Check failed: status.ok() Failed to create NeuronCore Group with GRPC Error. Status Error: 14, Error message: \"Connect Failed\"\n\nMultiple MMS workers die with “Backend worker process die.” message\n-------------------------------------------------------------------\n\n.. important ::\n\n ``NEURONCORE_GROUP_SIZES`` will no longer be supported starting Neuron 1.19.0 release if your application is using ``NEURONCORE_GROUP_SIZES`` please \n see :ref:`neuron-migrating-apps-neuron-to-libnrt` and :ref:`eol-ncgs-env_2` for more details.\n\nIf you run inference with MMS and get multiple messages “Backend worker process die\", please ensure that the number of workers (\"intial_workers\") passed during load model is less than or equal to number of NeuronCores available divided by number of NeuronCores required by model.\n\n.. code:: bash\n\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Backend worker process die.\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File \"/usr/local/lib/python3.6/site-packages/mxnet/symbol/symbol.py\", line 1524, in simple_bind\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ctypes.byref(exe_handle)))\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File \"/usr/local/lib/python3.6/site-packages/mxnet/base.py\", line 252, in check_call\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise MXNetError(py_str(_LIB.MXGetLastError()))\n com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mxnet.base.MXNetError: [00:26:32] src/operator/subgraph/neuron/./neuron_util.h:221: Check failed: 0 == create_eg_rsp.status().code() Failed to create NeuronCore Group with KRTD Error. KRTD Status Code: 4, details: \"\"\n\nAs indicated in :ref:`appnote-performance-tuning`, for greater flexibility user can use NEURONCORE_GROUP_SIZES to specify the groupings of NeuronCores into Neuron devices, each device consisting of one or more NeuronCores. Each worker would take a device. The total number of NeuronCores taken by all the workers should be less than or equal the total number of NeuronCores visible to neuron-rtd. This situation should be considered at full load (MMS scales up to max_workers). Additionally, to properly assign model to Neuron device, the environment NEURONCORE_GROUP_SIZES must be specified within the model server class (ie. mxnet_model_service.py in the example above). For example, add the following line within mxnet_model_service.py for model compiled to 1 NeuronCore:\n\n.. code:: python\n\n os.environ['NEURONCORE_GROUP_SIZES'] = '1'\n\nMore information about max_worker limit setting can be found at `MMS Management API Documentation`_. For example, to run up to 4 workers in inf1.xlarge where 4 NeuronCores are available by default to Neuron-RTD, set max_workers to 4:\n\n.. _MMS Management API Documentation: https://github.com/awslabs/multi-model-server/blob/master/docs/management_api.md#user-content-scale-workers\n\n.. code:: bash\n\n curl -v -X PUT \"http://localhost:8081/models/squeezenet_v1.1_compiled?min_worker=1?max_worker=4\"\n\nMMS throws a \"mxnet.base.MXNetError: array::at\" error\n-----------------------------------------------------\n\nIf you see “mxnet.base.MXNetError: array::at” when running MMS please check that NDArray/Gluon API is not used as they are not supported in MXNet-Neuron.\nIf you would like to use NDArray or Gluon API, please upgrade to MXNet 1.8.\n\n.. code:: bash\n\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - array::at\n [INFO ] W-9000-squeezenet_v1.1_compiled com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 30\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File \"/tmp/models/6606fa046f68a34df87f15362a7a2d9a49749878/model_handler.py\", line 82, in handle\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data = self.inference(data)\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File \"/tmp/models/6606fa046f68a34df87f15362a7a2d9a49749878/mxnet_model_service.py\", line 153, in inference\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - d.wait_to_read()\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File \"/home/user/regression_venv_p3.6/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py\", line 1819, in wait_to_read\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - check_call(_LIB.MXNDArrayWaitToRead(self.handle))\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File \"/home/user/regression_venv_p3.6/lib/python3.6/site-packages/mxnet/base.py\", line 253, in check_call\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise MXNetError(py_str(_LIB.MXGetLastError()))\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mxnet.base.MXNetError: array::at\n [INFO ] W-9000-squeezenet_v1.1_compiled-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Invoking custom service failed.\n\nMXNet Model Server is not able to clean up Neuron RTD states after model is unloaded\n------------------------------------------------------------------------------------\n\nNOTE: This issue is resolved in version 1.5.1.1.1.88.0 released 11/17/2020 and only applies for MXNet 1.5.\n\nMXNet Model Server is not able to clean up Neuron RTD states after model is unloaded (deleted) from model server. Restarting the model server may fail with \"Failed to create NEURONCORE_GROUP\" error:\n\n.. code:: bash\n\n mxnet.base.MXNetError: [00:26:59] src/operator/subgraph/neuron/./neuron_util.h:348: Check failed: 0 == create_eg_rsp.status().code(): Failed to create NEURONCORE_GROUP with Neuron-RTD Error. Neuron-RTD Status Code: 9, details: \"\"\n\nThe workaround is to run “`/opt/aws/neuron/bin/neuron-cli reset`“ to clear Neuron RTD states after all models are unloaded and server is shut down before restarting the model server.\n\nPipeline mode is not able to execute inferences requests in parallel\n--------------------------------------------------------------------\n\nIf you see that multiple executors in a neuron pipeline setup (one model compiled for more than one neuron-cores using `--neuroncore-pipeline-cores` option during compilation) are not running in parallel, please set the following MXNet's environment variables before inference to allow mxnet to execute the CPU ops in parallel. Otherwise it will be sequential and stall the executors.\n\n``MXNET_CPU_WORKER_NTHREADS`` is used to do that. Setting its value to ``__subgraph_opt_neuroncore__`` in the compiled model json will ensure that all the executors (threads) can be run in parallel.\n\n\nFeatures only in MXNet-Neuron 1.5\n---------------------------------\n- Shared memory for IFMaps transfer to neuron runtime (has higher performance compared to GRPC mode)\n- Neuron profiling using MXNet\n\nFeatures only in MXNet-Neuron 1.8\n---------------------------------\n- Gluon API support\n- Library mode neuron runtime\n" }, { "path": "archive/mxnet-neuron/tutorials/mxnet-tutorial-setup.rst", "content": ".. _mxnet-tutorial-setup:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nMXNet Tutorial Setup\n====================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n#. Launch an Inf1.6xlarge Instance:\n .. include:: /setup/install-templates/inf1/launch-inf1-dlami.rst\n\n#. Set up a development environment:\n * Enable or install MXNet-Neuron: :ref:`install-neuron-mxnet`.\n \n\n#. Run tutorial in Jupyter notebook:\n * Follow instruction at :ref:`Setup Jupyter notebook ` to:\n \n #. Start the Jupyter Notebook on the instance\n #. Run the Jupyter Notebook from your local browser\n\n * Connect to the instance from the terminal, clone the Neuron Github repository to the Inf1 instance and then change the working directory to the tutorial directory:\n\n .. code::\n\n git clone https://github.com/aws/aws-neuron-sdk.git\n cd aws-neuron-sdk/src/examples/mxnet\n\n * Locate the tutorial notebook file (.ipynb file) under ``aws-neuron-sdk/src/examples/mxnet``\n * From your local browser, open the tutorial notebook from the menu and follow the instructions.\n" }, { "path": "archive/mxnet-neuron/tutorials/tutorial-model-serving.rst", "content": ".. _mxnet-neuron-model-serving:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTutorial: Neuron Apache MXNet Model Serving\n=============================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThis MXNet Neuron Model Serving (MMS) example is adapted from the MXNet\nvision service example which uses pretrained squeezenet to perform image\nclassification:\nhttps://github.com/awslabs/multi-model-server/tree/master/examples/mxnet_vision.\n\nBefore starting this example, please ensure that Neuron-optimized MXNet\nversion mxnet-neuron is installed along with Neuron Compiler.\n\nWarning\n*******\nIf you are using MXNet-1.5, please note that MXNet-1.5 entered maintenance mode and require Neuron Runtime 1.x, please see :ref:`maintenance_mxnet_1_5`.\nTo setup development environment for MXNet-1.5 see installation instructions at :ref:`mxnet-setup`.\n\nIf using DLAMI, you can activate the environment aws_neuron_mxnet_p36\nand skip the installation part in the first step below.\n\n1. First, install Java runtime and multi-model-server:\n\n.. code:: bash\n\n cd ~/\n # sudo dnf -y install -q jre # for AL2023\n sudo apt-get install -y -q default-jre # for Ubuntu\n pip install multi-model-server\n\nDownload the example code:\n\n.. code:: bash\n\n git clone https://github.com/awslabs/multi-model-server\n cd ~/multi-model-server/examples/mxnet_vision\n\n2. Compile ResNet50 model to Inferentia target by saving the following\n Python script to compile_resnet50.py and run\n “\\ ``python compile_resnet50.py``\\ ”\n\n.. code:: python\n\n\n from packaging import version\n import numpy as np\n import mxnet as mx\n \n mxnet_version = version.parse(mx.__version__)\n if mxnet_version >= version.parse(\"1.8\"):\n import mx_neuron as neuron\n else: \n from mxnet.contrib import neuron\n\n path='http://data.mxnet.io/models/imagenet/'\n mx.test_utils.download(path+'resnet/50-layers/resnet-50-0000.params')\n mx.test_utils.download(path+'resnet/50-layers/resnet-50-symbol.json')\n mx.test_utils.download(path+'synset.txt')\n\n nn_name = \"resnet-50\"\n\n #Load a model\n sym, args, auxs = mx.model.load_checkpoint(nn_name, 0)\n\n #Define compilation parameters\n # - input shape and dtype\n inputs = {'data' : mx.nd.zeros([1,3,224,224], dtype='float32') }\n\n # compile graph to inferentia target\n csym, cargs, cauxs = neuron.compile(sym, args, auxs, inputs)\n\n # save compiled model\n mx.model.save_checkpoint(nn_name + \"_compiled\", 0, csym, cargs, cauxs)\n\n3. Prepare signature file ``signature.json`` to configure the input name\n and shape:\n\n.. code:: json\n\n {\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"data_shape\": [\n 1,\n 3,\n 224,\n 224\n ]\n }\n ]\n }\n\n4. Prepare ``synset.txt`` which is a list of names for ImageNet\n prediction classes:\n\n.. code:: bash\n\n curl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/synset.txt\n\n5. Create custom service class following template in\n model_server_template folder:\n\n.. code:: bash\n\n cp -r ../model_service_template/* .\n\nEdit ``mxnet_model_service.py`` to use the appropriate context. \n\nMake the following change:\n\n.. code:: bash\n\n from packaging import version\n \n mxnet_version = version.parse(mx.__version__)\n if mxnet_version >= version.parse(\"1.8\"):\n import mx_neuron as neuron\n self.mxnet_ctx = mx.neuron()\n\nComment out the existing context set:\n\n.. code:: bash\n\n #self.mxnet_ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n\nAlso, comment out unnecessary data copy for model_input in\n``mxnet_model_service.py``:\n\n.. code:: bash\n\n #model_input = [item.as_in_context(self.mxnet_ctx) for item in model_input]\n\n6. Package the model with model-archiver:\n\n.. code:: bash\n\n cd ~/multi-model-server/examples\n model-archiver --force --model-name resnet-50_compiled --model-path mxnet_vision --handler mxnet_vision_service:handle\n\n7. Start MXNet Model Server (MMS) and load model using RESTful API.\n Please ensure that Neuron RTD is running with default settings (see\n Neuron Runtime Getting Started):\n\n.. code:: bash\n\n cd ~/multi-model-server/\n multi-model-server --start --model-store examples\n # Pipe to log file if you want to keep a log of MMS\n curl -v -X POST \"http://localhost:8081/models?initial_workers=1&max_workers=1&synchronous=true&url=resnet-50_compiled.mar\"\n sleep 10 # allow sufficient time to load model\n\nEach worker requires a NeuronCore group that can accommodate the compiled\nmodel. Additional workers can be added by increasing max_workers\nconfiguration as long as there are enough NeuronCores available. Use\n``neuron-top`` to see which models are loaded on specific NeuronCores.\n\n8. Test inference using an example image:\n\n.. code:: bash\n\n curl -O https://raw.githubusercontent.com/awslabs/multi-model-server/master/docs/images/kitten_small.jpg\n curl -X POST http://127.0.0.1:8080/predictions/resnet-50_compiled -T kitten_small.jpg\n\nYou will see the following output:\n\n.. code:: bash\n\n [\n {\n \"probability\": 0.6375716328620911,\n \"class\": \"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\": 0.1692783385515213,\n \"class\": \"n02123159 tiger cat\"\n },\n {\n \"probability\": 0.12187337130308151,\n \"class\": \"n02124075 Egyptian cat\"\n },\n {\n \"probability\": 0.028840631246566772,\n \"class\": \"n02127052 lynx, catamount\"\n },\n {\n \"probability\": 0.019691042602062225,\n \"class\": \"n02129604 tiger, Panthera tigris\"\n }\n ]\n\n9. To cleanup after test, issue a delete command via RESTful API and\n stop the model server:\n\n.. code:: bash\n\n curl -X DELETE http://127.0.0.1:8081/models/resnet-50_compiled\n\n multi-model-server --stop\n" }, { "path": "archive/mxnet-neuron/tutorials/tutorials-mxnet-computervision.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nComputer Vision Tutorials (``mxnet-neuron``)\n============================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n* ResNet-50 tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n* Model Serving tutorial :ref:`[html] `\n* Getting started with Gluon tutorial :ref:`[html] ` :github:`[notebook] `\n" }, { "path": "archive/mxnet-neuron/tutorials/tutorials-mxnet-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTutorials (``mxnet-neuron``)\n=============================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Computer Vision Tutorials \n Natural Language Processing (NLP) Tutorials \n Utilizing Neuron Capabilities Tutorials \n\n\n.. include:: /archive/mxnet-neuron/tutorials/tutorials-mxnet-neuron.txt\n" }, { "path": "archive/mxnet-neuron/tutorials/tutorials-mxnet-neuron.txt", "content": ".. tab-set::\n\n .. tab-item:: Computer Vision Tutorials\n :name:\n\n * ResNet-50 tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n * Model Serving tutorial :ref:`[html] `\n * Getting started with Gluon tutorial :ref:`[html] ` :github:`[notebook] `\n\n\n .. tab-item:: Natural Language Processing (NLP) Tutorials\n :name:\n\n * MXNet 1.8: Using data parallel mode tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n\n\n .. tab-item:: Utilizing Neuron Capabilities Tutorials\n :name:\n\n * NeuronCore Groups tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n\n\n.. note::\n\n To use Jupyter Notebook see:\n\n * :ref:`setup-jupyter-notebook-steps-troubleshooting`\n * :ref:`running-jupyter-notebook-as-script`" }, { "path": "archive/mxnet-neuron/tutorials/tutorials-mxnet-nlp.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nNatural Language Processing (NLP) Tutorials (``mxnet-neuron``)\n==============================================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n* MXNet 1.8: Using data parallel mode tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n\n" }, { "path": "archive/mxnet-neuron/tutorials/tutorials-mxnet-utilizing-neuron-capabilities.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUtilizing Neuron Capabilities Tutorials (``mxnet-neuron``)\n==========================================================\n\n.. warning::\n\n This document is archived. MXNet is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n* NeuronCore Groups tutorial :ref:`[html] ` :mxnet-neuron-src:`[notebook] `\n\n" }, { "path": "archive/neuronperf/index.rst", "content": ".. _neuronperf:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n=================\nNeuronPerf (Beta)\n=================\n\nNeuronPerf is a lightweight Python library with a simple API that enables fast measurements of performance when running models using Neuron.\n\n.. _neuronperf_quickstart:\n\nNeuronPerf Quickstart\n---------------------\n\nTo install NeuronPerf in your Neuron environment, execute:\n\n.. code:: bash\n\n $ pip install neuronperf --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n\nRefer to the :ref:`neuronperf_examples` and :ref:`neuronperf_user_guide` to get started.\n\n\n.. _neuronperf_user_guide:\n\nNeuronPerf User Guide\n---------------------\n\n.. toctree::\n :maxdepth: 1\n\n Overview \n Terminology \n Examples \n Benchmark Guide \n Evaluate Guide \n Compile Guide \n Model Index Guide \n\n\nNeuronPerf API Reference\n------------------------\n\n.. toctree::\n :maxdepth: 1\n\n API \n Framework Notes \n\n\nFAQ\n---\n\n.. toctree::\n :maxdepth: 1\n\n FAQ \n\n\nTroubleshooting\n---------------\n\n.. toctree::\n :maxdepth: 1\n\n Troubleshooting \n\n\nRelease Notes\n-------------\n\n.. toctree::\n :maxdepth: 1\n\n rn\n\n\n\n\n\n" }, { "path": "archive/neuronperf/neuronperf_api.rst", "content": ".. _neuronperf_api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuronPerf API\n==============\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n.. note::\n Due to a bug in Sphinx, some of the type annotations may be incomplete. \n\n.. py:function:: compile(compile_fn, model, inputs, batch_sizes: Union[int, List[int]] = None, pipeline_sizes: Union[int, List[int]] = None, performance_levels: Union[str, List[int]] = None, models_dir: str = \"models\", filename: str = None, compiler_args: dict = None, verbosity: int = 1, *args, **kwargs) -> str:\n\n Compiles the provided model with each provided example input, pipeline size, and performance level.\n Any additional compiler_args passed will be forwarded to the compiler on every invocation.\n\n :param model: The model to compile.\n :param list inputs: A list of example inputs.\n :param batch_sizes: A list of batch sizes that correspond to the example inputs.\n :param pipeline_sizes: A list of pipeline sizes to use. See :ref:`neuroncore-pipeline`.\n :param performance_levels: A list of performance levels to try. Options are: 0 (max accuracy), 1, 2, 3 (max performance, default). See :ref:`neuron-cc-training-mixed-precision`.\n :param str models_dir: The directory where compilation artifacts will be stored.\n :param str model_name: An optional model name tag to apply to compiled artifacts.\n :param str filename: The name of the model index to write out. If not provided, a name will be generated and returned.\n :param dict compiler_args: Additional compiler arguments to be forwarded with every compilation.\n :param int verbosity: 0 = error, 1 = info, 2 = debug\n :return: A model index filename. If a configuration fails to compile, it will not be included in the index and an error will be logged.\n :rtype: str\n\n.. _neuronperf_api_benchmark:\n\n\n.. py:function:: benchmark(load_fn: Callable[[str, int], Any], model_filename: str, inputs: Any, batch_sizes: Union[int, List[int]] = None, duration: float = BENCHMARK_SECS, n_models: Union[int, List[int]] = None, pipeline_sizes: Union[int, List[int]] = None, cast_modes: Union[str, List[str]] = None, workers_per_model: Union[int, None] = None, env_setup_fn: Callable[[int, Dict], None] = None, setup_fn: Callable[[int, Dict, Any], None] = None, preprocess_fn: Callable[[Any], Any] = None, postprocess_fn: Callable[[Any], Any] = None, dataset_loader_fn: Callable[[Any, int], Any] = None, verbosity: int = 1, multiprocess: bool = True, multiinterpreter: bool = False, return_timers: bool = False, device_type: str = \"neuron\") -> List[Dict]:\n\n Benchmarks the model index or individiual model using the provided inputs.\n If a model index is provided, additional fields such as ``pipeline_sizes`` and\n ``performance_levels`` can be used to filter the models to benchmark. The default\n behavior is to benchmark all configurations in the model index.\n\n :param load_fn: A function that accepts a model filename and device id, and returns a loaded model. This is automatically passed through the subpackage calls (e.g. ``neuronperf.torch.benchmark``).\n :param str model_filename: A path to a model index from compile or path to an individual model. For CPU benchmarking, a class should be passed that can be instantiated with a default constructor (e.g. ``MyModelClass``).\n :param list inputs: A list of example inputs. If the list contains tuples, they will be destructured on inference to support multiple arguments.\n :param batch_sizes: A list of ints indicating batch sizes that correspond to the inputs. Assumes 1 if not provided.\n :param float duration: The number of seconds to benchmark each model.\n :param n_models: The number of models to run in parallel. Default behavior runs 1 model and the max number of models possible, determined by a best effort from ``device_type``, instance size, or other environment state.\n :param pipeline_sizes: A list of pipeline sizes to use. See :ref:`neuroncore-pipeline`.\n :param performance_levels: A list of performance levels to try. Options are: 0 (max accuracy), 1, 2, 3 (max performance, default). See :ref:`neuron-cc-training-mixed-precision`.\n :param workers_per_model: The number of workers to use per model loaded. If ``None``, this is automatically selected.\n :param env_setup_fn: A custom environment setup function to run in each subprocess before model loading. It will receive the benchmarker id and config.\n :param setup_fn: A function that receives the benchmarker id, config, and model to perform last minute configuration before inference.\n :param preprocess_fn: A custom preprocessing function to perform on each input before inference.\n :param postprocess_fn: A custom postprocessing function to perform on each input after inference.\n :param bool multiprocess: When True, model loading is dispatched to forked subprocesses. Should be left alone unless debugging.\n :param bool multiinterpreter: When True, benchmarking is performed in a new python interpreter per model. All parameters must be serializable. Overrides multiprocess.\n :param bool return_timers: When True, the return of this function is a list of tuples ``(config, results)`` with detailed information. This can be converted to reports with ``get_reports(results)``.\n :param float stats_interval: Collection interval (in seconds) for metrics during benchmarking, such as CPU and memory usage.\n :param str device_type: This will be set automatically to one of the ``SUPPORTED_DEVICE_TYPES``.\n :param float cost_per_hour: The price of this device / hour. Used to estimate cost / 1 million infs in reports.\n :param str model_name: A friendly name for the model to use in reports.\n :param str model_class_name: Internal use.\n :param str model_class_file: Internal use.\n :param int verbosity: 0 = error, 1 = info, 2 = debug\n :return: A list of benchmarking results.\n :rtype: list[dict]\n\n\n.. py:function:: get_reports(results)\n\n Summarizes and combines the detailed results from ``neuronperf.benchmark``, when run with ``return_timers=True``. One report dictionary is produced per model configuration benchmarked. The list of reports can be fed directly to other reporting utilities, such as ``neuronperf.write_csv``.\n\n :param list[tuple] results: The list of results from ``neuronperf.benchmark``.\n :param list[int] batch_sizes: The batch sizes that correspond to the `inputs` provided to ``compile`` and ``benchmark``. Used to correct throughput values in the reports.\n :return: A list of dictionaries that summarize the results for each model configuration.\n :rtype: list[dict]\n\n.. py:function:: print_reports(reports, cols=SUMMARY_COLS, sort_by=\"throughput_peak\", reverse=False)\n\n Print a report to the terminal.\n Example of default behavior:\n\n >>> neuronperf.print_reports(reports)\n throughput_avg latency_ms_p50 latency_ms_p99 n_models pipeline_size workers_per_model batch_size model_filename\n 329.667 6.073 6.109 1 1 2 1 models/model_b1_p1_83bh3hhs.pt\n\n :param reports: Results from `get_reports`.\n :param cols: The columns in the report to be displayed.\n :param sort_by: Sort the cols by the specified key.\n :param reverse: Sort order.\n\n.. py:function:: write_csv(reports: list[dict], filename: str = None, cols=REPORT_COLS)\n\n Write benchmarking reports to CSV file.\n\n :param list[dict] reports: Results from `neuronperf.get_reports`.\n :param str filename: Filename to write. If not provided, generated from model_name in report and current timestamp.\n :param list[str] cols: The columns in the report to be kept.\n :return: The filename written.\n :rtype: str\n\n.. py:function:: write_json(reports: list[dict], filename: str = None)\n\n Writes benchmarking reports to a JSON file.\n\n\t:param list[dict] reports: Results from `neuronperf.get_reports`.\n\t:param str filename: Filename to write. If not provided, generated from model_name in report and current timestamp.\n\t:return: The filename written.\n\t:rtype: str\n\n\n.. py:function:: model_index.append(*model_indexes: Union[str, dict]) -> dict:\n\n Appends the model indexes non-destructively into a new model index, without\n modifying any of the internal data.\n\n This is useful if you have benchmarked multiple related models and wish to\n combine their respective model indexes into a single index.\n\n Model name will be taken from the first index provided.\n Duplicate configs will be filtered.\n\n :param model_indexes: Model indexes or paths to model indexes to combine.\n :return: A new dictionary representing the combined model index.\n :rtype: dict\n\n\n.. py:function:: model_index.copy(old_index: Union[str, dict], new_index: str, new_dir: str) -> str:\n\n Copy an index to a new location. Will rename ``old_index``\n to ``new_index`` and copy all model files into ``new_dir``,\n updating the index paths.\n\n This is useful for pulling individual models out of a pool.\n\n Returns the path to the new index.\n\n\n.. py:function:: model_index.create(filename, input_idx=0, batch_size=1, pipeline_size=1, cast_mode=DEFAULT_CAST, compile_s=None)\n\n Create a new model index from a pre-compiled model.\n\n :param str filename: The path to the compiled model.\n :param int input_idx: The index in your inputs that this model should be run on.\n :param int batch_size: The batch size at compilation for this model.\n :param int pipeline_size: The pipeline size used at compilation for this model.\n :param str cast_mode: The casting option this model was compiled with.\n :param float compile_s: Seconds spent compiling.\n :return: A new dictionary representing a model index.\n :rtype: dict\n\n\n.. py:function:: model_index.delete(filename: str):\n\n Deletes the model index and all associated models referenced by the index.\n\n\n.. py:function:: model_index.filter(index: Union[str, dict], **kwargs) -> dict:\n\n Filters provided model index on provided criteria and returns a new index.\n Each kwarg is a standard (k, v) pair, where k is treated as a filter name\n and v may be one or more values used to filter model configs.\n\n\n.. py:function:: model_index.load(filename) -> dict:\n\n Load a NeuronPerf model index from a file.\n\n\n.. py:function:: model_index.move(old_index: str, new_index: str, new_dir: str) -> str:\n\n This is the same as ``copy`` followed by ``delete`` on the old index.\n\n\n.. py:function:: model_index.save(model_index, filename: str = None, root_dir=None) -> str:\n\n Save a NeuronPerf model index to a file.\n\n\n" }, { "path": "archive/neuronperf/neuronperf_benchmark_guide.rst", "content": ".. _neuronperf_benchmark_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n==========================\nNeuronPerf Benchmark Guide\n==========================\n\nThe call to ``neuronperf[torch/tensorflow/mxnet/cpu].benchmark`` is used to measure your model performance. It will choose reasonable defaults if none are provided, and will return back reports that summarize the benchmarking results.\n\nWhat is the default behavior of ``benchmark``?\n----------------------------------------------\n\nThat will depend how you provided your model and how your model was compiled.\n\nThe two most common ways to provide your model are:\n\n#. Provide the path to your compiled model\n#. Provide the path to a model index from ``neuronperf.compile`` (a JSON file)\n\n\nData Parallel\n~~~~~~~~~~~~~\n\nYour model is benchmarked on provided ``inputs`` in 4 different configurations:\n #. A single model on 1 NeuronCore with one worker (min. latency)\n #. A single model on 1 NeuronCore with two workers (max. throughput / NC)\n #. ``MAX`` models on ``MAX`` NeuronCores with one worker (min. latency + max. instance usage)\n #. ``MAX`` models on ``MAX`` NeuronCores with two workers (max. throughput + max. instance usage)\n\nThe value ``MAX`` is automatically determined by your instance size. If it can't be identified, those configurations will be skipped.\n\nThe primary benefit of (3) and (4) is to verify that your model scales well at maximum instance usage.\n\n.. note::\n\n If you provided the path to a model index from ``compile``:\n * Your input parameters to ``benchmark`` (``batch_sizes``, etc.) are treated as filters on the index\n * Each remaining model configuration is benchmarked as described in (1)\n\n\nPipeline\n~~~~~~~~\n\nPipeline mode is active when using a Neuron device and ``pipeline_sizes > 1``. The same behavior as described in Data Parallel applies, except that only one worker configuration is executed: the optimal number of workers for your pipeline size, unless manually overridden.\n\n\nParameters\n----------\n\nBelow are some useful and common parameters to tweak. Please see the :ref:`neuronperf_api` for full details.\n\n* ``n_models`` controls how many models to load. The default behavior is ``n_models=[1, MAX]``.\n* ``workers_per_model`` controls how many worker threads will be feeding inputs to each model. The default is automatically determined.\n* ``pipeline_sizes`` tells the benchmarker how many cores are needed for your model so that each model instance can be loaded properly. Default is 1.\n* ``duration`` controls how long to run each configuration.\n* ``batch_sizes`` is used to inform the benchmarker of your input shape so that throughput can be computed correctly.\n\nAlmost all NeuronPerf behaviors are controllable via arguments found in the :ref:`neuronperf_api`. This guide attempts to provide some context and examples for those arguments.\n\nInputs\n------\n\nModels accept one or more inputs to operate on. Since NeuronPerf needs to support multiple inputs for multiple models, as well as multi-input models, there are some details that may need your attention. See the :ref:`neuronperf_framework_notes` for details.\n\nMulti-input Models\n~~~~~~~~~~~~~~~~~~\n\nIf your model accepts multiple inputs, you must provide them in a ``tuple``. For example, suppose you have a model like this:\n\n.. code:: python\n\n\n\tclass Model(torch.nn.Module):\n\t\tdef forward(self, x, y, z):\n\t\t\t...\n\t\t\treturn output\n\n\nIn order for NeuronPerf to pass along your multiple inputs correctly, you should provide them as a ``tuple``:\n\n.. code:: python\n\n\tinputs = (x, y, z)\n\tnpf.torch.benchmark(model_filename, inputs, ...)\n\nIf you are compiling and/or benchmarking multiple models, you can pass different sized inputs as a list of tuples:\n\n.. code:: python\n\n\tinputs = [(x1, y1, z1), (x2, y2, z2), ...]\n\tnpf.torch.benchmark(model_filename, inputs, ...)\n\n\nPreprocessing and Postprocessing\n--------------------------------\n\nMany models have additional preprocessing and postprocessing steps involved that may add non-negligible overhead to inference time. NeuronPerf supports these use cases through the use of custom functions.\n\nPreprocessing\n~~~~~~~~~~~~~\n\nRecall that NeuronPerf expects (or wraps) each model input into a ``tuple``. These tuples will be unpacked before calling your model.\n\nHere is an example for a model with one input. The example multiples the input by 5 before inference.\n\n.. code:: python\n\n def preprocess_fn(x):\n return x * 5\n\n ...\n\n # Benchmark with custom preprocessing function\n reports = npf.torch.benchmark(\n filename,\n inputs,\n ...,\n preprocess_fn = preprocess_fn,\n )\n\nOr if your model expects multiple inputs:\n\n.. code:: python\n\n def preprocess_fn(x, y, z):\n return x / 255, y / 255, z / 255\n\n ...\n\n # Benchmark with custom preprocessing function\n reports = npf.torch.benchmark(\n filename,\n inputs,\n ...,\n preprocess_fn = preprocess_fn,\n )\n\nPostprocessing\n~~~~~~~~~~~~~~\n\nPostprocessing is almost identical to preprocessing, except that your function will receive whatever the output of your model is, exactly as returned without modification. There are no type guarantees.\n\n.. code:: python\n\n def postprocess_fn(x):\n return x.argmax()\n\n ...\n\n # Benchmark with custom preprocessing function\n reports = npf.torch.benchmark(\n filename,\n inputs,\n ...,\n postprocess_fn = postprocess_fn,\n )\n\nMinimal Latency\n---------------\n\nSuppose you are interested in the minimal latency achievable with your model. In this case, there is no need for more than one worker to execute at a time. We can manually specify the number of workers to use. See below :ref:`neuronperf_worker_threads`.\n\n\n.. _neuronperf_worker_threads:\n\nWorker Threads\n--------------\n\nThe argument ``workers_per_model`` controls the number of worker threads that are trying to prepare and load examples onto a single NeuronCore at a time. Therefore, a value of 1 corresponds to 1 thread / model. If ``n_models=16``, then there would be 16 worker threads, one per model. This number is selected based upon whether you are using DataParallel (i.e. ``pipeline_sizes == 1``), or Pipeline Mode (``pipeline_sizes != 1``).\n\nBy default, NeuronPerf will try to pick try multiple combinations of model copies and workers. You may be interested in controlling this manually.\n\n.. code:: python\n\n reports = npf.torch.benchmark('model_neuron_b1.pt', ..., workers_per_model=1)\n\n\nYou may also pass a list, as with other parameters:\n\n.. code:: python\n\n workers_per_model = [1, 2] # Same as the default for data parallel\n reports = npf.torch.benchmark('model_neuron_b1.pt', ..., workers_per_model=workers_per_model)\n\nWith the default number of :ref:`neuronperf_model_copies`, a call to ``print_results`` might look like this:\n\n.. code:: bash\n\n throughput_avg latency_ms_p50 latency_ms_p99 n_models pipeline_size workers_per_model batch_size model_filename\n 307.25 3.251 3.277 1 1 1 1 models/a5cff386-89ca-4bbf-9087-d0e624c3c604.pt\n 2746.0 5.641 6.82 16 1 1 1 models/a5cff386-89ca-4bbf-9087-d0e624c3c604.pt\n 329.5 6.053 6.108 1 1 2 1 models/a5cff386-89ca-4bbf-9087-d0e624c3c604.pt\n 2809.0 10.246 12.52 16 1 2 1 models/a5cff386-89ca-4bbf-9087-d0e624c3c604.pt\n\n\n.. _neuronperf_model_copies:\n\nModel Copies\n------------\n\nBy default, NeuronPerf will benchmark two settings for ``n_models``:\n 1. A single copy\n 2. The maximum number number of copies for your instance size\n\nYou can override this behavior by passing ``n_models`` to ``benchmark``, as shown below:\n\n.. code:: python\n\n reports = npf.torch.benchmark('model_neuron_b1.pt', ..., n_models=6)\n\nor\n\n.. code:: python\n\n n_models = list(range(1, 10))\n reports = npf.torch.benchmark('model_neuron_b1.pt', ..., n_models=n_models)\n\n.. _neuronperf_pipeline_mode:\n\nPipeline Mode\n-------------\n\nBy default, NeuronPerf will assume you intend to use DataParallel, with two exceptions:\n\n* You compiled your model using NeuronPerf for pipeline mode\n* You constructed a model index that uses pipeline mode\n\nYou can also manually tell NeuronPerf that your model was compiled for pipeline mode. It is similar to how other arguments are passed.\n\n.. code:: python\n\n reports = npf.torch.benchmark('model_neuron_b1.pt', ..., pipeline_sizes=2)\n\nIf you are passing multiple models in an index, then you should pass a list for ``pipeline_sizes``.\n\n.. code:: python\n\n reports = npf.torch.benchmark('model_index.json', ..., pipeline_sizes=[1, 2, 3])\n\n\nDuration\n--------\n\nNeuronPerf will benchmark each configuration specified for 60 seconds by default. You can control the duration by passing ``duration`` (in seconds).\n\n.. code:: python\n\n reports = npf.torch.benchmark('model_index.json', ..., duration=10)\n\n.. warning::\n\n If you make the duration too short, it may expire before all models are loaded and have had time to execute.\n\n\nCustom Datasets (Beta)\n----------------------\n\nCurrently, only PyTorch supports custom datasets, and the interface is subject to change. If you provide a custom dataset, it will be fully executed on each loaded model copy. So if you provide ``n_models=2``, your dataset will be run through twice in parallel.\n\nTo use this API, call ``benchmark`` passing a ``torch.utils.data.Dataset`` to ``inputs``. You can easily create your own ``Dataset`` by implementing the interface, or use one of the available datasets. For example:\n\n.. code:: python\n\n import torchvision\n\n dataset = torchvision.datasets.FashionMNIST(\n root=\"data\",\n train=False,\n download=True,\n transform=ToTensor()\n )\n\n reports = npf.torch.benchmark('model_index.json', inputs=dataset, batch_sizes=[8], preprocess_fn=lambda x: x[0], loop_dataset=False)\n\n.. note::\n\n The ``preprocess_fn`` is required here to extract image input from the ``(image, label)`` tuple generated by dataloader. If the length of dataset is not sufficient to get the runtime performance, one can set ``loop_dataset=True`` to rerun dataset until certain duration. \n\nResults\n-------\n\nViewing and Saving\n~~~~~~~~~~~~~~~~~~\n\nThere are currently three ways to view results.\n\n- ``neuronperf.print_reports(...)``\n - Dump abbrieviated results in your terminal\n- ``neuronperf.write_csv(...)``\n - Store metrics of interest as CSV\n- ``neuronperf.write_json(...)``\n - Store everything as JSON\n\nSee the :ref:`neuronperf_api` for full details.\n\nFull Timing Results\n~~~~~~~~~~~~~~~~~~~\n\nNeuronPerf automatically combines and summarizes the detailed timing information collecting during benchmarking. If you wish to receive everything back yourself, you can use:\n\n.. code:: python\n\n results = npf.torch.benchmark('model_index.json', ..., return_timers=True)\n\nIf you later wish to produce reports the same way that NeuronPerf does internally, you can call:\n\n.. code:: python\n\n reports = npf.get_reports(results)\n\nVerbosity\n---------\n\nVerbosity is an integer, currently one of ``{0, 1, 2}``, where:\n\n* 0 = SILENT\n* 1 = INFO (default)\n* 2 = VERBOSE / DEBUG\n\nExample:\n\n.. code:: python\n\n reports = npf.torch.benchmark(..., n_models=1, duration=5, verbosity=2)\n\n.. code:: bash\n\n DEBUG:neuronperf.benchmarking - Cast mode was not specified, assuming default.\n INFO:neuronperf.benchmarking - Benchmarking 'resnet50.json', ~5 seconds remaining.\n DEBUG:neuronperf.benchmarking - Running model config: {'model_filename': 'models/model_b1_p1_83bh3hhs.pt', 'device_type': 'neuron', 'input_idx': 0, 'batch_size': 1, 'n_models': 1, 'workers_per_model': 2, 'pipeline_size': 1, 'cast_mode': None, 'multiprocess': True, 'multiinterpreter': False, 'start_dts': '20211111-062818', 'duration': '5'}\n DEBUG:neuronperf.benchmarking - Benchmarker 0 started.\n DEBUG:neuronperf.benchmarking - Benchmarker 0, Worker 0 started.\n DEBUG:neuronperf.benchmarking - Benchmarker 0, Worker 1 started.\n DEBUG:neuronperf.benchmarking - Benchmarker 0, Worker 0 finished after 738 inferences.\n DEBUG:neuronperf.benchmarking - Benchmarker 0, Worker 1 finished after 738 inferences.\n DEBUG:neuronperf.benchmarking - Benchmarker 0 finished.\n throughput_avg latency_ms_p50 latency_ms_p99 n_models pipeline_size workers_per_model batch_size model_filename\n 329.667 6.073 6.109 1 1 2 1 models/model_b1_p1_83bh3hhs.pt\n\n\nInternal Process Model\n----------------------\n\nFor each model loaded (see :ref:`neuronperf_model_copies`), a process is spawned. Each process may use multiple threads (see :ref:`neuronperf_worker_threads`). The threads will continue to load examples and keep the hardware busy.\n\nNeuronPerf spawns processes slightly differently between frameworks. For PyTorch and Apache MXNet, processes are forked. For Tensorflow/Keras, a fresh interpreter is launched, and benchmarkers are serialized and run as a script.\n\nIf you suspect you are having trouble due to the way processes are managed, you have two mechanisms of control:\n\n.. code:: python\n\n reports = npf.torch.benchmark(..., multiprocess=False)\n\nDefault is ``True``, and ``False`` will disable multiprocessing and run everything inside a single parent process. This may not work for all frameworks beyond the first model configuration, because process teardown is used to safely deallocate models from the hardware. It is not recommeneded to benchmark this way.\n\n\n.. code:: python\n\n reports = npf.torch.benchmark(..., multiinterpreter=True)\n\nThis flag controls whether a fresh interpreter is used instead of forking. Defaults to ``False`` except with Tensorflow/Keras.\n\n\n.. _npf-cpu-gpu:\n\nBenchmark on CPU or GPU\n-----------------------\n\nWhen benchmarking on CPU or GPU, the API is slightly different. With CPU or GPU, there is no compiled model to benchmark, so instead we need to directly pass a reference to the model class that will be instantiated.\n\n.. note::\n\n GPU benchmarking is currently only available for PyTorch.\n\nCPU:\n\n.. code:: python\n\n cpu_reports = npf.cpu.benchmark(YourModelClass, ...)\n\nGPU:\n\n.. code:: python\n\n gpu_reports = npf.torch.benchmark(YourModelClass, ..., device_type=\"gpu\")\n\n\nYour model class will be instantiated in a subprocess, so there are some things to keep in mind.\n\n* Your model class must be defined at the top level inside a Python module\n * i.e. don't place your model class definition inside a function or other nested scope\n* If your model class has special Python module dependencies, consider importing them inside your class ``__init__``\n* If your model class expects constructor arguments, wrap your class so that it has no constructor arguments\n\n\nExample of a wrapped model class for CPU/GPU benchmarking:\n\n.. code:: python\n\n class ModelWrapper(torch.nn.Module):\n def __init__(self):\n super().__init__()\n from transformers import AutoModelForSequenceClassification\n model_name = \"bert-base-cased\"\n self.bert = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n self.add_module(model_name, self.bert)\n\n def forward(self, *inputs):\n return self.bert(*inputs)\n\n\n reports = npf.torch.benchmark(ModelWrapper, inputs, device_type=\"gpu\")\n" }, { "path": "archive/neuronperf/neuronperf_compile_guide.rst", "content": ".. _neuronperf_compile_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n========================\nNeuronPerf Compile Guide\n========================\n\nIf you wish to compile multiple configurations at once, NeuronPerf provides a simplified and uniform API across frameworks. The output is a model index that tracks the artifacts produces, and can be passed directly to the :ref:`benchmark ` routine for a streamlined end-to-end process. This may be useful if you wish to test multiple configurations of your model on Neuron hardware.\n\nYou can manually specify the model index filename by passing ``filename``, or let NeuronPerf generate one and return it for you. Compiled artifacts will be placed in a local ``models`` directory.\n\nHow does ``compile`` know which instance type to compile for?\n-------------------------------------------------------------\n\nNeuronPerf will assume that the instance type your are currently on is also the compile target. However, you may compile on a non-Neuron instance or choose to target a different instance type. In the case, you can pass ``compiler_target`` to the ``compile`` call.\n\nFor example:\n\n.. code:: python\n\n import neuronperf as npf\n import neuronperf.torch\n\n npf.torch.compile(model, inputs) # compile for current instance type\n npf.torch.compile(model, inputs, compiler_target=\"inf2\") # compile for inf2\n\n\n\nCompiling multiple variants\n---------------------------\n\nIf you provide multiple pipeline sizes, batch sizes, and/or cast modes, NeuronPerf will compile all of them.\n\n.. code:: python\n\n # Select a few batch sizes and pipeline configurations to test\n batch_sizes = [1, 5, 10]\n pipeline_sizes = [1, 2, 4]\n\n # Construct example inputs\n example_inputs = [torch.zeros([batch_size, 3, 224, 224], dtype=torch.float16) for batch_size in batch_sizes]\n\n # Compile all configurations\n index = npf.torch.compile(\n model,\n example_inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n )\n\n\nIf you wished to benchmark specific subsets of configurations, you could compile the specific configurations independently and later combine the results into a single index, as shown below.\n\n.. code:: python\n\n # Compile with pipeline size 1 and vary batch dimension\n batch_index = npf.torch.compile(\n model,\n example_inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=1,\n )\n\n # Compile with batch size 1 and vary pipeline dimension\n pipeline_index = npf.torch.compile(\n model,\n example_inputs[0],\n batch_sizes=1,\n pipeline_sizes=pipeline_sizes,\n )\n\n index = npf.model_index.append(batch_index, pipeline_index)\n npf.model_index.save(index, 'model_index.json')\n\nThe ``compile`` function supports ``batch_sizes``, ``pipeline_sizes``, ``cast_modes``, and custom ``compiler_args``. If there is an error during compilation for a requested configuration, it will be logged and compilation will continue onward without terminating. (This is to support long-running compile jobs with many configurations.)\n\n" }, { "path": "archive/neuronperf/neuronperf_evaluate_guide.rst", "content": ".. _neuronperf_evaluate_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n==========================\nNeuronPerf Evaluate Guide\n==========================\n\nNeuronPerf has a new API for evaluating model accuracy on Neuron hardware. This API is currently only available for PyTorch.\n\nYou can access the API through standard ``benchmark()`` by passing an additional kwarg, ``eval_metrics``.\n\nFor example:\n\n.. code:: python\n\n reports = npf.torch.benchmark(\n model_index_or_path,\n dataset,\n n_models=1,\n workers_per_model=2,\n duration=0,\n eval_metrics=['accuracy', 'precision']\n )\n\n\nIn this example, we fix ``n_models`` and ``n_workers`` because replicating the same model will not impact accuracy. We also set ``duration=0`` to allow benchmarking to run untimed through all dataset examples.\n\nBecause this call can be tedious to type, a convenience function is provided:\n\n.. code:: python\n\n reports = npf.torch.evaluate(model_index_or_path, dataset, metrics=['accuracy', 'precision'])\n\n\n.. note:\n\n Please note that ``eval_metrics`` becomes ``metrics`` when using ``evaluate``.\n\nThe ``dataset`` can be any iterable object that produces ``tuple(*INPUTS, TARGET)``.\n\nIf ``TARGET`` does not appear in the last column for your dataset, you can customize this by passing ``eval_target_col``.\n\nFor example:\n\n.. code:: python\n\n reports = npf.torch.evaluate(model_index_or_path, dataset, metrics='accuracy', eval_target_col=1)\n\n\nYou can list the currently available metrics.\n\n.. code:: python\n\n >>> npf.list_metrics() │·····\n Name Description │·····\n Accuracy (TP + TN) / (TP + TN + FP + FN) │·····\n TruePositiveRate TP / (TP + FN) │·····\n Sensitivity Alias for TruePositiveRate │·····\n Recall Alias for TruePositiveRate │·····\n Hit Rate Alias for TruePositiveRate │·····\n TrueNegativeRate TN / (TN + FP) │·····\n Specificity Alias for TrueNegativeRate │·····\n Selectivity Alias for TrueNegativeRate │·····\n PositivePredictiveValue TP / (TP + FP) │·····\n Precision Alias for PositivePredictiveValue │·····\n NegativePredictiveValue TN / (TN + FN) │·····\n FalseNegativeRate FN / (FN + TP) │·····\n FalsePositiveRate FP / (FP + TN) │·····\n FalseDiscoveryRate FP / (FP + TN) │·····\n FalseOmissionRate FP / (FP + TP) │·····\n PositiveLikelihoodRatio TPR / FPR │·····\n NegativeLikelihoodRatio FNR / TNR │·····\n PrevalenceThreshold sqrt(FPR) / (sqrt(FPR) + sqrt(TPR)) │·····\n ThreatScore TP / (TP + FN + FP) │·····\n F1Score 2TP / (2TP + FN + FP) │·····\n MeanAbsoluteError sum(|y - x|) / n │·····\n MeanSquaredError sum((y - x)^2) / n\n\n\nNew metrics may appear in the list after importing a submodule. For example, ``import neuronperf.torch`` will register a new ``topk`` metric.\n\nCustom Metrics\n--------------\n\nSimple Variants\n===============\n\nIf you wish to register a metric that is a slight tweak of an existing metric with different ``init`` args, you can use ``register_metric_from_existing()``:\n\n.. code:: python\n\n npf.register_metric_from_existing(\"topk\", \"topk_3\", k=3)\n\nThis example registers a new metric ``topk_3`` from existing metric ``topk``, passing ``k=3`` as at ``init`` time.\n\n\nNew Metrics\n===========\n\nYou can register your own metrics using ``register_metric()``.\n\nYou metrics must extend ``BaseEvalMetric``:\n\n.. code:: python\n\n class BaseEvalMetric(ABC):\n \"\"\"\n Abstract base class BaseEvalMetric from which other metrics inherit.\n \"\"\"\n\n @abstractmethod\n def process_record(self, output: Any = None, target: Any = None) -> None:\n \"\"\"Process an individual record and return the result.\"\"\"\n pass\n\n @staticmethod\n def aggregate(metrics: Iterable[\"BaseEvalMetric\"]) -> Any:\n \"\"\"Combine a sequence of metrics into a single result.\"\"\"\n raise NotImplementedError\n\nFor example:\n\n.. code:: python\n\n import neuronperf as npf\n\n class MyCustomMetric(npf.BaseEvalMetric):\n def __init__(self):\n super().__init__()\n self.passing = 0\n self.processed = 0\n\n def process_record(self, outputs, target):\n self.processed += 1\n if outputs == target:\n self.passing += 1\n \n @staticmethod\n def aggregate(metrics):\n passing = 0\n processed = 0\n for metric in metrics:\n passing += metric.passing\n processed += metric.processed\n return passing / processed if processed else 0\n\n\n npf.register_metric(\"MyCustomMetric\", MyCustomMetric)\n\n\n" }, { "path": "archive/neuronperf/neuronperf_examples.rst", "content": ".. _neuronperf_examples:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuronPerf Examples\n===================\n\nThis page walks through several examples of using NeuronPerf, starting with the simplest way---using a compiled model. We will also see how we can use NeuronPerf to perform a hyperparameter search, and manage the artifacts produced, as well as our results.\n\nBenchmark a Compiled Model\n--------------------------\n\nThis example assumes you have already compiled your model for Neuron and saved it to disk.\nYou will need to adapt the batch size, input shape, and filename for your model.\n\n.. code:: python\n\n import torch # or tensorflow, mxnet\n\n import neuronperf as npf\n import neuronperf.torch # or tensorflow, mxnet\n\n # Construct dummy inputs\n batch_sizes = 1\n input_shape = (batch_sizes, 3, 224, 224)\n inputs = torch.ones(input_shape) # or numpy array for TF, MX\n\n # Benchmark and save results\n reports = npf.torch.benchmark(\"your_model_file.pt\", inputs, batch_sizes)\n npf.print_reports(reports)\n npf.write_json(reports)\n\n\n.. code:: bash\n\n INFO:neuronperf.benchmarking - Benchmarking 'your_model_file.pt', ~8.0 minutes remaining.\n throughput_avg latency_ms_p50 latency_ms_p99 n_models pipeline_size workers_per_model batch_size model_filename\n 296766.5 0.003 0.003 1 1 1 1 your_model_file.pt\n 3616109.75 0.005 0.008 24 1 1 1 your_model_file.pt\n 56801.0 0.035 0.04 1 1 2 1 your_model_file.pt\n 3094419.4 0.005 0.051 24 1 2 1 your_model_file.pt\n\n\nLet's suppose you only wish to test two specific configurations. You wish to benchmark 1 model and 1 worker thread, and also with 2 worker threads for 15 seconds each. The call to ``benchmark`` becomes:\n\n.. code:: python\n\n reports = npf.torch.benchmark(filename, inputs, batch_sizes, n_models=1, workers_per_model=[1, 2], duration=15)\n\nYou can also add a custom model name to reports.\n\n.. code:: python\n\n reports = npf.torch.benchmark(..., model_name=\"MyFancyModel\")\n\nSee the :ref:`neuronperf_benchmark_guide` for further details.\n\n\nBenchmark a Model from Source\n-----------------------------\n\nIn this example, we define, compile, and benchmark a simple (dummy) model using PyTorch.\n\nWe'll assume you already have a PyTorch model compiled for Neuron with the filename ``model_neuron_b1.pt``. Furthermore, let's assume the model was traced with a batch size of 1, and has an input shape of (3, 224, 224).\n\n.. literalinclude:: test_simple_pt.py\n :language: python\n :caption: :download:`test_simple_pt.py `\n :linenos:\n\n\n.. code:: bash\n\n (aws_neuron_pytorch_p36) ubuntu@ip-172-31-11-122:~/tmp$ python test_simple_pt.py\n INFO:neuronperf.benchmarking - Benchmarking 'model_neuron_b1.pt', ~8.0 minutes remaining.\n throughput_avg latency_ms_p50 latency_ms_p99 n_models pipeline_size workers_per_model batch_size model_filename\n 296766.5 0.003 0.003 1 1 1 1 model_neuron_b1.pt\n 3616109.75 0.005 0.008 24 1 1 1 model_neuron_b1.pt\n 56801.0 0.035 0.04 1 1 2 1 model_neuron_b1.pt\n 3094419.4 0.005 0.051 24 1 2 1 model_neuron_b1.pt\n\nCompile and Benchmark a Model\n-----------------------------\n\nHere is an end-to-end example of compiling and benchmarking a ResNet-50 model from ``torchvision``.\n\n.. literalinclude:: test_resnet50_pt.py\n :language: python\n :caption: :download:`test_resnet50_pt.py `\n :linenos:\n\n\nBenchmark on CPU or GPU\n-----------------------\n\nWhen benchmarking on CPU or GPU, the API is slightly different. With CPU or GPU, there is no compiled model to benchmark, so instead we need to directly pass a reference to the model class that will be instantiated.\n\n.. note::\n\n GPU benchmarking is currently only available for PyTorch.\n\nCPU:\n\n.. code:: python\n\n cpu_reports = npf.cpu.benchmark(YourModelClass, ...)\n\nGPU:\n\n.. code:: python\n\n gpu_reports = npf.torch.benchmark(YourModelClass, ..., device_type=\"gpu\")\n\n\nPlease refer to :ref:`npf-cpu-gpu` for details and an example of providing your model class.\n" }, { "path": "archive/neuronperf/neuronperf_faq.rst", "content": ".. _neuronperf_faq:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuronPerf FAQ\n==============\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nWhen should I use NeuronPerf?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen you want to measure the highest achievable performance for your model with Neuron.\n\nWhen should I **not** use NeuronPerf?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen measuring end-to-end performance that includes your network serving stack. Instead, your should compare your e2e numbers to those obtained by NeuronPerf to optimize your serving overhead.\n\n\nWhich frameworks does NeuronPerf support?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nSee :ref:`neuronperf_framework_notes`.\n\nWhich Neuron instance types does NeuronPerf support?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nPyTorch and TensorFlow support all instance types.\nMXNet support is limited to inf1.\n\n\nWhat is the secret to obtaining the best numbers?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThere is no secret sauce. NeuronPerf follows best practices.\n\nWhat are the \"best practices\" that NeuronPerf uses?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n- These vary slightly by framework and how your model was compiled\n- For a model compiled for a single NeuronCore (DataParallel):\n\n\t- To maximize throughput, for ``N`` models, use ``2 * N`` worker threads\n\t- To minimize latency, use 1 worker thread per model\n- Use a new Python process for each model to avoid GIL contention\n- Ensure you benchmark long enough for your numbers to stabilize\n- Ignore outliers at the start and end of inference benchmarking\n\n" }, { "path": "archive/neuronperf/neuronperf_framework_notes.rst", "content": ".. _neuronperf_framework_notes:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n==========================\nNeuronPerf Framework Notes\n==========================\n\nPyTorch\n=======\n\n * Requires: ``torch-neuron`` or ``torch-neuronx``\n\t- Versions: 1.7.x, 1.8.x, 1.9.x, 1.10.x, 1.11.x, 1.12.x, 1.13.x\n * Input to ``compile``: ``torch.nn.Module``\n * Model inputs: ``Any``.\n\n\nTensorFlow 1.x\n==============\n\n * Requires: ``tensorflow-neuron``\n \t- Versions: All\n * Input to ``compile``: Path to uncompiled model dir from ``saved_model.simple_save``\n * Model inputs: Tensors must be provided as ``numpy.ndarray``\n\n.. note::\n\n\tAlthough TensorFlow *tensors* must be ``ndarray``, this doesn't stop you from wrapping them inside of data structures that traverse process boundaries safely. For example, you can still pass an input ``dict`` like ``{'input_0': np.zeros((2, 1))}``.\n\nTensorFlow 2.x\n==============\n\n * Requires: ``tensorflow-neuron`` or ``tensorflow-neuronx``\n \t- Versions: All\n * Input to ``compile``: ``tf.keras.Model``\n * Model inputs: Tensors must be provided as ``numpy.ndarray``\n\n.. note::\n\n\tAlthough TensorFlow *tensors* must be ``ndarray``, this doesn't stop you from wrapping them inside of data structures that traverse process boundaries safely. For example, you can still pass an input ``dict`` like ``{'input_0': np.zeros((2, 1))}``.\n\nApache MXNet\n=============\n\n * Requires: ``mxnet-neuron``\n \t- Versions 1.5, 1.8\n * Input to ``compile``: ``tuple(sym, args, aux)``\n * Inputs: Tensors must be provided as ``mxnet.ndarray`` or ``numpy.ndarray``\n" }, { "path": "archive/neuronperf/neuronperf_install.rst", "content": ".. _neuronperf_install:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuronPerf Install\n==================\n\nActivate your Neuron environment, and execute:\n\n.. code:: bash\n\n $ pip install neuronperf --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n" }, { "path": "archive/neuronperf/neuronperf_model_index_guide.rst", "content": ".. _neuronperf_model_index_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n============================\nNeuronPerf Model Index Guide\n============================\n\nA **model index** is a JSON file that tracks information about one or more compiled models. You can generate them using ``compile``, by using the API described here, or you may create them manually in a text editor.\n\nAfter a call to ``compile`` you may notice that you now have a ``models`` directory. You will also spot a new file named something like ``model_83b3raj2.json`` in your local directory, if you didn't provide a ``filename`` yourself.\n\nA model index is not intended to be opaque; you should feel free to open, inspect, and modify it yourself. It contains some information about the artifacts that were compiled. Individual models referenced by the index can be handed to ``benchmark`` directly along with an example input, or you may pass the entire index as in the basic example above. Here is an example index:\n\n.. code:: bash\n\n python3 -m json.tool model_index.json\n\n.. code:: json\n\n {\n \"version\": \"0.0.0.0+0bc220a\",\n \"model_configs\": [\n {\n \"filename\": \"models/model_b1_p1_38793jda.pt\",\n \"input_idx\": 0,\n \"batch_size\": 1,\n \"pipeline_size\": 1,\n \"compile_s\": 5.32\n }\n ]\n }\n\nAn index is useful for keeping track of your compiled artifacts and their parameters. The advantages of using ``neuronperf.[torch/tensorflow/mxnet].compile`` are clearer when we wish to compile multiple variants of our model and benchmark all of them at the same time. All of the model artifacts and the index can be destroyed using ``model_index.delete('model_index.json')``.\n\nBenchmarking\n============\n\nWhen benchmarking with an index, there are some important details to keep in mind. If you originally built the index using a set of inputs, the model index has associated the ``inputs`` with the compiled models by their positional index.\n\nFor example:\n\n.. code:: python\n\n batch_sizes = [1, 2]\n inputs = [torch.zeros((b, 100)) for b in batch_sizes]\n\nHere, ``inputs[0]`` corresponds to batch size 1. Therefore, the model index will contain a reference to input 0 for that model. When you call ``benchmark``, you must pass inputs with the same shape in the same positions as at compile time.\n\n.. note::\n\n It's only necessary that there is an input with the correct shape at``inputs[input_index]``. The example data itself is not important.\n\n\nWorking with Indexes\n--------------------\n\nThe API detail below describes utilities for working with indexes. An ``index`` can be either a loaded index (JSON) or the path to an index (it will be loaded automatically).\n\nCreating\n========\n\n.. code:: python\n\n index = neuronperf.model_index.create('/path/to/model', batch_size=1)\n filename = neuronperf.model_index.save(index)\n\nOnce you have an index, you can pass its path directly to ``benchmark``. You can also pass a custom filename instead:\n\n.. code:: python\n\n index = neuronperf.model_index.create('/path/to/model', batch_size=1)\n neuronperf.model_index.save(index, 'my_index.json')\n\nAppending\n=========\n\nIf **multiple models use the same inputs**, you can append them together. For example, if you have the same batch size with multiple pipeline sizes, the inputs are the same, but the model changes.\n\n.. code:: python\n\n pipeline_sizes = [1, 2, 3, 4]\n indexes = [neuronperf.model_index.create(f'/path/to/model_p{p}', pipeline_size=p, batch_size=5) for p in pipeline_sizes]\n index = neuronperf.model_index.append(*indexes)\n neuronperf.model_index.save(index, 'my_index.json')\n\nFiltering\n=========\n\nYou can construct a new model index that is filtered by some parameter. For example, to get a new index with only batch sizes [1, 2], you could do:\n\n.. code:: python\n\n new_index = neuronperf.model_index.filter(index, batch_sizes=[1, 2])\n\nYou can also benchmark subset of a model index by passing only the subset parameters of interest, but remember to ensure you provide the correct number of inputs for the index (even if some are not used).\n\nFor example, if you an index with models at ``batch_sizes = [1, 2, 3]``, but only wish to benchmark batch size 2:\n\n.. code:: python\n\n batch_sizes = [1, 2, 3]\n inputs = [torch.zeros((b, 100)) for b in batch_sizes]\n reports = neuronperf.torch.benchmark('model_index.json', inputs, batch_sizes=2)\n\nCopying\n=======\n\nYou can copy an index to a new location with ``neuronperf.model_index.copy(index, new_index_name, new_index_dir)``. This is mostly useful in combination with ``filter``/``append``.\n\nDeleting\n========\n\nIf you wish to keep your compiled models, just delete the model index file yourself. If you want to delete your model index and all associated artifacts, use:\n\n.. code:: python\n\n neuronperf.model_index.delete('my_index.json')" }, { "path": "archive/neuronperf/neuronperf_overview.rst", "content": ".. _neuronperf_overview:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\n===================\nNeuronPerf Overview\n===================\n\nNeuronPerf is a lightweight Python library that can help you easily benchmark your models with Neuron hardware.\n\nNeuronPerf supports Neuron releases for PyTorch, Tensorflow, and MXNet. It is used internally by the Neuron team to generate performance benchmarking numbers.\n\nWhen interacting with NeuronPerf, you will typically import the base package along with one of the submodule wrappers, for example:\n\n.. code:: python\n\n\timport neuronperf\n\timport neuronperf.torch\n\nYou may then benchmark and/or compile one or more models with NeuronPerf. For example,\n\n.. code:: python\n\n\treports = neuronperf.torch.benchmark(model, inputs, ...)\n\nThe ``compile`` and ``benchmark`` methods must be accessed through one of the supported framework submodules.\n\nBenchmarking\n============\n\nAll NeuronPerf ``benchmark`` calls require a minimum of two arguments:\n\n\t1. A filename\n\t2. Inputs\n\nThe filename may refer to:\n\n\t1. A Neuron-compiled model (e.g. ``my_model.pt``)\n\t2. A :ref:`Model Index `.\n\nA Model Index is useful for benchmarking more than one model in a single session.\n\nCompiling\n=========\n\nNeuronPerf also provides a standard interface to all Neuron frameworks through the ``compile`` API.\n\n.. code:: python\n\n\tmodel_index = neuronperf.torch.compile(model, inputs, ...)\n\nThis is completely optional. You may use the standard compilation guides for supported frameworks.\n\nNext Steps\n==========\n\nTake a look at the simple :ref:`neuronperf_examples`, :ref:`neuronperf_benchmark_guide`, :ref:`neuronperf_compile_guide`, and :ref:`neuronperf_api`." }, { "path": "archive/neuronperf/neuronperf_terminology.rst", "content": ".. _neuronperf_terminology:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuronPerf Terminology\n======================\n\n * Model Inputs\n - An individual input or ``list`` of inputs\n - Example: ``inputs = [(torch.ones((batch_size, 5))) for batch_size in batch_sizes]``\n - Each input is associated with the ``batch_sizes`` specified, in the same order\n - Each input is fed individually to a corresponding model\n - If an input is provided as a ``tuple``, it will be destructured to ``model(*input)`` to support multiple args\n - See :ref:`neuronperf_framework_notes` for framework-specific requirements\n * Latency\n \t- Time to execute a single ``model(input)``\n \t- Typically measured in milliseconds\n * Model\n \t- Your data model; varies by framework. See :ref:`neuronperf_framework_notes`\n \t- Models may be wrapped by submodules (``torch``, ``tensorflow``, ``mxnet``) as callables\n * Model Index\n \t- A JSON file that tracks compiled model artifacts\n * Model Inputs\n \t- A ``tuple`` of inputs passed to a model, i.e. a single complete example\n \t- Example: ``input = (torch.ones((5, 3, 224, 224)),)``\n * Throughput\n \t- Inferences / second" }, { "path": "archive/neuronperf/neuronperf_troubleshooting.rst", "content": ".. _neuronperf_troubleshooting:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuronPerf Troubleshooting\n==========================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nCompilation issues\n^^^^^^^^^^^^^^^^^^\n\nModel fails to compile\n~~~~~~~~~~~~~~~~~~~~~~\n\nPlease `file a bug `_ with as much information as possible.\n\nBenchmarking Issues\n^^^^^^^^^^^^^^^^^^^\n\nBenchmarking terminates early with errors\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Scroll up and read the output. Most likely causes are:\n - invalid input shapes or\n - not enough memory to load the requested number of model copies on the device. Try passing ``n_models=1`` to ``benchmark`` again to test for memory issues.\n\nOther Issues or Feature Requests\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nPlease file a bug on `Github `_." }, { "path": "archive/neuronperf/rn.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nWhat's New\n==========\n\n.. toctree::\n :maxdepth: 1\n\n /release-notes/components/dev-tools\n" }, { "path": "archive/neuronperf/setup.cfg", "content": "[aliases]\n# Define this so we don't resolve to the wrong setuptools 'test' entrypoint when\n# invoking brazil-build test.\ntest = brazil_test\n" }, { "path": "archive/neuronperf/setup.py", "content": "import collections\nimport os\nimport subprocess\n\nfrom setuptools import find_packages, setup\n\n# Read __version__.py\nversion_py = os.path.join(\"src\", \"neuronperf\", \"__version__.py\")\nwith open(version_py, \"rt\") as fp:\n lines = fp.readlines()\nmeta = collections.OrderedDict()\nfor line in lines:\n key, value = line.split(\"=\")\n meta[key.strip()] = value.strip()[1:-1]\n\n# Extract fields for packaging\nTITLE = meta[\"__title__\"]\nAUTHOR = meta[\"__author__\"]\nDESCRIPTION = meta[\"__description__\"]\nVERSION = os.getenv(\"BRAZIL_PACKAGE_VERSION\", \"0.0.0.0\")\nLICENSE = meta[\"__license__\"]\n\n# Compute release version and write back meta info for consistency.\nGIT_SHA = os.environ.get(\"BRAZIL_PACKAGE_CHANGE_ID\")\nif GIT_SHA:\n GIT_SHA = GIT_SHA.strip()[:9]\nelse:\n # This is probably a local build. Try to attach something meaningful.\n try:\n GIT_SHA = subprocess.check_output([\"git\", \"rev-parse\", \"--short\", \"HEAD\"]).decode().strip()\n except:\n GIT_SHA = \"0\" * 9\nVERSION = \"{}+{}\".format(VERSION.strip(), GIT_SHA)\nmeta[\"__version__\"] = VERSION\nwith open(version_py, \"wt\") as fp:\n for k, v in meta.items():\n fp.write('{} = \"{}\"\\n'.format(k, v))\n\n\nsetup(\n name=TITLE,\n version=VERSION,\n description=DESCRIPTION,\n author=AUTHOR,\n license=LICENSE,\n classifiers=[\n \"Development Status :: 4 - Beta\",\n \"Intended Audience :: Developers\",\n \"Topic :: Scientific/Engineering :: Artificial Intelligence\",\n \"License :: Other/Proprietary License\",\n \"Programming Language :: Python :: 3.6\",\n ],\n keywords=\"aws neuron\",\n packages=find_packages(where=\"src\", exclude=(\"test\",)),\n install_requires=[\"dill==0.3.4\", \"numpy\", \"psutil==5.9.0\"],\n python_requires=\">=3.6\",\n package_dir={\"\": \"src\"},\n data_files=[],\n package_data={\"\": [\"py.typed\"]},\n)\n" }, { "path": "archive/neuronperf/test_resnet50_pt.py", "content": "import torch\nimport torch_neuron\n\nimport neuronperf as npf\nimport neuronperf.torch\n\nfrom torchvision import models\n\n\n# Load a pretrained ResNet50 model\nmodel = models.resnet50(pretrained=True)\n\n# Select a few batch sizes to test\nfilename = 'resnet50.json'\nbatch_sizes = [5, 6, 7]\n\n# Construct example inputs\ninputs = [torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32) for batch_size in batch_sizes]\n\n# Compile\nnpf.torch.compile(\n\tmodel, \n\tinputs, \n\tbatch_sizes=batch_sizes, \n\tfilename=filename,\n)\n\n# Benchmark\nreports = npf.torch.benchmark(filename, inputs)\n\n# View and save results\nnpf.print_reports(reports)\nnpf.write_csv(reports, 'resnet50_results.csv')\nnpf.write_json(reports, 'resnet50_results.json')\n" }, { "path": "archive/neuronperf/test_simple_pt.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf as npf\nimport neuronperf.torch\n\n\n# Define a simple model\nclass Model(torch.nn.Module):\n def forward(self, x):\n x = x * 3\n return x + 1\n\n\n# Instantiate\nmodel = Model()\nmodel.eval()\n\n# Define some inputs\nbatch_sizes = [1]\ninputs = [torch.ones((batch_size, 3, 224, 224)) for batch_size in batch_sizes]\n\n# Compile for Neuron\nmodel_neuron = torch.neuron.trace(model, inputs)\nmodel_neuron.save(\"model_neuron_b1.pt\")\n\n# Benchmark\nreports = npf.torch.benchmark(\"model_neuron_b1.pt\", inputs, batch_sizes)\n\n# View and save results\nnpf.print_reports(reports)\nnpf.write_csv(reports, \"model_neuron_b1.csv\")\n" }, { "path": "archive/src/benchmark/pytorch/bert-base-cased_benchmark.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"bert-base-cased\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = neuronperf.torch.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)" }, { "path": "archive/src/benchmark/pytorch/bert-base-cased_compile.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"bert-base-cased\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n neuronperf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "archive/src/benchmark/pytorch/bert-base-uncased_benchmark.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"bert-base-uncased\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = neuronperf.torch.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)" }, { "path": "archive/src/benchmark/pytorch/bert-base-uncased_compile.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"bert-base-uncased\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n neuronperf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "archive/src/benchmark/pytorch/distilbert-base-uncased-finetuned-sst-2-english_benchmark.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"distilbert-base-uncased-finetuned-sst-2-english\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = neuronperf.torch.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)" }, { "path": "archive/src/benchmark/pytorch/distilbert-base-uncased-finetuned-sst-2-english_compile.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"distilbert-base-uncased-finetuned-sst-2-english\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n neuronperf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "archive/src/benchmark/pytorch/distilbert-base-uncased_benchmark.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"distilbert-base-uncased\"]\nsequence_lengths = [128]\nbatch_sizes = [9]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = neuronperf.torch.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)" }, { "path": "archive/src/benchmark/pytorch/distilbert-base-uncased_compile.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"distilbert-base-uncased\"]\nsequence_lengths = [128]\nbatch_sizes = [9]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n neuronperf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "archive/src/benchmark/pytorch/distilroberta-base_benchmark.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"distilroberta-base\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = neuronperf.torch.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)" }, { "path": "archive/src/benchmark/pytorch/distilroberta-base_compile.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf\nimport neuronperf.torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Add to these lists or change as needed\nmodel_names = [\"distilroberta-base\"]\nsequence_lengths = [128]\nbatch_sizes = [6]\npipeline_sizes = [1]\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence_0 = \"The company HuggingFace is based in New York City\"\n sequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n paraphrase = tokenizer.encode_plus(\n sequence_0,\n sequence_1,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"pt\",\n )\n inputs = (\n torch.cat([paraphrase[\"input_ids\"]] * batch_size, 0),\n torch.cat([paraphrase[\"attention_mask\"]] * batch_size, 0),\n )\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n neuronperf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "archive/src/benchmark/pytorch/hf-google-vit_benchmark.py", "content": "import torch\nimport neuronperf\nimport neuronperf.torch\nimport torch_neuronx\n\nfrom PIL import Image\nimport requests\nfrom transformers import ViTImageProcessor, ViTForImageClassification\n\ndef benchmark(batch_size):\n feature_extractor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')\n model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224', torchscript=True)\n model.eval()\n\n url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n image = Image.open(requests.get(url, stream=True).raw)\n inputs = feature_extractor(images=image, return_tensors=\"pt\")\n inputs = inputs['pixel_values'].repeat([batch_size, 1, 1, 1])\n example = (inputs,)\n\n traced = torch_neuronx.trace(model, example, compiler_args=\"--model-type=transformer\")\n filename = 'model.pt'\n torch.jit.save(traced, filename)\n reports = neuronperf.torch.benchmark(filename, [example], batch_sizes=[batch_size])\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)\n\nif __name__ == '__main__':\n # Use batch_size = 1 for best latency, batch_size = 2 for best throughput\n benchmark(batch_size=2)" }, { "path": "archive/src/benchmark/pytorch/hf-openai-clip_benchmark.py", "content": "import torch\nimport neuronperf\nimport neuronperf.torch\nimport torch_neuronx\nimport os\n\nfrom torchvision.datasets import CIFAR100\nfrom transformers import CLIPProcessor, CLIPModel\n\ndef benchmark(model_name, batch_size):\n # Build the model, preprocessor, and dataset\n cifar100 = CIFAR100(root=os.path.expanduser(\"~/.cache\"), download=True, train=False)\n processor = CLIPProcessor.from_pretrained(model_name)\n model = CLIPModel.from_pretrained(model_name, return_dict=False)\n\n # Prepare a sample input\n image = cifar100[0][0]\n text = []\n for c in cifar100.classes:\n text.append(f'a photo of a {c}')\n\n inputs = processor(text=text, images=image, return_tensors=\"pt\", padding=True)\n image = inputs['pixel_values']\n # (b, c, h, w)\n image = image.repeat(batch_size, 1, 1, 1)\n inputs = (inputs['input_ids'], image)\n\n # Trace the model\n model.eval()\n traced = torch_neuronx.trace(model, inputs, compiler_args='--enable-saturate-infinity')\n filename = 'model.pt'\n torch.jit.save(traced, filename)\n reports = neuronperf.torch.benchmark(filename, [inputs], batch_sizes=[batch_size])\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)\n\nif __name__ == '__main__':\n # Recommended batch sizes for throughput\n # openai/clip-vit-base-patch32: 64\n # openai/clip-vit-large-patch14: 4\n model_name = 'openai/clip-vit-base-patch32'\n batch_size = 64\n benchmark(model_name, batch_size)" }, { "path": "archive/src/benchmark/pytorch/hf_pretrained_wav2vec2_conformer_relpos_benchmark.py", "content": "import torch\nimport torch_neuronx\nfrom datasets import load_dataset\nfrom transformers import Wav2Vec2Processor, Wav2Vec2ConformerForCTC\nimport neuronperf as npf\nimport neuronperf.torch\n\nBATCH_SIZE = 1\ndef benchmark():\n processor = Wav2Vec2Processor.from_pretrained(\"facebook/wav2vec2-conformer-rel-pos-large-960h-ft\")\n model = Wav2Vec2ConformerForCTC.from_pretrained(\"facebook/wav2vec2-conformer-rel-pos-large-960h-ft\")\n model.eval()\n\n # take the first entry in the dataset as our input\n ds = load_dataset(\"patrickvonplaten/librispeech_asr_dummy\", \"clean\", split=\"validation\", trust_remote_code=True)\n inputs = processor(ds[0][\"audio\"][\"array\"], return_tensors=\"pt\", padding=\"longest\", sampling_rate=16_000).input_values\n inputs = inputs.repeat([BATCH_SIZE, 1])\n example = (inputs,)\n\n traced = torch_neuronx.trace(model, example, compiler_args='--model-type=transformer')\n filename = 'model.pt'\n torch.jit.save(traced, filename)\n \n model_neuron = torch.jit.load(filename)\n output = model_neuron(inputs)\n print(f\"output is {output}\")\n\n reports = neuronperf.torch.benchmark(filename, [example], multiprocess=False, batch_sizes=[BATCH_SIZE])\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)\n\nif __name__ == '__main__':\n benchmark()\n" }, { "path": "archive/src/benchmark/pytorch/hf_pretrained_wav2vec2_conformer_rope_benchmark.py", "content": "import torch\nimport torch_neuronx\nfrom datasets import load_dataset\nfrom transformers import Wav2Vec2Processor, Wav2Vec2ConformerForCTC\nimport neuronperf as npf\nimport neuronperf.torch\n\nBATCH_SIZE = 1\ndef benchmark():\n processor = Wav2Vec2Processor.from_pretrained(\"facebook/wav2vec2-conformer-rope-large-960h-ft\")\n model = Wav2Vec2ConformerForCTC.from_pretrained(\"facebook/wav2vec2-conformer-rope-large-960h-ft\")\n model.eval()\n\n # take the first entry in the dataset as our input\n ds = load_dataset(\"patrickvonplaten/librispeech_asr_dummy\", \"clean\", split=\"validation\", trust_remote_code=True)\n inputs = processor(ds[0][\"audio\"][\"array\"], return_tensors=\"pt\", padding=\"longest\", sampling_rate=16_000).input_values\n inputs = inputs.repeat([BATCH_SIZE, 1])\n example = (inputs,)\n\n traced = torch_neuronx.trace(model, example, compiler_args='--model-type=transformer')\n filename = 'model.pt'\n torch.jit.save(traced, filename)\n \n model_neuron = torch.jit.load(filename)\n output = model_neuron(inputs)\n print(f\"output is {output}\")\n\n reports = neuronperf.torch.benchmark(filename, [example], multiprocess=False, batch_sizes=[BATCH_SIZE])\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)\n\nif __name__ == '__main__':\n benchmark()\n" }, { "path": "archive/src/benchmark/pytorch/inf2_benchmark.py", "content": "# primary Script used for inf2 Benchmarking\n\nimport torch\nimport neuronperf\nimport neuronperf.torch\nimport torch_neuronx\nfrom transformers import (\n AutoModel, AutoModelForSequenceClassification # Any other model class respective to the model we want to infer on\n)\n\nclass GPT2Neuron(torch.nn.Module):\n def __init__(self, model) -> None:\n super().__init__()\n self.model = model\n\n def forward(self, input_ids, attention_mask):\n return self.model(input_ids=input_ids, attention_mask=attention_mask, use_cache=False)\n\ndef benchmark(model_name, batch_size, sequence_length):\n model = AutoModel.from_pretrained(model_name, torchscript=True)\n if 'gpt2' in model_name:\n model = GPT2Neuron(model)\n model.eval()\n\n example = (\n torch.zeros(batch_size, sequence_length, dtype=torch.int), # input_ids\n torch.zeros(batch_size, sequence_length, dtype=torch.int), # attention_mask\n )\n\n traced = torch_neuronx.trace(model, example)\n filename = 'model.pt'\n torch.jit.save(traced, filename)\n reports = neuronperf.torch.benchmark(filename, [example])\n # View and save results\n print(\"======== {} ========\".format(filename))\n neuronperf.print_reports(reports)\n neuronperf.write_csv(reports)\n neuronperf.write_json(reports)\n\nif __name__ == '__main__':\n # benchmark(model_name, batch_size, sequence_length)\n # Below are a few examples -\n # benchmark('bert-base-cased', 16, 128)\n # benchmark('bert-base-uncased', 4, 128)\n # benchmark('gpt2', 16, 256)\n" }, { "path": "archive/src/benchmark/pytorch/opt_benchmark.py", "content": "import os\nimport neuronperf as npf\nimport torch\nfrom transformers import AutoTokenizer\n\n\"\"\"\nRun the sample at this link to get the split model state_dict (opt-13b-split):\nhttps://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/transformers-neuronx/inference/facebook-opt-13b-sampling.ipynb\n\nMake sure transformers is installed\n\nChange the variables below for opt30b or opt66b models\n\"\"\"\n\n\nBATCH_SIZE = 2\nTP_DEGREE = 2\nSEQ_LEN = 2048\nTOKENIZER = AutoTokenizer.from_pretrained(\"facebook/opt-13b\")\nMODEL_DIR = \"./opt-13b-split\"\n\n\nclass Wrapper(torch.nn.Module):\n def __init__(self, filename):\n super().__init__()\n from transformers_neuronx.opt.model import OPTForSampling\n self.neuron_model = OPTForSampling.from_pretrained(\n filename, batch_size=BATCH_SIZE, tp_degree=TP_DEGREE, amp=\"f16\"\n )\n self.neuron_model.to_neuron()\n\n def forward(self, *inputs):\n return self.neuron_model.sample(torch.concat(inputs), sequence_length=SEQ_LEN)\n\n# Custom load to let our Wrapper class handle things\ndef load_fn(filename, **kwargs):\n return Wrapper(filename)\n\n# NeuronPerf can't see tp_degree at the moment, so just expose all cores\ndef env_setup_fn(*_):\n del os.environ[\"NEURON_RT_VISIBLE_CORES\"]\n\ndef preprocess_fn(inputs):\n return [TOKENIZER.encode(text, return_tensors=\"pt\") for text in inputs]\n\ndef postprocess_fn(outputs):\n return [TOKENIZER.decode(seq) for seq in outputs]\n\ndef benchmark():\n inputs = [\"Hello, I'm a language model,\"] * BATCH_SIZE\n reports = npf.benchmark(\n load_fn,\n MODEL_DIR,\n [inputs], # treat batch as 1 input and let Wrapper handle batching\n batch_sizes=1, # ^\n n_models=1, # only load 1 copy of model\n max_infers=5,\n max_duration=0, # sampling can take a while, so let's not timeout\n workers_per_model=1, # no bottleneck on model inputs, so 1 is fine\n env_setup_fn=env_setup_fn,\n preprocess_fn=preprocess_fn,\n postprocess_fn=postprocess_fn,\n )\n \n # grab the only report (we only benchmarked 1 config)\n report = reports[0]\n \n # let's update throughput to be tokens / second and add a new record\n new_tokens = sum(SEQ_LEN - len(TOKENIZER.encode(i)) for i in inputs)\n tokens_per_s = round(new_tokens / (report[\"latency_ms_avg\"] / 1000), 2)\n report[\"throughput_avg\"] = report[\"tokens_per_s\"] = tokens_per_s\n \n # display and save results\n npf.print_report(report)\n print(f\"Results saved to: {npf.write_json(report)}\")\n\n\nif __name__ == \"__main__\":\n benchmark()\n" }, { "path": "archive/src/benchmark/pytorch/perceiver-multimodal_benchmark.py", "content": "import base64\nimport os\nimport ssl\nimport re\nfrom urllib import request\nimport time\nimport random\nfrom tqdm import tqdm\nimport numpy as np\nimport math\n\nfrom typing import Optional, Tuple, Union\nfrom transformers import PerceiverForMultimodalAutoencoding\nfrom transformers.modeling_outputs import BaseModelOutputWithCrossAttentions\nfrom transformers.models.perceiver.modeling_perceiver import PerceiverBasicDecoder, PerceiverClassifierOutput\nfrom transformers.models.perceiver.modeling_perceiver import restructure\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E pipeline models.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n\n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector() \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n \nclass MultimodalPerceiverWrapper(nn.Module):\n def __init__(self, perceiver_model, nchunks, image_chunk_size, audio_chunk_size):\n super().__init__()\n self.perceiver_model = perceiver_model\n self.nchunks = nchunks\n self.image_chunk_size = image_chunk_size\n self.audio_chunk_size = audio_chunk_size\n \n def forward(self, inputs: torch.FloatTensor,\n neuron_decoder,\n attention_mask: Optional[torch.FloatTensor] = None,\n head_mask: Optional[torch.FloatTensor] = None,\n output_attentions: Optional[bool] = None,\n output_hidden_states: Optional[bool] = None,\n return_dict: Optional[bool] = None):\n\n\n output_attentions = output_attentions if output_attentions is not None else self.perceiver_model.config.output_attentions\n output_hidden_states = (\n output_hidden_states if output_hidden_states is not None else self.perceiver_model.config.output_hidden_states\n )\n return_dict = return_dict if return_dict is not None else self.perceiver_model.config.use_return_dict\n \n if self.perceiver_model.input_preprocessor is not None:\n inputs, modality_sizes, inputs_without_pos = self.perceiver_model.input_preprocessor(inputs)\n else:\n modality_sizes = None\n inputs_without_pos = None\n if inputs.size()[-1] != self.perceiver_model.config.d_model:\n raise ValueError(\n f\"Last dimension of the inputs: {inputs.size()[-1]} doesn't correspond to config.d_model:\"\n f\" {self.perceiver_model.config.d_model}. Make sure to set config.d_model appropriately.\"\n )\n\n batch_size, seq_length, _ = inputs.size()\n device = inputs.device\n\n # If no attention mask is provided, make them all ones\n if attention_mask is None:\n attention_mask = torch.ones((batch_size, seq_length), device=device)\n # Make the attention mask broadcastable to [batch_size, num_heads, seq_length, seq_length]\n extended_attention_mask = self.perceiver_model.invert_attention_mask(attention_mask)\n\n head_mask = self.perceiver_model.get_head_mask(head_mask, self.perceiver_model.config.num_blocks * self.perceiver_model.config.num_self_attends_per_block)\n embedding_output = self.perceiver_model.embeddings(batch_size=batch_size)\n\n encoder_outputs = self.perceiver_model.encoder(\n embedding_output,\n attention_mask=None,\n head_mask=head_mask,\n inputs=inputs,\n inputs_mask=extended_attention_mask,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n return_dict=return_dict,\n )\n sequence_output = encoder_outputs[0]\n\n logits = None\n reconstruction = {}\n for chunk_idx in tqdm(range(self.nchunks)):\n subsampled_output_points = {\n 'image': torch.arange(\n self.image_chunk_size * chunk_idx, self.image_chunk_size * (chunk_idx + 1)).to(device),\n 'audio': torch.arange(\n self.audio_chunk_size * chunk_idx, self.audio_chunk_size * (chunk_idx + 1)).to(device),\n 'label': None,\n }\n \n logits = neuron_decoder(sequence_output, extended_attention_mask, \n inputs, modality_sizes, inputs_without_pos, subsampled_points=subsampled_output_points)\n\n reconstruction['label'] = logits['label']\n if 'image' not in reconstruction:\n reconstruction['image'] = logits['image']\n reconstruction['audio'] = logits['audio']\n else:\n reconstruction['image'] = torch.cat(\n [reconstruction['image'], logits['image']], dim=1)\n reconstruction['audio'] = torch.cat(\n [reconstruction['audio'], logits['audio']], dim=1)\n \n del logits\n\n return reconstruction\n\ndef custom_model_forward(\n self,\n nchunks,\n image_chunk_size,\n audio_chunk_size,\n neuron_decoder,\n inputs: Optional[torch.Tensor] = None,\n attention_mask: Optional[torch.Tensor] = None,\n head_mask: Optional[torch.Tensor] = None,\n output_attentions: Optional[bool] = None,\n output_hidden_states: Optional[bool] = None,\n return_dict: Optional[bool] = None,\n ) -> Union[Tuple, PerceiverClassifierOutput]:\n\n return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n perceiver_wrapper = MultimodalPerceiverWrapper(self.perceiver, nchunks, image_chunk_size, audio_chunk_size)\n outputs = perceiver_wrapper(\n inputs,\n neuron_decoder,\n attention_mask=attention_mask,\n head_mask=head_mask,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n return_dict=return_dict,\n )\n return outputs\n\n\ndef custom_decoder_query(self, inputs, modality_sizes=None, inputs_without_pos=None, subsampled_points=None):\n if self.position_encoding_type == \"none\": # Queries come from elsewhere\n raise ValueError(\"You cannot construct decoder queries when position_encoding_type is set to none\")\n if subsampled_points is not None:\n # subsampled_points are the indices if the inputs would be flattened\n # however, the inputs aren't flattened, that's why we use unravel_index\n # to get the indices for the unflattened array\n # unravel_index returns a tuple (x_idx, y_idx, ...)\n # stack to get the [n, d] tensor of coordinates\n\n def unravel_indices(indices, shape):\n coord = []\n\n for dim in reversed(shape):\n coord.append(indices % dim)\n indices = indices // dim\n\n coord = torch.stack(coord[::-1], dim=-1)\n\n return coord\n\n pos = unravel_indices(subsampled_points, self.output_index_dims)\n\n batch_size = inputs.shape[0]\n # Map these coordinates to [-1, 1]\n pos = -1 + 2 * pos / torch.tensor(self.output_index_dims)[None, :]\n pos = torch.broadcast_to(pos[None], [batch_size, pos.shape[0], pos.shape[1]])\n # Construct the position encoding.\n if self.position_encoding_type == \"trainable\":\n pos_emb = self.output_position_encodings(batch_size)\n elif self.position_encoding_type == \"fourier\":\n pos_emb = self.output_position_encodings(\n self.output_index_dims, batch_size=batch_size, device=inputs.device, dtype=inputs.dtype, pos=pos\n )\n\n # Optionally project them to a target dimension.\n pos_emb = self.positions_projection(pos_emb)\n pos_emb = torch.reshape(pos_emb, [pos_emb.shape[0], -1, pos_emb.shape[-1]])\n else:\n batch_size = inputs.shape[0]\n index_dims = inputs.shape[2:]\n\n # Construct the position encoding.\n if self.position_encoding_type == \"trainable\":\n pos_emb = self.output_position_encodings(batch_size)\n elif self.position_encoding_type == \"fourier\":\n pos_emb = self.output_position_encodings(\n index_dims, batch_size, device=inputs.device, dtype=inputs.dtype\n )\n\n # Optionally project them to a target dimension.\n pos_emb = self.positions_projection(pos_emb)\n\n if self.concat_preprocessed_input:\n if inputs_without_pos is None:\n raise ValueError(\"Value is required for inputs_without_pos if concat_preprocessed_input is True\")\n pos_emb = torch.cat([inputs_without_pos, pos_emb], dim=-1)\n\n return pos_emb\n\n\n# Define wrapper for tracing encoder\nclass EncoderWrapper(nn.Module):\n def __init__(self, encoder):\n super().__init__()\n self.encoder = encoder\n \n def forward(self, embedding_output, inputs, extended_attention_mask):\n output = self.encoder(embedding_output, inputs=inputs, inputs_mask=extended_attention_mask)\n return output\n\nclass NeuronEncoder(nn.Module):\n def __init__(self, encoder_wrapper):\n super().__init__()\n self.encoder_wrapper = encoder_wrapper\n \n def forward(self,\n hidden_states: torch.Tensor,\n attention_mask: Optional[torch.FloatTensor] = None,\n head_mask: Optional[torch.FloatTensor] = None,\n inputs: Optional[torch.FloatTensor] = None,\n inputs_mask: Optional[torch.FloatTensor] = None,\n output_attentions: Optional[bool] = False,\n output_hidden_states: Optional[bool] = False,\n return_dict: Optional[bool] = True):\n\n last_hidden_states = self.encoder_wrapper(hidden_states, inputs, inputs_mask)['last_hidden_state']\n return BaseModelOutputWithCrossAttentions(last_hidden_state=last_hidden_states)\n\n\n# Define wrapper for tracing decoder\nclass DecoderWrapper(nn.Module):\n def __init__(self, decoder, decoder_query_audio, decoder_query_image, decoder_query_label, output_postprocessor):\n super().__init__()\n self.decoder = decoder\n self.decoder_query_audio = decoder_query_audio\n self.decoder_query_image = decoder_query_image\n self.decoder_query_label = decoder_query_label\n self.output_postprocessor = output_postprocessor\n self.num_query_channels = decoder.num_query_channels\n \n def forward(self, z, query_mask,\n audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding,\n image_input, image_input_without_pos, image_subsampled_point, image_padding,\n label_input, label_input_without_pos, label_padding):\n audio_query = self.decoder_query_audio(inputs=audio_input, inputs_without_pos=audio_input_without_pos, subsampled_points=audio_subsampled_point)\n image_query = self.decoder_query_image(inputs=image_input, inputs_without_pos=image_input_without_pos, subsampled_points=image_subsampled_point)\n label_query = self.decoder_query_label(inputs=label_input, inputs_without_pos=label_input_without_pos)\n\n def embed(x, pos):\n x = torch.reshape(x, [x.shape[0], np.prod(x.shape[1:-1]), x.shape[-1]])\n pos = torch.broadcast_to(pos, [x.shape[0], x.shape[1], self.num_query_channels - x.shape[2]])\n return torch.cat([x, pos], dim=2)\n\n audio_padded = embed(audio_query, audio_padding)\n image_padded = embed(image_query, image_padding)\n label_padded = embed(label_query, label_padding)\n\n decoder_query = torch.cat([audio_padded, image_padded, label_padded], dim=1)\n logits = self.decoder(decoder_query, z, query_mask).logits\n \n output_modality_sizes = {\"audio\": audio_subsampled_point.shape[0],\n \"image\": image_subsampled_point.shape[0],\n \"label\": 1}\n logits = self.output_postprocessor(logits, modality_sizes=output_modality_sizes)\n return logits\n\nclass NeuronDecoder(nn.Module):\n def __init__(self, decoder_wrapper):\n super().__init__()\n self.decoder_wrapper = decoder_wrapper\n self.modalities = decoder_wrapper.decoder.modalities\n self.padding = decoder_wrapper.decoder.padding\n\n def forward(self, z, query_mask, inputs, modality_sizes, inputs_without_pos=None, subsampled_points=None, output_attentions=False):\n # Partition the flat inputs among the different modalities\n inputs = restructure(modality_sizes, inputs)\n\n assert(subsampled_points is not None)\n assert(inputs_without_pos is not None)\n\n for modality, decoder in self.modalities.items():\n if modality == \"audio\":\n audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding = inputs[modality], inputs_without_pos[modality], subsampled_points[modality].to(torch.float32), self.padding[modality]\n elif modality == \"image\":\n image_input, image_input_without_pos, image_subsampled_point, image_padding = inputs[modality], inputs_without_pos[modality], subsampled_points[modality].to(torch.float32), self.padding[modality]\n else:\n # label doesn't have subsampled point\n label_input, label_input_without_pos, label_padding = inputs[modality], inputs_without_pos[modality], self.padding[modality]\n\n assert(audio_input_without_pos is not None)\n assert(audio_subsampled_point is not None)\n assert(image_input_without_pos is not None)\n assert(image_subsampled_point is not None)\n assert(label_input_without_pos is not None)\n\n output = self.decoder_wrapper(z, query_mask, \n audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding,\n image_input, image_input_without_pos, image_subsampled_point, image_padding,\n label_input, label_input_without_pos, label_padding)\n return output\n\n\n# -- Load compiled models --\nmodel = PerceiverForMultimodalAutoencoding.from_pretrained(\"deepmind/multimodal-perceiver\", \n low_cpu_mem_usage=True)\n\nPerceiverForMultimodalAutoencoding.forward = custom_model_forward\nPerceiverBasicDecoder.decoder_query = custom_decoder_query\n\nCOMPILER_WORKDIR_ROOT=\"perceiver_multimodal_compile_dir\"\nCOMPILER_WORKDIR_DECODER = os.path.join(COMPILER_WORKDIR_ROOT, \"decoder\")\nCOMPILER_WORKDIR_ENCODER = os.path.join(COMPILER_WORKDIR_ROOT, \"encoder\")\n\n\n# load saved encoder from disk\nencoder_fname = os.path.join(COMPILER_WORKDIR_ENCODER, 'model.pt')\nneuron_encoder = NeuronEncoder(EncoderWrapper(model.perceiver.encoder))\nneuron_encoder.encoder_wrapper = torch.jit.load(encoder_fname)\nmodel.perceiver.encoder = neuron_encoder\n\n# load saved decoder from disk\ndecoder_fname = os.path.join(COMPILER_WORKDIR_DECODER, 'model.pt')\nneuron_decoder = NeuronDecoder(DecoderWrapper(model.perceiver.decoder, model.perceiver.decoder.modalities['audio'].decoder_query, \\\n model.perceiver.decoder.modalities['image'].decoder_query, model.perceiver.decoder.modalities['label'].decoder_query, \\\n model.perceiver.output_postprocessor))\nneuron_decoder.decoder_wrapper = torch.jit.load(decoder_fname)\n\n\n\n# Inference function\ndef autoencode_video(images, audio, nchunks, image_chunk_size, audio_chunk_size):\n input_image = torch.from_numpy(np.moveaxis(images, -1, 2)).to(torch.float32)\n input_audio = torch.from_numpy(audio).to(torch.float32)\n input_label = torch.zeros((images.shape[0], 700))\n\n inputs = {'image': input_image, 'audio': input_audio, 'label':input_label}\n\n reconstruction = {}\n with torch.no_grad():\n reconstruction = model(nchunks, image_chunk_size, audio_chunk_size, neuron_decoder, inputs=inputs)\n\n # reshape image and audio modalities back to original shape\n reconstruction['image'] = torch.reshape(reconstruction['image'], images.shape)\n reconstruction['audio'] = torch.reshape(reconstruction['audio'], audio.shape)\n return reconstruction\n\n# Generate random image for benchmarking\nAUDIO_SAMPLES_PER_PATCH = 16\nimage = np.random.random(size=(1, 16, 224, 224, 3))\naudio = np.random.random(size=(1, 30720, 1))\n\nnchunks = 128\nimage_chunk_size = np.prod(image.shape[1:-1]) // nchunks\naudio_chunk_size = audio.shape[1] // AUDIO_SAMPLES_PER_PATCH // nchunks\n\nn_runs = 20\nmodel_inputs = (image, audio, nchunks, image_chunk_size, audio_chunk_size)\nbenchmark(n_runs, \"perceiver-multimodal\", autoencode_video, model_inputs)" }, { "path": "archive/src/benchmark/pytorch/perceiver-multimodal_compile.py", "content": "import base64\nimport os\nimport ssl\nimport re\nfrom urllib import request\nimport time\nimport random\nfrom tqdm import tqdm\nimport numpy as np\n\nfrom typing import Optional, Tuple, Union\nfrom transformers import PerceiverForMultimodalAutoencoding\nfrom transformers.modeling_outputs import BaseModelOutputWithCrossAttentions\nfrom transformers.models.perceiver.modeling_perceiver import PerceiverBasicDecoder, PerceiverClassifierOutput\nfrom transformers.models.perceiver.modeling_perceiver import restructure\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nclass MultimodalPerceiverWrapper(nn.Module):\n def __init__(self, perceiver_model, nchunks, image_chunk_size, audio_chunk_size):\n super().__init__()\n self.perceiver_model = perceiver_model\n self.nchunks = nchunks\n self.image_chunk_size = image_chunk_size\n self.audio_chunk_size = audio_chunk_size\n \n def forward(self, inputs: torch.FloatTensor,\n neuron_decoder,\n attention_mask: Optional[torch.FloatTensor] = None,\n head_mask: Optional[torch.FloatTensor] = None,\n output_attentions: Optional[bool] = None,\n output_hidden_states: Optional[bool] = None,\n return_dict: Optional[bool] = None):\n\n\n output_attentions = output_attentions if output_attentions is not None else self.perceiver_model.config.output_attentions\n output_hidden_states = (\n output_hidden_states if output_hidden_states is not None else self.perceiver_model.config.output_hidden_states\n )\n return_dict = return_dict if return_dict is not None else self.perceiver_model.config.use_return_dict\n \n if self.perceiver_model.input_preprocessor is not None:\n inputs, modality_sizes, inputs_without_pos = self.perceiver_model.input_preprocessor(inputs)\n else:\n modality_sizes = None\n inputs_without_pos = None\n if inputs.size()[-1] != self.perceiver_model.config.d_model:\n raise ValueError(\n f\"Last dimension of the inputs: {inputs.size()[-1]} doesn't correspond to config.d_model:\"\n f\" {self.perceiver_model.config.d_model}. Make sure to set config.d_model appropriately.\"\n )\n\n batch_size, seq_length, _ = inputs.size()\n device = inputs.device\n\n # If no attention mask is provided, make them all ones\n if attention_mask is None:\n attention_mask = torch.ones((batch_size, seq_length), device=device)\n # Make the attention mask broadcastable to [batch_size, num_heads, seq_length, seq_length]\n extended_attention_mask = self.perceiver_model.invert_attention_mask(attention_mask)\n\n head_mask = self.perceiver_model.get_head_mask(head_mask, self.perceiver_model.config.num_blocks * self.perceiver_model.config.num_self_attends_per_block)\n embedding_output = self.perceiver_model.embeddings(batch_size=batch_size)\n\n encoder_outputs = self.perceiver_model.encoder(\n embedding_output,\n attention_mask=None,\n head_mask=head_mask,\n inputs=inputs,\n inputs_mask=extended_attention_mask,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n return_dict=return_dict,\n )\n sequence_output = encoder_outputs[0]\n\n logits = None\n reconstruction = {}\n for chunk_idx in tqdm(range(self.nchunks)):\n subsampled_output_points = {\n 'image': torch.arange(\n self.image_chunk_size * chunk_idx, self.image_chunk_size * (chunk_idx + 1)).to(device),\n 'audio': torch.arange(\n self.audio_chunk_size * chunk_idx, self.audio_chunk_size * (chunk_idx + 1)).to(device),\n 'label': None,\n }\n \n logits = neuron_decoder(sequence_output, extended_attention_mask, \n inputs, modality_sizes, inputs_without_pos, subsampled_points=subsampled_output_points)\n\n reconstruction['label'] = logits['label']\n if 'image' not in reconstruction:\n reconstruction['image'] = logits['image']\n reconstruction['audio'] = logits['audio']\n else:\n reconstruction['image'] = torch.cat(\n [reconstruction['image'], logits['image']], dim=1)\n reconstruction['audio'] = torch.cat(\n [reconstruction['audio'], logits['audio']], dim=1)\n \n del logits\n\n return reconstruction\n\ndef custom_model_forward(\n self,\n nchunks,\n image_chunk_size,\n audio_chunk_size,\n neuron_decoder,\n inputs: Optional[torch.Tensor] = None,\n attention_mask: Optional[torch.Tensor] = None,\n head_mask: Optional[torch.Tensor] = None,\n output_attentions: Optional[bool] = None,\n output_hidden_states: Optional[bool] = None,\n return_dict: Optional[bool] = None,\n ) -> Union[Tuple, PerceiverClassifierOutput]:\n\n return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n perceiver_wrapper = MultimodalPerceiverWrapper(self.perceiver, nchunks, image_chunk_size, audio_chunk_size)\n outputs = perceiver_wrapper(\n inputs,\n neuron_decoder,\n attention_mask=attention_mask,\n head_mask=head_mask,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n return_dict=return_dict,\n )\n return outputs\n\n\ndef custom_decoder_query(self, inputs, modality_sizes=None, inputs_without_pos=None, subsampled_points=None):\n if self.position_encoding_type == \"none\": # Queries come from elsewhere\n raise ValueError(\"You cannot construct decoder queries when position_encoding_type is set to none\")\n if subsampled_points is not None:\n # subsampled_points are the indices if the inputs would be flattened\n # however, the inputs aren't flattened, that's why we use unravel_index\n # to get the indices for the unflattened array\n # unravel_index returns a tuple (x_idx, y_idx, ...)\n # stack to get the [n, d] tensor of coordinates\n\n def unravel_indices(indices, shape):\n coord = []\n\n for dim in reversed(shape):\n coord.append(indices % dim)\n indices = indices // dim\n\n coord = torch.stack(coord[::-1], dim=-1)\n\n return coord\n\n pos = unravel_indices(subsampled_points, self.output_index_dims)\n\n batch_size = inputs.shape[0]\n # Map these coordinates to [-1, 1]\n pos = -1 + 2 * pos / torch.tensor(self.output_index_dims)[None, :]\n pos = torch.broadcast_to(pos[None], [batch_size, pos.shape[0], pos.shape[1]])\n # Construct the position encoding.\n if self.position_encoding_type == \"trainable\":\n pos_emb = self.output_position_encodings(batch_size)\n elif self.position_encoding_type == \"fourier\":\n pos_emb = self.output_position_encodings(\n self.output_index_dims, batch_size=batch_size, device=inputs.device, dtype=inputs.dtype, pos=pos\n )\n\n # Optionally project them to a target dimension.\n pos_emb = self.positions_projection(pos_emb)\n pos_emb = torch.reshape(pos_emb, [pos_emb.shape[0], -1, pos_emb.shape[-1]])\n else:\n batch_size = inputs.shape[0]\n index_dims = inputs.shape[2:]\n\n # Construct the position encoding.\n if self.position_encoding_type == \"trainable\":\n pos_emb = self.output_position_encodings(batch_size)\n elif self.position_encoding_type == \"fourier\":\n pos_emb = self.output_position_encodings(\n index_dims, batch_size, device=inputs.device, dtype=inputs.dtype\n )\n\n # Optionally project them to a target dimension.\n pos_emb = self.positions_projection(pos_emb)\n\n if self.concat_preprocessed_input:\n if inputs_without_pos is None:\n raise ValueError(\"Value is required for inputs_without_pos if concat_preprocessed_input is True\")\n pos_emb = torch.cat([inputs_without_pos, pos_emb], dim=-1)\n\n return pos_emb\n\n\n# Define wrapper for tracing encoder\nclass EncoderWrapper(nn.Module):\n def __init__(self, encoder):\n super().__init__()\n self.encoder = encoder\n \n def forward(self, embedding_output, inputs, extended_attention_mask):\n output = self.encoder(embedding_output, inputs=inputs, inputs_mask=extended_attention_mask)\n return output\n\nclass NeuronEncoder(nn.Module):\n def __init__(self, encoder_wrapper):\n super().__init__()\n self.encoder_wrapper = encoder_wrapper\n \n def forward(self,\n hidden_states: torch.Tensor,\n attention_mask: Optional[torch.FloatTensor] = None,\n head_mask: Optional[torch.FloatTensor] = None,\n inputs: Optional[torch.FloatTensor] = None,\n inputs_mask: Optional[torch.FloatTensor] = None,\n output_attentions: Optional[bool] = False,\n output_hidden_states: Optional[bool] = False,\n return_dict: Optional[bool] = True):\n\n last_hidden_states = self.encoder_wrapper(hidden_states, inputs, inputs_mask)['last_hidden_state']\n return BaseModelOutputWithCrossAttentions(last_hidden_state=last_hidden_states)\n\n\n# Define wrapper for tracing decoder\nclass DecoderWrapper(nn.Module):\n def __init__(self, decoder, decoder_query_audio, decoder_query_image, decoder_query_label, output_postprocessor):\n super().__init__()\n self.decoder = decoder\n self.decoder_query_audio = decoder_query_audio\n self.decoder_query_image = decoder_query_image\n self.decoder_query_label = decoder_query_label\n self.output_postprocessor = output_postprocessor\n self.num_query_channels = decoder.num_query_channels\n \n def forward(self, z, query_mask,\n audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding,\n image_input, image_input_without_pos, image_subsampled_point, image_padding,\n label_input, label_input_without_pos, label_padding):\n audio_query = self.decoder_query_audio(inputs=audio_input, inputs_without_pos=audio_input_without_pos, subsampled_points=audio_subsampled_point)\n image_query = self.decoder_query_image(inputs=image_input, inputs_without_pos=image_input_without_pos, subsampled_points=image_subsampled_point)\n label_query = self.decoder_query_label(inputs=label_input, inputs_without_pos=label_input_without_pos)\n\n def embed(x, pos):\n x = torch.reshape(x, [x.shape[0], np.prod(x.shape[1:-1]), x.shape[-1]])\n pos = torch.broadcast_to(pos, [x.shape[0], x.shape[1], self.num_query_channels - x.shape[2]])\n return torch.cat([x, pos], dim=2)\n\n audio_padded = embed(audio_query, audio_padding)\n image_padded = embed(image_query, image_padding)\n label_padded = embed(label_query, label_padding)\n\n decoder_query = torch.cat([audio_padded, image_padded, label_padded], dim=1)\n logits = self.decoder(decoder_query, z, query_mask).logits\n \n output_modality_sizes = {\"audio\": audio_subsampled_point.shape[0],\n \"image\": image_subsampled_point.shape[0],\n \"label\": 1}\n logits = self.output_postprocessor(logits, modality_sizes=output_modality_sizes)\n return logits\n\nclass NeuronDecoder(nn.Module):\n def __init__(self, decoder_wrapper):\n super().__init__()\n self.decoder_wrapper = decoder_wrapper\n self.modalities = decoder_wrapper.decoder.modalities\n self.padding = decoder_wrapper.decoder.padding\n\n def forward(self, z, query_mask, inputs, modality_sizes, inputs_without_pos=None, subsampled_points=None, output_attentions=False):\n # Partition the flat inputs among the different modalities\n inputs = restructure(modality_sizes, inputs)\n\n assert(subsampled_points is not None)\n assert(inputs_without_pos is not None)\n\n for modality, decoder in self.modalities.items():\n if modality == \"audio\":\n audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding = inputs[modality], inputs_without_pos[modality], subsampled_points[modality].to(torch.float32), self.padding[modality]\n elif modality == \"image\":\n image_input, image_input_without_pos, image_subsampled_point, image_padding = inputs[modality], inputs_without_pos[modality], subsampled_points[modality].to(torch.float32), self.padding[modality]\n else:\n # label doesn't have subsampled point\n label_input, label_input_without_pos, label_padding = inputs[modality], inputs_without_pos[modality], self.padding[modality]\n\n assert(audio_input_without_pos is not None)\n assert(audio_subsampled_point is not None)\n assert(image_input_without_pos is not None)\n assert(image_subsampled_point is not None)\n assert(label_input_without_pos is not None)\n\n output = self.decoder_wrapper(z, query_mask, \n audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding,\n image_input, image_input_without_pos, image_subsampled_point, image_padding,\n label_input, label_input_without_pos, label_padding)\n return output\n\n\nmodel = PerceiverForMultimodalAutoencoding.from_pretrained(\"deepmind/multimodal-perceiver\", \n low_cpu_mem_usage=True)\nCOMPILER_WORKDIR_ROOT=\"perceiver_multimodal_compile_dir\"\n\nPerceiverForMultimodalAutoencoding.forward = custom_model_forward\nPerceiverBasicDecoder.decoder_query = custom_decoder_query\n\n\n# --- Compile Encoder ---\n# Define sample inputs for tracing encoder\nembedding_output = torch.randn(1, 784, 512)\nsample_inputs = torch.randn(1, 52097, 704)\nextended_attention_mask = torch.zeros(1, 1, 1, 52097)\n\n# Wrap and trace the encoder, save the traced encoder\nCOMPILER_WORKDIR_ENCODER = os.path.join(COMPILER_WORKDIR_ROOT, \"encoder\")\nneuron_encoder = NeuronEncoder(EncoderWrapper(model.perceiver.encoder))\n\n# You might see a warning from trace about unused input - these are safe to ignore.\nprint(\"Compiling Encoder...\")\nneuron_encoder.encoder_wrapper = torch_neuronx.trace(\n neuron_encoder.encoder_wrapper,\n (embedding_output, sample_inputs, extended_attention_mask),\n compiler_workdir=COMPILER_WORKDIR_ENCODER,\n compiler_args=[f\"--temp-dir={COMPILER_WORKDIR_ENCODER}\", \"--auto-cast=none\"] # --auto-cast=none is needed to avoid numerical error.\n)\n\n# Save compiled encoder\nencoder_fname = os.path.join(COMPILER_WORKDIR_ENCODER, 'model.pt')\ntorch.jit.save(neuron_encoder.encoder_wrapper, encoder_fname)\n\n\n# --- Compile Decoder ---\n# Define sample inputs for tracing decoder\nz = torch.randn(1, 784, 512)\nquery_mask = torch.zeros(1, 1, 1, 52097)\n\naudio_input = torch.randn(1, 1920, 704)\naudio_input_without_pos = torch.randn(1, 1920, 16)\naudio_subsampled_point = torch.arange(0, 15, dtype=torch.float32) # 15 = 1920/128\naudio_padding = torch.randn(1, 641)\n\nimage_input = torch.randn(1, 50176, 704)\nimage_input_without_pos = torch.randn(1, 50176, 48)\nimage_subsampled_point = torch.arange(0, 6272, dtype=torch.float32) # 6272 = 224*224*16/128\nimage_padding = torch.randn(1, 831)\n\nlabel_input = torch.randn(1, 1, 704)\nlabel_input_without_pos = torch.randn(1, 1, 700)\nlabel_padding = torch.randn(1, 2)\n\n# Wrap and trace the decoder, save the traced decoder\nCOMPILER_WORKDIR_DECODER = os.path.join(COMPILER_WORKDIR_ROOT, \"decoder\")\nneuron_decoder = NeuronDecoder(DecoderWrapper(model.perceiver.decoder, model.perceiver.decoder.modalities['audio'].decoder_query, \\\n model.perceiver.decoder.modalities['image'].decoder_query, model.perceiver.decoder.modalities['label'].decoder_query, \\\n model.perceiver.output_postprocessor))\n\n# You might see a warning from trace about unused input - these are safe to ignore.\nprint(\"Compiling decoder...\")\nneuron_decoder.decoder_wrapper = torch_neuronx.trace(\n neuron_decoder.decoder_wrapper,\n (z, query_mask, audio_input, audio_input_without_pos, audio_subsampled_point, audio_padding,\n image_input, image_input_without_pos, image_subsampled_point, image_padding,\n label_input, label_input_without_pos, label_padding),\n compiler_workdir=COMPILER_WORKDIR_DECODER,\n compiler_args=[f\"--temp-dir={COMPILER_WORKDIR_DECODER}\", \"--auto-cast=none\"] # --auto-cast=none is needed to avoid numerical error.\n)\n\n# Save compiled decoder\ndecoder_fname = os.path.join(COMPILER_WORKDIR_DECODER, 'model.pt')\ntorch.jit.save(neuron_decoder.decoder_wrapper, decoder_fname)\n\nprint(\"Done\")" }, { "path": "archive/src/benchmark/pytorch/perceiver-vision_benchmark.py", "content": "import torch\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodels_list = [\n (\"PerceiverForImageClassificationLearned\", \"deepmind/vision-perceiver-learned\"),\n (\"PerceiverForImageClassificationFourier\", \"deepmind/vision-perceiver-fourier\"),\n (\"PerceiverForImageClassificationConvProcessing\", \"deepmind/vision-perceiver-conv\"),\n]\nbatch_sizes = [1]\nn_models = [1, 2]\nworkers_per_model = [1, 2] # optimized for latency or throughput\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n for class_name, pretrained_name in models_list:\n model_name = pretrained_name.split(\"/\")[1]\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = npf.torch.benchmark(filename, inputs, n_models=n_models, workers_per_model=workers_per_model) \n\n # View and save results\n print(\"======== {} ========\".format(filename))\n npf.print_reports(reports)\n npf.write_csv(reports)\n npf.write_json(reports)\n" }, { "path": "archive/src/benchmark/pytorch/perceiver-vision_compile.py", "content": "import torch\nimport transformers # ==4.32.0\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodels_list = [\n (\"PerceiverForImageClassificationLearned\", \"deepmind/vision-perceiver-learned\"),\n (\"PerceiverForImageClassificationFourier\", \"deepmind/vision-perceiver-fourier\"),\n (\"PerceiverForImageClassificationConvProcessing\", \"deepmind/vision-perceiver-conv\"),\n]\nbatch_sizes = [1]\npipeline_sizes = [1]\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n for class_name, pretrained_name in models_list:\n model_name = pretrained_name.split(\"/\")[1]\n\n model = getattr(transformers, class_name).from_pretrained(pretrained_name)\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n npf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )" }, { "path": "archive/src/benchmark/pytorch/pixart_alpha_benchmark.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\nos.environ[\"NEURON_CUSTOM_SILU\"] = \"1\"\n\nimport copy\nimport diffusers\nimport math\nimport numpy as npy\nimport time\nimport torch\nimport torch_neuronx\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom diffusers import PixArtAlphaPipeline\nfrom diffusers import Transformer2DModel\nfrom IPython.display import clear_output\nfrom matplotlib import image as mpimg\nfrom matplotlib import pyplot as plt\nfrom torch import nn\n\nimport torch\nfrom torch import nn\nfrom transformers.models.t5.modeling_t5 import T5EncoderModel\nfrom diffusers import Transformer2DModel\n\n# Define datatype\nDTYPE = torch.bfloat16\n\n# Specialized benchmarking class for PixArt models.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E PixArt performance,\n# because the top-level PixArt pipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because PixArt pipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\nclass InferenceTextEncoderWrapper(nn.Module):\n def __init__(self, dtype, t: T5EncoderModel, seqlen: int):\n super().__init__()\n self.dtype = dtype\n self.device = t.device\n self.t = t\n def forward(self, text_input_ids, attention_mask=None):\n return [self.t(text_input_ids, attention_mask)['last_hidden_state'].to(self.dtype)]\n\nclass InferenceTransformerWrapper(nn.Module):\n def __init__(self, transformer: Transformer2DModel):\n super().__init__()\n self.transformer = transformer\n self.config = transformer.config\n self.dtype = transformer.dtype\n self.device = transformer.device\n def forward(self, hidden_states, encoder_hidden_states=None, timestep=None, \n encoder_attention_mask=None, added_cond_kwargs=None,\n return_dict=False):\n output = self.transformer(\n hidden_states, \n encoder_hidden_states, \n timestep, \n encoder_attention_mask)\n return output\n\nclass SimpleWrapper(nn.Module):\n def __init__(self, model):\n super().__init__()\n self.model = model\n def forward(self, x):\n output = self.model(x)\n return output\n\n# --- Load all compiled models and benchmark pipeline ---\ndef get_pipe(resolution, dtype):\n if resolution == 256:\n transformer: Transformer2DModel = Transformer2DModel.from_pretrained(\n \"PixArt-alpha/PixArt-XL-2-256x256\", \n subfolder=\"transformer\", \n torch_dtype=dtype)\n return PixArtAlphaPipeline.from_pretrained(\n \"PixArt-alpha/PixArt-XL-2-512x512\", \n transformer=transformer, \n torch_dtype=dtype)\n elif resolution == 512:\n return PixArtAlphaPipeline.from_pretrained(\n \"PixArt-alpha/PixArt-XL-2-512x512\", \n torch_dtype=dtype)\n else:\n raise Exception(f\"Unsupport resolution {resolution} for pixart alpha\")\n\nCOMPILER_WORKDIR_ROOT = 'pixart_alpha_compile_dir'\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntransformer_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'transformer/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\n# Select the desired resolution ()\nresolution = 256\n# resolution = 512\n\npipe = get_pipe(resolution, DTYPE)\nseqlen = 120\n\n_neuronTextEncoder = InferenceTextEncoderWrapper(DTYPE, pipe.text_encoder, seqlen)\n_neuronTextEncoder.t = torch.jit.load(text_encoder_filename)\npipe.text_encoder = _neuronTextEncoder\nassert pipe._execution_device is not None\n\ndevice_ids = [0, 1]\n_neuronTransformer = InferenceTransformerWrapper(pipe.transformer)\n_neuronTransformer.transformer = torch_neuronx.DataParallel(torch.jit.load(transformer_filename), device_ids, set_dynamic_batching=False)\npipe.transformer = _neuronTransformer\npipe.vae.decoder = SimpleWrapper(torch.jit.load(decoder_filename))\npipe.vae.post_quant_conv = SimpleWrapper(torch.jit.load(post_quant_conv_filename))\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\nn_runs = 20\nbenchmark(n_runs, \"pixart_alpha\", pipe, prompt)\n" }, { "path": "archive/src/benchmark/pytorch/pixart_sigma_benchmark.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\nos.environ[\"NEURON_CUSTOM_SILU\"] = \"1\"\n\nimport copy\nimport diffusers\nimport math\nimport numpy as npy\nimport time\nimport torch\nimport torch_neuronx\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom diffusers import PixArtSigmaPipeline\nfrom IPython.display import clear_output\nfrom matplotlib import image as mpimg\nfrom matplotlib import pyplot as plt\nfrom torch import nn\n\nimport torch\nfrom torch import nn\nfrom transformers.models.t5.modeling_t5 import T5EncoderModel\nfrom diffusers import Transformer2DModel\n\n# Define datatype\nDTYPE = torch.bfloat16\n\n# Specialized benchmarking class for PixArt models.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E PixArt performance,\n# because the top-level PixArt pipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because PixArt pipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\nclass InferenceTextEncoderWrapper(nn.Module):\n def __init__(self, dtype, t: T5EncoderModel, seqlen: int):\n super().__init__()\n self.dtype = dtype\n self.device = t.device\n self.t = t\n def forward(self, text_input_ids, attention_mask=None):\n return [self.t(text_input_ids, attention_mask)['last_hidden_state'].to(self.dtype)]\n\nclass InferenceTransformerWrapper(nn.Module):\n def __init__(self, transformer: Transformer2DModel):\n super().__init__()\n self.transformer = transformer\n self.config = transformer.config\n self.dtype = transformer.dtype\n self.device = transformer.device\n def forward(self, hidden_states, encoder_hidden_states=None, timestep=None, \n encoder_attention_mask=None, added_cond_kwargs=None,\n return_dict=False):\n output = self.transformer(\n hidden_states, \n encoder_hidden_states, \n timestep, \n encoder_attention_mask)\n return output\n\nclass SimpleWrapper(nn.Module):\n def __init__(self, model):\n super().__init__()\n self.model = model\n def forward(self, x):\n output = self.model(x)\n return output\n\n# --- Load all compiled models and benchmark pipeline ---\ndef get_pipe(resolution, dtype):\n if resolution == 256:\n transformer = Transformer2DModel.from_pretrained(\n \"PixArt-alpha/PixArt-Sigma-XL-2-256x256\", \n subfolder='transformer', \n torch_dtype=dtype,\n )\n return PixArtSigmaPipeline.from_pretrained(\n \"PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers\",\n transformer=transformer,\n torch_dtype=dtype,\n )\n elif resolution == 512:\n transformer = Transformer2DModel.from_pretrained(\n \"PixArt-alpha/PixArt-Sigma-XL-2-512-MS\",\n subfolder='transformer', \n torch_dtype=dtype,\n )\n return PixArtSigmaPipeline.from_pretrained(\n \"PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers\",\n transformer=transformer,\n torch_dtype=dtype,\n )\n else:\n raise Exception(f\"Unsupport resolution {resolution} for PixArt Sigma\")\n\nCOMPILER_WORKDIR_ROOT = 'pixart_sigma_compile_dir'\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntransformer_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'transformer/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\n# Select the desired resolution ()\nresolution = 256\n# resolution = 512\n\npipe = get_pipe(resolution, DTYPE)\nseqlen = 300\n\n_neuronTextEncoder = InferenceTextEncoderWrapper(DTYPE, pipe.text_encoder, seqlen)\n_neuronTextEncoder.t = torch.jit.load(text_encoder_filename)\npipe.text_encoder = _neuronTextEncoder\nassert pipe._execution_device is not None\n\ndevice_ids = [0, 1]\n_neuronTransformer = InferenceTransformerWrapper(pipe.transformer)\n_neuronTransformer.transformer = torch_neuronx.DataParallel(torch.jit.load(transformer_filename), device_ids, set_dynamic_batching=False)\npipe.transformer = _neuronTransformer\npipe.vae.decoder = SimpleWrapper(torch.jit.load(decoder_filename))\npipe.vae.post_quant_conv = SimpleWrapper(torch.jit.load(post_quant_conv_filename))\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\nn_runs = 20\nbenchmark(n_runs, \"pixart_alpha\", pipe, prompt)\n" }, { "path": "archive/src/benchmark/pytorch/resnet50_benchmark.py", "content": "import torch\nimport torch.neuron\n\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_name = \"resnet50\"\nbatch_sizes = [1, 6]\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = npf.torch.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n npf.print_reports(reports)\n npf.write_csv(reports)\n npf.write_json(reports)\n" }, { "path": "archive/src/benchmark/pytorch/resnet50_compile.py", "content": "import torch\nimport torch.neuron\nimport torchvision\n\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_name = \"resnet50\"\nbatch_sizes = [1, 6]\npipeline_sizes = [1]\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n model = torchvision.models.resnet50(pretrained=True)\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n npf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "archive/src/benchmark/pytorch/resnet_benchmark.py", "content": "import torch\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_names = [\"resnet18\", \"resnet34\", \"resnet50\", \"resnet101\", \"resnet152\"]\nbatch_sizes = [1, 8, 64]\nn_models = [1, 2]\nworkers_per_model = [1, 2] # optimized for latency or throughput\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = npf.torch.benchmark(filename, inputs, n_models=n_models, workers_per_model=workers_per_model) \n\n # View and save results\n print(\"======== {} ========\".format(filename))\n npf.print_reports(reports)\n npf.write_csv(reports)\n npf.write_json(reports)\n" }, { "path": "archive/src/benchmark/pytorch/resnet_compile.py", "content": "import torch\nimport torchvision\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_names = [\"resnet18\", \"resnet34\", \"resnet50\", \"resnet101\", \"resnet152\"]\nbatch_sizes = [1, 8, 64]\npipeline_sizes = [1]\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n model = getattr(torchvision.models, model_name)(pretrained=True)\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n npf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )" }, { "path": "archive/src/benchmark/pytorch/sd2_512_benchmark.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nfrom diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nimport time\nimport math\n\n# Define datatype\nDTYPE = torch.bfloat16\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None, return_dict=False):\n sample = self.unetwrap(sample, timestep.to(dtype=DTYPE).expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n \ndef decode_latents(self, latents):\n latents = latents.to(torch.float)\n latents = 1 / self.vae.config.scaling_factor * latents\n image = self.vae.decode(latents).sample\n image = (image / 2 + 0.5).clamp(0, 1)\n image = image.cpu().permute(0, 2, 3, 1).float().numpy()\n return image\n\nStableDiffusionPipeline.decode_latents = decode_latents\n\n# --- Load all compiled models and benchmark pipeline ---\nCOMPILER_WORKDIR_ROOT = 'sd2_compile_dir_512'\nmodel_id = \"stabilityai/stable-diffusion-2-1-base\"\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\npipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)\n\n# Load the compiled UNet onto two neuron cores.\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\ndevice_ids = [0,1]\npipe.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_filename), device_ids, set_dynamic_batching=False)\n\nclass NeuronTypeConversionWrapper(nn.Module):\n def __init__(self, network):\n super().__init__()\n self.network = network\n\n def forward(self, x):\n return self.network(x.float())\n\n# Load other compiled models onto a single neuron core.\npipe.text_encoder = NeuronTextEncoder(pipe.text_encoder)\npipe.text_encoder.neuron_text_encoder = torch.jit.load(text_encoder_filename)\npipe.vae.decoder = NeuronTypeConversionWrapper(torch.jit.load(decoder_filename))\npipe.vae.post_quant_conv = NeuronTypeConversionWrapper(torch.jit.load(post_quant_conv_filename))\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\nn_runs = 20\nbenchmark(n_runs, \"stable_diffusion_512\", pipe, prompt)\n" }, { "path": "archive/src/benchmark/pytorch/sd2_512_compile.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nimport copy\nfrom diffusers import StableDiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n# Compatibility for diffusers<0.18.0\nfrom packaging import version\nimport diffusers\ndiffusers_version = version.parse(diffusers.__version__)\nuse_new_diffusers = diffusers_version >= version.parse('0.18.0')\nif use_new_diffusers:\n from diffusers.models.attention_processor import Attention\nelse:\n from diffusers.models.cross_attention import CrossAttention\n\n# Define datatype\nDTYPE = torch.bfloat16\n\n# Have to do this double wrapper trick to compile the unet, because\n# of the special UNet2DConditionOutput output type.\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n sample = self.unetwrap(sample, timestep.to(dtype=DTYPE).expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n \nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n\n\n# Optimized attention\ndef get_attention_scores(self, query, key, attn_mask): \n dtype = query.dtype\n\n if self.upcast_attention:\n query = query.float()\n key = key.float()\n\n # Check for square matmuls\n if(query.size() == key.size()):\n attention_scores = custom_badbmm(\n key,\n query.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = attention_scores.softmax(dim=1).permute(0,2,1)\n attention_probs = attention_probs.to(dtype)\n\n else:\n attention_scores = custom_badbmm(\n query,\n key.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = attention_scores.softmax(dim=-1)\n attention_probs = attention_probs.to(dtype)\n \n return attention_probs\n\n# In the original badbmm the bias is all zeros, so only apply scale\ndef custom_badbmm(a, b):\n bmm = torch.bmm(a, b)\n scaled = bmm * 0.125\n return scaled\n\n\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'sd2_compile_dir_512'\n\n# Model ID for SD version pipeline\nmodel_id = \"stabilityai/stable-diffusion-2-1-base\"\n\n# --- Compile UNet and save ---\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\n\n# Replace original cross-attention module with custom cross-attention module for better performance\nif use_new_diffusers:\n Attention.get_attention_scores = get_attention_scores\nelse:\n CrossAttention.get_attention_scores = get_attention_scores\n\n# Apply double wrapper to deal with custom return type\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe.unet.unetwrap)\ndel pipe\n\n# Compile unet - FP32\nsample_1b = torch.randn([1, 4, 64, 64], dtype=DTYPE)\ntimestep_1b = torch.tensor(999, dtype=DTYPE).expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 1024], dtype=DTYPE)\nexample_inputs = sample_1b, timestep_1b, encoder_hidden_states_1b\n\nunet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet'),\n compiler_args=[\"--model-type=unet-inference\", \"--enable-fast-loading-neuron-binaries\"]\n)\n\n# Enable asynchronous and lazy loading to speed up model load\ntorch_neuronx.async_load(unet_neuron)\ntorch_neuronx.lazy_load(unet_neuron)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\ndel unet_neuron\n\n\n\n# --- Compile CLIP text encoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\ntext_encoder = copy.deepcopy(pipe.text_encoder)\ndel pipe\n\n# Apply the wrapper to deal with custom return type\ntext_encoder = NeuronTextEncoder(text_encoder)\n\n# Compile text encoder\n# This is used for indexing a lookup table in torch.nn.Embedding,\n# so using random numbers may give errors (out of range).\nemb = torch.tensor([[49406, 18376, 525, 7496, 49407, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\ntext_encoder_neuron = torch_neuronx.trace(\n text_encoder.neuron_text_encoder, \n emb, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n )\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(text_encoder_neuron)\n\n# Save the compiled text encoder\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntorch.jit.save(text_encoder_neuron, text_encoder_filename)\n\n# delete unused objects\ndel text_encoder\ndel text_encoder_neuron\n\n\n\n# --- Compile VAE decoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\ndecoder = copy.deepcopy(pipe.vae.decoder)\ndel pipe\n\n# Compile vae decoder\ndecoder_in = torch.randn([1, 4, 64, 64], dtype=torch.float32)\ndecoder_neuron = torch_neuronx.trace(\n decoder, \n decoder_in, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(decoder_neuron)\n\n# Save the compiled vae decoder\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntorch.jit.save(decoder_neuron, decoder_filename)\n\n# delete unused objects\ndel decoder\ndel decoder_neuron\n\n\n\n# --- Compile VAE post_quant_conv and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\npost_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\ndel pipe\n\n# # Compile vae post_quant_conv\npost_quant_conv_in = torch.randn([1, 4, 64, 64], dtype=torch.float32)\npost_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv, \n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(post_quant_conv_neuron)\n\n# # Save the compiled vae post_quant_conv\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntorch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n# delete unused objects\ndel post_quant_conv\ndel post_quant_conv_neuron\n" }, { "path": "archive/src/benchmark/pytorch/sd2_768_benchmark.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nfrom diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nimport time\nimport math\n\n# Define datatype\nDTYPE = torch.float32\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None, return_dict=False):\n sample = self.unetwrap(sample, timestep.to(dtype=DTYPE).expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n \n \n# --- Load all compiled models and run pipeline ---\nCOMPILER_WORKDIR_ROOT = 'sd2_compile_dir_768'\nmodel_id = \"stabilityai/stable-diffusion-2-1\"\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\npipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)\n\n# Load the compiled UNet onto two neuron cores.\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\ndevice_ids = [0,1]\npipe.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_filename), device_ids, set_dynamic_batching=False)\n\n# Load other compiled models onto a single neuron core.\npipe.text_encoder = NeuronTextEncoder(pipe.text_encoder)\npipe.text_encoder.neuron_text_encoder = torch.jit.load(text_encoder_filename)\npipe.vae.decoder = torch.jit.load(decoder_filename)\npipe.vae.post_quant_conv = torch.jit.load(post_quant_conv_filename)\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\nn_runs = 20\nbenchmark(n_runs, \"stable_diffusion_768\", pipe, prompt)\n" }, { "path": "archive/src/benchmark/pytorch/sd2_768_compile.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nimport copy\nfrom diffusers import StableDiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n# Compatibility for diffusers<0.18.0\nfrom packaging import version\nimport diffusers\ndiffusers_version = version.parse(diffusers.__version__)\nuse_new_diffusers = diffusers_version >= version.parse('0.18.0')\nif use_new_diffusers:\n from diffusers.models.attention_processor import Attention\nelse:\n from diffusers.models.cross_attention import CrossAttention\n\n# Define datatype\nDTYPE = torch.float32\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n sample = self.unetwrap(sample, timestep.to(dtype=DTYPE).expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n \nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n\n\n# Optimized attention\ndef get_attention_scores(self, query, key, attn_mask): \n dtype = query.dtype\n\n if self.upcast_attention:\n query = query.float()\n key = key.float()\n\n # Check for square matmuls\n if(query.size() == key.size()):\n attention_scores = custom_badbmm(\n key,\n query.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = attention_scores.softmax(dim=1).permute(0,2,1)\n attention_probs = attention_probs.to(dtype)\n\n else:\n attention_scores = custom_badbmm(\n query,\n key.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = attention_scores.softmax(dim=-1)\n attention_probs = attention_probs.to(dtype)\n \n return attention_probs\n\n# In the original badbmm the bias is all zeros, so only apply scale\ndef custom_badbmm(a, b):\n bmm = torch.bmm(a, b)\n scaled = bmm * 0.125\n return scaled\n\n\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'sd2_compile_dir_768'\n\n# Model ID for SD version pipeline\nmodel_id = \"stabilityai/stable-diffusion-2-1\"\n\n# --- Compile UNet and save ---\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\n\n# Replace original cross-attention module with custom cross-attention module for better performance\nif use_new_diffusers:\n Attention.get_attention_scores = get_attention_scores\nelse:\n CrossAttention.get_attention_scores = get_attention_scores\n\n# Apply double wrapper to deal with custom return type\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe.unet.unetwrap)\ndel pipe\n\n# Compile unet\nsample_1b = torch.randn([1, 4, 96, 96], dtype=DTYPE)\ntimestep_1b = torch.tensor(999, dtype=DTYPE).expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 1024], dtype=DTYPE)\nexample_inputs = sample_1b, timestep_1b, encoder_hidden_states_1b\n\nunet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet'),\n compiler_args=[\"--model-type=unet-inference\", \"--enable-fast-loading-neuron-binaries\"]\n)\n\n# Enable asynchronous and lazy loading to speed up model load\ntorch_neuronx.async_load(unet_neuron)\ntorch_neuronx.lazy_load(unet_neuron)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\ndel unet_neuron\n\n\n\n# --- Compile CLIP text encoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\ntext_encoder = copy.deepcopy(pipe.text_encoder)\ndel pipe\n\n# Apply the wrapper to deal with custom return type\ntext_encoder = NeuronTextEncoder(text_encoder)\n\n# Compile text encoder\n# This is used for indexing a lookup table in torch.nn.Embedding,\n# so using random numbers may give errors (out of range).\nemb = torch.tensor([[49406, 18376, 525, 7496, 49407, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\ntext_encoder_neuron = torch_neuronx.trace(\n text_encoder.neuron_text_encoder, \n emb, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n )\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(text_encoder_neuron)\n\n# Save the compiled text encoder\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntorch.jit.save(text_encoder_neuron, text_encoder_filename)\n\n# delete unused objects\ndel text_encoder\ndel text_encoder_neuron\n\n\n\n# --- Compile VAE decoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\ndecoder = copy.deepcopy(pipe.vae.decoder)\ndel pipe\n\n# Compile vae decoder\ndecoder_in = torch.randn([1, 4, 96, 96], dtype=DTYPE)\ndecoder_neuron = torch_neuronx.trace(\n decoder, \n decoder_in, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(decoder_neuron)\n\n# Save the compiled vae decoder\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntorch.jit.save(decoder_neuron, decoder_filename)\n\n# delete unused objects\ndel decoder\ndel decoder_neuron\n\n\n\n# --- Compile VAE post_quant_conv and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\npost_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\ndel pipe\n\n# # Compile vae post_quant_conv\npost_quant_conv_in = torch.randn([1, 4, 96, 96], dtype=DTYPE)\npost_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv, \n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(post_quant_conv_neuron)\n\n# # Save the compiled vae post_quant_conv\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntorch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n# delete unused objects\ndel post_quant_conv\ndel post_quant_conv_neuron\n" }, { "path": "archive/src/benchmark/pytorch/sd2_inpainting_benchmark.py", "content": "import torch\nimport torch.nn as nn\nimport torch_neuronx\nimport os\nfrom diffusers import StableDiffusionInpaintPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nfrom diffusers.models.attention_processor import Attention\n\nimport argparse\nimport copy\n\ntorch.manual_seed(0)\n\ndef parse_argsuments():\n parser = argparse.ArgumentParser()\n parser.add_argument('--prompt', type=str, default='Face of a yellow cat, high resolution, sitting on a park bench', help=\"user input for text to image use case\")\n parser.add_argument('--target_dir', type=str, default='./sd21_inpainting_512_neuron', help=\"directory to save neuron compield model\")\n args=parser.parse_args()\n return args\n\n# Have to do this double wrapper trick to compile the unet, because\n# of the special UNet2DConditionOutput output type.\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n sample = self.unetwrap(sample, timestep.bfloat16().expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n\n# Optimized attention\ndef get_attention_scores(self, query, key, attn_mask): \n dtype = query.dtype\n\n if self.upcast_attention:\n query = query.float()\n key = key.float()\n\n # Check for square matmuls\n if(query.size() == key.size()):\n attention_scores = custom_badbmm(\n key,\n query.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = torch.nn.functional.softmax(attention_scores, dim=1).permute(0,2,1)\n attention_probs = attention_probs.to(dtype)\n\n else:\n attention_scores = custom_badbmm(\n query,\n key.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = torch.nn.functional.softmax(attention_scores, dim=-1)\n attention_probs = attention_probs.to(dtype)\n \n return attention_probs\n\ndef custom_badbmm(a, b):\n bmm = torch.bmm(a, b)\n scaled = bmm * 0.125\n return scaled\n\ninputs=parse_argsuments()\nprint(inputs.target_dir)\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = inputs.target_dir\n\ndef trace_vae_encoder(model_id, height, width):\n # Only keep the model being compiled in RAM to minimze memory pressure\n pipe = StableDiffusionInpaintPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n vae_encoder = copy.deepcopy(pipe.vae.encoder)\n del pipe\n\n sample_input = torch.randn([1, 3, height, width])\n vae_encoder_neuron = torch_neuronx.trace(\n vae_encoder, \n sample_input, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_encoder'),\n )\n\n # Save the compiled text encoder\n vae_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_encoder/model.pt')\n torch.jit.save(vae_encoder_neuron, vae_encoder_filename)\n\n # delete unused objects\n del vae_encoder\n del vae_encoder_neuron\n\ndef trace_unet(model_id, height, width):\n # --- Compile UNet and save ---\n DTYPE = torch.bfloat16\n pipe = StableDiffusionInpaintPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\n\n # Replace original cross-attention module with custom cross-attention module for better performance\n Attention.get_attention_scores = get_attention_scores\n\n # Apply double wrapper to deal with custom return type\n pipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n\n # Only keep the model being compiled in RAM to minimze memory pressure\n unet = copy.deepcopy(pipe.unet.unetwrap)\n del pipe\n\n sample_1b = torch.randn([1, 9, height, width], dtype=DTYPE)\n timestep_1b = torch.tensor(999, dtype=DTYPE).expand((1,))\n encoder_hidden_states_1b = torch.randn([1, 77, 1024], dtype=DTYPE)\n example_inputs = sample_1b, timestep_1b, encoder_hidden_states_1b\n\n unet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet'),\n compiler_args=[\"--model-type=unet-inference\", \"--verbose=info\"],\n )\n\n # save compiled unet\n unet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\n torch.jit.save(unet_neuron, unet_filename)\n\n # delete unused objects\n del unet\n del unet_neuron\n \n\ndef main():\n \n model_id = \"stabilityai/stable-diffusion-2-inpainting\"\n height = 624\n width = 936\n\n trace_unet(model_id, height // 8, width // 8)\n trace_vae_encoder(model_id, height, width)\n\n # Only keep the model being compiled in RAM to minimze memory pressure\n pipe = StableDiffusionInpaintPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n text_encoder = copy.deepcopy(pipe.text_encoder)\n del pipe\n # Apply the wrapper to deal with custom return type\n text_encoder = NeuronTextEncoder(text_encoder)\n\n # Compile text encoder\n # This is used for indexing a lookup table in torch.nn.Embedding,\n # so using random numbers may give errors (out of range).\n emb = torch.tensor([[49406, 18376, 525, 7496, 49407, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\n text_encoder_neuron = torch_neuronx.trace(\n text_encoder.neuron_text_encoder, \n emb, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n )\n\n # Save the compiled text encoder\n text_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\n torch.jit.save(text_encoder_neuron, text_encoder_filename)\n\n # delete unused objects\n del text_encoder\n del text_encoder_neuron\n\n # --- Compile VAE decoder and save ---\n\n # Only keep the model being compiled in RAM to minimze memory pressure\n pipe = StableDiffusionInpaintPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n decoder = copy.deepcopy(pipe.vae.decoder)\n del pipe\n\n # Compile vae decoder\n decoder_in = torch.randn([1, 4, height // 8, width // 8])\n decoder_neuron = torch_neuronx.trace(\n decoder, \n decoder_in, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder'),\n compiler_args=[\"--verbose\", \"info\"]\n )\n\n # Save the compiled vae decoder\n decoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\n torch.jit.save(decoder_neuron, decoder_filename)\n\n # delete unused objects\n del decoder\n del decoder_neuron\n \n # --- Compile VAE post_quant_conv and save ---\n\n # Only keep the model being compiled in RAM to minimze memory pressure\n pipe = StableDiffusionInpaintPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n post_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\n del pipe\n\n # Compile vae post_quant_conv\n post_quant_conv_in = torch.randn([1, 4, height // 8 , width // 8])\n post_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv, \n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n compiler_args=[\"--verbose\", \"info\"]\n )\n\n # Save the compiled vae post_quant_conv\n post_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n torch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n # delete unused objects\n del post_quant_conv\n del post_quant_conv_neuron\n \n\nif __name__ == \"__main__\":\n main()" }, { "path": "archive/src/benchmark/pytorch/sd2_inpainting_inference.py", "content": "import torch\nimport torch.nn as nn\nimport torch_neuronx\nimport os\nimport time\nfrom diffusers import StableDiffusionInpaintPipeline, DPMSolverMultistepScheduler\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nfrom diffusers.models.attention_processor import Attention\n\nimport threading\nimport argparse\nimport sys\nimport copy\nimport PIL\nimport math\n\ntorch.manual_seed(0)\n\ndef parse_argsuments():\n parser = argparse.ArgumentParser()\n parser.add_argument('--prompt', type=str, default='Face of a yellow cat, high resolution, sitting on a park bench', help=\"user input for text to image use case\")\n parser.add_argument('--target_dir', type=str, default='./sd21_inpainting_512_neuron', help=\"directory to save neuron compield model\")\n args=parser.parse_args()\n return args\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\nDTYPE = torch.bfloat16\n\n# Have to do this double wrapper trick to compile the unet, because\n# of the special UNet2DConditionOutput output type.\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, timestep_cond=None, added_cond_kwargs=None, cross_attention_kwargs=None, return_dict=False):\n sample = self.unetwrap(sample.to(dtype=DTYPE), timestep.to(dtype=DTYPE).expand((sample.shape[0],)), encoder_hidden_states.to(dtype=DTYPE))[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n\n# Optimized attention\ndef get_attention_scores(self, query, key, attn_mask): \n dtype = query.dtype\n\n if self.upcast_attention:\n query = query.float()\n key = key.float()\n\n # Check for square matmuls\n if(query.size() == key.size()):\n attention_scores = custom_badbmm(\n key,\n query.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = torch.nn.functional.softmax(attention_scores, dim=1).permute(0,2,1)\n attention_probs = attention_probs.to(dtype)\n\n else:\n attention_scores = custom_badbmm(\n query,\n key.transpose(-1, -2)\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = torch.nn.functional.softmax(attention_scores, dim=-1)\n attention_probs = attention_probs.to(dtype)\n \n return attention_probs\n\ndef custom_badbmm(a, b):\n bmm = torch.bmm(a, b)\n scaled = bmm * 0.125\n return scaled\n\n\ndef main():\n \n inputs=parse_argsuments()\n print(inputs.target_dir)\n\n # For saving compiler artifacts\n COMPILER_WORKDIR_ROOT = inputs.target_dir\n\n model_id = \"stabilityai/stable-diffusion-2-inpainting\"\n \n pipe = StableDiffusionInpaintPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n \n text_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\n unet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\n vae_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_encoder/model.pt')\n decoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\n post_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\n # Load the compiled UNet onto two neuron cores.\n pipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n device_ids = [0,1]\n pipe.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_filename), device_ids, set_dynamic_batching=False)\n\n # Load other compiled models onto a single neuron core.\n pipe.text_encoder = NeuronTextEncoder(pipe.text_encoder)\n pipe.text_encoder.neuron_text_encoder = torch.jit.load(text_encoder_filename)\n pipe.vae.encoder = torch.jit.load(vae_encoder_filename)\n pipe.vae.decoder = torch.jit.load(decoder_filename)\n pipe.vae.post_quant_conv = torch.jit.load(post_quant_conv_filename)\n \n height = 624\n width = 936\n base_image = PIL.Image.open('sd2_inpainting_photo.png')\n mask = PIL.Image.open('sd2_inpainting_mask.png')\n image = pipe(prompt=inputs.prompt, image=base_image, mask_image=mask, height=height, width=width).images[0]\n image.save(\"sd2_inpainting_output.png\")\n \n n_runs = 10\n benchmark(n_runs, \"stable_diffusion_inpainting\", pipe, (inputs.prompt, base_image, mask, None, height, width))\n\nif __name__ == \"__main__\":\n main()" }, { "path": "archive/src/benchmark/pytorch/sd_15_512_benchmark.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\n\nimport copy\nimport time\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nfrom diffusers import StableDiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nimport time\nimport math\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n \nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None, return_dict=False):\n sample = self.unetwrap(sample, timestep.float().expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = torch.float32\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n\n\nclass NeuronSafetyModelWrap(nn.Module):\n def __init__(self, safety_model):\n super().__init__()\n self.safety_model = safety_model\n\n def forward(self, clip_inputs):\n return list(self.safety_model(clip_inputs).values())\n\n\n\n# # For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'sd_1_5_fp32_512_compile_workdir'\n\n# Model ID for SD version pipeline\nmodel_id = \"runwayml/stable-diffusion-v1-5\"\n\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\nsafety_model_neuron_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'safety_model/model.pt')\n\n\n# Load the compiled UNet onto two neuron cores.\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\ndevice_ids = [0,1]\npipe.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_filename), device_ids, set_dynamic_batching=False)\n\n# Load other compiled models onto a single neuron core.\npipe.text_encoder = NeuronTextEncoder(pipe.text_encoder)\npipe.text_encoder.neuron_text_encoder = torch.jit.load(text_encoder_filename)\npipe.vae.decoder = torch.jit.load(decoder_filename)\npipe.vae.post_quant_conv = torch.jit.load(post_quant_conv_filename)\npipe.safety_checker.vision_model = NeuronSafetyModelWrap(torch.jit.load(safety_model_neuron_filename))\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\nn_runs = 20\nbenchmark(n_runs, \"stable_diffusion_15_512\", pipe, prompt)" }, { "path": "archive/src/benchmark/pytorch/sd_15_512_compile.py", "content": "import os\nos.environ[\"NEURON_FUSE_SOFTMAX\"] = \"1\"\n\nimport copy\nimport time\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nfrom diffusers import StableDiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\n# Compatibility for diffusers<0.18.0\nfrom packaging import version\nimport diffusers\ndiffusers_version = version.parse(diffusers.__version__)\nuse_new_diffusers = diffusers_version >= version.parse('0.18.0')\nif use_new_diffusers:\n from diffusers.models.attention_processor import Attention\nelse:\n from diffusers.models.cross_attention import CrossAttention\n\n\ndef get_attention_scores(self, query, key, attn_mask): \n dtype = query.dtype\n\n if self.upcast_attention:\n query = query.float()\n key = key.float()\n\n if(query.size() == key.size()):\n attention_scores = cust_badbmm(\n key,\n query.transpose(-1, -2),\n self.scale\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = torch.nn.functional.softmax(attention_scores, dim=1).permute(0,2,1)\n attention_probs = attention_probs.to(dtype)\n\n else:\n attention_scores = cust_badbmm(\n query,\n key.transpose(-1, -2),\n self.scale\n )\n\n if self.upcast_softmax:\n attention_scores = attention_scores.float()\n\n attention_probs = torch.nn.functional.softmax(attention_scores, dim=-1)\n attention_probs = attention_probs.to(dtype)\n \n return attention_probs\n\ndef cust_badbmm(a, b, scale):\n bmm = torch.bmm(a, b)\n scaled = bmm * scale\n return scaled\n\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None):\n out_tuple = self.unet(sample, timestep, encoder_hidden_states, return_dict=False)\n return out_tuple\n \nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(self, sample, timestep, encoder_hidden_states, cross_attention_kwargs=None, return_dict=False):\n sample = self.unetwrap(sample, timestep.float().expand((sample.shape[0],)), encoder_hidden_states)[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = torch.float32\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask = None):\n return [self.neuron_text_encoder(emb)['last_hidden_state']]\n\n\nclass NeuronSafetyModelWrap(nn.Module):\n def __init__(self, safety_model):\n super().__init__()\n self.safety_model = safety_model\n\n def forward(self, clip_inputs):\n return list(self.safety_model(clip_inputs).values())\n\n\n\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'sd_1_5_fp32_512_compile_workdir'\n\n# Model ID for SD version pipeline\nmodel_id = \"runwayml/stable-diffusion-v1-5\"\n\n\n# --- Compile CLIP text encoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\ntext_encoder = copy.deepcopy(pipe.text_encoder)\ndel pipe\n\n# Apply the wrapper to deal with custom return type\ntext_encoder = NeuronTextEncoder(text_encoder)\n\n# Compile text encoder\n# This is used for indexing a lookup table in torch.nn.Embedding,\n# so using random numbers may give errors (out of range).\nemb = torch.tensor([[49406, 18376, 525, 7496, 49407, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\n\nwith torch.no_grad():\n start_time = time.time()\n text_encoder_neuron = torch_neuronx.trace(\n text_encoder.neuron_text_encoder, \n emb, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n )\n text_encoder_neuron_compile_time = time.time() - start_time\n print('text_encoder_neuron_compile_time:', text_encoder_neuron_compile_time)\n\n# Save the compiled text encoder\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntorch_neuronx.async_load(text_encoder_neuron)\ntorch.jit.save(text_encoder_neuron, text_encoder_filename)\n\n# delete unused objects\ndel text_encoder\ndel text_encoder_neuron\ndel emb\n\n# --- Compile VAE decoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\ndecoder = copy.deepcopy(pipe.vae.decoder)\ndel pipe\n\n# Compile vae decoder\ndecoder_in = torch.randn([1, 4, 64, 64])\nwith torch.no_grad():\n start_time = time.time()\n decoder_neuron = torch_neuronx.trace(\n decoder, \n decoder_in, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n )\n vae_decoder_compile_time = time.time() - start_time\n print('vae_decoder_compile_time:', vae_decoder_compile_time)\n\n# Save the compiled vae decoder\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntorch_neuronx.async_load(decoder_neuron)\ntorch.jit.save(decoder_neuron, decoder_filename)\n\n# delete unused objects\ndel decoder\ndel decoder_in\ndel decoder_neuron\n\n# --- Compile UNet and save ---\n\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n\n# Replace original cross-attention module with custom cross-attention module for better performance\nif use_new_diffusers:\n Attention.get_attention_scores = get_attention_scores\nelse:\n CrossAttention.get_attention_scores = get_attention_scores\n\n# Apply double wrapper to deal with custom return type\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe.unet.unetwrap)\ndel pipe\n\n# Compile unet - FP32\nsample_1b = torch.randn([1, 4, 64, 64])\ntimestep_1b = torch.tensor(999).float().expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 768])\nexample_inputs = sample_1b, timestep_1b, encoder_hidden_states_1b\n\nwith torch.no_grad():\n start_time = time.time()\n unet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet'),\n compiler_args=[\"--model-type=unet-inference\", \"--enable-fast-loading-neuron-binaries\"]\n )\n unet_compile_time = time.time() - start_time\n print('unet_compile_time:', unet_compile_time)\n\n# Enable asynchronous and lazy loading to speed up model load\ntorch_neuronx.async_load(unet_neuron)\ntorch_neuronx.lazy_load(unet_neuron)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\ndel unet_neuron\ndel sample_1b\ndel timestep_1b\ndel encoder_hidden_states_1b\n\n\n# --- Compile VAE post_quant_conv and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\npost_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\ndel pipe\n\n# Compile vae post_quant_conv\npost_quant_conv_in = torch.randn([1, 4, 64, 64])\nwith torch.no_grad():\n start_time = time.time()\n post_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv, \n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n )\n vae_post_quant_conv_compile_time = time.time() - start_time\n print('vae_post_quant_conv_compile_time:', vae_post_quant_conv_compile_time)\n\n# Save the compiled vae post_quant_conv\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntorch_neuronx.async_load(post_quant_conv_neuron)\ntorch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n# delete unused objects\ndel post_quant_conv\n\n\n\n# --- Compile safety checker and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\nsafety_model = copy.deepcopy(pipe.safety_checker.vision_model)\ndel pipe\n\nclip_input = torch.randn([1, 3, 224, 224])\nwith torch.no_grad():\n start_time = time.time()\n safety_model = torch_neuronx.trace(\n safety_model, \n clip_input,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'safety_model'),\n compiler_args=[\"--enable-fast-loading-neuron-binaries\"]\n )\n safety_model_compile_time = time.time() - start_time\n print('safety_model_compile_time:', safety_model_compile_time)\n\n# Save the compiled safety checker\nsafety_model_neuron_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'safety_model/model.pt')\ntorch_neuronx.async_load(safety_model)\ntorch.jit.save(safety_model, safety_model_neuron_filename)\n\n# delete unused objects\ndel safety_model\n\nprint('Total compile time:', text_encoder_neuron_compile_time + vae_decoder_compile_time + unet_compile_time + vae_post_quant_conv_compile_time + safety_model_compile_time)\n" }, { "path": "archive/src/benchmark/pytorch/sd_4x_upscaler_benchmark.py", "content": "import os\n\nimport time\nimport requests\nimport copy\nimport math\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch_neuronx\nimport numpy as np\n\nfrom PIL import Image\nfrom io import BytesIO\n\nimport diffusers\nfrom diffusers import StableDiffusionUpscalePipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(\n self,\n sample,\n timestep,\n encoder_hidden_states,\n class_labels,\n cross_attention_kwargs=None,\n ):\n out_tuple = self.unet(\n sample, timestep, encoder_hidden_states, class_labels, return_dict=False\n )\n return out_tuple\n\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(\n self,\n sample,\n timestep,\n encoder_hidden_states,\n class_labels,\n cross_attention_kwargs=None,\n return_dict=False,\n ):\n sample = self.unetwrap(\n sample,\n timestep.float().expand((sample.shape[0],)),\n encoder_hidden_states,\n class_labels,\n )[0]\n return UNet2DConditionOutput(sample=sample)\n\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask=None):\n return [self.neuron_text_encoder(emb)[\"last_hidden_state\"]]\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\n\n\n\n# --- Load all compiled models ---\nCOMPILER_WORKDIR_ROOT = 'stable_diffusion_upscaler_fp32'\nmodel_id = \"stabilityai/stable-diffusion-x4-upscaler\"\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\npipe = StableDiffusionUpscalePipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n\n# Load the compiled UNet onto two neuron cores.\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\ndevice_ids = [0,1]\npipe.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_filename), device_ids, set_dynamic_batching=False)\n\n# Load other compiled models onto a single neuron core.\npipe.text_encoder = NeuronTextEncoder(pipe.text_encoder)\npipe.text_encoder.neuron_text_encoder = torch.jit.load(text_encoder_filename)\npipe.vae.decoder = torch.jit.load(decoder_filename)\npipe.vae.post_quant_conv = torch.jit.load(post_quant_conv_filename)\n\n# Run pipeline\nprompt = [\"a white cat\"]\nurl = \"https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd2-upscale/low_res_cat.png\"\nresponse = requests.get(url)\nlow_res_img = Image.open(BytesIO(response.content)).convert(\"RGB\")\nlow_res_img = low_res_img.resize((128, 128))\nupscaled_image = pipe(prompt=prompt, image=low_res_img).images[0]\nos.makedirs(\"misc\", exist_ok=True)\nupscaled_image.save(\"upsampled_cat.png\")\n\n# Benchmark\nn_runs = 20\nbenchmark(n_runs, \"stable_diffusion_512\", pipe, (prompt, low_res_img))\n" }, { "path": "archive/src/benchmark/pytorch/sd_4x_upscaler_compile.py", "content": "import os\n\nimport requests\nimport copy\nimport math\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch_neuronx\n\nfrom PIL import Image\nfrom io import BytesIO\n\nimport diffusers\nfrom diffusers import StableDiffusionUpscalePipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nfrom packaging import version\n\ndef apply_neuron_attn_override(\n diffusers_pkg, get_attn_scores_func, neuron_scaled_dot_product_attention\n):\n diffusers_version = version.parse(diffusers_pkg.__version__)\n use_new_diffusers = diffusers_version >= version.parse(\"0.18.0\")\n if use_new_diffusers:\n diffusers_pkg.models.attention_processor.Attention.get_attention_scores = (\n get_attn_scores_func\n )\n else:\n diffusers_pkg.models.cross_attention.CrossAttention.get_attention_scores = (\n get_attn_scores_func\n )\n\n # If Pytorch 2 is available, a F.scaled_dot_product_attention will be used, so we need to\n # monkey patch that too to be Neuron optimized attention\n if hasattr(F, \"scaled_dot_product_attention\"):\n F.scaled_dot_product_attention = neuron_scaled_dot_product_attention\n\n\ndef get_attention_scores_neuron(self, query, key, attn_mask):\n if query.size() == key.size():\n attention_scores = cust_badbmm(key, query.transpose(-1, -2), self.scale)\n attention_probs = attention_scores.softmax(dim=1).permute(0, 2, 1)\n\n else:\n attention_scores = cust_badbmm(query, key.transpose(-1, -2), self.scale)\n attention_probs = attention_scores.softmax(dim=-1)\n\n return attention_probs\n\n\ndef cust_badbmm(a, b, scale):\n bmm = torch.bmm(a, b)\n scaled = bmm * scale\n return scaled\n\n\ndef neuron_scaled_dot_product_attention(\n query, key, value, attn_mask=None, dropout_p=None, is_causal=None\n):\n orig_shape = None\n if len(query.shape) == 4:\n orig_shape = query.shape\n\n def to3d(x):\n return x.reshape(-1, x.shape[2], x.shape[3])\n\n query, key, value = map(to3d, [query, key, value])\n\n if query.size() == key.size():\n attention_scores = torch.bmm(key, query.transpose(-1, -2)) * (\n 1 / math.sqrt(query.size(-1))\n )\n attention_probs = attention_scores.softmax(dim=1).permute(0, 2, 1)\n\n else:\n attention_scores = torch.bmm(query, key.transpose(-1, -2)) * (\n 1 / math.sqrt(query.size(-1))\n )\n attention_probs = attention_scores.softmax(dim=-1)\n\n attn_out = torch.bmm(attention_probs, value)\n\n if orig_shape:\n attn_out = attn_out.reshape(\n orig_shape[0], orig_shape[1], attn_out.shape[1], attn_out.shape[2]\n )\n\n return attn_out\n\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(\n self,\n sample,\n timestep,\n encoder_hidden_states,\n class_labels,\n cross_attention_kwargs=None,\n ):\n out_tuple = self.unet(\n sample, timestep, encoder_hidden_states, class_labels, return_dict=False\n )\n return out_tuple\n\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.device = unetwrap.unet.device\n\n def forward(\n self,\n sample,\n timestep,\n encoder_hidden_states,\n class_labels,\n cross_attention_kwargs=None,\n return_dict=False,\n ):\n sample = self.unetwrap(\n sample,\n timestep.float().expand((sample.shape[0],)),\n encoder_hidden_states,\n class_labels,\n )[0]\n return UNet2DConditionOutput(sample=sample)\n\n\nclass NeuronTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.neuron_text_encoder = text_encoder\n self.config = text_encoder.config\n self.dtype = text_encoder.dtype\n self.device = text_encoder.device\n\n def forward(self, emb, attention_mask=None):\n return [self.neuron_text_encoder(emb)[\"last_hidden_state\"]]\n\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'stable_diffusion_upscaler_fp32'\n\n# Model ID for SD version pipeline\nmodel_id = \"stabilityai/stable-diffusion-x4-upscaler\"\n\n# --- Compile CLIP text encoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionUpscalePipeline.from_pretrained(\nmodel_id, torch_dtype=torch.float32\n)\ntext_encoder = copy.deepcopy(pipe.text_encoder)\ndel pipe\n\n# Apply the wrapper to deal with custom return type\ntext_encoder = NeuronTextEncoder(text_encoder)\n\n# Compile text encoder\n# This is used for indexing a lookup table in torch.nn.Embedding,\n# so using random numbers may give errors (out of range).\nemb = torch.tensor([[49406, 18376, 525, 7496, 49407, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\n\ntext_encoder_neuron = torch_neuronx.trace(\n text_encoder.neuron_text_encoder,\n emb,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n )\n\n# Save the compiled text encoder\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntorch.jit.save(text_encoder_neuron, text_encoder_filename)\n\n# delete unused objects\ndel text_encoder\n\n# --- Compile VAE decoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionUpscalePipeline.from_pretrained(\nmodel_id, torch_dtype=torch.float32\n)\ndecoder = copy.deepcopy(pipe.vae.decoder)\ndel pipe\n\n# # Compile vae decoder\ndecoder_in = torch.randn([1, 4, 128, 128])\ndecoder_neuron = torch_neuronx.trace(\n decoder,\n decoder_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder'),\n)\n\n# Save the compiled vae decoder\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntorch.jit.save(decoder_neuron, decoder_filename)\n\n# delete unused objects\ndel decoder\n\n# --- Compile UNet and save ---\n\npipe = StableDiffusionUpscalePipeline.from_pretrained(\nmodel_id, torch_dtype=torch.float32\n)\n\n# Replace original cross-attention module with custom cross-attention module for better performance\napply_neuron_attn_override(\ndiffusers, get_attention_scores_neuron, neuron_scaled_dot_product_attention\n)\n\n# Apply double wrapper to deal with custom return type\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe.unet.unetwrap)\ndel pipe\n\n# Compile unet - FP32\nsample_1b = torch.randn([1, 7, 128, 128])\ntimestep_1b = torch.tensor(999).float().expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 1024])\nclass_labels = torch.tensor([20])\nexample_inputs = sample_1b, timestep_1b, encoder_hidden_states_1b, class_labels\n\nunet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet'),\n compiler_args=[\"--model-type=unet-inference\"]\n)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\n\n# --- Compile VAE post_quant_conv and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = StableDiffusionUpscalePipeline.from_pretrained(\nmodel_id, torch_dtype=torch.float32\n)\npost_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\ndel pipe\n\n# # # Compile vae post_quant_conv\npost_quant_conv_in = torch.randn([1, 4, 128, 128])\npost_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv,\n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n)\n\n\n# # Save the compiled vae post_quant_conv\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntorch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n# delete unused objects\ndel post_quant_conv\n" }, { "path": "archive/src/benchmark/pytorch/sdxl_base_1024_benchmark.py", "content": "import os\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nfrom diffusers import DiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\nfrom transformers.models.clip.modeling_clip import CLIPTextModelOutput\n\nimport time\nimport math\n\n# Define datatype\nDTYPE = torch.float32\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n \n def forward(self, sample, timestep, encoder_hidden_states, text_embeds=None, time_ids=None):\n out_tuple = self.unet(sample,\n timestep,\n encoder_hidden_states,\n added_cond_kwargs={\"text_embeds\": text_embeds, \"time_ids\": time_ids},\n return_dict=False)\n return out_tuple\n \n \nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.add_embedding = unetwrap.unet.add_embedding\n self.device = unetwrap.unet.device\n \n def forward(self, sample, timestep, encoder_hidden_states, added_cond_kwargs=None, return_dict=False, cross_attention_kwargs=None):\n sample = self.unetwrap(sample,\n timestep.to(dtype=DTYPE).expand((sample.shape[0],)),\n encoder_hidden_states,\n added_cond_kwargs[\"text_embeds\"],\n added_cond_kwargs[\"time_ids\"])[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass TextEncoderOutputWrapper(nn.Module):\n def __init__(self, traceable_text_encoder, original_text_encoder):\n super().__init__()\n self.traceable_text_encoder = traceable_text_encoder\n self.config = original_text_encoder.config\n self.dtype = original_text_encoder.dtype\n self.device = original_text_encoder.device\n\n def forward(self, text_input_ids, output_hidden_states=True):\n out_tuple = self.traceable_text_encoder(text_input_ids)\n return CLIPTextModelOutput(text_embeds=out_tuple[0], last_hidden_state=out_tuple[1], hidden_states=out_tuple[2])\n \n \nclass TextEncoderOutputWrapper(nn.Module):\n def __init__(self, traceable_text_encoder, original_text_encoder):\n super().__init__()\n self.traceable_text_encoder = traceable_text_encoder\n self.config = original_text_encoder.config\n self.dtype = original_text_encoder.dtype\n self.device = original_text_encoder.device\n\n def forward(self, text_input_ids, output_hidden_states=True):\n out_tuple = self.traceable_text_encoder(text_input_ids)\n return CLIPTextModelOutput(text_embeds=out_tuple[0], last_hidden_state=out_tuple[1], hidden_states=out_tuple[2])\n \nclass TraceableTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.text_encoder = text_encoder\n\n def forward(self, text_input_ids):\n out_tuple = self.text_encoder(text_input_ids, output_hidden_states=True, return_dict=False)\n return out_tuple\n\n\n \n# --- Load all compiled models and run pipeline ---\nCOMPILER_WORKDIR_ROOT = 'sdxl_base_compile_dir_1024'\nmodel_id = \"stabilityai/stable-diffusion-xl-base-1.0\"\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntext_encoder_2_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder_2/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntext_encoder_2_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder_2/model.pt')\n\npipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\n\n# Load the compiled UNet onto two neuron cores.\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\ndevice_ids = [0,1]\npipe.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_filename), device_ids, set_dynamic_batching=False)\n\n# Load other compiled models onto a single neuron core.\npipe.vae.decoder = torch.jit.load(decoder_filename)\npipe.vae.post_quant_conv = torch.jit.load(post_quant_conv_filename)\npipe.text_encoder = TextEncoderOutputWrapper(torch.jit.load(text_encoder_filename), pipe.text_encoder)\npipe.text_encoder_2 = TextEncoderOutputWrapper(torch.jit.load(text_encoder_2_filename), pipe.text_encoder_2)\n\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\nn_runs = 20\nbenchmark(n_runs, \"stable_diffusion_1024\", pipe, prompt)\n\n\n" }, { "path": "archive/src/benchmark/pytorch/sdxl_base_1024_compile.py", "content": "import os\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch_neuronx\n\nimport math\nimport copy\nimport diffusers\nfrom diffusers import DiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\nfrom diffusers.models.attention_processor import Attention\nfrom transformers.models.clip.modeling_clip import CLIPTextModelOutput\n\nfrom packaging import version\n\ndef apply_neuron_attn_override(\n diffusers_pkg, get_attn_scores_func, neuron_scaled_dot_product_attention\n):\n diffusers_version = version.parse(diffusers_pkg.__version__)\n use_new_diffusers = diffusers_version >= version.parse(\"0.18.0\")\n if use_new_diffusers:\n diffusers_pkg.models.attention_processor.Attention.get_attention_scores = (\n get_attn_scores_func\n )\n else:\n diffusers_pkg.models.cross_attention.CrossAttention.get_attention_scores = (\n get_attn_scores_func\n )\n\n # If Pytorch 2 is available, a F.scaled_dot_product_attention will be used, so we need to\n # monkey patch that too to be Neuron optimized attention\n if hasattr(F, \"scaled_dot_product_attention\"):\n F.scaled_dot_product_attention = neuron_scaled_dot_product_attention\n\n# Define datatype\nDTYPE = torch.float32\n\n# Optimized attention\ndef get_attention_scores_neuron(self, query, key, attn_mask): \n if query.size() == key.size():\n attention_scores = custom_badbmm(\n key,\n query.transpose(-1, -2),\n self.scale\n )\n attention_probs = attention_scores.softmax(dim=1).permute(0,2,1)\n\n else:\n attention_scores = custom_badbmm(\n query,\n key.transpose(-1, -2),\n self.scale\n )\n attention_probs = attention_scores.softmax(dim=-1)\n \n return attention_probs\n \ndef custom_badbmm(a, b, scale):\n bmm = torch.bmm(a, b)\n scaled = bmm * scale\n return scaled\n \ndef neuron_scaled_dot_product_attention(\n query, key, value, attn_mask=None, dropout_p=None, is_causal=None\n):\n orig_shape = None\n if len(query.shape) == 4:\n orig_shape = query.shape\n\n def to3d(x):\n return x.reshape(-1, x.shape[2], x.shape[3])\n\n query, key, value = map(to3d, [query, key, value])\n\n if query.size() == key.size():\n attention_scores = torch.bmm(key, query.transpose(-1, -2)) * (\n 1 / math.sqrt(query.size(-1))\n )\n attention_probs = attention_scores.softmax(dim=1).permute(0, 2, 1)\n\n else:\n attention_scores = torch.bmm(query, key.transpose(-1, -2)) * (\n 1 / math.sqrt(query.size(-1))\n )\n attention_probs = attention_scores.softmax(dim=-1)\n\n attn_out = torch.bmm(attention_probs, value)\n\n if orig_shape:\n attn_out = attn_out.reshape(\n orig_shape[0], orig_shape[1], attn_out.shape[1], attn_out.shape[2]\n )\n\n return attn_out\n\n# Replace original cross-attention module with custom cross-attention module for better performance\napply_neuron_attn_override(\n diffusers, get_attention_scores_neuron, neuron_scaled_dot_product_attention\n)\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n\n def forward(\n self, sample, timestep, encoder_hidden_states, text_embeds=None, time_ids=None\n ):\n out_tuple = self.unet(\n sample,\n timestep,\n encoder_hidden_states,\n added_cond_kwargs={\"text_embeds\": text_embeds, \"time_ids\": time_ids},\n return_dict=False,\n )\n return out_tuple\n\n\nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.add_embedding = unetwrap.unet.add_embedding\n self.device = unetwrap.unet.device\n\n def forward(\n self,\n sample,\n timestep,\n encoder_hidden_states,\n added_cond_kwargs=None,\n return_dict=False,\n cross_attention_kwargs=None,\n ):\n sample = self.unetwrap(\n sample,\n timestep.float().expand((sample.shape[0],)),\n encoder_hidden_states,\n added_cond_kwargs[\"text_embeds\"],\n added_cond_kwargs[\"time_ids\"],\n )[0]\n return UNet2DConditionOutput(sample=sample)\n\nclass TextEncoderOutputWrapper(nn.Module):\n def __init__(self, traceable_text_encoder, original_text_encoder):\n super().__init__()\n self.traceable_text_encoder = traceable_text_encoder\n self.config = original_text_encoder.config\n self.dtype = original_text_encoder.dtype\n self.device = original_text_encoder.device\n\n def forward(self, text_input_ids, output_hidden_states=True):\n out_tuple = self.traceable_text_encoder(text_input_ids)\n return CLIPTextModelOutput(text_embeds=out_tuple[0], last_hidden_state=out_tuple[1], hidden_states=out_tuple[2])\n \nclass TraceableTextEncoder(nn.Module):\n def __init__(self, text_encoder):\n super().__init__()\n self.text_encoder = text_encoder\n\n def forward(self, text_input_ids):\n out_tuple = self.text_encoder(text_input_ids, output_hidden_states=True, return_dict=False)\n return out_tuple\n\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'sdxl_base_compile_dir_1024'\n\n# Model ID for SD XL version pipeline\nmodel_id = \"stabilityai/stable-diffusion-xl-base-1.0\"\n\n\n# --- Compile Text Encoders and save ---\n\npipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\n\n\n# Apply wrappers to make text encoders traceable\ntraceable_text_encoder = copy.deepcopy(TraceableTextEncoder(pipe.text_encoder))\ntraceable_text_encoder_2 = copy.deepcopy(TraceableTextEncoder(pipe.text_encoder_2))\n\ndel pipe\n\ntext_input_ids_1 = torch.tensor([[49406, 736, 1615, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407]])\n\n\ntext_input_ids_2 = torch.tensor([[49406, 736, 1615, 49407, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\n\n\n# Text Encoder 1\nneuron_text_encoder = torch_neuronx.trace(\n traceable_text_encoder,\n text_input_ids_1,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n)\n\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntorch.jit.save(neuron_text_encoder, text_encoder_filename)\n\n\n# Text Encoder 2\nneuron_text_encoder_2 = torch_neuronx.trace(\n traceable_text_encoder_2,\n text_input_ids_2,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder_2'),\n)\n\ntext_encoder_2_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder_2/model.pt')\ntorch.jit.save(neuron_text_encoder_2, text_encoder_2_filename)\n\n\n\n# --- Compile Text Encoders and save ---\n\npipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)\n\n# Apply wrappers to make text encoders traceable\ntraceable_text_encoder = copy.deepcopy(TraceableTextEncoder(pipe.text_encoder))\ntraceable_text_encoder_2 = copy.deepcopy(TraceableTextEncoder(pipe.text_encoder_2))\n\ndel pipe\n\ntext_input_ids_1 = torch.tensor([[49406, 736, 1615, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,\n 49407, 49407, 49407, 49407, 49407, 49407, 49407]])\n\n\ntext_input_ids_2 = torch.tensor([[49406, 736, 1615, 49407, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0]])\n\n\n# Text Encoder 1\nneuron_text_encoder = torch_neuronx.trace(\n traceable_text_encoder,\n text_input_ids_1,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder'),\n)\n\ntext_encoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder/model.pt')\ntorch.jit.save(neuron_text_encoder, text_encoder_filename)\n\n# Text Encoder 2\nneuron_text_encoder_2 = torch_neuronx.trace(\n traceable_text_encoder_2,\n text_input_ids_2,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder_2'),\n)\n\ntext_encoder_2_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'text_encoder_2/model.pt')\ntorch.jit.save(neuron_text_encoder_2, text_encoder_2_filename)\n\n\n\n# --- Compile UNet and save ---\n\npipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\n\n\n# Replace original cross-attention module with custom cross-attention module for better performance\nAttention.get_attention_scores = get_attention_scores_neuron\n\n# Apply double wrapper to deal with custom return type\npipe.unet = NeuronUNet(UNetWrap(pipe.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe.unet.unetwrap)\ndel pipe\n\n# Compile unet - FP32\nsample_1b = torch.randn([1, 4, 128, 128], dtype=DTYPE)\ntimestep_1b = torch.tensor(999, dtype=DTYPE).expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 2048], dtype=DTYPE)\nadded_cond_kwargs_1b = {\"text_embeds\": torch.randn([1, 1280], dtype=DTYPE),\n \"time_ids\": torch.randn([1, 6], dtype=DTYPE)}\nexample_inputs = (sample_1b, timestep_1b, encoder_hidden_states_1b, added_cond_kwargs_1b[\"text_embeds\"], added_cond_kwargs_1b[\"time_ids\"],)\n\nunet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet'),\n compiler_args=[\"--model-type=unet-inference\"]\n)\n\n# Enable asynchronous and lazy loading to speed up model load\ntorch_neuronx.async_load(unet_neuron)\ntorch_neuronx.lazy_load(unet_neuron)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\ndel unet_neuron\n\n\n\n# --- Compile VAE decoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\ndecoder = copy.deepcopy(pipe.vae.decoder)\ndel pipe\n\n# Compile vae decoder\ndecoder_in = torch.randn([1, 4, 128, 128], dtype=DTYPE)\ndecoder_neuron = torch_neuronx.trace(\n decoder, \n decoder_in, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder')\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(decoder_neuron)\n\n# Save the compiled vae decoder\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntorch.jit.save(decoder_neuron, decoder_filename)\n\n# delete unused objects\ndel decoder\ndel decoder_neuron\n\n\n\n# --- Compile VAE post_quant_conv and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=DTYPE)\npost_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\ndel pipe\n\n# Compile vae post_quant_conv\npost_quant_conv_in = torch.randn([1, 4, 128, 128], dtype=DTYPE)\npost_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv, \n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(post_quant_conv_neuron)\n\n# Save the compiled vae post_quant_conv\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntorch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n# delete unused objects\ndel post_quant_conv\ndel post_quant_conv_neuron\n" }, { "path": "archive/src/benchmark/pytorch/sdxl_base_and_refiner_1024_benchmark.py", "content": "import os\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nfrom diffusers import DiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\n\nimport time\nimport math\n\n# Define datatype\nDTYPE = torch.float32\n\n# Specialized benchmarking class for stable diffusion.\n# We cannot use any of the pre-existing benchmarking utilities to benchmark E2E stable diffusion performance,\n# because the top-level StableDiffusionPipeline cannot be serialized into a single Torchscript object.\n# All of the pre-existing benchmarking utilities (in neuronperf or torch_neuronx) require the model to be a\n# traced Torchscript.\ndef benchmark(n_runs, test_name, model, model_inputs):\n if not isinstance(model_inputs, tuple):\n model_inputs = (model_inputs,)\n \n warmup_run = model(*model_inputs)\n\n latency_collector = LatencyCollector()\n # can't use register_forward_pre_hook or register_forward_hook because StableDiffusionPipeline is not a torch.nn.Module\n \n for _ in range(n_runs):\n latency_collector.pre_hook()\n res = model(*model_inputs)\n latency_collector.hook()\n \n p0_latency_ms = latency_collector.percentile(0) * 1000\n p50_latency_ms = latency_collector.percentile(50) * 1000\n p90_latency_ms = latency_collector.percentile(90) * 1000\n p95_latency_ms = latency_collector.percentile(95) * 1000\n p99_latency_ms = latency_collector.percentile(99) * 1000\n p100_latency_ms = latency_collector.percentile(100) * 1000\n\n report_dict = dict()\n report_dict[\"Latency P0\"] = f'{p0_latency_ms:.1f}'\n report_dict[\"Latency P50\"]=f'{p50_latency_ms:.1f}'\n report_dict[\"Latency P90\"]=f'{p90_latency_ms:.1f}'\n report_dict[\"Latency P95\"]=f'{p95_latency_ms:.1f}'\n report_dict[\"Latency P99\"]=f'{p99_latency_ms:.1f}'\n report_dict[\"Latency P100\"]=f'{p100_latency_ms:.1f}'\n\n report = f'RESULT FOR {test_name}:'\n for key, value in report_dict.items():\n report += f' {key}={value}'\n print(report)\n\nclass LatencyCollector:\n def __init__(self):\n self.start = None\n self.latency_list = []\n\n def pre_hook(self, *args):\n self.start = time.time()\n\n def hook(self, *args):\n self.latency_list.append(time.time() - self.start)\n\n def percentile(self, percent):\n latency_list = self.latency_list\n pos_float = len(latency_list) * percent / 100\n max_pos = len(latency_list) - 1\n pos_floor = min(math.floor(pos_float), max_pos)\n pos_ceil = min(math.ceil(pos_float), max_pos)\n latency_list = sorted(latency_list)\n return latency_list[pos_ceil] if pos_float - pos_floor > 0.5 else latency_list[pos_floor]\n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n \n def forward(self, sample, timestep, encoder_hidden_states, text_embeds=None, time_ids=None):\n out_tuple = self.unet(sample,\n timestep,\n encoder_hidden_states,\n added_cond_kwargs={\"text_embeds\": text_embeds, \"time_ids\": time_ids},\n return_dict=False)\n return out_tuple\n \n \nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.add_embedding = unetwrap.unet.add_embedding\n self.device = unetwrap.unet.device\n \n def forward(self, sample, timestep, encoder_hidden_states, added_cond_kwargs=None, return_dict=False, cross_attention_kwargs=None):\n sample = self.unetwrap(sample,\n timestep.to(dtype=DTYPE).expand((sample.shape[0],)),\n encoder_hidden_states,\n added_cond_kwargs[\"text_embeds\"],\n added_cond_kwargs[\"time_ids\"])[0]\n return UNet2DConditionOutput(sample=sample)\n \n# Helper function to run both refiner and base pipes and return the final image\ndef run_refiner_and_base(base, refiner, prompt, n_steps=40, high_noise_frac=0.8, generator=None):\n image = base(\n prompt=prompt,\n num_inference_steps=n_steps,\n denoising_end=high_noise_frac,\n output_type=\"latent\",\n generator=generator,\n ).images\n\n image = refiner(\n prompt=prompt,\n num_inference_steps=n_steps,\n denoising_start=high_noise_frac,\n image=image,\n ).images[0]\n\n return image\n \n \n# --- Load all compiled models and run pipeline ---\nCOMPILER_WORKDIR_ROOT = 'sdxl_base_and_refiner_compile_dir_1024'\nbase_model_id = \"stabilityai/stable-diffusion-xl-base-1.0\"\nrefiner_model_id = \"stabilityai/stable-diffusion-xl-refiner-1.0\"\n\nunet_base_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet_base/model.pt')\nunet_refiner_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet_refiner/model.pt')\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\n\n# ------- Load base -------\npipe_base = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=DTYPE, low_cpu_mem_usage=True)\n\n# Load the compiled UNet onto two neuron cores.\npipe_base.unet = NeuronUNet(UNetWrap(pipe_base.unet))\ndevice_ids = [0,1]\npipe_base.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_base_filename), device_ids, set_dynamic_batching=False)\n\n# Load other compiled models onto a single neuron core.\npipe_base.vae.decoder = torch.jit.load(decoder_filename)\npipe_base.vae.post_quant_conv = torch.jit.load(post_quant_conv_filename)\n\n\n# ------- Load refiner -------\n# refiner shares text_encoder_2 and vae with the base\npipe_refiner = DiffusionPipeline.from_pretrained(\n refiner_model_id,\n text_encoder_2=pipe_base.text_encoder_2,\n vae=pipe_base.vae,\n torch_dtype=torch.float32,\n low_cpu_mem_usage=True,\n)\n\n# Refiner - load the compiled UNet onto two neuron cores.\npipe_refiner.unet = NeuronUNet(UNetWrap(pipe_refiner.unet))\ndevice_ids = [0,1]\npipe_refiner.unet.unetwrap = torch_neuronx.DataParallel(torch.jit.load(unet_refiner_filename), device_ids, set_dynamic_batching=False)\n\n\n\n# Define how many steps and what % of steps to be run on each experts (80/20) here\nn_steps = 40\nhigh_noise_frac = 0.8\n\n\nprompt = \"a photo of an astronaut riding a horse on mars\"\ninputs = (pipe_base, pipe_refiner, prompt, n_steps, high_noise_frac, torch.manual_seed(0),)\n\nn_runs = 50\nbenchmark(n_runs, \"stable_diffusion_1024\", run_refiner_and_base, inputs)\n" }, { "path": "archive/src/benchmark/pytorch/sdxl_base_and_refiner_1024_compile.py", "content": "import os\n\nimport torch\nimport torch.nn as nn\nimport torch_neuronx\n\nimport copy\nfrom diffusers import DiffusionPipeline\nfrom diffusers.models.unet_2d_condition import UNet2DConditionOutput\nfrom diffusers.models.attention_processor import Attention\n\n# Define datatype\nDTYPE = torch.float32\n\n# Optimized attention\ndef get_attention_scores_neuron(self, query, key, attn_mask): \n if query.size() == key.size():\n attention_scores = custom_badbmm(\n key,\n query.transpose(-1, -2),\n self.scale\n )\n attention_probs = attention_scores.softmax(dim=1).permute(0,2,1)\n\n else:\n attention_scores = custom_badbmm(\n query,\n key.transpose(-1, -2),\n self.scale\n )\n attention_probs = attention_scores.softmax(dim=-1)\n \n return attention_probs\n \n\ndef custom_badbmm(a, b, scale):\n bmm = torch.bmm(a, b)\n scaled = bmm * scale\n return scaled\n \n\nclass UNetWrap(nn.Module):\n def __init__(self, unet):\n super().__init__()\n self.unet = unet\n \n def forward(self, sample, timestep, encoder_hidden_states, text_embeds=None, time_ids=None):\n out_tuple = self.unet(sample,\n timestep,\n encoder_hidden_states,\n added_cond_kwargs={\"text_embeds\": text_embeds, \"time_ids\": time_ids},\n return_dict=False)\n return out_tuple\n \n \nclass NeuronUNet(nn.Module):\n def __init__(self, unetwrap):\n super().__init__()\n self.unetwrap = unetwrap\n self.config = unetwrap.unet.config\n self.in_channels = unetwrap.unet.in_channels\n self.add_embedding = unetwrap.unet.add_embedding\n self.device = unetwrap.unet.device\n \n def forward(self, sample, timestep, encoder_hidden_states, added_cond_kwargs=None, return_dict=False, cross_attention_kwargs=None):\n sample = self.unetwrap(sample,\n timestep.expand((sample.shape[0],)),\n encoder_hidden_states,\n added_cond_kwargs[\"text_embeds\"],\n added_cond_kwargs[\"time_ids\"])[0]\n return UNet2DConditionOutput(sample=sample)\n \n\n# For saving compiler artifacts\nCOMPILER_WORKDIR_ROOT = 'sdxl_base_and_refiner_compile_dir_1024'\n\n# Model IDs for SD XL version pipeline\nbase_model_id = \"stabilityai/stable-diffusion-xl-base-1.0\"\nrefiner_model_id = \"stabilityai/stable-diffusion-xl-refiner-1.0\"\n\n# All components we compile in this script:\n# 1. unet (base, in fp32)\n# 2. unet (refiner, in fp32)\n# 3. vae.decoder (base & refiner)\n# 4. vae.post_quant_conv (base & refiner)\n\n# --- Compile UNet in fp32 (base) and save ---\n\npipe_base = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=DTYPE, low_cpu_mem_usage=True)\n\n# Replace original cross-attention module with custom cross-attention module for better performance\nAttention.get_attention_scores = get_attention_scores_neuron\n\n# Apply double wrapper to deal with custom return type\npipe_base.unet = NeuronUNet(UNetWrap(pipe_base.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe_base.unet.unetwrap)\ndel pipe_base\n\n# Compile unet - fp32 (note these tensors are cast to fp32 in UNetWrap)\nsample_1b = torch.randn([1, 4, 128, 128])\ntimestep_1b = torch.tensor(999).float().expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 2048])\nadded_cond_kwargs_1b = {\"text_embeds\": torch.randn([1, 1280]),\n \"time_ids\": torch.randn([1, 6])}\nexample_inputs = (sample_1b, timestep_1b, encoder_hidden_states_1b, added_cond_kwargs_1b[\"text_embeds\"], added_cond_kwargs_1b[\"time_ids\"],)\n\nunet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet_base'),\n compiler_args=[\"--model-type=unet-inference\"]\n)\n\n# Enable asynchronous and lazy loading to speed up model load\ntorch_neuronx.async_load(unet_neuron)\ntorch_neuronx.lazy_load(unet_neuron)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet_base/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\ndel unet_neuron\n\n\n\n# --- Compile UNet in fp32 (refiner) and save ---\n\npipe_refiner = DiffusionPipeline.from_pretrained(refiner_model_id, torch_dtype=DTYPE, low_cpu_mem_usage=True)\n\n# Replace original cross-attention module with custom cross-attention module for better performance\nAttention.get_attention_scores = get_attention_scores_neuron\n\n# Apply double wrapper to deal with custom return type\npipe_refiner.unet = NeuronUNet(UNetWrap(pipe_refiner.unet))\n\n# Only keep the model being compiled in RAM to minimze memory pressure\nunet = copy.deepcopy(pipe_refiner.unet.unetwrap)\ndel pipe_refiner\n\n# Compile unet - fp32 - some input shapes are different from base\nsample_1b = torch.randn([1, 4, 128, 128])\ntimestep_1b = torch.tensor(999).float().expand((1,))\nencoder_hidden_states_1b = torch.randn([1, 77, 1280])\nadded_cond_kwargs_1b = {\"text_embeds\": torch.randn([1, 1280]),\n \"time_ids\": torch.randn([1, 5])}\nexample_inputs = (sample_1b, timestep_1b, encoder_hidden_states_1b, added_cond_kwargs_1b[\"text_embeds\"], added_cond_kwargs_1b[\"time_ids\"],)\n\nunet_neuron = torch_neuronx.trace(\n unet,\n example_inputs,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'unet_refiner'),\n compiler_args=[\"--model-type=unet-inference\"]\n)\n\n# Enable asynchronous and lazy loading to speed up model load\ntorch_neuronx.async_load(unet_neuron)\ntorch_neuronx.lazy_load(unet_neuron)\n\n# save compiled unet\nunet_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'unet_refiner/model.pt')\ntorch.jit.save(unet_neuron, unet_filename)\n\n# delete unused objects\ndel unet\ndel unet_neuron\n\n\n\n# --- Compile VAE decoder and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=DTYPE, low_cpu_mem_usage=True)\ndecoder = copy.deepcopy(pipe.vae.decoder)\ndel pipe\n\n# Compile vae decoder\ndecoder_in = torch.randn([1, 4, 128, 128], dtype=DTYPE)\ndecoder_neuron = torch_neuronx.trace(\n decoder, \n decoder_in, \n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder')\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(decoder_neuron)\n\n# Save the compiled vae decoder\ndecoder_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_decoder/model.pt')\ntorch.jit.save(decoder_neuron, decoder_filename)\n\n# delete unused objects\ndel decoder\ndel decoder_neuron\n\n\n\n# --- Compile VAE post_quant_conv and save ---\n\n# Only keep the model being compiled in RAM to minimze memory pressure\npipe = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=DTYPE, low_cpu_mem_usage=True)\npost_quant_conv = copy.deepcopy(pipe.vae.post_quant_conv)\ndel pipe\n\n# Compile vae post_quant_conv\npost_quant_conv_in = torch.randn([1, 4, 128, 128], dtype=DTYPE)\npost_quant_conv_neuron = torch_neuronx.trace(\n post_quant_conv, \n post_quant_conv_in,\n compiler_workdir=os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv'),\n)\n\n# Enable asynchronous loading to speed up model load\ntorch_neuronx.async_load(post_quant_conv_neuron)\n\n# Save the compiled vae post_quant_conv\npost_quant_conv_filename = os.path.join(COMPILER_WORKDIR_ROOT, 'vae_post_quant_conv/model.pt')\ntorch.jit.save(post_quant_conv_neuron, post_quant_conv_filename)\n\n# delete unused objects\ndel post_quant_conv\ndel post_quant_conv_neuron" }, { "path": "archive/src/benchmark/pytorch/unet_benchmark.py", "content": "import torch\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_name = \"UNet\"\nbatch_sizes = [1, 4]\nn_models = [1, 2]\nworkers_per_model = [1, 2] # optimized for latency or throughput\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = npf.torch.benchmark(filename, inputs, n_models=n_models, workers_per_model=workers_per_model) \n\n # View and save results\n print(\"======== {} ========\".format(filename))\n npf.print_reports(reports)\n npf.write_csv(reports)\n npf.write_json(reports)\n" }, { "path": "archive/src/benchmark/pytorch/unet_compile.py", "content": "import torch\n\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_name = \"UNet\"\nbatch_sizes = [1, 4]\npipeline_sizes = [1]\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\nif __name__ == \"__main__\":\n # UNet Implementation from https://github.com/milesial/Pytorch-UNet\n # load the model\n model = torch.hub.load('milesial/Pytorch-UNet', 'unet_carvana', pretrained=False)\n # load the weights\n state_dict = torch.hub.load_state_dict_from_url('https://github.com/milesial/Pytorch-UNet/releases/download/v3.0/unet_carvana_scale0.5_epoch2.pth', map_location=\"cpu\")\n model.load_state_dict(state_dict)\n\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n npf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )" }, { "path": "archive/src/benchmark/pytorch/vgg_benchmark.py", "content": "import torch\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_names = [\"vgg11\", \"vgg16\"]\nbatch_sizes = [1, 8, 64]\nn_models = [1, 2]\nworkers_per_model = [1, 2] # optimized for latency or throughput\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = npf.torch.benchmark(filename, inputs, n_models=n_models, workers_per_model=workers_per_model) \n\n # View and save results\n print(\"======== {} ========\".format(filename))\n npf.print_reports(reports)\n npf.write_csv(reports)\n npf.write_json(reports)\n" }, { "path": "archive/src/benchmark/pytorch/vgg_compile.py", "content": "import torch\nimport torchvision\nimport neuronperf as npf\nimport neuronperf.torch\n\n# Add to these lists or change as needed\nmodel_names = [\"vgg11\", \"vgg16\"]\nbatch_sizes = [1, 8, 64]\npipeline_sizes = [1]\n\n\ndef get_batch(batch_size):\n return torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n model = getattr(torchvision.models, model_name)(pretrained=True)\n inputs = [get_batch(batch_size) for batch_size in batch_sizes]\n filename = f\"{model_name}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n npf.torch.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )" }, { "path": "archive/tensorboard/getting-started-tensorboard-neuron-plugin.rst", "content": ".. _neuron-plugin-tensorboard:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This page for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n :date-modified: 12-02-2025\n\nNeuron Plugin for TensorBoard (Inf1)\n====================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nThis guide is for developers who want to better understand how their\nmodel is executed using Neuron SDK through TensorBoard.\n\nThe Neuron plugin for TensorBoard provides metrics to the performance of machine learning tasks accelerated using the Neuron SDK. It is\ncompatible with TensorBoard versions 1.15 and higher. It provides visualizations and profiling results for graphs executed on NeuronCores.\n\n.. note::\n\n The following information is compatible with Neuron SDK for Inf1. For a walkthrough on the latest version, please check out the guide\n :ref:`neuronx-plugin-tensorboard`.\n\n.. note:: \n\n Graph visualization is currently only supported for TensorFlow-Neuron. Support\n for MXNet-Neuron and PyTorch-Neuron visualization will be added in a future\n release.\n\n\nCompile the neural network\n--------------------------\n\n3. Refer to the following guides on how to compile a graph using Neuron SDK.\n\n- TensorFlow-Neuron\n - :ref:`/src/examples/tensorflow/tensorflow_resnet50/resnet50.ipynb`\n- PyTorch-Neuron:\n - \"Compile model for Neuron\" in `PyTorch-Neuron Resnet50 Tutorial`_\n- MXNet-Neuron:\n - :ref:`/src/examples/mxnet/resnet50/resnet50.ipynb`\n\nEnable profiling \n-----------------\n\nIn this step, we enable Neuron profile data collection and collect results\nfrom executing an inference.\n\n4.1. To start profiling the neural network and collect inference traces, create a\ndirectory where profile data will be dumped and set the ``NEURON_PROFILE`` environment\nvariable. In this example, we will assume this directory is ``$HOME/profile``\n\n.. code:: bash\n\n mkdir -p $HOME/profile\n export NEURON_PROFILE=$HOME/profile\n\n4.2. Ensure Neuron Tools are executable by setting the ``PATH`` environment variable.\n\n.. code:: bash\n\n export PATH=/opt/aws/neuron/bin:$PATH\n\n4.3. Execute inference!\n\n.. note::\n\n Please run the inference script outside of Jupyter notebook. Profiling in\n Jupyter notebook is not supported at this time.\n\n.. note::\n\n Please ensure the inference script executes only one inference, as profiling\n results are currently only supported for a single inference.\n\nFor more info on how to execute inference, refer to the following guides:\n\n- TensorFlow-Neuron\n - :ref:`/src/examples/tensorflow/tensorflow_resnet50/resnet50.ipynb`\n- PyTorch-Neuron\n - \"Run inference on Single Core\" in :ref:`/src/examples/pytorch/resnet50.ipynb`\n- MXNet-Neuron\n - :ref:`/src/examples/mxnet/resnet50/resnet50.ipynb`\n\n4.4. Check if profiling results were successfully saved. In the directory\npointed to by ``NEURON_PROFILE`` environment variable set in Step 4.1, there\nshould be at least two files, one with the ``.neff`` extension and one with the\n``.ntff`` extension. For TensorFlow-Neuron users, the graph file (``.pb``) will\nalso be in this directory.\n\n.. code:: bash\n\n ls $NEURON_PROFILE\n\nLaunch TensorBoard\n------------------\n\nIn this step, we will process the Neuron profile data and launch TensorBoard.\n\n5.1. Install the Neuron plugin for Tensorboard.\n\n.. include:: /setup/install-templates/inf1/tensorboard-plugin-neuron-pip-install.rst\n\n5.2. After collecting the raw profile data, we need to post-process it to create the\nlog files used by the Neuron plugin. This can be done when launching TensorBoard\nby passing an extra flag ``--run_neuron_profiler``. Using this flag will create the\ndirectory specified by ``--logdir`` and populate it with Neuron plugin data. Please\nnote that the ``NEURON_PROFILE`` environment variable set in Step 4.1 must still point\nto the same directory as before.\n\n.. code:: bash\n\n tensorboard --logdir results --run_neuron_profiler\n\n.. note::\n\n If using TensorBoard >= 2.5, please use the ``--load_fast=false`` option when launching.\n ``tensorboard --logdir results --run_neuron_profiler --load_fast=false``\n\n5.3. After you see the following message, TensorBoard is ready to use. By default,\nTensorBoard will be launched at ``localhost:6006`` on the Deployment Instance.\n\n::\n\n ...\n Running neuron-profile\n Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n TensorBoard 2.4.1 at http://localhost:6006/ (Press CTRL+C to quit)\n\nView results in TensorBoard\n---------------------------\n\nIn this step, we will view the Neuron plugin for TensorBoard from a browser on your local\ndevelopment machine.\n\n6.1. Connect to the Deployment Instance while enabling port forwarding. In this example, we\nassume TensorBoard has been launched using the default address ``localhost:6006`` on the\nDeployment Instance.\n\n.. code:: bash\n\n # if Ubuntu-based AMI\n ssh -i ubuntu@ -L 6006:localhost:6006\n\n # if AL2-based AMI\n ssh -i ec2-user@ -L 6006:localhost:6006\n\n6.2. In a browser, visit |tensorboard_address|.\n\n6.3. In the top navigation bar, switch from ``Graphs`` to ``Neuron``. If it does not show up,\nplease wait a while and refresh the page while the plugin loads. If the issue persists, check\nthe ``Inactive`` dropdown list on the right and check for ``Neuron``.\n\n|image1|\n\n6.4. If TensorBoard failed to find the generated logs, you will see the following message:\n\n|image10|\n\n\nIn this case, please check the console output on the Deployment Instance where TensorBoard was\nlaunched for any warnings or error messages, and make sure the version of the ``aws-neuron-tools``\npackage is compatible.\n\n\n.. _tensorboard-plugin-visualize-graph:\n\nVisualize graphs executed on Neuron\n-----------------------------------\n\n.. _tensorboard-plugin-graph-device:\n\nShow how the graph was partition to run on NeuronCores\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo view how the graph was partitioned to run on NeuronCores, select \"Device\" under \"Graph Color\nSchemes\" in the left navigation bar.\n\n|image2|\n\nEach operator will be colored according to the device used. In this example, light blue indicates\nan operator was executed on CPU, and orange indicates the operator was executed on NeuronCores.\nOperators that are white may have been optimized by the Neuron compiler and fused into another\noperation.\n\n.. _tensorboard-plugin-graph-time:\n\nInspect which operators consumes the most time\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nYou can also view how long each operator took by changing to the \"Compute time\" color scheme.\n\n|image3|\n\nThis view will show time taken by each layer and will be colored according to how much relative\ntime the layer took to compute. A lighter shade of red means that a relatively small portion of\ncompute time was spent in this layer, while a darker red shows that more compute time was used.\n\n.. _tensorboard-plugin-graph-supported-ops:\n\nCheck out Neuron support operators for each framework\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe \"Compatibility\" color scheme allows you to better understand what operators are currently\nsupported by the Neuron compiler - green for compatible ops, red for incompatible ops, and yellow\nfor subgraphs that contain both compatible and incompatible ops.\n\n|image4|\n\n.. _tensorboard-plugin-graph-filter-device:\n\nFilter view by device\n^^^^^^^^^^^^^^^^^^^^^\n\nAdditionally, you can choose to filter by CPU and NeuronCores, which will only color ops that\nmatch the selected device(s).\n\n|image5|\n\nExpand/collapse subgraphs and view operator details\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nEach rectangular node in the graph represents a subgraph that can be expanded or collapse by\nclicking on the name. Operators will be represented by ellipses, and can be clicked to reveal\nmore information on that operator, such as inputs and execution device.\n\n|image11|\n\nThe ``Expand All`` and ``Collapse All`` buttons can be used to expand or collapse every subgraph.\nWhen using these features, the positioning of the graph may change when redrawing the new graph.\nTry using ``Reset Position`` button and zoom out by scrolling if the graph appears to be missing.\n\n.. _tensorboard-plugin-view-profile:\n\nViewing the Neuron profile data\n-------------------------------\n\nOn the right side of the Neuron plugin, information on the profiled inference will be displayed.\n\n.. _tensorboard-plugin-profile-summary:\n\nSee performance summary\n^^^^^^^^^^^^^^^^^^^^^^^\n\nFirst is the \"Neuron Performance Summary,\" which gives a quick overview on how Neuron executed the graph,\nincluding information on the number of NeuronCores and both on-NeuronCore time and on-CPU time.\n\n|image6|\n\n.. _tensorboard-plugin-profile-nc:\n\nGet a breakdown of time spent per NeuronCore\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNext, the \"Neuron Execution\" will give more details on how a graph was partitioned for Neuron.\nEach entry in the table will show the order it was executed in, what type of device was used, the compute\ntime (in microseconds), and the percentage of total time spent. To dive deeper into subgraphs, you can\ncheck the \"Show Details\" box to display the breakdown per NeuronCore.\n\n|image7|\n\n.. _tensorboard-plugin-profile-op:\n\nGet a breakdown of time spent per operator\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe \"Op Time Table\" section shows the cycle count per operator, much like the \"Compute time\" coloring\nfor graph visualization. This table can be sorted by clicking the column names, and searched using the \nprovided text box in the top right corner. Due to Neuron compiler optimizations, some of the compute may\nnot be associated with any specific operator and will be categorized as ``unknown``. Additionally, time\nspent moving data to and from NeuronCores will fall under ``(ND_ENGINE_LOAD)``.\n\n|image8|\n\n\n\n.. |image1| image:: /images/tb-plugin-img1.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image2| image:: /images/tb-plugin-img2.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image3| image:: /images/tb-plugin-img3.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image4| image:: /images/tb-plugin-img4.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image5| image:: /images/tb-plugin-img5.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image6| image:: /images/tb-plugin-img6.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image7| image:: /images/tb-plugin-img7.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image8| image:: /images/tb-plugin-img8.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image9| image:: /images/tb-plugin-img9.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image10| image:: /images/tb-plugin-img10.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image11| image:: /images/tb-plugin-img11.png\n :height: 2826\n :width: 5341\n :scale: 10%\n.. _PyTorch-Neuron Resnet50 Tutorial: ../../src/examples/pytorch/resnet50.ipynb\n.. |tensorboard_address| raw:: html\n\n localhost:6006\n" }, { "path": "archive/tensorflow/index.rst", "content": ".. _tensorflow-neuron-main:\n.. _tensorflow-neuron:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow Neuron\n=================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\nTensorFlow Neuron unlocks high-performance and cost-effective deep learning acceleration on AWS Trainium-based and Inferentia-based Amazon EC2 instances.\n\nTensorFlow Neuron enables native TensorFlow models to be accelerated on Neuron devices, so you can use your existing framework application and get started easily with minimal code changes.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n /archive/tensorflow/tensorflow-setup\n\n.. toctree::\n :maxdepth: 2\n :hidden:\n\n Inference (Inf2 & Trn1) \n Inference (Inf1) \n\n.. card:: Tensorflow NeuronX for Inference on ``Inf2`` & ``Trn1`` / ``Trn1n``\n :link: inference-tensorflow-neuronx\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n.. card:: Tensorflow Neuron for Inference on ``Inf1``\n :link: inference-tensorflow-neuron\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n" }, { "path": "archive/tensorflow/setup-legacy-inf1-tensorflow.rst", "content": ".. meta::\n :description: Legacy TensorFlow installation guide for AWS Inferentia 1 (Inf1) instances\n :keywords: tensorflow, neuron, inf1, legacy, installation, tensorflow-neuron\n :framework: tensorflow\n :instance-types: inf1\n :status: legacy\n :content-type: legacy-guide\n :date-modified: 2026-03-30\n\nTensorFlow on Inf1 (legacy)\n=============================\n\n.. warning::\n \n **Legacy hardware**: Inf1 instances use NeuronCore v1 with TensorFlow 2.x (``tensorflow-neuron``).\n \n For new projects, use **Inf2, Trn1, Trn2, or Trn3** with PyTorch 2.9+ or JAX 0.7+.\n See :ref:`setup-guide-index` for current setup options.\n\n.. note::\n \n TensorFlow support for Inf2 has reached end of support as of Neuron SDK 2.29.\n See :ref:`announce-eos-tensorflow-inf2` for details.\n\nSetup instructions\n------------------\n\nFor complete Inf1 TensorFlow setup instructions, see the original setup guides:\n\n- :doc:`/archive/tensorflow/tensorflow-neuron/setup/tensorflow-update` - TensorFlow Neuron setup and updates\n- :doc:`/archive/tensorflow/tensorflow-neuron-inference` - Inference on Inf1\n\nThe setup guides cover:\n\n- Ubuntu 20, Ubuntu 22, and Amazon Linux 2 installation\n- DLAMI-based installation\n- Manual pip installation\n- TensorFlow 2.10.1, 2.9.3, and 2.8.4 versions\n\nVerification\n------------\n\nAfter installation, verify with:\n\n.. code-block:: python\n \n import tensorflow as tf\n import tensorflow_neuron\n \n print(f\"TensorFlow version: {tf.__version__}\")\n\n.. code-block:: bash\n \n neuron-ls\n\nNext steps\n----------\n\n- :doc:`/archive/tensorflow/tensorflow-neuron-inference` - Inference tutorials for Inf1\n- :ref:`setup-guide-index` - Current setup options (Inf2, Trn1, Trn2, Trn3)\n" }, { "path": "archive/tensorflow/tensorflow-neuron/additional-examples.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nAdditional Examples (``tensorflow-neuron``)\n===========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n AWS Neuron Samples GitHub Repository \n\n\n.. include:: /archive/tensorflow/tensorflow-neuron/additional-examples.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuron/additional-examples.txt", "content": "* `AWS Neuron Samples GitHub Repository `_\n" }, { "path": "archive/tensorflow/tensorflow-neuron/api-auto-replication-api.rst", "content": ".. _tensorflow-ref-auto-replication-python-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuron``) Auto Multicore Replication (Beta)\n===================================================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe Neuron auto multicore replication Python API enables modifying TensorFlow 2.x\ntraced models so that they can be automatically replicated across multiple cores.\nFor Tensorflow-Serving models and TensorFlow 1.x models, see :ref:`tensorflow-ref-auto-replication-cli-api`\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nTensorFlow 2.x (``tensorflow-neuron TF2.x``) Auto Multicore Replication Python API (Beta)\n-----------------------------------------------------------------------------------------------------------\n\nMethod\n^^^^^^\n\n``tensorflow.neuron.auto_multicore``\n\nDescription\n^^^^^^^^^^^\n\nConverts an existing AWS-Neuron-optimized ``keras.Model`` and returns an auto-replication tagged\nAWS-Multicore-Neuron-optimized ``keras.Model`` that can execute on AWS Machine Learning Accelerators.\nLike the traced model, the returned ``keras.Model`` will support inference only. Attributes or\nvariables held by the original function or ``keras.Model`` will be dropped.\n\nThe auto model replication feature in TensorFlow-Neuron enables you to\ncreate a model once and the model parallel replication would happen\nautomatically. The desired number of cores can be less than the total available NeuronCores\non an Inf1 instance but not less than 1. This reduces framework memory usage as you are not\nloading the same model multiple times manually. Calls to the returned model will execute the call\non each core in a round-robin fashion.\n\nThe returned ``keras.Model`` can be exported as SavedModel and served using\nTensorFlow Serving. Please see the TensorFlow Serving documentation for more\ninformation about exporting to saved model and serving using TensorFlow\nServing.\n\nNote that the automatic replication will only work on models compiled with pipeline size 1:\nvia ``--neuroncore-pipeline-cores=1``. If auto replication is not enabled, the model will default to\nreplicate on up to 4 cores.\n\nSee :ref:`neuron-compiler-cli-reference` for more information about compiler options.\n\nArguments\n^^^^^^^^^\n\n- **func:** The ``keras.Model`` or function to be traced.\n- **example_inputs:** A ``tf.Tensor`` or a tuple/list/dict of\n ``tf.Tensor`` objects for tracing the function. When ``example_inputs``\n is a ``tf.Tensor`` or a list of ``tf.Tensor`` objects, we expect\n ``func`` to have calling signature ``func(example_inputs)``. Otherwise,\n the expectation is that inference on ``func`` is done by calling\n ``func(*example_inputs)`` when ``example_inputs`` is a ``tuple``,\n or ``func(**example_inputs)`` when ``example_inputs`` is a ``dict``.\n The case where ``func`` accepts mixed positional and keyword arguments\n is currently unsupported.\n- **num_cores:** The desired number of cores where the model will be automatically\n replicated across\n\nReturns\n^^^^^^^\n\n- An AWS-Multicore-Neuron-optimized ``keras.Model``.\n\n\nExample Python API Usage for TF2.x traced models:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code :: python\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n inputs = [input0]\n outputs = [dense0]\n model = tf.keras.Model(inputs=inputs, outputs=outputs)\n input0_tensor = tf.random.uniform([1, 3])\n model_neuron = tfn.trace(model, input0_tensor)\n\n num_cores = 4\n multicore_model = tfn.auto_multicore(model_neuron, input0_tensor, num_cores=num_cores)\n multicore_model(input0_tensor)\n\nExample Python API Usage for TF2.x saved models:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code :: python\n\n from tensorflow.python import saved_model\n\n input0_tensor = tf.random.uniform([1, 3])\n num_cores = 4\n reload_model = saved_model.load(model_dir)\n multicore_model = tfn.auto_multicore(reload_model, input0_tensor, num_cores=num_cores)\n\n.. _tensorflow-ref-auto-replication-cli-api:\n\nTensorFlow Neuron 2.x (``tensorflow-neuron``) Auto Multicore Replication CLI (Beta)\n---------------------------------------------------------------------------------------------------------------\n\nThe Neuron auto multicore replication CLI enables modifying TensorFlow 1.x and Tensorflow 2.x\ntraced saved models so that they can be automatically replicated across multiple cores. By performing\nthis call on Tensorflow Saved Models, we can support both Tensorflow-Serving and Tensorflow 1.x\nwithout significant modifications to the code. Note that the python API does not support Tensorflow 1.x.\n\nMethod\n^^^^^^\n\n``tf-neuron-auto-multicore MODEL_DIR --num_cores NUM_CORES --new_model_dir NEW_MODEL_DIR``\n\nArguments\n^^^^^^^^^\n\n- **MODEL_DIR:** The directory of a saved AWS-Neuron-optimized ``keras.Model``.\n- **NUM_CORES:** The desired number of cores where the model will be automatically\n replicated across\n- **NEW_MODEL_DIR:** The directory of where the AWS-Multicore-Neuron-optimized\n ``keras.Model`` will be saved\n" }, { "path": "archive/tensorflow/tensorflow-neuron/api-compilation-python-api.rst", "content": ".. _tensorflow-ref-neuron-compile-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 1.x (``tensorflow-neuron``) Compilation API\n=======================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe Neuron compilation API for TensorFlow 1.x enables compilation of saved\nmodel to an Inferentia target.\n\nMethod\n------\n\n``tensorflow.neuron.saved_model.compile``\n\nDescription\n-----------\n\nWithin the graph or subgraph, the compile method selects and send\nNeuron-supported operations to Neuron-Compiler for compilation and saves\nthe compiled artifacts in the graph. Uncompilable operations are kept as\noriginal operations for framework execution.\n\nThe compiled graph can be exported to saved model and served using\nTensorFlow Serving. Please see the TensorFlow Serving documentation for more\ninformation about exporting to saved model and serving using TensorFlow\nServing.\n\nOptions can be passed to Neuron compiler via the compile function. For\nexample, the “\\ ``--neuroncore-pipeline-cores``\\ ” option directs Neuron\ncompiler to compile each subgraph to fit in the specified number of\nNeuronCores. This number can be less than the total available\nNeuronCores on an Inf1 instance. See :ref:`neuron-compiler-cli-reference`\nfor more information about compiler options.\n\nArguments\n---------\n\n- **model_dir:** The path of the original ``SavedModel``.\n- **new_model_dir:** The path to which the Neuron-optimized\n ``SavedModel`` will be stored.\n- **batch_size:** (Optional) Positive integer representing batch size\n used in inference. The default value is 1.\n- **model_shape_feed_dict:** (Optional) Dictionary {str: list} used for\n inferring tensor shapes. Keys should match model input names. Values\n are lists of positive integers representing model input tensor\n shapes.\n- **model_feed_dict:** (Optional) Dictionary {str: numpy.array} used\n for inference. Useful for inferring tensor shapes. Keys should match\n model input names. Values are numpy arrays that can be fed as inputs\n to the ``SavedModel``.\n- **tags:** (Optional) Iterable of strings to identify the required\n ``MetaGraphDef``. These should correspond to the tags used when\n saving the variables using the ``SavedModel`` ``save()`` API. Default\n is to use the first ``tag_set`` available in the ``SavedModel``.\n- **signature_def_key:** (Optional) String specifying the\n ``signature_def`` to use. Default is to use 'serving_default' or the\n first ``signature_def`` corresponding to ``tags``.\n- **minimum_segment_size:** (Optional) Integer indicating the minimum\n number of operations in an NeuronOp.\n- **no_fuse_ops:** (Optional) None or iterable of strings (unordered)\n representing names of operations that are forcibly placed on CPU.\n- **compiler_args:** (Optional) List of strings representing neuron-cc\n compiler arguments. Note that these arguments apply to all subgraphs\n generated by whitelist partitioning. For example, use\n ``compiler_args=['--neuroncore-pipeline-cores', '4']`` to set number\n of NeuronCores per subgraph to 4. See :ref:`neuron-compiler-cli-reference`\n for more information about compiler options.\n- **compiler_workdir:** (Optional) String representing work directory\n of the neuron-cc compiler.\n\nReturns\n-------\n\n- Dictionary with operator counts before/after optimization.\n- Operator count statistics are displayed to show original count,\n post-optimization count, and the number placed on Neuron runtime. For\n example:\n\n::\n\n INFO:tensorflow:Number of operations in TensorFlow session: 3978\n INFO:tensorflow:Number of operations after tf.neuron optimizations: 555\n INFO:tensorflow:Number of operations placed on Neuron runtime: 554\n\nExample Usage\n-------------\n\n.. code:: python\n\n import shutil\n import tensorflow.neuron as tfn\n saved_model_path = \"\"\n compiled_saved_model_path = \"\"\n shutil.rmtree(compiled_saved_model_path, ignore_errors=True)\n tfn.saved_model.compile(saved_model_path, compiled_saved_model_path)\n" }, { "path": "archive/tensorflow/tensorflow-neuron/api-reference-guide.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nAPI Reference Guide (``tensorflow-neuron``)\n===========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /archive/tensorflow/tensorflow-neuron/api-tracing-python-api\n /archive/tensorflow/tensorflow-neuron/api-tfn-analyze-model-api\n /archive/tensorflow/tensorflow-neuron/api-auto-replication-api\n\n\n.. include:: /archive/tensorflow/tensorflow-neuron/api-reference-guide.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuron/api-reference-guide.txt", "content": "* :ref:`tensorflow-ref-neuron-tracing-api`\n* :ref:`tensorflow-ref-neuron-analyze_model-api`\n* :ref:`tensorflow-ref-auto-replication-python-api`" }, { "path": "archive/tensorflow/tensorflow-neuron/api-tfn-analyze-model-api.rst", "content": ".. _tensorflow-ref-neuron-analyze_model-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuron``) analyze_model API\n========================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nMethod\n------\n\n``tensorflow.neuron.analyze_model``\n\nDescription\n-----------\n\nAnalyzes a ``keras.Model`` or a Python callable that can be decorated by\n``tf.function`` for it's compatibility with Neuron. It displays supported \nvs. unsupported operators in the model as well as percentages and counts of \neach operator and returns a dictionary with operator statistics.\n\nArguments\n---------\n\n- **func:** The ``keras.Model`` or function to be analyzed.\n- **example_inputs:** A ``tf.Tensor`` or a tuple/list/dict of\n ``tf.Tensor`` objects for tracing the function. When ``example_inputs``\n is a ``tf.Tensor`` or a list of ``tf.Tensor`` objects, we expect\n ``func`` to have calling signature ``func(example_inputs)``. Otherwise,\n the expectation is that inference on ``func`` is done by calling\n ``func(*example_inputs)`` when ``example_inputs`` is a ``tuple``,\n or ``func(**example_inputs)`` when ``example_inputs`` is a ``dict``.\n The case where ``func`` accepts mixed positional and keyword arguments\n is currently unsupported.\n\nReturns\n-------\n\n- A results ``dict`` with these keys: ``'percent_supported', 'supported_count', \n 'total_count', 'supported_operators', 'unsupported_operators', 'operators', \n 'operator_count'``.\n\nExample Usage\n-------------\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n model = tf.keras.Model(inputs=[input0], outputs=[dense0])\n example_inputs = tf.random.uniform([1, 3])\n results = tfn.analyze_model(model, example_inputs)\n print(results)\n\n # expected output\n '''\n BiasAdd\n\tMatMul\n\t100.00% of all operations (2 of 2) are supported\n\t{'percent_supported': 100.0, 'supported_count': 2, 'total_count': 2, \n\t'supported_operators': {'BiasAdd', 'MatMul'}, 'unsupported_operators': [], \n\t'operators': ['BiasAdd', 'MatMul'], 'operator_count': {'MatMul': 1, 'BiasAdd': 1}}\n\t'''\n" }, { "path": "archive/tensorflow/tensorflow-neuron/api-tracing-python-api.rst", "content": ".. _tensorflow-ref-neuron-tracing-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuron``) Tracing API\n===================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe Neuron tracing API enables tracing TensorFlow 2.x models for deployment\non AWS Machine Learning Accelerators.\n\nMethod\n------\n\n``tensorflow.neuron.trace``\n\nDescription\n-----------\n\nTrace a ``keras.Model`` or a Python callable that can be decorated by\n``tf.function``, and return an AWS-Neuron-optimized ``keras.Model`` that\ncan execute on AWS Machine Learning Accelerators. Tracing is ideal for\n``keras.Model`` that accepts a list of ``tf.Tensor`` objects and returns\na list of ``tf.Tensor`` objects. It is expected that users will provide\nexample inputs, and the ``trace`` function will execute ``func``\nsymbolically and convert it to a ``keras.Model``.\n\nThe returned ``keras.Model`` will support inference only. Attributes or\nvariables held by the original function or ``keras.Model`` will be dropped.\n\nThe returned ``keras.Model`` can be exported as SavedModel and served using\nTensorFlow Serving. Please see the TensorFlow Serving documentation for more\ninformation about exporting to saved model and serving using TensorFlow\nServing.\n\nThe returned ``keras.Model`` has an ``.on_neuron_ratio`` attribute\nwhich shows the percentage of ops mapped to neuron hardware. This calculation\nignores PlaceholerOp, IdentityOp, ReadVariableOp and NoOp.\n\nOptions can be passed to Neuron compiler via the environment variable\n``NEURON_CC_FLAGS``. For example, the syntax\n``env NEURON_CC_FLAGS=\"--neuroncore-pipeline-cores=4\"`` directs Neuron\ncompiler to compile each subgraph to fit in the specified number of\nNeuronCores. This number can be less than the total available NeuronCores\non an Inf1 instance. See :ref:`neuron-compiler-cli-reference` for more\ninformation about compiler options.\n\nArguments\n---------\n\n- **func:** The ``keras.Model`` or function to be traced.\n- **example_inputs:** A ``tf.Tensor`` or a tuple/list/dict of\n ``tf.Tensor`` objects for tracing the function. When ``example_inputs``\n is a ``tf.Tensor`` or a list of ``tf.Tensor`` objects, we expect\n ``func`` to have calling signature ``func(example_inputs)``. Otherwise,\n the expectation is that inference on ``func`` is done by calling\n ``func(*example_inputs)`` when ``example_inputs`` is a ``tuple``,\n or ``func(**example_inputs)`` when ``example_inputs`` is a ``dict``.\n The case where ``func`` accepts mixed positional and keyword arguments\n is currently unsupported.\n- **subgraph_builder_function:** (Optional) A callable with signature\n\n ``subgraph_builder_function(node : NodeDef) -> bool``\n (``NodeDef`` is defined in tensorflow/core/framework/node_def.proto)\n\n that is used as a call-back function to determine which part of\n the tensorflow GraphDef given by tracing ``func`` will be placed on\n Machine Learning Accelerators.\n\n If ``subgraph_builder_function`` is not provided, then ``trace`` will\n automatically place operations on Machine Learning Accelerators or\n on CPU to maximize the execution efficiency.\n\n If it is provided, and ``subgraph_builder_function(node)`` returns\n ``True``, and placing ``node`` on Machine Learning Accelerators\n will not cause deadlocks during execution, then ``trace`` will place\n ``node`` on Machine Learning Accelerators. If\n ``subgraph_builder_function(node)`` returns ``False``, then ``trace``\n will place ``node`` on CPU.\n\nSpecial Flags\n-------------\n\nThese are flags that get passed directly to the Neuron tracing API\n(rather than the Neuron Compiler). The flags are still passed\nvia the environment variable ``NEURON_CC_FLAGS``.\n\n- **workdir:** example usage - ``NEURON_CC_FLAGS='--workdir ./artifacts'``\n will create a folder named artifacts in the current directory and\n save artifacts that can be used for debug.\n- **dynamic-batch-size:** example usage -\n ``NEURON_CC_FLAGS='--dynamic-batch-size'`` A flag to allow Neuron graphs to\n consume variable sized batches of data. Dynamic sizing is restricted to the\n 0th dimension of a tensor.\n- **extract-weights (Beta):** example usage -\n ``NEURON_CC_FLAGS='--extract-weights inf1.2xlarge'`` will reduce the compiled\n model's protobuf size by taking the weights out of the protobuf.\n Useful for compiling large models that would exceed the 2GB protobuf\n size limit. This feature is in beta. Model performance is not\n guaranteed and the flag does not work in combination with\n ``--neuroncore-pipeline-cores``, ``--dynamic-batch-size``, models with\n multiple NEFFs, and models that are 4GB or greater. \n Compiles models for different neuron instances depending on the instance type passed.\n Supports all inf1 instance types.\n\nReturns\n-------\n\n- An AWS-Neuron-optimized ``keras.Model``.\n\n\nExample Usage\n-------------\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n model = tf.keras.Model(inputs=[input0], outputs=[dense0])\n example_inputs = tf.random.uniform([1, 3])\n model_neuron = tfn.trace(model, example_inputs) # trace\n # check to see how much of the model was compiled successfully\n print(model_neuron.on_neuron_ratio) \n\n model_dir = './model_neuron'\n model_neuron.save(model_dir)\n model_neuron_reloaded = tf.keras.models.load_model(model_dir)\n\n\nExample Usage with Manual Device Placement Using ``subgraph_builder_function``\n------------------------------------------------------------------------------\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n reshape0 = tf.keras.layers.Reshape([1, 3])(dense0)\n output0 = tf.keras.layers.Dense(2)(reshape0)\n model = tf.keras.Model(inputs=[input0], outputs=[output0])\n example_inputs = tf.random.uniform([1, 3])\n\n def subgraph_builder_function(node):\n return node.op == 'MatMul'\n\n model_neuron = tfn.trace(\n model, example_inputs,\n subgraph_builder_function=subgraph_builder_function,\n )\n\n.. important ::\n\n Although the old API ``tensorflow.neuron.saved_model.compile`` is still available under tensorflow-neuron 2.x,\n it supports only the limited capabilities of ``tensorflow.neuron.trace`` and will be deprecated in future releases.\n" }, { "path": "archive/tensorflow/tensorflow-neuron/dlc-then-ec2-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/dlc-then-ec2-devflow.rst\n" }, { "path": "archive/tensorflow/tensorflow-neuron/dlc-then-ecs-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/dlc-then-ecs-devflow.rst\n" }, { "path": "archive/tensorflow/tensorflow-neuron/dlc-then-eks-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/dlc-then-eks-devflow.rst\n" }, { "path": "archive/tensorflow/tensorflow-neuron/ec2-then-ec2-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/ec2-then-ec2-devflow.rst\n" }, { "path": "archive/tensorflow/tensorflow-neuron/misc-tensorflow-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nMisc (``tensorflow-neuron``)\n============================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /release-notes/archive/tensorflow/tensorflow-neuron/tensorflow-neuron-v2 \n /archive/tensorflow/tensorflow-neuron/tensorflow2-accelerated-ops\n\n\n.. include:: /archive/tensorflow/tensorflow-neuron/misc-tensorflow-neuron.txt\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/misc-tensorflow-neuron.txt", "content": "* :ref:`tensorflow-neuron-rn-v2`\n* :ref:`tensorflow-ref-neuron-accelerated-ops`" }, { "path": "archive/tensorflow/tensorflow-neuron/neo-then-hosting-devflow.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n.. include:: /devflows/inference/neo-then-hosting-devflow.rst\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.14.2-tensorflow-install.rst", "content": ".. _install-neuron-1.14.2-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron (Neuron 1.14.2)\n======================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.15.0-tensorflow-install.rst", "content": ".. _install-neuron-1.15.0-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron (Neuron 1.15.0)\n======================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n \n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.15.1-tensorflow-install.rst", "content": ".. _install-neuron-1.15.1-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron (Neuron 1.15.1)\n=========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n \n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.15.2-tensorflow-install.rst", "content": ".. _install-neuron-1.15.2-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron (Neuron 1.15.2)\n=========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n \n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.4.2\n\n\n .. tab-item:: TensorFlow 2.3.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.3.3\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.16.3-tensorflow-install.rst", "content": ".. _install-neuron-1.16.3-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n \n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.17.0-tensorflow-install.rst", "content": ".. _install-neuron-1.17.0-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.0\n\n\n\n \n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.0\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.0\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.17.1-tensorflow-install.rst", "content": ".. _install-neuron-1.17.1-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.1\n\n\n\n \n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.17.2-tensorflow-install.rst", "content": ".. _install-neuron-1.17.2-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n \n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.5.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n\n\n .. tab-item:: TensorFlow 2.4.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.4.3\n\n\n .. tab-item:: TensorFlow 2.3.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.3.4\n\n\n .. tab-item:: TensorFlow 2.2.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.2.3\n\n\n .. tab-item:: TensorFlow 2.1.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-2.1.4 \n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.18.0-tensorflow-install.rst", "content": ".. _install-neuron-1.18.0-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n \n .. tab-item:: TensorFlow 2.6.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n\n .. tab-item:: TensorFlow 2.5.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n\n\n .. tab-item:: TensorFlow 2.6.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n\n .. tab-item:: TensorFlow 2.5.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n\n\n .. tab-item:: TensorFlow 2.6.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.6.3\n\n\n .. tab-item:: TensorFlow 2.5.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-2.5.3\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.19.0-tensorflow-install.rst", "content": ".. _install-neuron-1.19.0-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n\n \n .. tab-item:: TensorFlow 2.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n\n\n \n .. tab-item:: TensorFlow 2.6.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n\n .. tab-item:: TensorFlow 2.5.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5 \n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: TensorFlow 2.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n\n\n\n\n .. tab-item:: TensorFlow 2.6.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n\n .. tab-item:: TensorFlow 2.5.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.8.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n\n\n .. tab-item:: TensorFlow 2.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.7.1\n\n\n\n\n\n .. tab-item:: TensorFlow 2.6.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.6.3\n\n\n .. tab-item:: TensorFlow 2.5.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-2.5.3\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install tensorflow --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=tensorflow-1.15.5 \n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-al2023.rst", "content": ".. _tensorflow-neuron-install-prev-al2023:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous Tensorflow Neuron Releases for Ubuntu (``tensorflow-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n \n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-u20.rst", "content": ".. _tensorflow-neuron-install-prev-u20:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous Tensorflow Neuron Releases for Ubuntu (``tensorflow-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n \n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-u22.rst", "content": ".. _tensorflow-neuron-install-prev-u20:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous Tensorflow Neuron Releases for Ubuntu (``tensorflow-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev.rst", "content": ".. _install-prev-neuron-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall previous TensorFlow Neuron releases\n===========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. toctree::\n :maxdepth: 1\n\n Neuron 1.19.0 \n Neuron 1.18.0 \n Neuron 1.17.2 \n Neuron 1.17.1 \n Neuron 1.17.0 \n Neuron 1.16.3 \n Neuron 1.15.2 \n Neuron 1.15.1 \n Neuron 1.15.0 \n Neuron 1.14.2 \n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-install.rst", "content": ".. _install-neuron-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow Neuron\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n .. tab-item:: TensorFlow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n .. tab-item:: TensorFlow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n .. tab-item:: TensorFlow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-update-u20.rst", "content": "\n.. _tensorflow-neuron-u20-update:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest TensorFlow Neuron (``tensorflow-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.10.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-update-u22.rst", "content": "\n.. _tensorflow-neuron-u20-update:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest TensorFlow Neuron (``tensorflow-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: TensorFlow 2.10.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuron/setup/tensorflow-update.rst", "content": ".. _update-neuron-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest TensorFlow Neuron\n===============================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n .. tab-item:: TensorFlow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n \n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n .. tab-item:: TensorFlow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n .. tab-item:: TensorFlow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n .. tab-item:: TensorFlow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=2.7.4 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: TensorFlow 1.15.5\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=tensorflow --framework-version=1.15.5 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tensorflow2-accelerated-ops.rst", "content": ".. _tensorflow-ref-neuron-accelerated-ops:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuron``) Accelerated (``torch-neuron``) Python APIs and Graph Ops\n======================================================================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThis page lists TensorFlow 2.x Python APIs and graph operators that are\naccelerated by AWS Neuron. The lists are not exhaustive. TensorFlow 2.x Python\nAPIs or graph operators that are not listed here may still be accelerated if\nthey are composed of accelerated primitives, or they will be executed on CPU\nwithout significant acceleration. The TensorFlow Neuron integration contains\nan automatic operator-device-placement mechanism that strives to maximize\nthe execution efficiency of your deep learning models on AWS Machine Learning\nASIC instances.\n\nAccelerated Python APIs\n--------------------------------\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| Module | Accelerated Python API | Comments |\n+===============+===================================+===========================================================+\n| ``tf`` | ``tf.abs`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.add`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.add_n`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.broadcast_static_shape`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.cast`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.constant`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.convert_to_tensor`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.cumsum`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.einsum`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.erf`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.exp`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.identity`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.matmul`` | Uses float16/bfloat16 matmul with float32 accumulation. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.maximum`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.minimum`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.multiply`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.negative`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.range`` | ``start``, ``limit`` and ``delta`` arguments must be |\n| | | compile-time constants. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.realdiv`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reciprocal`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reduce_all`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reduce_any`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reduce_max`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reduce_min`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reduce_prod`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reduce_sum`` | ``axis`` must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.reshape`` | ``shape`` argument must be a compile-time constant. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.rsqrt`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.scalar_mul`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.shape`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.shape_n`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.sigmoid`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.size`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.slice`` | ``size`` must be a compile-time constant. In addition, |\n| | | |\n| | | either ``begin`` must be a compile-time constant or |\n| | | |\n| | | ``size`` must be non-negative. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.sqrt`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.square`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.squared_difference`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.squeeze`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.stack`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.stop_gradient`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.strided_slice`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.tanh`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.tensordot`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.to_bfloat16`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.to_float`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.truediv`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| ``tf.layers`` | ``tf.layers.batch_normalization`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.layers.dense`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.layers.flatten`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| ``tf.nn`` | ``tf.nn.batch_normalization`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.bias_add`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.dropout`` | Always treated as ``tf.identity`` during inference. |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.fused_batch_norm`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.leaky_relu`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.relu`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.relu6`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.relu_layer`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n| | ``tf.nn.softmax`` | |\n+---------------+-----------------------------------+-----------------------------------------------------------+\n\nAccelerated graph operators\n--------------------------------\n.. code:: python\n\n Add\n AddN\n AddV2\n BatchMatMul\n BatchMatMulV2\n BiasAdd\n Cast\n Const\n Cumsum\n Einsum\n Erf\n Exp\n ExpandDims\n FusedBatchNorm\n FusedBatchNormV2\n FusedBatchNormV3\n Greater\n Identity\n LeakyRelu\n MatMul\n Max\n Maximum\n Minimum\n Mean\n Mul\n Neg\n Pack\n RealDiv\n Relu\n Relu6\n Reshape\n Rsqrt\n Sigmoid\n Softmax\n Split\n SplitV\n Sqrt\n Square\n SquaredDifference\n Squeeze\n StridedSlice\n Sub\n Sum\n Tanh\n Transpose\n Unpack\n\n\nThe lists share many commonalities with `Available TensorFlow Ops `_. Portions of this page are modifications based on work created and `shared by Google `_ and used according to terms described in the `Creative Commons 4.0 Attribution License `_.\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tf2_faq.rst", "content": ".. _tf2_faq:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x FAQ\n===================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\n\nHow do I get started with TensorFlow?\n-------------------------------------\n\nThe easiest entry point is the tutorials offered by the AWS Neuron team. For beginners, the :ref:`HuggingFace DistilBERT Tutorial ` is a good place to start.\n\nWhat TensorFlow versions are supported by Neuron?\n-------------------------------------------------\n\nThe AWS Neuron provide well-tested tensorflow-neuron packages that work with a range of tensorflow official releases, as long as the version of tensorflow-neuron matches that of tensorflow. For example, you may install ``tensorflow-neuron==2.3.3.1.0.9999.0`` on top of ``tensorflow==2.3.3`` and expect them to work together.\n\nCurrently, tensorflow-neuron can work with tensorflow versions 2.1.4, 2.2.3, 2.3.3, 2.4.2, 2.5.0.\n\nIn a fresh Python environment, ``pip install tensorflow-neuron`` would bring in the highest version (2.5.0 as of 07/13/2021), which then pulls ``tensorflow==2.5.0`` into the current environment.\n\nIf you already have a particular version of tensorflow 2.x installed, then it is recommended to pay attention to the precise version of tensorflow-neuron and only install the desired one. For example, in an existing Python environment with ``tensorflow==2.3.3`` installed, you may install tensorflow-neuron by pip install ``tensorflow-neuron==2.3.3``, which will reuse the existing tensorflow installation.\n\nWhat operators are supported?\n-----------------------------\n\nDue to fundamental backend design changes in the TensorFlow 2.x framework, the concept of \"supported graph operators\" is no longer well-defined. Please refer to :ref:`Accelerated Python APIs and graph operators ` for a guide to the set of TensorFlow 2.x Python APIs and graph operators that can be accelerated by Neuron.\n\nHow do I compile my model?\n--------------------------\n\nIt is achieved by a new public API called tfn.trace, which resembles the compilation API of AWS PyTorch Neuron integration. Programmatically, customers would be able to execute the following code.\n\n.. code::\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n\n ...\n model = tf.keras.Model(inputs=inputs, outputs=outputs)\n model_neuron = tfn.trace(model, example_inputs)\n model_neuron.save('./model_neuron_dir')\n ...\n model_loaded = tf.saved_model.load('./model_dir')\n predict_func = model_loaded['serving_default']\n model_loaded_neuron = tfn.trace(predict_func, example_inputs2)\n model_loaded_neuron.save('./model_loaded_neuron_dir')\n ...\n\nHow do I deploy my model?\n-------------------------\n\nPython tensorflow\n^^^^^^^^^^^^^^^^^\n\nPre-compiled models can be saved and reloaded back into a Python environment using regular tensorflow model loading APIs, as long as tensorflow-neuron is installed.\n\n.. code::\n\n import tensorflow as tf\n\n model = tf.keras.models.load_model('./model_loaded_neuron_dir')\n example_inputs = ...\n output = model(example_inputs)\n\ntensorflow-serving\n^^^^^^^^^^^^^^^^^^\n\nPre-compiled models can be saved into SavedModel format via tensorflow SavedModel APIs\n\n.. code::\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n\n ...\n model = tf.keras.Model(inputs=inputs, outputs=outputs)\n model_neuron = tfn.trace(model, example_inputs)\n tf.saved_model.save(model_neuron, './model_neuron_dir/1')\n\nThe generated SavedModel './model_neuron_dir' can be loaded into tensorflow-model-server-neuron, which can be installed through apt or yum based on the type of the operating system. For example, on Ubuntu 18.04 LTS the following command installs and launches a tensorflow-model-server-neuron on a pre-compiled SavedModel.\n\n.. code::\n\n sudo apt install tensorflow-model-server-neuron\n # --model_base_path needs to be an absolute path\n tensorflow_model_server_neuron --model_base_path=$(pwd)/model_neuron_dir\n\nWhere can I find tutorials and examples ?\n-----------------------------------------\n\n:ref:`HuggingFace DistilBERT Tutorial ` is a good place to start.\n\n\nHow to debug or profile my model?\n---------------------------------\n\n:ref:`AWS Neuron TensorBoard integration ` provides visibility into what is happening inside of the Neuron runtime, and allows a more fine-grained (but also more hardware-awared) reasoning on where to improve the performance of machine learning applications.\n\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/bert_demo/bert_demo.rst", "content": ".. _tensorflow-bert-demo:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n[Broken] Running TensorFlow BERT-Large with AWS Neuron\n=============================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThis example shows a Neuron compatible BERT-Large implementation that is\nfunctionally equivalent to open source BERT-Large model. This demo uses\nTensorFlow-Neuron, BERT-Large weights fine tuned for MRPC and also shows\nthe performance achieved by the Inf1 instance. For users who want to use\npublic BERT SavedModels please also follow the steps described :ref:`using-public-bert-savedmodels`.\n\nLaunch EC2 instances\n--------------------\n\nFor this demo, launch two EC2 instances :\n\n- a c5.4xlarge instance for compiling the BERT-Large Model and\n- an inf1.xlarge instance for running inference\n\nFor both of these instances choose the latest Ubuntu 18 Deep Learning\nAMI (DLAMI).\n\n.. _compiling-neuron-compatible-bert-large:\n\nCompiling Neuron compatible BERT-Large\n--------------------------------------\n\nFirst connect to a c5.4xlarge instance and update tensorflow-neuron and\nneuron-cc\n\nUpdate compilation EC2 instance\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUpdate to the latest neuron software by executing the instructions at :ref:`install-neuron-tensorflow`.\n\nNote: if your tensorflow-neuron version on the inference instance is\nlower than 1.15.0.1.0.1333.0, you will need to run this demo on\ninf1.2xlarge instead of inf1.xlarge.\n\nCompile open source BERT-Large saved model using Neuron compatible BERT-Large implementation\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron software works with TensorFlow saved models. Users should bring\ntheir own BERT-Large saved model for this section. This demo will run\ninference for the MRPC task and the saved model should be fine tuned for\nMRPC. Users who need additional help to fine-tune the model for MRPC or\nto create a saved model can refer to :ref:`bert-tensorflow-demo-appendix1`.\n\nIn the same environment and directory bert_demo scripts, run the\nfollowing :\n\n.. code:: bash\n\n git clone https://github.com/aws/aws-neuron-sdk\n cd ~/aws-neuron-sdk/src/examples/tensorflow/bert_demo/\n export BERT_LARGE_SAVED_MODEL=\"/path/to/user/bert-large/savedmodel\"\n pip install tensorflow_neuron==1.15.5.2.8.9.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com/\n pip install neuron_cc==1.13.5.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com\n python bert_model.py --input_saved_model $BERT_LARGE_SAVED_MODEL --output_saved_model ./bert-saved-model-neuron --batch_size=6 --aggressive_optimizations\n\nThis compiles BERT-Large pointed to by $BERT_LARGE_SAVED_MODEL for an\ninput size of 128 and batch size of 6. The compilation output is stored\nin bert-saved-model-neuron. Copy this to your Inf1 instance for\ninferencing.\n\nThe bert_model.py script encapsulates all the steps necessary for this\nprocess. For details on what is done by bert_model.py please refer to\n:ref:`bert-tensorflow-demo-appendix2`.\n\nRunning the inference demo\n--------------------------\n\nConnect to your inf1.xlarge instance and update tensorflow-neuron,\naws-neuron-runtime and aws-neuron-tools.\n\nUpdate inference EC2 instance\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUpdate to the latest neuron software by executing the instructions at :ref:`install-neuron-tensorflow`.\n\nLaunching the BERT-Large demo server\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCopy the compiled model (bert-saved-model-neuron) from your c5.4xlarge\nto your inf1.xlarge instance. Place the model in the same directory as\nthe bert_demo scripts. Then from the same conda environment launch the\nBERT-Large demo server :\n\n.. code:: bash\n\n cd ~/aws-neuron-sdk/src/examples/tensorflow/bert_demo/\n pip install tensorflow_neuron==1.15.5.2.8.9.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com/\n python bert_server.py --dir bert-saved-model-neuron --batch 6 --parallel 4\n\nThis loads 4 BERT-Large models, one into each of the 4 NeuronCores found\nin an inf1.xlarge instance. For each of the 4 models, the BERT-Large\ndemo server opportunistically stitches together asynchronous requests\ninto batch 6 requests. When there are insufficient pending requests, the\nserver creates dummy requests for batching.\n\nWait for the bert_server to finish loading the BERT-Large models to\nInferentia memory. When it is ready to accept requests it will print the\ninferences per second once every second. This reflects the number of\nreal inferences only. Dummy requests created for batching are not\ncredited to inferentia performance. Once the inferences are done you can send\na keyboard interrupt to print out the average throughput of your run.\n\nSending requests to server from multiple clients\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWait until the bert demo server is ready to accept requests. Then on the\nsame inf1.xlarge instance, launch a separate linux terminal. From the\nbert_demo directory execute the following commands :\n\n.. code:: bash\n\n source activate aws_neuron_tensorflow_p36\n cd ~/aws-neuron-sdk/src/examples/tensorflow/bert_demo/\n for i in {1..96}; do python bert_client.py --cycle 128 & done\n\nThis spins up 96 clients, each of which sends 128 inference requests.\n\n\nPrinting latency metrics\n~~~~~~~~~~~~~~~~~~~~~~~~\nAfter all your requests have been sent to your server you can\nrun the following command:\n\n.. code:: bash\n\n python latency_printer.py\n\n.. _using-public-bert-savedmodels:\n\nUsing public BERT SavedModels\n-----------------------------\n\nWe are now providing a compilation script that has better compatibility\nwith various flavors of BERT SavedModels generated from\nhttps://github.com/google-research/bert. Here are the current\nlimitations:\n\n1. You did not change\n `modeling.py `__\n2. BERT SavedModel is generated using ``estimator.export_saved_model``\n3. BERT SavedModel uses fixed sequence length 128 (you may check by\n ``saved_model_cli show --dir /path/to/user/bert/savedmodel --all``)\n4. ``neuron-cc`` version is at least 1.0.12000.0\n5. ``aws-neuron-runtime`` version is at least 1.0.7000.0\n6. The ``--batch_size`` argument specified in this script is at most 4\n\nExample usage is shown below:\n\n.. code:: bash\n\n export BERT_LARGE_SAVED_MODEL=\"/path/to/user/bert-large/savedmodel\"\n cd ~/aws-neuron-sdk/src/examples/tensorflow/bert_demo/\n python bert_no_model.py --input_saved_model $BERT_LARGE_SAVED_MODEL --output_saved_model ./bert-saved-model-neuron --batch_size=1\n\n.. _bert-tensorflow-demo-appendix1:\n\nAppendix 1\n----------\n\nUsers who need help finetuning BERT-Large for MRPC and creating a saved\nmodel may follow the instructions here.\n\nConnect to the c5.4xlarge compilation EC2 instance you started above and\ndownload these three items :\n\n1. clone `this `__ github repo.\n2. download GLUE data as described\n `here `__.\n Do not run the finetuning command.\n3. download a desired pre-trained BERT-Large checkpoint from\n `here `__.\n This is the model we will fine tune.\n\nNext edit run_classifier.py in the cloned bert repo to apply the patch\ndescribed in the following git diff.\n\n::\n\n diff --git a/run_classifier.py b/run_classifier.py\n index 817b147..c9426bc 100644\n --- a/run_classifier.py\n +++ b/run_classifier.py\n @@ -955,6 +955,18 @@ def main(_):\n drop_remainder=predict_drop_remainder)\n\n result = estimator.predict(input_fn=predict_input_fn)\n + features = {\n + \"input_ids\": tf.placeholder(shape=[None, FLAGS.max_seq_length], dtype=tf.int32, name='input_ids'),\n + \"input_mask\": tf.placeholder(shape=[None, FLAGS.max_seq_length], dtype=tf.int32, name='input_mask'),\n + \"segment_ids\": tf.placeholder(shape=[None, FLAGS.max_seq_length], dtype=tf.int32, name='segment_ids'),\n + \"label_ids\": tf.placeholder(shape=[None], dtype=tf.int32, name='label_ids'),\n + \"is_real_example\": tf.placeholder(shape=[None], dtype=tf.int32, name='is_real_example'),\n + }\n + serving_input_fn = tf.estimator.export.build_raw_serving_input_receiver_fn(features)\n + estimator._export_to_tpu = False ## !!important to add this\n + estimator.export_saved_model(\n + export_dir_base='./bert_classifier_saved_model',\n + serving_input_receiver_fn=serving_input_fn)\n\n output_predict_file = os.path.join(FLAGS.output_dir, \"test_results.tsv\")\n with tf.gfile.GFile(output_predict_file, \"w\") as writer:\n\nNOTE : Users who are interested may refer to this\n`link `__\nfor additional background information on the patch but it is not\nnecessary for running this demo.\n\nThen from the bert_demo directory run the following :\n\n.. code:: bash\n\n source activate aws_neuron_tensorflow_p36\n cd ~/aws-neuron-sdk/src/examples/tensorflow/bert_demo/\n export BERT_REPO_DIR=\"/path/to/cloned/bert/repo/directory\"\n export GLUE_DIR=\"/path/to/glue/data/directory\"\n export BERT_BASE_DIR=\"/path/to/pre-trained/bert-large/checkpoint/directory\"\n ./tune_save.sh\n\nThe a saved model will be created in\n$BERT_REPO_DIR/bert-saved-model/*random_number*/. Where, *random_number*\nis a random number generated for every run. Use this saved model to\ncontinue with the rest of the demo.\n\n.. _bert-tensorflow-demo-appendix2:\n\nAppendix 2\n----------\n\nFor all BERT variants, we currently need to augment the standard Neuron\ncompilation process for performance tuning. In the future, we intend to\nautomate this tuning process. This would allow users to use the standard\nNeuron compilation process, which requires only a one line change in\nuser source code. The standard compilation process is described :ref:`/src/examples/mxnet/resnet50/resnet50.ipynb`.\n\nThe augmented Neuron compilation process is encapsulated by the\nbert_model.py script, which performs the following things :\n\n1. Define a Neuron compatible implementation of BERT-Large. For\n inference, this is functionally equivalent to the open source\n BERT-Large. The changes needed to create a Neuron compatible\n BERT-Large implementation is described in :ref:`bert-tensorflow-demo-appendix3`.\n2. Extract BERT-Large weights from the open source saved model pointed\n to by --input_saved_model and associates it with the Neuron\n compatible model\n3. Invoke TensorFlow-Neuron to compile the Neuron compatible model for\n Inferentia using the newly associated weights\n4. Finally, the compiled model is saved into the location given by\n --output_saved_model\n\n.. _bert-tensorflow-demo-appendix3:\n\nAppendix 3\n----------\n\nThe Neuron compatible implementation of BERT-Large is functionally\nequivalent to the open source version when used for inference. However,\nthe detailed implementation does differ and here are the list of changes\n:\n\n1. Data Type Casting : If the original BERT-Large an FP32 model,\n bert_model.py contains manually defined cast operators to enable\n mixed-precision. FP16 is used for multi-head attention and\n fully-connected layers, and fp32 everywhere else. This will be\n automated in a future release.\n2. Remove Unused Operators: A model typically contains training\n operators that are not used in inference, including a subset of the\n reshape operators. Those operators do not affect inference\n functionality and have been removed.\n3. Reimplementation of Selected Operators : A number of operators\n (mainly mask operators), has been reimplemented to bypass a known\n compiler issue. This will be fixed in a planned future release.\n4. Manually Partition Embedding Ops to CPU : The embedding portion of\n BERT-Large has been partitioned manually to a subgraph that is\n executed on the host CPU, without noticable performance impact. In\n near future, we plan to implement this through compiler\n auto-partitioning without the need for user intervention.\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/bert_demo/glue_mrpc_dev.tsv", "content": "Quality\t#1 ID\t#2 ID\t#1 String\t#2 String\n1\t1355540\t1355592\tHe said the foodservice pie business doesn 't fit the company 's long-term growth strategy .\t\" The foodservice pie business does not fit our long-term growth strategy .\n0\t2029631\t2029565\tMagnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war .\tHis wife said he was \" 100 percent behind George Bush \" and looked forward to using his years of training in the war .\n0\t487993\t487952\tThe dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat .\tThe dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .\n1\t1989515\t1989458\tThe AFL-CIO is waiting until October to decide if it will endorse a candidate .\tThe AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .\n0\t1783137\t1782659\tNo dates have been set for the civil or the criminal trial .\tNo dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .\n1\t3039165\t3039036\tWal-Mart said it would check all of its million-plus domestic workers to ensure they were legally employed .\tIt has also said it would review all of its domestic employees more than 1 million to ensure they have legal status .\n0\t1490811\t1490840\tWhile dioxin levels in the environment were up last year , they have dropped by 75 percent since the 1970s , said Caswell .\tThe Institute said dioxin levels in the environment have fallen by as much as 76 percent since the 1970s .\n1\t426112\t426210\tThis integrates with Rational PurifyPlus and allows developers to work in supported versions of Java , Visual C # and Visual Basic .NET.\tIBM said the Rational products were also integrated with Rational PurifyPlus , which allows developers to work in Java , Visual C # and VisualBasic .Net.\n1\t1439663\t1439808\tThe top rate will go to 4.45 percent for all residents with taxable incomes above $ 500,000 .\tFor residents with incomes above $ 500,000 , the income-tax rate will increase to 4.45 percent .\n1\t3147370\t3147525\tThe results appear in the January issue of Cancer , an American Cancer Society journal , being published online today .\tThe results appear in the January issue of Cancer , an American Cancer Society ( news - web sites ) journal , being published online Monday .\n1\t3300040\t3299992\tThe delegates said raising and distributing funds has been complicated by the U.S. crackdown on jihadi charitable foundations , bank accounts of terror-related organizations and money transfers .\tBin Laden ’ s men pointed out that raising and distributing funds has been complicated by the U.S. crackdown on jihadi charitable foundations , bank accounts of terror-related organizations and money transfers .\n0\t524136\t524119\t\" Sanitation is poor ... there could be typhoid and cholera , \" he said .\t\" Sanitation is poor , drinking water is generally left behind . . . there could be typhoid and cholera . \"\n0\t969512\t969295\tThe broader Standard & Poor 's 500 Index .SPX gave up 11.91 points , or 1.19 percent , at 986.60 .\tThe technology-laced Nasdaq Composite Index was down 25.36 points , or 1.53 percent , at 1,628.26 .\n1\t1685339\t1685429\tThe only announced Republican to replace Davis is Rep. Darrell Issa of Vista , who has spent $ 1.71 million of his own money to force a recall .\tSo far the only declared major party candidate is Rep. Darrell Issa , a Republican who has spent $ 1.5 million of his own money to fund the recall .\n1\t1967578\t1967664\tThe decision to issue new guidance has been prompted by intelligence passed to Britain by the FBI in a secret briefing in late July .\tScotland Yard 's decision to issue new guidance has been prompted by new intelligence passed to Britain by the FBI in late July .\n1\t2047034\t2046820\tUnable to find a home for him , a judge told mental health authorities they needed to find supervised housing and treatment for DeVries somewhere in California .\tThe judge had told the state Department of Mental Health to find supervised housing and treatment for DeVries somewhere in California .\n1\t2046630\t2046644\tThe decision came a year after Whipple ended federal oversight of the district 's racial balance , facilities , budget , and busing .\tThe decision came a year after Whipple ended federal oversight of school busing as well as the district 's racial balance , facilities and budget .\n0\t2221603\t2221633\tIn midafternoon trading , the Nasdaq composite index was up 8.34 , or 0.5 percent , to 1,790.47 .\tThe Nasdaq Composite Index .IXIC dipped 8.59 points , or 0.48 percent , to 1,773.54 .\n1\t129995\t129864\tMorgan Stanley raised its rating on the beverage maker to \" overweight \" from \" equal-weight \" saying in part that pricing power with its bottlers should improve in 2004 .\tMorgan Stanley raised its rating on the company to \" overweight \" from \" equal-weight , \" saying the beverage maker 's pricing power with bottlers should improve in 2004 .\n0\t919683\t919782\tThe pound also made progress against the dollar , reached fresh three-year highs at $ 1.6789 .\tThe British pound flexed its muscle against the dollar , last up 1 percent at $ 1.6672 .\n0\t970740\t971209\tFriday , Stanford ( 47-15 ) blanked the Gamecocks 8-0 .\tStanford ( 46-15 ) has a team full of such players this season .\n1\t2745055\t2745022\tLast month Intel raised its revenue guidance for the quarter to between $ 7.6 billion and $ 7.8 billion .\tAt the end of the second quarter , Intel initially predicted sales of between $ 6.9 billion and $ 7.5 billion .\n0\t2199097\t2199072\tThe driver , Eugene Rogers , helped to remove children from the bus , Wood said .\tAt the accident scene , the driver was \" covered in blood \" but helped to remove children , Wood said .\n1\t1609290\t1609098\tONG KONG , July 9 Tens of thousands of demonstrators gathered tonight before the legislature building here to call for free elections and the resignation of Hong Kong 's leader .\tTens of thousands of demonstrators gathered yesterday evening to stand before this city 's legislature building and call for free elections and the resignation of Hong Kong 's leader .\n1\t1597193\t1597119\tSaddam loyalists have been blamed for sabotaging the nation 's infrastructure , as well as frequent attacks on U.S. soldiers .\tHussein loyalists have been blamed for sabotaging the nation 's infrastructure and attacking US soldiers .\n1\t2758944\t2758975\tIts closest living relatives are a family frogs called sooglossidae that are found only in the Seychelles in the Indian Ocean .\tIts closest relative is found in the Seychelles Archipelago , near Madagascar in the Indian Ocean .\n0\t2584416\t2584653\tCooley said he expects Muhammad will similarly be called as a witness at a pretrial hearing for Malvo .\tLee Boyd Malvo will be called as a witness Wednesday in a pretrial hearing for fellow sniper suspect John Allen Muhammad .\n1\t86007\t86373\t\" Instead of pursuing the most imminent and real threats - international terrorists , \" Graham said , \" this Bush administration chose to settle old scores . \"\t\" Instead of pursuing the most imminent and real threats - international terrorists - this Bush administration has chosen to settle old scores , \" Graham said .\n1\t1602860\t1602844\tHe said they lied on a sworn affidavit that requires them to list prior marriages .\tMorgenthau said the women , all U.S. citizens , lied on a sworn affidavit that requires them to list prior marriages .\n1\t1201306\t1201329\tThe association said 28.2 million DVDs were rented in the week that ended June 15 , compared with 27.3 million VHS cassettes .\tThe Video Software Dealers Association said 28.2 million DVDs were rented out last week , compared to 27.3 million VHS cassettes .\n0\t461779\t461815\tWith these assets , Funny Cide has a solid chance to become the first Triple Crown winner since Affirmed in 1978 .\tFunny Cide is looking to become horse racing 's first Triple Crown winner in a generation .\n1\t1438666\t1438643\tIntel was disappointed and assessing its \" options in the event Mr. Hamidi resumes his spamming activity against Intel , \" spokesman Chuck Mulloy said .\tIntel spokesman Chuck Mulloy said the company was disappointed and assessing its \" options in the event Mr. Hamidi resumes his spamming activity against Intel . \"\n1\t3261484\t3261306\tMr Annan also warned the US should not use the war on terror as an excuse to suppress \" long-cherished freedoms \" .\tAnnan warned that the dangers of extremism after September 11 should not be used as an excuse to suppress \" long-cherished \" freedoms .\n1\t1277539\t1277527\tAt community colleges , tuition will jump to $ 2,800 from $ 2,500 .\tCommunity college students will see their tuition rise by $ 300 to $ 2,800 or 12 percent .\n1\t3035788\t3035918\tHe made a point of saying during Tuesdays debate that the Confederate flag was a racist symbol .\tThough Dean made a point of saying during the debate that the Confederate flag is a racist symbol .\n0\t132553\t132725\tBush wanted \" to see an aircraft landing the same way that the pilots saw an aircraft landing , \" White House press secretary Ari Fleischer said yesterday .\tOn Tuesday , before Byrd 's speech , Fleischer said Bush wanted ' ' to see an aircraft landing the same way that the pilots saw an aircraft landing .\n0\t2259788\t2259747\tOn Monday the Palestinian Prime Minister , Mahmoud Abbas , will report to the Palestinian parliament on his Government 's achievements in its first 100 days in office .\tPalestinian Prime Minister Mahmoud Abbas must defend the record of his first 100 days in office before Parliament today as the death toll in the occupied territories continues to rise .\n0\t2307064\t2307235\tThe civilian unemployment rate improved marginally last month -- slipping to 6.1 percent -- even as companies slashed payrolls by 93,000 .\tThe civilian unemployment rate improved marginally last month _ sliding down to 6.1 percent _ as companies slashed payrolls by 93,000 amid continuing mixed signals about the nation 's economic health .\n1\t3046488\t3046824\tPer-user pricing is $ 29 for Workplace Messaging , $ 89 for Team Collaboration and $ 35 for Collaborative Learning .\tWorkplace Messaging is $ 29 , Workplace Team Collaboration is $ 89 , and Collaborative Learning is $ 35 .\n1\t86020\t86007\t\" Instead of pursuing the most imminent and real threats – international terrorism – this Bush administration chose to settle old scores , \" Mr. Graham said .\t\" Instead of pursuing the most imminent and real threats - international terrorists , \" Graham said , \" this Bush administration chose to settle old scores . \"\n0\t1100998\t1100441\tSARS has killed about 800 people and affected more than 8400 since being detected in China in November .\tSARS has killed about 800 people and sickened more than 8,400 worldwide , mostly in Asia .\n1\t2268396\t2268480\tAuthorities had no evidence to suggest the two incidents were connected .\tThere was no immediate evidence that the two incidents were connected , police said .\n0\t1984039\t1983986\t\" Jeremy 's a good guy , \" Barber said , adding : \" Jeremy is living the dream life of the New York athlete .\tHe also said Shockey is \" living the dream life of a New York athlete .\n0\t2697659\t2697747\tRatliff 's daughters , Margaret and Martha Ratliff , were adopted by Peterson after their mother 's death .\tPeterson helped raise Ratliff 's two daughters , Margaret and Martha Ratliff , who supported him throughout the trial .\n0\t2175939\t2176090\tAfter losing as much as 84.56 earlier , the Dow Jones industrial average closed up 22.81 , or 0.2 percent , at 9,340.45 .\tIn midday trading , the Dow Jones industrial average lost 68.84 , or 0.7 percent , to 9,248.80 .\n1\t886618\t886456\tRumsfeld , who has been feuding for two years with Army leadership , passed over nine active-duty four-star generals .\tRumsfeld has been feuding for a long time with Army leadership , and he passed over nine active-duty four-star generals .\n1\t588637\t588864\tConsumers who said jobs are difficult to find jumped from 29.4 to 32.6 , while those claiming work was plentiful slipped from 13 to 12.6 .\tConsumers who said jobs are difficult to find jumped to 32.6 from 29.4 , while those saying work was plentiful slipped to 12.6 from 13 in April .\n0\t2252795\t2252970\tHe has no immediate plans for television advertising , believing it is unnecessary this early .\tA Lieberman aide said there were no immediate plans for television advertising .\n1\t1756329\t1756394\t\" I think it happened very quickly , \" Houston Police Department homicide investigator Phil Yochum said of the crime .\t\" I think it happened very quickly , \" said Investigator Phil Yochum of the Houston Police Department 's homicide division .\n1\t1673112\t1673068\tUnited issued a statement saying it will \" work professionally and cooperatively with all its unions . \"\tSenior vice president Sara Fields said the airline \" will work professionally and cooperatively with all our unions . \"\n1\t2357324\t2357271\t\" But they never climb out of the pot of beer again . \"\tIt 's just that they never climb out of the beer again . \"\n1\t780408\t780363\tChief financial officer Andy Bryant has said that hike had a greater affect volume than officials expected .\tBryant has said that hike had a greater effect on demand than officials expected .\n1\t821523\t821385\tRobert Liscouski , the Assistant Secretary of Homeland Security for Infrastructure Protection , will oversee NCSD .\tNCSD 's chief will be Robert Liscouski , the assistant secretary of Homeland Security for Infrastructure Protection .\n1\t2304696\t2304863\tHP 's shipments increased 48 percent year-over-year , compared to an increase of 31 percent for Dell .\tHPs shipments increased 48 per cent year-on-year , compared to an increase of 31 per cent for Dell .\n1\t2531749\t2531607\tChirac , who can pardon a law-breaker , refused Humbert 's request last year but kept in close touch with the family .\tChirac , who has the authority to pardon law-breakers , refused Humbert 's request to be allowed to die last year but kept in close touch with the family .\n1\t3180014\t3179967\tThe charges allege that he was part of the conspiracy to kill and kidnap persons in a foreign country .\tThe government now charges that Sattar conspired with Rahman to kill and kidnap individuals in foreign countries .\n1\t726966\t726945\tIn the 2002 study , the margin of error ranged from 1.8 to 4.4 percentage points .\tIt has a margin of error of plus or minus three to four percentage points .\n1\t2638861\t2638982\tMr. Clinton 's national security adviser , Sandy Berger , said that the White House wasn 't informed of the FBI activities .\tClinton ’ s national security adviser , Sandy Berger , said in an interview that the White House was not informed of the FBI activities .\n1\t2495223\t2495307\t\" This decision is clearly incorrect , \" FTC Chairman Timothy Muris said in a written statement .\tThe decision is \" clearly incorrect , \" FTC Chairman Tim Muris said .\n1\t55187\t54831\tProsecutors allege that Nichols and co-conspirator Timothy McVeigh worked together to prepare a bomb that destroyed the Alfred P. Murrah Federal Building .\tProsecutors allege that Nichols and coconspirator Timothy McVeigh worked together to prepare a 4,000-pound fuel-and-fertilizer bomb that destroyed the Murrah building .\n0\t2763381\t2763517\tTerri Schiavo , 39 , is expected to die sometime in the next two weeks in the Tampa-area hospice where she has spent the past several years .\tTerri Schiavo , 39 , underwent the procedure at the Tampa Bay area hospice where she has been living for several years , said her father , Bob Schindler .\n1\t1990975\t1991132\tSecretary of State Colin Powell designated the Chechen leader believed responsible for last year 's hostage standoff in a Moscow theater as a threat to U.S. security Friday .\tU.S. Secretary of State Colin Powell on Friday designated Chechen rebel leader Shamil Basayev a threat to the security of the United States and to U.S. citizens .\n1\t2204353\t2204418\t\" Today , we are trying to convey this problem to Russian President Vladimir Putin and US President George W Bush . \"\t\" Today , we are trying to convey this problem to Russian President Vladimir Putin ( news - web sites ) and President Bush ( news - web sites ) . \"\n1\t60122\t60445\tThat would be a potential setback to Chief Executive Phil Condit 's strategy of bolstering defense-related sales during a slump in jetliner deliveries .\tThe inquiry may hinder Chief Executive Phil Condit 's strategy of bolstering defense-related sales during a slump in jetliner deliveries .\n1\t961836\t962243\tPeopleSoft also said its board had officially rejected Oracle 's offer .\tThursday morning , PeopleSoft 's board rejected the Oracle takeover offer .\n0\t3140260\t3140288\tThe Dow Jones industrial average ended the day down 10.89 at 9,837.94 , after advancing 111.04 Wednesday .\tThe Dow Jones industrial average fell 10.89 points , or 0.11 percent , to 9,837.94 .\n1\t1720166\t1720115\tCortisol levels in the saliva of day care children were highest and rose most steeply in those judged by day care center personnel to be the shyest .\tCortisol levels in the saliva of day-care children were highest and rose most steeply in those whom day-care centre staffed judged to be the shyest .\n1\t2573262\t2573319\t\" The idea that Tony Abbott is in some way a one-dimensional political head-kicker couldn 't be more wrong , \" Mr Howard said .\t\" The idea that Tony Abbott is in some way a one-dimensional political head kicker couldn 't be more wrong . \"\n0\t1353356\t1353174\t\" Biotech products , if anything , may be safer than conventional products because of all the testing , \" Fraley said , adding that 18 countries have adopted biotechnology .\t\" Biotech products , if anything , may be safer than conventional products because of all the testing , \" said Robert Fraley , Monsanto 's executive vice president .\n1\t2738677\t2738741\tThe rate of skin cancer has tripled since the 1950s in Norway and Sweden , according to the study .\tThe study also found that skin cancer nearly tripled in Norway and Sweden since the 1950s .\n1\t1638813\t1639087\tWe acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , \" Rumsfeld said .\tRather , the US acted because the administration saw \" existing evidence in a new light , through the prism of our experience on September 11 \" .\n1\t1605350\t1605425\tTrans fat makes up only 1 percent to 3 percent of the total fat Americans consume , compared with 14 percent for saturated fat .\tTrans fat accounts for 2.5 percent of Americans ' daily calories , compared to 11 percent to 12 percent for saturated fat .\n1\t2494149\t2494073\tHowever , a recent slide in prices and OPEC 's expectations of a surge in oil inventories have compounded its fears about a further softening of the market .\tA 14 percent slide in crude prices this month and expectations of a build up in oil inventories compounded OPEC 's fears of a further softening of the market .\n1\t3023029\t3023229\tPeterson , 31 , is now charged with murder in the deaths of his 27-year-old wife and their unborn son .\tPeterson , 31 , is charged with two counts of first-degree murder in the slayings of his wife , Laci , and their unborn son , Conner .\n1\t1351550\t1351155\tCarlson on Tuesday said he would not recuse himself from the case .\tService officials said Carlson refused to recuse himself from the case .\n1\t981185\t981234\tThe program will grow to include ports in Dubai , Turkey and Malaysia , among others .\tThe program will be expanded to include areas of the Middle East such as Dubai , Turkey and Malaysia , Mr. Ridge said .\n0\t2111629\t2111786\tMcCabe said he was considered a witness , not a suspect .\t\" He is not considered a suspect , \" McCabe said .\n1\t655498\t655391\tThe woman was exposed to the SARS virus while in the hospital but was not a health care worker , said Dr. Colin D ’ Cunha , Ontario ’ s commissioner of public health .\tThe woman was exposed to the SARS virus while in the hospital but was not a health-care worker , said Dr Colin D 'Cunha , Ontario 's commissioner of public health .\n1\t533823\t533909\tHe added that those \" are not solely American principles , nor are they exclusively Western . \"\t\" These are not solely American principles nor are they exclusively Western , \" Rumsfeld said .\n1\t581592\t581570\t\" If we don 't march into Tehran , I think we will be in pretty good shape , \" he said .\t\" As long as we don 't march on Tehran , I think we are going to be in pretty good shape , \" he said .\n0\t1010655\t1010430\tOn Saturday , a 149mph serve against Agassi equalled Rusedski 's world record .\tOn Saturday , Roddick equalled the world record with a 149 m.p.h. serve in beating Andre Agassi .\n1\t2241925\t2242066\tChad Kolton , emergency management spokesman with the Department of Homeland Security , said the government is open to new technologies and methods to communicate more quickly and efficiently .\tChad Kolton , emergency management spokesman with the Department of Homeland Security , said the government is open to new ways to communicate .\n1\t2796978\t2797024\t\" APEC leaders are painfully aware that security and prosperity are inseparable , \" Thai Prime Minister Thaksin Shinawatra told business leaders .\t\" APEC leaders are painfully aware that security and prosperity are inseparable , \" Thaksin said .\n0\t101746\t101775\tDanbury prosecutor Warren Murray could not be reached for comment Monday .\tProsecutors could not be reached for comment after the legal papers were obtained late Monday afternoon .\n1\t327839\t327748\tWittig resigned last year after being indicted on federal bank fraud charges involving a real estate loan unrelated to Westar business .\tWittig resigned in late November about two weeks after being indicted on bank fraud charges in a real estate case unrelated to the company .\n0\t2988297\t2988555\tShattered Glass , \" starring Hayden Christensen as Stephen Glass , debuted well with $ 80,000 in eight theaters .\t\" Shattered Glass \" _ starring Hayden Christensen as Stephen Glass , The New Republic journalist fired for fabricating stories _ debuted well with $ 80,000 in eight theaters .\n1\t2217613\t2217659\tHe was arrested Friday night at an Alpharetta seafood restaurant while dining with his wife , singer Whitney Houston .\tHe was arrested again Friday night at an Alpharetta restaurant where he was having dinner with his wife .\n0\t2128530\t2128455\tHowever , EPA officials would not confirm the 20 percent figure .\tOnly in the past few weeks have officials settled on the 20 percent figure .\n1\t2208376\t2208198\tUniversity of Michigan President Mary Sue Coleman said in a statement on the university 's Web site , \" Our fundamental values haven 't changed .\t\" Our fundamental values haven 't changed , \" Mary Sue Coleman , president of the university , said in a statement in Ann Arbor .\n1\t1980654\t1980641\tThe first products are likely to be dongles costing between US $ 100 and US $ 150 that will establish connections between consumer electronics devices and PCs .\tThe first products will likely be dongles costing $ 100 to $ 150 that will establish connections between consumer electronics devices and PCs .\n0\t589579\t589557\tHowever , Lapidus expects foreign brands ' sales to be up 4 percent , driven by strong truck sales at Honda Motor Co .\tLapidus expects Ford to be down 5 percent , Chrysler down 10 percent and foreign brands up 4 percent driven by strong truck sales at Honda .\n1\t1636060\t1635946\tMichel , who remains in the government , denied that US pressure had provoked the government 's move .\tMichel , who has stayed in the new government , denied that it was U.S. pressure which had provoked the government 's move .\n1\t1630585\t1630657\tSome of the computers also are used to send spam e-mail messages to drum up traffic to the sites .\tSome are also used to send spam e-mail messages to boost traffic to the sites .\n0\t447728\t447699\tIndonesia 's army has often been accused of human rights abuses during GAM 's battle for independence , charges it has generally denied while accusing the separatists of committing rights violations .\tIndonesia 's army has been accused of human rights abuses during its earlier battles with GAM , charges it has generally denied .\n1\t1606495\t1606619\tBush also hoped to polish his anti-AIDS credentials in Uganda , which has been hailed as an African pioneer in fighting the killer disease .\tPresident Bush flies to Uganda Friday hoping to polish his anti- AIDS credentials in a country hailed as an African pioneer in fighting the epidemic .\n1\t1550897\t1550977\tLater this year , the command will send trainers with soldiers from four North African nations on patrolling and intelligence gathering missions .\tThis fall the command will send trainers to work with soldiers from four North African nations on patrolling and gathering intelligence .\n0\t490376\t490490\tThe reports helped overcome investor jitters after the euro briefly hit an all-time high against the dollar Tuesday .\tStocks slipped at the open after the euro hit record highs against the dollar .\n1\t3084554\t3084612\tSales for the quarter beat expectations , rising 37 percent year-on-year to 1.76 billion euros .\tSales rose 37 per cent year-on-year to 1.76bn , beating expectations .\n1\t315647\t315778\tIf the MTA 's appeal to a higher court is successful , the $ 2 bus and subway base fare won 't be rolled back .\tIf the MTA 's appeal is successful , the $ 2 bus and subway base fare won 't change .\n1\t3428298\t3428362\tRobert Walsh , 40 , remained in critical but stable condition Friday at Staten Island University Hospital 's north campus .\tWalsh , also 40 , was in critical but stable condition at Staten Island University Hospital last night .\n1\t2523564\t2523358\tThe Guru microcontroller serves four functions : hardware monitoring , overclocking management , BIOS ( Basic Input Output System ) update and a troubleshooting-assistance feature called Black Box .\tThe µGuru microcontroller serves four functions : hardware monitoring , overclocking management , BIOS update and a troubleshooting-assistance feature called Black Box .\n1\t2079200\t2079131\tU.S. corporate bond yield spreads tightened in spotty trading on Friday as Wall Street labored to get back on its feet after the largest power outage ever in North America .\tU.S. stocks rose slightly on feather-light volume on Friday , as Wall Street regrouped after the biggest-ever power outage in North America .\n1\t818091\t817811\tThe company said it would issue revised guidance for the full fiscal year next month when it releases its Q2 results .\tThe company said it would renew its guidance for 2003 when it announces its second quarter results in mid-July .\n1\t1580638\t1580663\t\" I stand 100 percent by it , and I think our intelligence services gave us the correct information at the time . \"\tI stand 100 percent by it , and I think that our intelligence services gave us the correct intelligence and information at the time , \" Blair said .\n0\t1919740\t1919926\t\" I don 't know if the person I 'm talking to now may end up being someone else at another time that may not follow the rules , \" Parrish said .\t\" I don 't know whether the person I 'm talking to now may end up being someone else , \" Parrish said .\n1\t2748287\t2748550\t\" I think it 's going to be a close vote , but I think the grant proposal is going to win , \" McConnell said .\t\" I think it 's going to be a close vote , but I think the grant proposal 's going to win , \" said Sen. Mitch McConnell , assistant majority leader .\n1\t3394891\t3394775\tTwenty-eight people were believed to have been spending Christmas Day with the caretaker of the St Sophia 's camp , when the mudslide smashed into two cabins .\tTwenty-seven people were believed to have been spending Christmas Day with the caretaker of Saint Sophia Camp , a Greek Orthodox facility , when the mudslide roared through .\n0\t2963943\t2963880\tOne , Capt. Doug McDonald , remained hospitalized in critical condition on Thursday .\tHer 20-year-old sister , Allyson , was severely burned and remained hospitalized in critical condition .\n0\t1865364\t1865251\tThe United States finally relented during President Bush 's visit to Africa earlier this month .\tDuring President Bush 's trip to Africa earlier this month , however , Washington said it would support the increase .\n1\t263690\t263819\t\" There is no conscious policy of the United States , I can assure you of this , to move the dollar at all , \" he said .\tHe also said there is no conscious policy by the United States to move the value of the dollar .\n1\t283751\t283290\tIt 's the first such drill since the September 11 terrorist attacks on New York and Washington .\tIt is the nation 's first large-scale counterterrorism exercise since the Sept . 11 terrorist attacks .\n1\t2517014\t2516995\tMyanmar 's pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital , her personal physician said .\tMyanmar 's pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery , her personal physician said Friday .\n1\t1330643\t1330622\tAccording to the Merchant Marine Ministry , the 37-year-old ship is registered to Alpha Shipping Inc. based in the Pacific Ocean nation of Marshall Islands .\tThe Baltic Sky is a 37-year-old ship registered to Alpha Shipping Inc. based in the Pacific Ocean nation of Marshall Islands .\n1\t3111452\t3111428\tIn an unusual move , the U.S. Patent and Trademark Office is reconsidering a patent affecting Internet pages that critics contend could disrupt millions of Web sites .\tIn an unusual move that critics contend could disrupt millions of Web sites , the U.S. Patent and Trademark Office is reconsidering a patent affecting Internet pages .\n0\t1167835\t1167651\tKansas Department of Health and Environment records show there were 88 abortions performed on girls age 14 and younger last year .\tStatistics from the Kansas Department of Health and Environment show that 11,844 abortions were performed in the state last year .\n0\t1423836\t1423708\tA European Union spokesman said the Commission was consulting EU member states \" with a view to taking appropriate action if necessary \" on the matter .\tLaos 's second most important export destination - said it was consulting EU member states ' ' with a view to taking appropriate action if necessary ' ' on the matter .\n1\t2090911\t2091154\tWaiting crowds filling the streets on both sides overwhelmed the peacekeepers soon after daylight , sweeping past the barbed wire barricades .\tBut waiting crowds filling the streets rushed the bridges soon after daylight , overrunning razor-wire barricades .\n1\t2265271\t2265152\tBarry Callebaut will be able to use Brach 's retail network to sell products made from its German subsidiary Stollwerck , which makes chocolate products not sold in the United States .\tBarry Callebaut will be able to use Brach 's retail network to sell products made from its German subsidiary Stollwerck , which makes chocolate products unknown to the American market .\n1\t3062202\t3062308\tBy skirting the FDA 's oversight , Eagan said , the quality of the imported drugs is \" less predictable \" than for those obtained in the United States .\tBy skirting the FDA 's oversight , Eagan said the quality of the imported drugs is \" less predictable \" than U.S. drugs .\n1\t2155514\t2155377\tHe said : \" For the first time there is an easy and affordable way of making this treasure trove of BBC content available to all . \"\t\" For the first time , there is an easy and affordable way of making this treasure trove of BBC content available to all , \" Dyke said .\n1\t1552068\t1551928\tThree such vigilante-style attacks forced the hacker organizer , who identified himself only as \" Eleonora [ 67 ] , \" to extend the contest until 7 p.m. EST Sunday .\tThree such vigilante-style attacks forced the hacker organiser , who identified himself only as \" Eleonora67 ] , \" to extend the contest until 8am ( AEST ) today .\n1\t936978\t937500\tEric Gagne pitched a perfect ninth for his 23rd save in as many opportunities .\tGagne struck out two in a perfect ninth inning for his 23rd save .\n0\t985015\t984975\tOne way or another , Harry Potter And The Order Of The Phoenix will be in your hands by Saturday .\tJust about everything about \" Harry Potter and the Order of the Phoenix \" will set records .\n1\t1430357\t1430425\t\" Allison just proves you don 't need to wait until August or September to have a disaster , \" said Josh Lichter , a meteorologist with the Houston-Galveston weather office .\t\" Allison just proves you don 't need to wait until August or September to have a disaster , \" Lichter said .\n1\t3039310\t3039413\tToday , analysts say , UN members can no longer ignore the shifts since the September 11 2001 attacks .\tOn Wednesday , analysts say , UN members can no longer ignore the shifts since the attacks in the US of September 11 2001 .\n1\t34513\t34742\tPolice say CIBA was involved in the importation of qat , a narcotic substance legal in Britain but banned in the United States .\tMr McKinlay said that CIBA was involved in the importation of qat , a narcotic substance legal in Britain but banned in the US .\n1\t368067\t368018\tChiron already has nearly 20 percent acceptances from PowderJect 's shareholders .\tChiron has acceptances from holders of nearly 20 percent of PowderJect shares .\n0\t611663\t611716\tErnst & Young has denied any wrongdoing and plans to fight the allegations .\tErnst & Young has denied the SEC 's claims , and called its recommendations \" irresponsible \" .\n1\t98432\t98657\tThe attack followed several days of disturbances in the city where American soldiers exchanged fire with an unknown number of attackers as civilians carried out demonstrations against the American presence .\tThe attack came after several days of disturbance in the city in which U.S. soldiers exchanged fire with an unknown number of attackers as civilians protested the American presence .\n1\t3039007\t3038845\tNo company employee has received an individual target letter at this time .\tShe said no company official had received \" an individual target letter at this time . \"\n1\t1708040\t1708062\tSecond-quarter results reflected a gain of 10 cents per diluted share , while the 2002 results included a loss of 19 cents per diluted share .\tThe second-quarter results had a non-operating gain of 10 cents a share while the 2002 second-quarter performance had a net non-operating loss of 19 cents a share .\n0\t1757264\t1757375\tHe allegedly told his ex-wife in an angry phone call that he had no intention of following their new custody agreement .\tThe two had battled over custody and he allegedly told her in an angry phone call that he had no intention of following their new custody agreement .\n1\t383417\t383558\tWorldwide , more than 50 million people have seen \" Les Miz , \" with gross receipts of $ 1.8 billion .\tWorldwide , Les Misérables has been seen by over 50 million people , with a total gross of over $ 2 billion .\n0\t2766112\t2766084\tIn fiction : Edward P. Jones ( \" The Known World \" ) and Scott Spencer ( \" A Ship Made of Paper \" ) .\tThe fifth nominee for fiction is Scott Spencer , for A Ship Made of Paper .\n1\t1261116\t1261234\t\" Overwhelmingly the Windows brand really resonated with them . \"\t\" Windows was the part of the experience that really resonated with people . \"\n1\t3028143\t3028234\tThe Centers for Medicare and Medicaid Services , the federal agency that runs Medicare , last year began a similar effort for nursing homes .\tThe Centers for Medicare and Medicaid launched a similar consumer tool for nursing homes last year .\n0\t249699\t249623\tVivace was founded in 1999 and has raised over $ 118 million in three rounds of venture financing .\tDuring difficult times for technology venture capital , Vivace raised over $ 118 million in three rounds of venture financing .\n0\t3448488\t3448449\tThe Dow Jones industrial average < .DJI > added 28 points , or 0.27 percent , at 10,557 , hitting its highest level in 21 months .\tThe Dow Jones industrial average < .DJI > rose 49 points , or 0.47 percent , to 10,578 .\n1\t2749322\t2749663\tThe Democratic candidates also began announcing their fund-raising totals before Wednesday 's deadline to file quarterly reports with the Federal Election Commission .\tThe Democratic candidates also began announcing their fund-raising totals in advance of the deadline today to file quarterly reports with the Federal Election Commission .\n0\t2204592\t2204588\tSun Microsystems Inc. on Thursday said it had added 100 new third-party systems and 100 new components to its Hardware Compatibility List for the Solaris x86 operating system Platform Edition .\tThe vendor has added 100 new third-party systems and 100 new components to the operating system 's Hardware Compatibility List ( HCL ) .\n1\t2889005\t2888954\tProsecutors said PW Marketing violated the state 's 1998 anti-spam law by sending unsolicited e-mail without a toll-free number for recipients to call to stop additional mailings .\tProsecutors said PW Marketing violated the 1998 anti-spam law because these unsolicited e-mails were sent without a free call number for recipients to phone to stop additional mailings .\n0\t1657632\t1657619\tThe Neighbours star and singer spent yesterday resting at her family home in Sydney and will have more tests today .\tGoodrem spent yesterday resting in her family home in Sydney and will have more tests today to determine her exact treatment .\n0\t555617\t555528\tThe 3 rd Armored Cavalry Regiment is 5,200 strong and the largest combat unit at Fort Carson .\tBroomhead , 34 , was assigned to the 2nd Squadron , 3rd Armored Cavalry Regiment .\n1\t2396937\t2396818\t\" The risk of inflation becoming undesirably low remains the predominant concern for the foreseeable future , \" the Fed said in a statement accompanying the unanimous decision .\t\" The risk of inflation becoming undesirably low remains the predominant concern for the foreseeable future , \" the policy-setting Federal Open Market Committee said .\n0\t2339738\t2339771\t\" It is bad for Symbian , \" said Per Lindberg , analyst at Dresdner Kleinwort Wasserstein .\t\" Motorola has displayed clear disloyalty \" to Symbian , said Per Lindberg , an analyst at Dresdner Kleinwort Wasserstein in London .\n0\t1616174\t1616206\tBob Richter , a spokesman for House Speaker Tom Craddick , had no comment about the ruling .\tBob Richter , spokesman for Craddick , R-Midland , said the speaker had not seen the ruling and could not comment .\n1\t635783\t635802\tBut Ms Ward said the headroom under its financial covenants was \" tight \" and that there could be another downgrade if Southcorp breached any of its banking covenants .\tBut Ms Ward said the headroom under its financial covenants was \" tight \" and that there could be a rating downgrade if Southcorp did breach any banking covenants .\n1\t3444633\t3444733\tHe added : ``I 've never heard of more reprehensiblebehaviour by a doctor .\tThe Harrisons ’ lawyer Paul LiCalsi said : “ I ’ ve never heard of more reprehensible behaviour by a doctor .\n1\t555553\t555528\tBroomhead was assigned to 2nd Squadron , 3rd Armor Cavalry Regiment , based at Fort Carson .\tBroomhead , 34 , was assigned to the 2nd Squadron , 3rd Armored Cavalry Regiment .\n1\t1112021\t1111925\tOther staff members , however , defended the document , saying it would still help policy-makers and the agency improve efforts to address the climate issue .\tSome E.P.A. staff members defended the document , saying that although pared down it would still help policy makers and the agency address the climate issue .\n0\t2749410\t2749625\tPresident Bush raised a record-breaking $ 49.5 million for his re-election campaign over the last three months , with contributions from 262,000 Americans , the president 's campaign chairman said Tuesday .\tPresident Bush has raised $ 83.9 million since beginning his re-election campaign in May , and has $ 70 million of that left to spend , his campaign said Tuesday .\n1\t1629064\t1629043\tAn episode is declared when the ozone reaches .20 parts per million parts of air for one hour .\tA Stage 1 episode is declared when ozone levels reach 0.20 parts per million .\n1\t789691\t789665\t\" He may not have been there , \" the defence official said on Thursday .\t\" He may not have been there , \" said a defence official speaking on condition of anonymity .\n1\t844421\t844679\tThe U.N. troops are in Congo to protect U.N. installations and personnel , and they can only fire in self defense and have been unable to stem the violence .\tThe troops - whose mandate is to protect U.N. installations and personnel - can only fire in self-defense and have been unable to stem the violence .\n1\t58540\t58567\tNorth American markets grabbed early gains Monday morning , as earnings season begins to slow and economic indicators take the spotlight .\tNorth American futures pointed to a strong start to the first trading session of the week Monday , as earnings season slows and economic indicators take the spotlight .\n1\t781439\t781461\tXerox itself paid a $ 10 million fine last year to settle similar SEC charges .\tXerox itself previously paid a $ 10-million penalty to settle the SEC accusations .\n1\t1909579\t1909408\t\" This deal makes sense for both companies , \" said National Chief Executive Brian Halla .\t\" This deal makes sense for both companies , \" Halla said in a prepared statement .\n0\t787432\t787464\tThe blasts killed two people and injured more than 150 others .\tThe Atlanta Olympic Games attack killed one woman and injured more than 100 other people .\n0\t52758\t52343\tMorrill 's wife , Ellie , sobbed and hugged Bondeson 's sister-in-law during the service .\tAt the service Morrill 's widow , Ellie , sobbed and hugged Bondeson 's sister-in-law as people consoled her .\n1\t1675025\t1675047\tSpansion products are to be available from both AMD and Fujitsu , AMD said .\tSpansion Flash memory solutions are available worldwide from AMD and Fujitsu .\n1\t2131318\t2131372\tAbout 1,500 police will be deployed for the visit .\tAround 1,500 police are to be deployed at Niigata for the ferry 's visit .\n1\t325763\t325928\tGamarekian told The News she remembers only the woman 's first name - and refused to reveal it .\tShe told the New York Daily News she remembers only the intern 's first name , which she refused to reveal .\n1\t2638975\t2638855\tOne of the FBI ’ s key operatives , who had a falling out with the bureau , provided an account of the operation at a friend ’ s closed immigration court proceeding .\tOne of the FBI 's key operatives , who has had a falling-out with the bureau , provided an account of the operation at a friend 's closed immigration court proceeding .\n1\t2198694\t2198937\tA nationally board certified teacher with a master 's degree , Kelley makes a salary of $ 65,000 in his 30th year .\tA nationally board certified teacher with a master 's degree , Kelley , in his 30th year teaching , makes $ 65,000 .\n1\t1825432\t1825301\tA man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $ 100,000 bail during an early morning court appearance Saturday .\tThe Queens man arrested for allegedly threatening to shoot City Councilman Hiram Monserrate was held on $ 100,000 bail Saturday , a spokesman for the Queens district attorney said .\n1\t2906104\t2906322\tThey were being held Sunday in the Camden County Jail on $ 100,000 bail .\tThey remained in Camden County Jail on Sunday on $ 100,000 bail .\n1\t722278\t722383\tMs Stewart , the chief executive , was not expected to attend .\tMs Stewart , 61 , its chief executive officer and chairwoman , did not attend .\n0\t101747\t101777\tChristina 's aunt , Shelley Riling , said the defense 's claims were preposterous .\tChristina 's aunt , Shelley Riling , said she will address the court .\n1\t2224884\t2224819\tThe Justice Department Aug. 19 gave pre-clearance for the Oct. 7 date for the election to recall Gov. Gray Davis , saying it would not affect minority voting rights .\tThe Justice Department on Aug. 19 sanctioned the Oct. 7 date for recall election , saying it would not affect voting rights .\n0\t977938\t978162\tLord Falconer hailed the changes as \" a new beginning as far as the courts , Crown Prosecution Service and police are concerned \" .\t\" It 's a new beginning as far as the courts , Crown Prosecution Service and police are concerned , making the criminal justice system work better . \"\n0\t1015010\t1014963\tGE stock closed at $ 30.65 a share , down about 42 cents , on the New York Stock Exchange .\tGE 's shares closed at $ 30.65 on Friday on the New York Stock Exchange .\n1\t1513190\t1513246\tAt least 27 US troops have been killed in hostile fire since Bush 's statement .\tAt least 26 American troops have been killed in hostile fire since major combat was officially declared over on May 1 .\n1\t2385348\t2385394\tA recent poll showed Edwards with a narrow lead in South Carolina , and he plans a rally there later on Tuesday .\tA recent poll showed Edwards in a virtual four-way tie at the top in South Carolina , and he plans a rally there later on Tuesday .\n1\t2317018\t2317252\tNovember 17 's last victim was British defence attache Stephen Saunders , who was shot on an Athens road in June 2000 .\tNovember 17 's last victim was British defense attache Stephen Saunders , who was shot and killed at point-blank range on a busy Athens road in June 2000 .\n0\t1831696\t1831660\tThe agency charged that one WD Energy worker discussed false reporting with traders at two other energy companies .\tThe agency found further that a WD Energy employee discussed false reporting with traders at two other energy companies , which the CFTC didn 't identify .\n1\t1528383\t1528083\tZulifquar Ali , a worshipper slightly wounded by shrapnel , said the assailants first targeted the mosque 's security guards .\tWitness Zulfiqar Ali , who was slightly wounded by shrapnel , said the attackers had focused on the mosque 's guards .\n1\t917965\t918315\tFor the second year in a row , rises in hospital costs accounted for much of the inflation , accounting for 51 percent of the overall cost increase .\tFor the second year in a row , rises in hospital costs dominated the increase , accounting for 51 percent of the overall cost spiral .\n0\t3218713\t3218830\tQ : Can I buy coverage for prescription drugs right away ?\tCongress has added a new benefit - an option to buy insurance coverage for prescription drugs .\n1\t221079\t221003\tThe airline also said it has the option to buy 380 more airplanes , orders that would be split evenly between the two manufacturers .\tThe airline has the option to buy 380 more , split evenly between the two manufacturers .\n1\t2546175\t2546198\tDr Mark McClean , Jonathan 's family doctor , said if the drug had been administered earlier Jonathan would have retained more of his brain functions .\tDr Mark McClean , the family 's GP , said had the drug been administered to Jonathan earlier , he would have retained more of his brain function .\n0\t799346\t799268\tThe chain operates more than 3,400 stores , and has annual revenue of about $ 15.8 billion .\tThe chain , which has been under new management since late 1999 , has more than 3,400 stores and $ 15.8 billion in annual revenue .\n0\t2673104\t2673130\tAll patients developed some or all of the symptoms of E. coli food poisoning : bloody diarrhea , vomiting , abdominal cramping and nausea .\tSymptoms of the E. coli infection include bloody diarrhea , nausea , vomiting and abdominal cramping .\n1\t1354501\t1354476\tFederal regulators have turned from sour to sweet on a proposed $ 2.8 billion merger of ice cream giants Nestle Holdings Inc. and Dreyer 's Grand Ice Cream Inc .\tFederal regulators have changed their minds on a proposed $ 2.8 billion merger of ice cream giants Nestle Holdings and Dreyer 's Grand Ice Cream .\n1\t3070979\t3070949\tEnvironmental campaigners are using this weekend ’ s lunar eclipse to highlight the huge increase in light pollution across the UK .\tEnvironmental campaigners used the eclipse to highlight the surge in light pollution across Britain .\n0\t1264509\t1264471\tAvailable July 7 , the software supports the Solaris , IBM AIX , Red Hat Linux and Windows operating systems .\tThe OpForce product currently works with Solaris , AIX , Red Hat Linux and Windows servers .\n1\t103280\t103431\tJustice Minister Martin Cauchon and Prime Minister Jean Chrétien have both said the Liberal government will introduce legislation soon to decriminalize possession of small amounts of pot for personal use .\tJustice Minister Martin Cauchon and Prime Minister Jean Chretien both have said the government will introduce legislation to decriminalize possession of small amounts of pot .\n0\t110731\t110648\tBut Chauncey Billups demonstrated he 's also capable of big games , scoring 77 points over the final two games against the Magic .\tBillups scored 77 points in the final two games of the first-round series against the Magic .\n1\t2274844\t2274714\tKelly killed himself after being exposed as the source for a BBC report which claimed the government had embellished evidence of Iraq 's banned weapons to justify the war .\tHe killed himself after being exposed as the source for a BBC report which claimed the government exaggerated the case for war against Iraq .\n0\t1050307\t1050144\tAnd it 's going to be a wild ride , \" said Allan Hoffenblum , a Republican consultant .\tNow the rest is just mechanical , \" said Allan Hoffenblum , a Republican consultant .\n1\t2810634\t2810670\tWhile the Ibrahims had one separation operation , Goodrich and Dr. David Staffenberg plan about three for the Aguirres , with several weeks between each .\tInstead of one long operation to separate the twins , Goodrich and Dr. David Staffenberg plan about three , with several weeks between each .\n1\t3073773\t3073779\tLay had contended that turning over the documents would violate his Fifth Amendment right against self-incrimination .\tLay had refused to turn over the papers , asserting his Fifth Amendment right against self-incrimination .\n0\t261202\t260995\tThe WHO experts didn 't say how many cases in Hebei were in rural areas .\tHebei has reported 191 cases and eight deaths , though the WHO experts did not say how many were in rural areas .\n1\t1824224\t1824209\tNearly 300 mutinous troops who seized a Manila shopping and apartment complex demanding the government resign gave up and retreated peacefully after some 19 hours .\tMutinous troops who seized a Manila shopping and apartment complex demanding the government resign ended a 19-hour standoff late Sunday and returned to barracks without a shot fired .\n1\t548867\t548785\tIn three years , Lend Lease has slipped from a top-five stock , when its share price was around $ 24 , to 37th .\tIn the space of three years , Lend Lease has slipped from a top-five 5 stock when its share price hovered around $ 24 to 37th on the list .\n0\t2796658\t2796682\tAbout two hours later , his body , wrapped in a blanket , was found dumped a few blocks away .\tThen his body was dumped a few blocks away , found in a driveway on Argyle Road .\n1\t1808166\t1808434\tColumbia broke up over Texas upon re-entry on Feb. 1 .\tColumbia broke apart in the skies above Texas on Feb. 1 .\n1\t853475\t853342\tA year or two later , 259 , or 10 per cent , of the youths reported that they had started to smoke , or had taken just a few puffs .\tWithin two years , 259 , or 10 percent , of the youths reported they had started to smoke or had at least taken a few puffs .\n0\t977772\t977804\tThe Lord Chancellor was guardian of the Great Seal , used to stamp all official documents from the sovereign .\tFalconer will hold on , for now , to the Lord Chancellor 's Great Seal , used to sign off instructions from the sovereign .\n1\t577854\t578500\tCindy Yeast , a 50-year-old Washington-area publicist , says she began taking supplements two years ago in part to avoid mild dementia that affects her elderly parents .\tShe started taking supplements two years ago - partly to stave off mild dementia that affects her elderly parents .\n1\t2829194\t2829229\tThe two are not related , but have referred to each other as father and son .\tHe 's not related to Malvo , but the two have referred to each other as father and son .\n1\t2074182\t2074668\tGibson said last month in a press statement that \" neither I nor my film are anti-Semitic .\tGibson said in a June statement that he and his film are not anti-Semitic .\n0\t2758265\t2758282\tThe world 's largest software company said it recognized the difficulty the multiple patches posed for companies , and set out to make it easier for them to apply the updates .\tThe world 's largest software company said it recognized the difficulty the multiple patches posed for companies trying to apply them .\n1\t1958079\t1958143\tThe Dow Jones industrial average .DJI ended up 64.64 points , or 0.71 percent , at 9,191.09 , according to the latest available data .\tThe blue-chip Dow Jones industrial average .DJI added 38 points , or 0.42 percent , to 9,165 .\n1\t544217\t544325\tThe vote came just two days after Kurds swept City Council elections , taking the largest single block of votes on the 30-seat council .\tThe vote for mayor followed City Council elections that gave Kurds the largest block of votes on the 30-seat council .\n1\t2385288\t2385256\tLarge swells and dangerous surf already were being felt along sections of the coast .\tAlready large swells and dangerous surf have arrived along the mid-Atlantic .\n0\t2324708\t2325028\tBased on a separate survey of households , the unemployment rate fell in August to 6.1 percent from 6.2 percent .\tLabor Department analysts discounted a slight improvement in the national unemployment rate , which fell in August to 6.1 percent from 6.2 percent .\n1\t2139506\t2139427\t\" We will work with the board to ensure a smooth transition . \"\tHe said federal regulators would work with the corporation to ensure a \" smooth transition . \"\n1\t2965576\t2965701\tGasps could be heard in the courtroom when the photo was displayed .\tGasps could be heard as the photo was projected onto the screen .\n1\t2931098\t2931144\tGilead had earnings of $ 73.1 million , or 33 cents a share , compared with $ 20.8 million , or 10 cents , in the year-ago quarter .\tQuarterly profit climbed to $ 73.1 million , or 33 cents a share , from $ 20.8 million , or 10 cents , a year earlier , the company said .\n0\t644788\t644816\t\" I had one bad stretch of holes that put me out of contention to win , \" Woods said .\t\" I had one bad stretch of holes that put me out of contention , \" Woods said , referring to his 42 on the front nine Saturday .\n0\t2551891\t2551563\tThe poll had a margin of error of plus or minus 2 percentage points .\tIt had a margin of sampling error of plus or minus four percentage points and was conducted Thursday through Saturday .\n1\t1089053\t1089297\tSen. Patrick Leahy of Vermont , the committee 's senior Democrat , later said the problem is serious but called Hatch 's suggestion too drastic .\tSen. Patrick Leahy , the committee 's senior Democrat , later said the problem is serious but called Hatch 's idea too drastic a remedy to be considered .\n1\t3435735\t3435717\tThe broad Standard & Poor 's 500 < .SPX > eased 0.37 of a point , or 0.03 percent , at 1,121 .\tThe Standard & Poor 's 500 Index < .SPX > slipped 0.26 point , or 0.02 percent , to 1,121.96 .\n0\t1954\t2142\tWatertown , Saugus and Framingham also are going smoke-free Monday , joining a growing number of cities around the country .\tAlong with Boston , Watertown , Saugus and Framingham also are going smoke-free Monday .\n1\t3400796\t3400822\tThat is evident from their failure , three times in a row , to get a big enough turnout to elect a president .\tThree times in a row , they failed to get a big _ enough turnout to elect a president .\n1\t1220668\t1220801\tWe firmly believe we have an absolute right to use the common word ' spike ' as the name of our network . \"\tWe firmly believe that we have an absolute right to use the common word ' spike ' to name our network .\n1\t1889954\t1889847\tSources who knew of the bidding said last week that cable TV company Comcast Corp. was also looking at VUE .\tLate last week , sources told Reuters cable TV company Comcast Corp. CMCSA.O also was looking at buying VUE assets .\n1\t315785\t315653\tBut MTA officials appropriated the money to the 2003 and 2004 budgets without notifying riders or even the MTA board members considering the 50-cent hike , Hevesi found .\tMTA officials appropriated the surplus money to later years ' budgets without notifying riders or the MTA board members when the 50-cent hike was being considered , he said .\n0\t1521034\t1520582\tWhite , who had suffered kidney failure from years of high blood pressure , died at Cedars-Sinai Medical Center around 9 : 30 a.m. , said manager Ned Shankman .\tWhite , who had kidney failure from years of high blood pressure , had been undergoing dialysis and had been hospitalized since a September stroke .\n1\t2083598\t2083810\tAbout 10 percent of high school and 16 percent of elementary students must be proficient at math .\tIn math , 16 percent of elementary and middle school students and 9.6 percent of high school students must be proficient .\n1\t1910610\t1910455\tThe legal ruling follows three days of intense speculation Hewlett-Packard Co. may be bidding for the company .\tThe legal ruling follows three days of wild volatility in RIM 's stock over speculation that PC giant Hewlett-Packard Co. may be bidding for the company .\n1\t3113791\t3113782\tThe European Commission , the EU 's antitrust enforcer , is expected to issue its decision next spring — unless a settlement is reached .\tThe European Commission is expected to issue its decision in the case next spring — unless a settlement is reached .\n1\t3214517\t3214483\t\" So Sebastian did his best to convincingly confess to a crime that he didn 't commit in order to survive , \" she told jurors .\t\" Sebastian did his best to confess convincingly to a crime he didn 't do in order to survive , \" Ms. Richardson declared .\n0\t2083612\t2083810\tTwenty percent of Latino students and 23 percent of black students performed at proficient or higher .\tIn math , 16 percent of elementary and middle school students and 9.6 percent of high school students must be proficient .\n1\t661390\t661218\tHe is charged in three bombings in Atlanta including a blast at the 1996 Olympics and one in Alabama .\tHe is charged in three bombings in Atlanta - including a blast at the 1996 Olympics - along with the bombing in Alabama .\n1\t1269572\t1269682\tThe men were remanded in custody and are due to appear again before court on July 8 .\tThey were remanded in custody and will appear in court again on July 8 .\n1\t1095780\t1095652\t\" No matter who becomes the sponsor for stock-car racing 's top series , NASCAR will need an all-star event , \" Wheeler said in a statement .\tNo matter who becomes the sponsor for stock-car racings top series , NASCAR will need an all-star event , Wheeler said Tuesday .\n1\t116294\t116332\tThe Phillies were upset that Counsell had stolen second in the sixth inning with Arizona leading 7-1 .\tThe Phillies were apparently upset when Counsell stole during the sixth with the Diamondbacks up 7-1 .\n1\t941617\t941673\tHe said his hatred for such people grew from these discussions and had helped convince him violence was the answer .\tHis hatred for these people had germinated from these discussions and helped cement his belief that violence was the panacea .\n1\t2640607\t2640576\t\" There is no need for one deadline for all to create the ASEAN Economic Community , \" Thaksin said .\tThus , he said , there did not have to one deadline to create the economic community .\n1\t3310210\t3310286\tThe announcement was made during the recording of a Christmas concert attended by top Vatican cardinals , bishops , and many elite from Italian society , witnesses said .\tThe broadside came during the recording on Saturday night of a Christmas concert attended by top Vatican cardinals , bishops and many elite of Italian society , witnesses said .\n1\t3376093\t3376101\tThe additional contribution brings total U.S. food aid to North Korea this year to 100,000 tonnes .\tThe donation of 60,000 tons brings the total of U.S. contributions for the year to 100,000 .\n1\t1549586\t1549609\tLeon Williams ' body was found inside his third-floor apartment at 196 Bay St. , in Tompkinsville .\tThe dead man , Leon Williams , was found in his third-floor apartment .\n1\t460211\t460445\tThe player 's eyes were bloodshot and a blood-alcohol test produced a reading of 0.18 - well above Tennessee 's level of presumed intoxication of 0.10 , the report said .\tHe failed a field sobriety test and a blood-alcohol test produced a reading of 0.18 – well above Tennessee 's level of presumed intoxication of 0.10 , the report said .\n1\t1196962\t1197061\tBut Virgin wants to operate Concorde on routes to New York , Barbados and Dubai .\tBranson said that his preference would be to operate a fully commercial service on routes to New York , Barbados and Dubai .\n0\t862804\t862715\tHe tried to fight off officers and was taken to a hospital after a police dog bit him but was later released .\tCruz tried to fight off officers and was hospitalized after a police dog bit him , Sgt. Steve Dixon said .\n1\t1726935\t1726879\tThe announcement , which economists said was not a surprise , may be bittersweet for the millions of Americans without jobs .\tEconomists said the announcement was not a surprise , and politicians said it offered little comfort to the millions of Americans without jobs .\n0\t331980\t332110\tAsked if the delegates could leave on Friday , police intelligence chief in Aceh , Surya Dharma , told reporters they could not because they did not have proper permission .\tAsked if the delegates could leave on Friday , police intelligence chief Surya Dharma told reporters : \" Of course they may not go .\n1\t173879\t173832\tDealers said the dollar also drew some downside support as Japanese investors are expected to keep snapping up foreign bonds amid the yen 's rise against the dollar .\tDealers said the dollar also drew some downside support as Japanese investors are expected to keep snapping up foreign bonds amid ever-falling domestic interest rates .\n0\t2834988\t2835026\tIran has until the end of the month to satisfy the agency it has no plans for nuclear weapons .\tThe Iranians have until the end of the month to answer all the agency 's questions about their past nuclear activities .\n1\t2587300\t2587243\tHer father , Florin Cioaba , the king of Transylvania 's Gypsies , had her brought back and she was married against her will .\tHer father , Roma King Florin Cioaba , had her brought back and she was promptly married against her will .\n0\t554905\t554627\tClaire had advanced to the third round of the 76th annual Scripps Howard National Spelling Bee .\tOne by one they strolled to the microphone , all 251 youngsters in the 76th Scripps Howard National Spelling Bee .\n1\t1912524\t1912648\tCitigroup Inc . C.N , the world 's largest financial services company , on Wednesday promoted Marjorie Magner to chairman and chief executive of its global consumer group .\tCitigroup ( C ) on Wednesday named Marjorie Magner chairman and chief executive of its colossal global consumer business .\n1\t3255597\t3255668\t\" They 've been in the stores for over six weeks , \" says Carney .\tThe quarterlies usually stay in stores for between six to eight weeks , \" Carney added .\n1\t629316\t629289\tLet me just say this : the evidence that we have of weapons of mass destruction was evidence drawn up and accepted by the joint intelligence community .\t\" The evidence that we had of weapons of mass destruction was drawn up and accepted by the Joint Intelligence Committee , \" he said .\n1\t54181\t53570\tRidge said no actual explosives or other harmful substances will be used .\tRidge said no real explosives or harmful devices will be used in the exercise .\n1\t723557\t724115\tThus far , Stewart 's company appears ready to stand behind her .\tFor now , the company 's management appears to be standing behind Stewart .\n0\t2607718\t2607708\tBut late Thursday night , the campaign issued a statement saying there would be no news conference and no big announcement .\tBut late yesterday , the campaign and the state Democratic Party said there would be no news conference .\n1\t753858\t753890\tThere 's also a flaw that results because IE does not implement an appropriate block on a file download dialog box .\tThe second vulnerability is a result of IE not implementing a block on a file download dialog box .\n1\t587009\t586969\tAnother $ 100-million in savings will come from management layoffs and pay cuts .\tThe airline expects to save another $ 100-million a year through management layoffs and pay cuts .\n1\t308567\t308525\tHe called on Prime Minister John Howard to establish a royal commission on child sex abuse .\tThe Senate motion also called on Prime Minister John Howard to hold a royal commission into child sex abuse .\n0\t665419\t665612\t\" We think that the United States of America should support the free speech of all groups , \" Mr. White said , objecting to Mr. Olson 's recommendation .\tWe think that the United States of America should support the free speech of all groups , he said .\n1\t2763517\t2763576\tTerri Schiavo , 39 , underwent the procedure at the Tampa Bay area hospice where she has been living for several years , said her father , Bob Schindler .\tThe tube was removed Wednesday from Terri Schiavo , 39 , at the Tampa Bay-area hospice where she has lived for several years .\n0\t3107118\t3107136\tAfter 18 months , Nissen found that Lipitor stopped plaque buildup in the patients ' arteries .\tAfter 18 months , the atorvastatin patients had no change in the plaque in their arteries .\n1\t780604\t780466\tToll , Australia 's second-largest transport company , last week offered NZ75 a share for Tranz Rail .\tToll last week offered to buy the company for NZ75c a share , or $ NZ158 million .\n0\t1989213\t1989116\t\" This child was literally neglected to death , \" Armstrong County District Attorney Scott Andreassi said .\tArmstrong County District Attorney Scott Andreassi said the many family photos in the home did not include Kristen .\n1\t1462409\t1462504\tWal-Mart , the nation 's largest private employer , has expanded its antidiscrimination policy to protect gay and lesbian employees , company officials said Tuesday .\tWal-Mart Stores Inc . , the nation 's largest private employer , will now include gays and lesbians in its anti-discrimination policy , company officials said Wednesday .\n1\t260952\t260924\tMetro , bus and local rail services in France 's four largest towns -- Paris , Lyon , Lille and Marseille -- were severely disrupted , Europe 1 radio reported .\tSubway , bus and suburban rail services in France 's four largest cities -- Paris , Lyon , Lille and Marseille -- were severely disrupted , transport authorities said .\n1\t1224743\t1225510\tIn the undergraduate case , Rehnquist said the use of race was not \" narrowly tailored \" to achieve the university 's asserted interest in diversity .\tRehnquist wrote that the system was not narrowly tailored to achieve the interest in educational diversity .\n0\t3329379\t3329416\tSP2 is basically about security enhancements to Windows , such as the improved Internet Connection Firewall ( ICF ) .\tThe firewall in the current Windows XP was known as the Internet Connection Firewall ( ICF ) .\n1\t2362761\t2362698\tA landslide in central Chungchong province derailed a Seoul-bound train and 28 passengers were injured , television said .\tIn central Chungchong province , a landslide caused a Seoul-bound Saemaeul Express train to derail , injuring 28 people , local television said .\n0\t1465073\t1464854\tThey will help draft a plan to attack obesity that Kraft will implement over three to four years .\tThe team will help draft a plan by the end of the year to attack obesity .\n1\t195728\t196099\tBut that amount would probably be impossible to pass in the Senate , where Republican moderates have refused to go above $ 350 billion .\tSuch an amount would probably be unable to summon a majority of the Senate , where Republican moderates have refused to go above $ 350 billion .\n1\t2587767\t2587673\tIn the clash with police , Lt. Mothana Ali said about 1,000 demonstrators had gone to the station demanding jobs .\tIn Baghdad , police Lieut . Mothana Ali said about 1,000 demonstrators arrived at the station demanding jobs .\n0\t1490044\t1489975\tCorixa shares rose 54 cents to $ 7.74 yesterday on the Nasdaq Stock Market .\tShares of Corixa rose 54 cents , or about 8 percent , to close at $ 7.74 .\n1\t958161\t957782\tCommittee approval , expected today , would set the stage for debate on the Senate floor beginning Monday .\tThat would clear the way for debate in the full Senate beginning on Monday .\n1\t1033204\t1033365\tO 'Brien was charged with leaving the scene of a fatal accident , a felony .\tBishop Thomas O 'Brien , 67 , was booked on a charge of leaving the scene of a fatal accident .\n0\t2996241\t2996734\tTom Hamilton said his daughter was conscious and alert and in stable condition after the attack Friday morning .\tBethany , who remained in stable condition after the attack Friday morning , talked of the attack Saturday .\n0\t2015389\t2015410\tThe Calgary woman , who is in her twenties , donated blood on Aug. 7 .\tThe woman -- who has no symptoms of illness -- donated blood Aug. 7 .\n1\t221515\t221509\tQuattrone lawyer John W. Keker said his client is innocent .\tIn a statement Monday , his lawyer John Keker said ``Frank Quattrone is innocent .\n0\t2283737\t2283794\tIn the weeks leading up to the execution , several Florida officials received anonymous threatening letters .\tSeveral Florida officials connected to the case have received threatening letters , accompanied by rifle bullets .\n1\t2826681\t2826474\tThe disagreement over online music sales was disclosed in documents filed last week with the judge and made available by the court yesterday .\tThe fight over online music sales was disclosed in documents made available Monday by the court .\n1\t2249237\t2249305\tParson was charged with intentionally causing and attempting to cause damage to protected computers .\tParson is charged with one count of intentionally causing damage to a protected computer .\n1\t389239\t389299\t\" The court and the public need to know much more of the details of the defendant 's seemingly massive fraud , \" the judge said .\t\" The court and the public need to know more of the defendants ' seemingly massive fraud , \" he said .\n1\t2652187\t2652218\tThe U.S. Supreme Court will hear arguments on Wednesday on whether companies can be sued under the Americans with Disabilities Act for refusing to rehire rehabilitated drug users .\tThe high court will hear arguments today on whether companies can be sued under the ADA for refusing to rehire rehabilitated drug users .\n1\t2945693\t2945847\tThe IRS said taxpayers can avoid undelivered checks by having refunds deposited directly into their checking or savings accounts .\tThe IRS said taxpayers can avoid problems with lost or stolen refunds by having refunds deposited directly into personal checking or savings accounts .\n1\t2065523\t2065836\t\" More than 70,000 men and women from bases in Southern California were deployed in Iraq .\tIn all , more than 70,000 troops based in Southern California were deployed to Iraq .\n1\t2222998\t2223097\tBP shares slipped 0.8 percent to 433.50 pence ( $ 6.85 ) each in afternoon trading on the London Stock Exchange .\tBP shares slipped 48 cents to $ 41.72 Friday in trading on the New York Stock Exchange .\n1\t2561999\t2561941\tBecause of the accounting charge , the company now says it lost $ 1.04 billion , or 32 cents a share , in the quarter ended June 30 .\tIncluding the charge , the Santa Clara , Calif.-based company said Monday it lost $ 1.04 billion , or 32 cents per share , in the period ending June 30 .\n0\t2324704\t2325023\tFriday 's report raised new worries that a weak job market could shackle the budding economic recovery despite a slight improvement in the overall unemployment rate .\tU.S. companies slashed payrolls for a seventh straight month in August , raising new worries that a weak jobs market could shackle the budding economic recovery .\n1\t2336453\t2336545\tFederal Emergency Management Administration designated $ 20 million to establish the registry .\tThe registry was launched with $ 20 million from the Federal Emergency Management Agency .\n1\t720572\t720486\tBREAST cancer cases in the UK have hit an all-time high with more than 40,000 women diagnosed with the disease each year , Cancer Re-search UK revealed yesterday .\tCases of breast cancer in Britain have reached a record high , with the number of women diagnosed with the disease passing the 40,000 mark for the first time .\n1\t1605818\t1605806\t\" It was never our intention to sell the product , \" said Health Minister Anne McClellan , a skeptic of medical marijuana use .\t\" It was never the intention of us to sell product , \" federal Health Minister Anne McLellan said yesterday in Edmonton .\n0\t2440680\t2440474\tGM , the world 's largest automaker , has 115,000 active UAW workers and another 340,000 retirees and spouses .\tThey cover more than 300,000 UAW workers and 500,000 retirees and spouses .\n0\t726399\t726078\tRosenthal is hereby sentenced to custody of the Federal Bureau of prisons for one day with credit for time served , \" Breyer said to tumultuous cheers in the courtroom .\t\" Rosenthal is hereby sentenced to custody of the Federal Bureau of Prisons for one day with credit for time served . \"\n1\t533903\t533818\t\" We are committed to helping the Iraqi people get on the path to a free society , \" Rumsfeld said in a speech to the Council on Foreign Relations .\t\" We are committed to helping the Iraqi people get on the path to a free society , \" he said .\n1\t1166473\t1166857\tMr. Young said he was disappointed that the government didn 't see the severe acute respiratory syndrome crisis as worthy of federal disaster-relief money .\tYoung said he was disappointed the government didn 't see the SARS crisis as worthy of federal disaster relief money .\n1\t144089\t143697\tThe 12-nation currency has risen by 33 percent against the dollar over the past 15 months .\tThe euro is up 9 percent against the dollar in the past six weeks .\n1\t3439854\t3439874\tIn February 2000 , the officers — Kenneth Boss , Sean Carroll , Edward McMellon and Richard Murphy — were acquitted of all charges in the killing .\tThe officers -- Kenneth Boss , Sean Carroll , Edward McMellon and Richard Murphy -- were acquitted in 2000 of state murder charges .\n1\t3464314\t3464302\tI was surprised it turned out me talking and the president just listening .\t\" I was surprised it turned out me talking and the president just listening . . . It was mostly a monologue . \"\n1\t2008984\t2009175\tThe state 's House delegation currently consists of 17 Democrats and 15 Republicans .\tDemocrats hold a 17-15 edge in the state 's U.S. House delegation .\n0\t816867\t816831\tFreddie also said Leland C. Brendsel will retire as chairman and chief executive and resign from the board .\tHe replaces Leland Brendsel , 61 , who retired as chairman and chief executive .\n1\t192285\t192327\tWe 'll be listening carefully to the [ IAEA ] director general 's report at the next board meeting .\t\" We 'll be listening carefully to the ( IAEA ) director-general 's report at the next board meeting . \"\n1\t2688145\t2688162\tIn that position , Elias will report to Joe Tucci , president and CEO of EMC .\tAs executive vice president of new ventures , Elias will report to Joe Tucci , EMC 's president and chief executive .\n1\t3294207\t3294290\tBut with the PM due to leave tomorrow afternoon for personal reasons there was a risk he might not be present when the final decision was made .\tBut with the Prime Minister due to leave tomorrow , a day early , he may not be present when the final decision is made .\n0\t205100\t205145\tA pro-independence radical , Miodrag Zivkovic , of the Liberal Alliance , came in second with 31 percent of the vote .\tMiodrag Zivkovic , of the Liberal Alliance of Montenegro , won 31 percent of the vote while the independent Dragan Hajdukovic got four percent .\n0\t3242051\t3241897\tMr. Kerkorian tried unsuccessfully to take over Chrysler in 1995 , but did win representation on its board .\tKerkorian and Tracinda had also tried to take over Chrysler in 1995 .\n0\t1076861\t1077018\tGlover spoke at a news conference that included about 20 relatives of the victims .\tAbout 20 family members of the victims were invited to the news conference .\n1\t2095803\t2095786\tDrax faced a financial crisis late last year after it lost its most lucrative sales contract , held with insolvent utility TXU Europe .\tDrax ’ s troubles began late last year when it lost its most lucrative sales contract , with the insolvent utility TXU Europe .\n1\t2112330\t2112376\tBut I would rather be talking about high standards than low standards . \"\t\" I would rather be talking about positive numbers rather than negative .\n1\t3389318\t3389271\tIt was not immediately known how many people were on flight UTA 141 , which could carry 141 passengers and crew .\tIt was still not known exactly how many people were on the plane , which could carry 141 passengers and crew .\n1\t698948\t698933\tThe market remains pinned in a narrow range after a powerful rally drove the broad Standard & Poor 's 500 index .SPX up more than 20 percent since mid-March .\tThe market remains pinned in a narrow range after a powerful rally pushed the broad S & P 500 index up more than 20 percent since mid-March .\n1\t539585\t539355\tWitnesses said they believed the man planned to crash the Launceston-bound Qantas flight 1737 , which was carrying 47 passengers and six crew .\tWitnesses believe he wanted to crash Flight 1737 , which had 47 passengers and six crew .\n1\t684848\t684557\tAs Samudra sat down to hear the indictment , he looked over to his nine lawyers and shouted ``God is Great ' ' three times .\tAs he sat down to hear the indictment , Samudra looked over to his nine lawyers and shouted \" Takbir ! \" , or \" Proclaim ! \" , a religious rallying cry .\n1\t347017\t347002\tIn hardest-hit Taipei , traffic has disappeared from once bustling streets , ubiquitous department stores stand mostly empty and restaurants are eerily quiet .\tIn hardest-hit Taipei , traffic has disappeared from once-bustling streets and department stores and restaurants are virtually empty .\n1\t1592037\t1592076\tIn a statement , Lee said he \" no longer believes that Viacom deliberately intended to trade on my name when naming Spike TV . \"\tSpike Lee no longer believes that Viacom deliberately intended to trade on his name by calling its own venture \" Spike TV , \" according to a statement read in court Tuesday .\n0\t3013483\t3013540\tSingapore Prime Minister Goh Chok Tong says China plays an important role in the integration of Asia , including managing the stresses and strains both within and between countries .\tHAINAN PROVINCE , China : Singapore Prime Minister Goh Chok Tong said China plays an important role in the integration of Asia .\n1\t2020252\t2020081\tThe worm attacks Windows computers via a hole in the operating system , an issue Microsoft on July 16 had warned about .\tThe worm attacks Windows computers via a hole in the operating system , which Microsoft warned of 16 July .\n0\t2614947\t2614904\tThe premium edition adds OfficeFront Page 2003 , Acceleration Server 2000 , and SQL Server 2000 .\tThe premium edition adds ISA Server , SQL Server and a specialized edition of BizTalk 2004 .\n0\t1744257\t1744378\tIn the year-ago quarter , the steelmaker recorded a profit of $ 16.2 million , or 15 cents per share , on sales of $ 1.14 billion .\tIn the second quarter last year , AK Steel reported a profit of $ 16.2 million , or 15 cents a share .\n0\t1119721\t1119714\tSony claimed that the reader 's capacitance sensing technology cannot be fooled by paper copies and does not require cleaning .\tIts capacitance sensing technology electronically reads a fingerprint ; Sony says it can 't be fooled by paper copies and doesn 't require cleaning .\n1\t1186754\t1187056\tAmazon.com shipped out more than a million copies of the new book , making Saturday the largest distribution day of a single item in e-commerce history .\tAmazon.com shipped more than a million copies by Saturday afternoon , making Saturday the largest distribution day of a single item in e-commerce history .\n1\t2842562\t2842582\tThe show 's closure affected third-quarter earnings per share by a penny .\tThe company said this impacted earnings by a penny a share .\n0\t431076\t431242\tAfter the two-hour meeting on May 14 , publisher Arthur O. Sulzberger Jr . , executive editor Howell Raines and managing editor Gerald Boyd pledged quick remedies to staff grievances .\tThe committee will make recommendations to Publisher Arthur Sulzberger , Executive Editor Howell Raines and Managing Editor Gerald Boyd .\n1\t1393764\t1393984\tIt 's been a busy couple of days for security gurus assigned to keep their companies safe and sound .\tIt 's been a busy couple of days for enterprise security gurus tasked with the job of keeping their companies safe and sound .\n0\t2916199\t2916164\tLu reclined in a soft chair wearing a woolly coat near the blackened capsule .\t\" It 's great to be back home , \" said Lu , dressed in a woolly coat near the blackened capsule .\n1\t2530671\t2530542\tGov. Bob Riley proposed the budget cuts after Alabama voters rejected his $ 1.2 billion tax plan Sept . 9 .\tAfter Alabama voters rejected his $ 1.2 billion tax plan Sept . 9 , Riley forecast significant cuts in state programs .\n1\t219064\t218969\t\" It is probably not the easiest time to come in and take over the shuttle program , but then again , I look forward to the challenge , \" he said .\t\" It 's probably not the easiest time to come in and take over the shuttle program , but I look forward to the challenge , \" Parsons told reporters at NASA headquarters .\n0\t2377289\t2377259\tEstonia 's place in the European mainstream and safeguard its independence regained in 1991 .\tEstonia was forcibly incorporated in the Soviet Union in 1940 and regained its independence only in 1991 .\n0\t2110220\t2110199\tFranklin County Judge-Executive Teresa Barton said a firefighter was struck by lightning and was taken to the Frankfort Regional Medical Center .\tA county firefighter , was struck by lightning and was in stable condition at Frankfort Regional Medical Center .\n0\t1864253\t1863810\tPolice suspected that Shaichat , 20 , had been abducted either by Palestinians or by Israeli Arabs .\tNobody claimed responsibility for Schaichat 's death , but police suspect that the 20-year-old soldier was abducted either by Palestinians or Israeli Arabs .\n0\t3150803\t3150839\tDuring this year 's August to October quarter , Lowe 's opened 38 new stores , including two relocations .\tDuring the third quarter , Lowe 's opened 38 new stores and now has 932 stores in 45 states .\n0\t969381\t969512\tThe technology-laced Nasdaq Composite Index < .IXIC > declined 25.78 points , or 1.56 percent , to 1,627.84 .\tThe broader Standard & Poor 's 500 Index .SPX gave up 11.91 points , or 1.19 percent , at 986.60 .\n1\t271891\t271839\tSony said the PSP would also feature a 4.5-inch LCD screen , Memory Stick expansion slots .\tIt also features a 4.5 in back-lit LCD screen and memory expansion facilities .\n0\t2829648\t2829613\tClinton did not mention that two Democratic senators , Charles Robb of Virginia and Wendell Ford of Kentucky , voted to shelve the McCain bill .\tTwo Democrats , Sen. Charles Robb of Virginia and Wendell Ford of Kentucky , voted with the 40 Republicans .\n1\t886904\t887158\tSome of the company 's software developers will join Microsoft , but details haven 't been finalized , said Mike Nash , corporate vice president of Microsoft 's security business unit .\tSome of the companys software developers will join Microsoft , but details havent been finalized , said Mike Nash , corporate vice president of Microsofts security business unit .\n0\t2632692\t2632767\tWal-Mart has said it plans to open at least 40 Supercenters in the state in the coming years ; analysts expect four or more to be in San Diego County .\tAt least 40 of the outlets will be in California , and analysts expect four or more to be in San Diego County .\n1\t2240399\t2240149\tCintas is battling efforts to unionize 17,000 of its workers and to let unions organize the workers by signing cards , rather than by a lengthy election process .\tCintas is battling efforts to unionize 17,000 of its workers and labor 's demands to let its workers organize by signing cards , rather than by a lengthy election process .\n1\t805457\t805985\tThe opposition would resort to rolling mass action \" at strategic times of our choice and without warning to the dictatorship , \" he said .\t\" From now onwards we will embark on rolling mass action at strategic times of our choice and without any warning to the dictatorship , \" he said .\n1\t2896308\t2896334\tFederal Agriculture Minister Warren Truss said the Government still did not know the real reason the sheep were rejected at the Saudi port of Jeddah on August 21 .\tHe said the Government still did not know the real reason the original Saudi buyer pulled out on August 21 .\n1\t2110775\t2110924\tTom Kraynak , manager of operations and resources for the Canton , Ohio-based East Central Area Reliability Council , said that scenario is one among many that investigators are considering .\tTom Kraynak , manager of operations and resources for the Canton , Ohio-based East Central Area Reliability Council , said investigators are considering the scenario .\n1\t1762569\t1762526\tHester said Sanmina was the best fit among several purchase offers the company received from electronics manufacturers and computer makers .\tHester said Sanmina 's offer was the best among several Newisys received from electronics manufacturers and computer makers .\n0\t2706154\t2706185\tThe other inmate fell but Selenski shimmed down the makeshift rope to a second-story roof and used the mattress to scale a razor-wire fence , Fischi said .\tAfter the other inmate fell , Selenski used the mattress to scale a 10-foot , razor-wire fence , Fischi said .\n1\t1057995\t1057778\tThe hearing , expected to last a week , will determine whether Akbar faces a court-martial .\tThe purpose of the hearing is to determine whether Akbar should be court-martialled .\n1\t1386884\t1386857\tHe said he has begun a court action to seize Beacon Hill 's assets and has frozen more than $ 13 million Beacon Hill had when it closed .\tHe said he has initiated a forfeiture action in court and frozen more than $ 13 million Beacon Hill had when it closed .\n1\t3093023\t3092996\tSpeaking for the first time yesterday , Brigitte 's maternal aunt said his family was unaware he had was in prison or that he had remarried .\tBrigitte 's maternal aunt said his family was unaware he had been sent to prison , or that he had remarried in Sydney .\n1\t1661381\t1661317\t\" Close co-operation between our law enforcement agencies , close co-operation between our intelligence services lie at the heart of the ongoing fight against terrorism . \"\tClose cooperation between regional law enforcement agencies and intelligence services was at the heart of the fight against terrorism , he said .\n0\t2926039\t2925982\tThe mother of a Briton held by Colombian guerrillasspoke of her relief yesterday after hearing that he might be freed in the next few weeks .\tThe parents of a Briton being held hostage by Colombian rebels spoke yesterday of their optimism that he would be freed in time for his birthday next month .\n0\t637168\t637447\tWe strongly disagree with Novell 's position and view it as a desperate measure to curry favor with the Linux community .\tMcBride characterized Novell 's move as \" a desperate measure to curry favor with the Linux community . \"\n1\t696677\t696932\tAfter more than two years ' detention under the State Security Bureau , the four were found guilty of subversion in Beijing 's No. 1 Intermediate Court last Wednesday .\tAfter more than two years in detention by the State Security Bureau , the four were found guilty last Wednesday of subversion .\n1\t3122429\t3122305\tMr Russell , 46 , a coal miner from Brisbane , said : \" They are obviously hurting , so we are basically going over there to help them . \"\t\" They are obviously hurting so we are basically going over there to help them , \" Russell , 46 , said .\n1\t1348909\t1348954\tThe New York Democrat and former first lady has said she will not run for the White House in 2004 , but has not ruled out a race in later years .\tThe former first lady has said she will not run for the White House in 2004 but has not ruled out a race later on .\n0\t162203\t162101\tIt does not affect the current Windows Media Player 9.0 Series .\tWindows Media Player has had security problems before .\n0\t71501\t71627\tThe seizure took place at 4 a.m. on March 18 , just hours before the first American air assault .\tThe time was about 4 a.m. on March 18 , just hours before the first pinpoint missiles rained down on the capital .\n1\t2907762\t2907649\tDonations stemming from the Sept . 11 attacks helped push up contributions to human service organizations and large branches of the United Way by 15 percent and 28.6 percent , respectively .\tDonations stemming from the Sept . 11 attacks helped push up contributions to human service organizations by 15 percent and to large branches of the United Way by 28.6 percent .\n1\t2167771\t2167744\tIn May , Mr. Hatfill said he was struck by a vehicle being driven by an FBI employee who was tailing him in Georgetown .\tLast May , Hatfill was struck by a vehicle being driven by an FBI employee who was tailing him in Washington 's Georgetown neighborhood .\n1\t3320577\t3320553\t\" I will support a constitutional amendment which would honor marriage between a man and a woman , codify that , \" he said .\t\" If necessary , I will support a constitutional amendment which would honour marriage between a man and a woman , codify that . \"\n1\t849291\t849442\tIBM of the US and Infineon Technologies of Germany will today announce a technological development that could threaten multi-billion dollar memory chip markets .\tIBMof the US andInfineon Technologies of Germany willon Tuesdayannounce a technological development that could threaten multi-billion dollar memory chip markets .\n0\t763948\t763991\tCosta 's semifinal opponent is Spaniard Juan Carlos Ferrero , whom he beat in last year 's final .\tCosta will play Juan Carlos Ferrero next in a rematch of last year 's final .\n1\t1908763\t1908744\tA former employee of a local power company pleaded guilty Wednesday to setting off a bomb that knocked out a power substation during the Winter Olympics last year .\tA former Utah Power meter reader pleaded guilty Wednesday to bombing a power substation during the 2002 Winter Olympics .\n0\t1876120\t1876059\tThyroid hormones are known to help in weight loss by stimulating metabolism - and cutting cholesterol - but come with the unwanted side effect of speeding up the heartbeat .\tThyroid hormones are known to help in weight loss by stimulating metabolism , and they can help cut cholesterol too .\n1\t518089\t518133\tJudge Craig Doran said it wasn 't his role to determine if Hovan was \" an evil man \" but maintained that \" he has committed an evil act . \"\tJudge Craig Doran said he couldn 't determine if Hovan was \" an evil man \" but said he \" has committed an evil act . \"\n0\t224932\t224868\tThe Hartford shares rose $ 2.88 , or 6.6 percent , to close Monday at $ 46.50 on the New York Stock Exchange .\tShares of Hartford rose $ 2.88 to $ 46.50 in New York Stock Exchange composite trading .\n1\t1771131\t1771091\tIt also offers a built-in NAND flash boot loader so that high-density NAND flash memory can be used without having to install an additional support chip .\tThe S3C2440 has a built-in NAND flash boot loader , for example , so that high-density NAND flash memory can be installed without an additional support chip .\n0\t2728425\t2728251\tIt decided instead to issue them before the stock market opened Monday after the downgrade of its debt late Friday by Moody 's , the credit rating agency .\tIt decided instead to issue them before the stock market opened Monday to counteract the downgrade of its debt late Friday by Moody 's to one step above junk status .\n0\t953733\t953537\tAltria shares fell 2.5 percent or $ 1.11 to $ 42.57 and were the Dow 's biggest percentage loser .\tIts shares fell $ 9.61 to $ 50.26 , ranking as the NYSE 's most-active issue and its biggest percentage loser .\n1\t349215\t349241\tIt will be followed in November by a third movie , \" The Matrix Revolutions . \"\tThe film is the second of a trilogy , which will wrap up in November with \" The Matrix Revolutions . \"\n1\t2919853\t2919804\tMassachusetts regulators and the Securities and Exchange Commission on Tuesday pressed securities fraud charges against Putnam Investments and two of its former portfolio managers for alleged improper mutual fund trading .\tState and federal securities regulators filed civil charges against Putnam Investments and two portfolio managers in the ever-expanding mutual fund trading scandal .\n1\t954526\t954607\tHe is blocking them until the Air Force assigns four additional C-130 cargo planes to Gowen Field , an Idaho Air National Guard base in Boise .\tHe is holding them up until the Air Force agrees to assign four additional C-130 cargo planes to the Idaho Air National Guard .\n1\t69773\t69792\tCisco pared spending to compensate for sluggish sales .\tIn response to sluggish sales , Cisco pared spending .\n0\t2823575\t2823513\tThe study , published Monday in the journal Molecular Brain Research , is likely to also apply to humans , its authors said .\tThe study , conducted on the brains of developing mice , was being published today in the journal Molecular Brain Research .\n1\t2455942\t2455978\tMy decision today is not based on any one event . \"\tGovernor Rowland said his decision was \" not based on any one event . \"\n1\t131979\t131957\tNelson , 27 , is being retried on civil-rights charges stemming from the disturbance which led to Rosenbaum 's death .\tNelson , 27 , is being retried on civil rights charges stemming from the disturbance that led to Rosenbaum 's death .\n0\t2010705\t2010779\t\" The government elements who have been causing trouble are still in place .\tThe government elements who have been causing trouble are still in place , they are attacking us . \"\n1\t54142\t53641\tNext Monday at about 2 p.m. ( CST ) , hospital officials in and near Chicago will notice a sudden increase in people complaining of flu-like symptoms .\tAround the same time , hospital officials in and near Chicago will notice a sudden increase in people complaining of flu-like symptoms .\n1\t1015249\t1015204\tWal-Mart Stores Inc . , Kohl 's Corp. , Family Dollar Stores Inc. and Big Lots Inc. were among the merchants posting May sales that fell below Wall Street 's modest expectations .\tWal- Mart , Kohl 's Corp. , Family Dollar Stores Inc . , and Big Lots Inc. posted May sales that fell below Wall Street 's modest expectations .\n0\t753928\t753890\tThe patch also fixes a vulnerability that results because IE does not implement an appropriate block on a file download dialog box .\tThe second vulnerability is a result of IE not implementing a block on a file download dialog box .\n1\t3022833\t3023029\tPeterson , a former fertilizer salesman , is charged with murder in the deaths of his 27-year-old wife and the baby boy she was carrying .\tPeterson , 31 , is now charged with murder in the deaths of his 27-year-old wife and their unborn son .\n0\t751520\t751373\tSPOT products run a Microsoft operating system and the company 's DirectBand radio technology developed with SCA Data Systems .\tThe DirectBand network was developed with the assistance of SCA Data Systems .\n0\t218848\t218851\tHe replaces Ron Dittemore , who announced his resignation in April .\tDittemore announced his plans to resign on April 23 .\n1\t3181118\t3181443\tDetectives told Deasean 's father , Stelly Chisolm , a college student , and mother , Kimberly Hill , of the arrest shortly after Perry was apprehended .\tShortly after his arrest , detectives told Deasean 's father , Stelly Chisolm , a college student , and mother , Kimberly Hill , a medical assistant , about the development .\n1\t515581\t515752\tThey were among about 40 people attending the traditional Jewish ceremony colored by some non-traditional touches .\tHe said about 40 people attended the traditional Jewish ceremony colored by some nontraditional touches .\n1\t347022\t347003\tTaiwan had been relatively free of the viral infection until a fiasco at a Taipei hospital in late April caused the number of infections to skyrocket .\tTaiwan had been relatively free of the viral infection until a severe outbreak at a Taipei hospital in late April .\n1\t3311600\t3311633\tMr. Rowland attended a party in South Windsor for the families of Connecticut National Guard soldiers called to active duty .\tRowland was making an appearance at a holiday party for families of Connecticut National Guard soldiers assigned to duty in Iraq and Afghanistan .\n0\t3439114\t3439084\tRoss Garber , Rowland 's lawyer , said Tuesday he would attend the meeting and would ask to speak on the issue .\tRoss Garber , Rowland 's legal counsel , said the governor would have no comment on the condo deal .\n0\t487951\t488007\tThe euro was at 1.5281 versus the Swiss franc EURCHF = , up 0.2 percent on the session , after hitting its highest since mid-2001 around 1.5292 earlier in the session .\tThe euro was steady versus the Swiss franc after hitting its highest since mid-2001 of 1.5261 earlier in the session .\n0\t314997\t315030\tOn the stand Wednesday , she said she was referring only to the kissing .\tOn the stand Wednesday , she testified that she was referring to the kissing before the alleged rape .\n0\t4733\t4557\tGarner said the group would probably be expanded to include , for example , a Christian and perhaps another Sunni leader .\tThe group has already met several times and Gen. Garner said it probably will be expanded to include a Christian and perhaps another Sunni Muslim leader .\n1\t2820371\t2820525\tBlair 's Foreign Secretary Jack Straw was to take his place on Monday to give a statement to parliament on the European Union .\tBlair 's office said his Foreign Secretary Jack Straw would take his place on Monday to give a statement to parliament on the EU meeting the prime minister attended last week .\n1\t801552\t801516\t\" There were more people surrounding the clubhouse than the Unabomber 's house up in the hills , \" Baker said .\t\" There are more people surrounding the clubhouse than surrounded the Unabomber 's home in the hills .\n1\t1704987\t1705268\tCharles O. Prince , 53 , was named as Mr. Weill 's successor .\tMr. Weill 's longtime confidant , Charles O. Prince , 53 , was named as his successor .\n1\t396041\t396188\tOfficials are also meeting with the International Organization for Epizootics ( OIE ) , which establishes animal-health standards for the world .\tCanadian officials were also expected to meet yesterday with the International Organization for Epizootics ( OIE ) , which establishes animal-health standards for the world .\n0\t1014983\t1014963\tGE stock closed Friday at $ 30.65 a share , down about 42 cents , on the New York Stock Exchange .\tGE 's shares closed at $ 30.65 on Friday on the New York Stock Exchange .\n1\t2320654\t2320666\tThe Midwestern research center will focus on the development of diagnostic , therapeutic and vaccine products for anthrax , botulism , tularemia , hemorrhagic fever viruses and plague .\tThe Midwestern center will focus on diagnosis , treatment and vaccines for anthrax , botulism , tularemia , hemorrhagic fever viruses and plague .\n1\t1057876\t1057778\tThe hearing is to determine whether there is enough evidence to order Akbar to a general court-martial proceeding .\tThe purpose of the hearing is to determine whether Akbar should be court-martialled .\n0\t2116843\t2116883\tIn the United States , heart attacks kill about 460,000 year , in Canada about 80,000 .\tIn the United States , heart attacks kill about 460,000 yearly , according to the National Institutes of Health .\n1\t1461629\t1461781\tNinety-five percent of international cargo to the United States is carried by ship .\tShips carry 95 percent of international cargo to the United States .\n0\t374015\t374162\t\" It 's a major victory for Maine , and it 's a major victory for other states .\tThe Maine program could be a model for other states .\n1\t2493369\t2493428\tNews that oil producers were lowering their output starting in November exacerbated a sell-off that was already under way on Wall Street .\tNews that the Organization of Petroleum Exporting Countries was lowering output starting in November exacerbated a stock sell-off already under way yesterday .\n1\t490355\t490378\tThey note that after several weeks of rallies on upbeat earnings , investors are looking for stronger evidence of a recovery before sending stocks higher .\tAfter several weeks of market rallies on upbeat earnings , many investors are looking for more concrete signs of an economic recovery .\n1\t2691044\t2691264\tMost economists had expected a more dire report , with many anticipating the fifth month of job losses in six months .\tMost economists had been expecting a far more dire report , with many expecting to see the fifth month of job losses in six months in September .\n1\t1831453\t1831491\tBut software license revenues , a measure financial analysts watch closely , decreased 21 percent to $ 107.6 million .\tLicense sales , a key measure of demand , fell 21 percent to $ 107.6 million .\n1\t2380695\t2380822\tKing , brand-name writer , master of the horror story and e-book pioneer , is receiving this year 's medal for Distinguished Contributions to American Letters .\tStephen King , master of the horror story and e-book pioneer , is receiving this year 's medal for Distinguished Contributions to American Letters from the National Book Foundation .\n1\t2577517\t2577531\tThe Denver-based natural gas producer and marketer said the inaccurate reporting was discovered after it received a subpoena from the U.S. Commodity Futures Trading Commission .\tThe natural gas producer and marketer said the inaccurate reporting was discovered in response to a subpoena from the U.S. Commodity Futures Trading Commission , or CFTC .\n1\t3267026\t3266930\tThe steel tariffs , which the U.S. president imposed in March 2002 , will officially end at midnight , instead of March 2005 as initially planned .\tThe U.S. steel tariffs , which Bush imposed in March 2002 , were to officially end at midnight Thursday ( 0500 GMT ) , instead of March 2005 as initially planned .\n1\t360875\t360943\tBusiness Week 's online edition reported on Friday that WorldCom and the SEC could announce a settlement as early as Monday .\tBusinessWeek Online has learned that the settlement could come as early as Monday , May 19 .\n1\t162632\t162653\tOnly one of the five buildings in the Baghdad compound of the United Nations Development Program escaped being burned , the UN said on its Web site .\tOnly one of the five buildings in the compound in Baghdad run by the UN Development Program , escaped being burned , the UN said on its Web site .\n1\t1128884\t1128865\tShares of Salix have rocketed 64 percent since Axcan made its first offer on April 10 .\tSince the initial takeover offer , Salix shares have risen about 35 percent .\n1\t3264732\t3264648\tThe jury verdict , reached Wednesday after less than four hours of deliberation , followed a 2 week trial , during which Waagner represented himself .\tThe quick conviction followed a 2 1 / 2 week trial , during which the Venango County man represented himself .\n1\t1721433\t1721267\tIt 's happened five times in the last 11 years : A disaster puts this Southwestern town in the headlines during the summer tourist season .\tIt 's happened five times in the last decade : A disaster puts this tourist town in the headlines during summer , its busiest season .\n0\t146112\t146127\tThe broader Standard & Poor 's 500 Index .SPX edged down 9 points , or 0.98 percent , to 921 .\tThe technology-laced Nasdaq Composite Index < .IXIC > shed 15 points , or 0.98 percent , to 1,492 .\n1\t389117\t389052\tThe company emphasized that McDonald 's USA does not import any raw beef or hamburger patties from Canada for McDonald 's use in the United States .\tMcDonald 's said in a statement that it does not import any raw beef or hamburger patties from Canada for use in the United States .\n1\t872784\t872834\tGregory Parseghian , a former investment banker , was appointed chief executive .\tGreg Parseghian was appointed the new chief executive .\n0\t2977500\t2977547\tTheir contract will expire at 12 : 01 a.m. Wednesday instead of 12 : 01 a.m. Sunday , said Rian Wathen , organizing director for United Food and Commercial Workers Local 700 .\t\" It has outraged the membership , \" said Rian Wathen , organizing director of United Food and Commercial Workers Local 700 .\n1\t3107137\t3107119\tBut plaque volume increased by 2.7 percent in pravastatin patients .\tThe volume of plaque in Pravachol patients ' arteries rose by 3 % .\n1\t1619244\t1619274\tToday in the US , the book - kept under wraps by its publishers , G. P. Putnam 's Sons , since its inception - will appear in bookstores .\tTomorrow the book , kept under wraps by G. P. Putnam 's Sons since its inception , will appear in bookstores .\n0\t3061836\t3062031\tThe S & P / TSX composite rose 87.74 points on the week , while the TSX Venture Exchange composite gained 44.49 points .\tOn the week , the Dow Jones industrial average rose 11.56 points , while the Nasdaq Stock Market gained 39.42 points .\n1\t485999\t486011\tEx-KGB agent Putin added that the Beatles were considered ' propaganda of an alien ideology ' .\tIn Soviet times the Beatles ' music \" was considered propaganda of an alien ideology .\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/bert_demo/mrpc.proto", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\nsyntax = \"proto3\";\n\npackage mrpc;\n\nservice mrpc {\n rpc paraphrase (TextPair) returns (YesNo) {}\n}\n\nmessage TextPair {\n bytes text_a = 1;\n bytes text_b = 2;\n}\n\nmessage YesNo {\n bytes message = 1;\n bytes prediction = 2;\n}\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/index.rst", "content": ".. _tensorflow-tutorials:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow Tutorials\n====================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nBefore running a tutorial\n-------------------------\n\nYou will run the tutorials on an inf1.6xlarge instance running Deep Learning AMI (DLAMI) to enable both compilation and deployment (inference) on the same instance. In a production environment we encourage you to try different instance sizes to optimize to your specific deployment needs.\n\nFollow instructions at :ref:`tensorflow-tutorial-setup` before running a TensorFlow tutorial on Inferentia. We recommend new users start with the ResNet-50 tutorial.\n\n\n.. toctree::\n :hidden:\n\n /archive/tensorflow/tensorflow-neuron/tutorials/tensorflow-tutorial-setup\n\n.. _tensorflow-nlp:\n\nNatural Language Processing\n---------------------------\n\n* Tensorflow 2.x - HuggingFace DistilBERT with Tensorflow2 Neuron :ref:`[html] ` :github:`[notebook] `\n\n.. toctree::\n :hidden:\n\n /archive/tensorflow/tensorflow-neuron/tutorials/bert_demo/bert_demo\n /src/examples/tensorflow/huggingface_bert/huggingface_bert\n\n.. _tensorflow-utilize-neuron:\n\nUtilizing Neuron Capabilities\n-----------------------------\n\n* Tensorflow 2.x - Using NEURON_RT_VISIBLE_CORES with TensorFlow Serving :ref:`[html] `\n\n.. toctree::\n :hidden:\n\n /src/examples/tensorflow/tensorflow_serving_tutorial.rst\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/k8s_bert_demo/Dockerfile.tfserving_example", "content": "From ubuntu:16.04\nRUN apt-get update\nRUN apt-get install -y wget apt-transport-https ca-certificates awscli\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com xenial main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\nRUN apt-get update\nRUN apt-get install -y tensorflow-model-server-neuron" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/tensorflow-tutorial-setup.rst", "content": ".. _tensorflow-tutorial-setup:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow Tutorial Setup\n=========================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n#. Launch an Inf1.6xlarge Instance:\n .. include:: /setup/install-templates/inf1/launch-inf1-dlami.rst\n\n#. Set up a development environment:\n * Enable or install TensorFlow-Neuron: :ref:`install-neuron-tensorflow`.\n \n#. Run tutorial in Jupyter notebook:\n * Follow instruction at :ref:`Setup Jupyter notebook ` to:\n \n #. Start the Jupyter Notebook on the instance\n #. Run the Jupyter Notebook from your local browser\n\n * Connect to the instance from the terminal, clone the Neuron Github repository to the Inf1 instance and then change the working directory to the tutorial directory:\n\n .. code::\n\n git clone https://github.com/aws/aws-neuron-sdk.git\n cd aws-neuron-sdk/src/examples/tensorflow\n\n * Locate the tutorial notebook file (.ipynb file) under ``aws-neuron-sdk/src/examples/tensorflow``\n * From your local browser, open the tutorial notebook from the menu and follow the instructions.\n\n \n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTutorials (``tensorflow-neuron``)\n===================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Natural Language Processing (NLP) Tutorials \n Utilizing Neuron Capabilities Tutorials \n\n\n.. include:: /archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-neuron.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-neuron.txt", "content": ".. tab-set::\n \n .. tab-item:: Natural Language Processing (NLP) Tutorials\n \n * Tensorflow 2.x - HuggingFace Pipelines distilBERT with Tensorflow2 Neuron :ref:`[html] ` :github:`[notebook] `\n\n \n \n .. tab-item:: Utilizing Neuron Capabilities Tutorials\n \n * Tensorflow 2.x - Using NEURON_RT_VISIBLE_CORES with TensorFlow Serving :ref:`[html] `\n\n.. note::\n\n To use Jupyter Notebook see:\n\n * :ref:`setup-jupyter-notebook-steps-troubleshooting`\n * :ref:`running-jupyter-notebook-as-script` " }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-nlp.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nNatural Language Processing (NLP) Tutorials (``tensorflow-neuron``)\n===================================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n* Tensorflow 2.x - HuggingFace DistilBERT with Tensorflow2 Neuron :ref:`[html] ` :github:`[notebook] `\n\n.. toctree::\n :hidden:\n\n /archive/tensorflow/tensorflow-neuron/tutorials/bert_demo/bert_demo\n /src/examples/tensorflow/huggingface_bert/huggingface_bert\n" }, { "path": "archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-utilizing-neuron-capabilities.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUtilizing Neuron Capabilities Tutorials (``tensorflow-neuron``)\n===============================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n* Using NEURON_RT_VISIBLE_CORES with TensorFlow Serving :ref:`[html] `\n\n.. note::\n\n To use Jupyter Notebook see:\n\n * :ref:`setup-jupyter-notebook-steps-troubleshooting`\n * :ref:`running-jupyter-notebook-as-script` \n" }, { "path": "archive/tensorflow/tensorflow-neuron-inference.rst", "content": ".. _inference-tensorflow-neuron:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInference on Inf1 (``tensorflow-neuron``)\n=========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Tutorials \n Additional Examples \n API Reference Guide \n Misc \n\n\n.. include:: tensorflow-neuron-inference.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuron-inference.txt", "content": ".. card:: Setup (``tensorflow-neuron``)\n :class-body: sphinx-design-class-title-small\n\n See :doc:`TensorFlow Neuron setup `.\n\n\n.. dropdown:: Tutorials (``tensorflow-neuron``)\n :class-title: sphinx-design-class-title-med\n :animate: fade-in\n \n .. include:: /archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-neuron.txt\n\n\n.. dropdown:: Additional Examples (``tensorflow-neuron``)\n :class-title: sphinx-design-class-title-med\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /archive/tensorflow/tensorflow-neuron/additional-examples.txt\n\n\n.. dropdown:: API Reference Guide (``tensorflow-neuron``)\n :class-title: sphinx-design-class-title-med\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /archive/tensorflow/tensorflow-neuron/api-reference-guide.txt\n\n\n.. dropdown:: Misc (``tensorflow-neuron``)\n :class-title: sphinx-design-class-title-med\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n \n .. include:: /archive/tensorflow/tensorflow-neuron/misc-tensorflow-neuron.txt\n\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/api-reference-guide.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nAPI Reference Guide (``tensorflow-neuronx``)\n===========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /archive/tensorflow/tensorflow-neuronx/tfneuronx-python-tracing-api\n /archive/tensorflow/tensorflow-neuronx/tf-neuronx-auto-replication-api\n /archive/tensorflow/tensorflow-neuronx/tfnx-analyze-model-api\n\n\n.. include:: /archive/tensorflow/tensorflow-neuronx/api-reference-guide.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/api-reference-guide.txt", "content": "* :ref:`tfneuronx-ref-neuron-tracing-api`\n* :ref:`tf-neuronx-ref-auto-replication-python-api`\n* :ref:`tf-neuronx-ref-analyze-model-api`" }, { "path": "archive/tensorflow/tensorflow-neuronx/misc-tensorflow-neuronx.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nMisc (``tensorflow-neuronx``)\n============================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /release-notes/archive/tensorflow/tensorflow-neuronx/tensorflow-neuronx\n\n\n\n.. include:: /archive/tensorflow/tensorflow-neuronx/misc-tensorflow-neuronx.txt\n \n" }, { "path": "archive/tensorflow/tensorflow-neuronx/misc-tensorflow-neuronx.txt", "content": "* :ref:`tensorflow-neuronx-release-notes`" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/index.rst", "content": ".. _tensorflow-neuron-setup:\n.. _tensorflow-neuronx-main:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow Setup Guide for Inf2 & Trn1\n======================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n Fresh install \n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/prev-releases/neuronx-2.8.0-tensorflow-install.rst", "content": ".. _install-neuronx-2.8.0-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Tensorflow Neuron (Neuron 2.8.0)\n========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.0\n\n .. tab-set::\n\n .. tab-item:: Amazon Linux 2 AMI\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.0 --neuron-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Ubuntu 20 AMI\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.0 --neuron-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/prev-releases/neuronx-2.9.0-tensorflow-install.rst", "content": ".. _install-neuronx-2.9.0-tensorflow:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Tensorflow Neuron (Neuron 2.9.0)\n========================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.0\n\n .. tab-set::\n\n .. tab-item:: Amazon Linux 2 AMI\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.0 --neuron-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Ubuntu 20 AMI\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.0 --neuron-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-al2.rst", "content": ".. _tensorflow-neuronx-install-prev-al2:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous TensorFlow Neuron Releases for Amazon Linux (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.17.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.17.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.16.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.16.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-al2023.rst", "content": ".. _tensorflow-neuronx-install-prev-al2023:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous TensorFlow NeuronX Releases for Amazon Linux 2023 (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-u20.rst", "content": ".. _tensorflow-neuronx-install-prev-u20:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous TensorFlow Neuron Releases for Ubuntu (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-u22.rst", "content": ".. _tensorflow-neuronx-install-prev-u20:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous TensorFlow Neuron Releases for Ubuntu (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n \n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-neuronx-install.rst", "content": ".. _install-tensorflow-neuronx:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall TensorFlow 2.x (``tensorflow-neuronx``)\n===============================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.1\n\n .. tab-set::\n\n .. tab-item:: Amazon Linux 2\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 13\n :end-line: 16\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 32\n :end-line: 33\n\n .. tab-item:: Ubuntu 20\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 19\n :end-line: 22\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 35\n :end-line: 36\n\n .. tab-item:: Tensorflow 2.9.3\n\n .. tab-set::\n\n .. tab-item:: Amazon Linux 2\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 13\n :end-line: 16\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 74\n :end-line: 75\n\n .. tab-item:: Ubuntu 20\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 19\n :end-line: 22\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 77\n :end-line: 78\n\n .. tab-item:: Tensorflow 2.8.4\n\n .. tab-set::\n\n .. tab-item:: Amazon Linux 2\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 13\n :end-line: 16\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 80\n :end-line: 81\n\n .. tab-item:: Ubuntu 20\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 19\n :end-line: 22\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 83\n :end-line: 84\n\n .. tab-item:: Tensorflow 2.7.4\n\n .. tab-set::\n\n .. tab-item:: Amazon Linux 2\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 13\n :end-line: 16\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 86\n :end-line: 87\n\n .. tab-item:: Ubuntu 20\n\n .. include :: /setup/install-templates/trn1/dlami-notes.rst\n :start-line: 19\n :end-line: 22\n\n .. include :: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 89\n :end-line: 90\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-al2-dlami.rst", "content": "\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n.. tensorflow-neuronx-al2-update:\n\nUpdate to latest TensorFlow Neuron (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.1\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 122\n :end-line: 123\n\n\n .. tab-item:: Tensorflow 2.9.3\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 125\n :end-line: 126\n\n\n .. tab-item:: Tensorflow 2.8.4\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 128\n :end-line: 129\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-al2.rst", "content": "\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n.. tensorflow-neuronx-al2-update:\n\nUpdate to latest TensorFlow Neuron (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.1\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 56\n :end-line: 57\n\n\n .. tab-item:: Tensorflow 2.9.3\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 62\n :end-line: 63\n\n\n .. tab-item:: Tensorflow 2.8.4\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 68\n :end-line: 69\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u20-dlami.rst", "content": "\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n.. tensorflow-neuronx-u20-update:\n\nUpdate to latest TensorFlow Neuron (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.1\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 131\n :end-line: 132\n\n\n .. tab-item:: Tensorflow 2.9.3\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 134\n :end-line: 135\n\n\n .. tab-item:: Tensorflow 2.8.4\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 137\n :end-line: 138\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u20.rst", "content": "\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n.. tensorflow-neuronx-u20-update:\n\nUpdate to latest TensorFlow NeuronX (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.1\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 59\n :end-line: 60\n\n\n .. tab-item:: Tensorflow 2.9.3\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 65\n :end-line: 66\n\n\n .. tab-item:: Tensorflow 2.8.4\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. include:: /src/helperscripts/installationScripts/python_instructions.txt\n :start-line: 71\n :end-line: 72\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u22.rst", "content": "\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n.. tensorflow-neuronx-u22-update:\n\nUpdate to latest TensorFlow Neuron (``tensorflow-neuronx``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: Tensorflow 2.10.1\n\n .. include:: /frameworks/torch/torch-neuronx/setup/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami --category=compiler_framework\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/tf-neuronx-auto-replication-api.rst", "content": ".. _tf-neuronx-ref-auto-replication-python-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuronx``) Auto Multicore Replication (Beta)\n===========================================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe Neuron auto multicore replication Python API enables modifying TensorFlow 2.x\nmodels trace by ```tensorflow_neuronx.trace``` so that they can be automatically replicated across multiple cores.\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nTensorFlow 2.x (``tensorflow-neuron TF2.x``) Auto Multicore Replication Python API (Beta)\n-------------------------------------------------------------------------------------------\n\nMethod\n^^^^^^\n\n``tensorflow.neuron.auto_multicore``\non models traced by\n``tensorflow_neuronx.trace``\n\nDescription\n^^^^^^^^^^^\n\nConverts an existing AWS-Neuron-optimized ``keras.Model`` and returns an auto-replication tagged\nAWS-Multicore-Neuron-optimized ``keras.Model`` that can execute on AWS Machine Learning Accelerators.\nLike the traced model, the returned ``keras.Model`` will support inference only. Attributes or\nvariables held by the original function or ``keras.Model`` will be dropped.\n\nThe auto model replication feature in TensorFlow-Neuron enables you to\ncreate a model once and the model parallel replication would happen\nautomatically. The desired number of cores can be less than the total available NeuronCores\non an trn1 or inf2 instance but not less than 1. This reduces framework memory usage as you are not\nloading the same model multiple times manually. Calls to the returned model will execute the call\non each core in a round-robin fashion.\n\nThe returned ``keras.Model`` can be exported as SavedModel and served using\nTensorFlow Serving. Please see the TensorFlow Serving documentation for more\ninformation about exporting to saved model and serving using TensorFlow\nServing.\n\nNote that the automatic replication will only work on models compiled with pipeline size 1:\nvia ``--neuroncore-pipeline-cores=1``. If auto replication is not enabled, the model will default to\nreplicate on up to 4 cores.\n\nSee :ref:`neuron-compiler-cli-reference-guide` for more information about compiler options.\n\nArguments\n^^^^^^^^^\n\n- **func:** The ``keras.Model`` or function to be traced.\n- **example_inputs:** A ``tf.Tensor`` or a tuple/list/dict of\n ``tf.Tensor`` objects for tracing the function. When ``example_inputs``\n is a ``tf.Tensor`` or a list of ``tf.Tensor`` objects, we expect\n ``func`` to have calling signature ``func(example_inputs)``. Otherwise,\n the expectation is that inference on ``func`` is done by calling\n ``func(*example_inputs)`` when ``example_inputs`` is a ``tuple``,\n or ``func(**example_inputs)`` when ``example_inputs`` is a ``dict``.\n The case where ``func`` accepts mixed positional and keyword arguments\n is currently unsupported.\n- **num_cores:** The desired number of cores where the model will be automatically\n replicated across\n\nReturns\n^^^^^^^\n\n- An AWS-Multicore-Neuron-optimized ``keras.Model``.\n\n\nExample Python API Usage for TF2.x traced models:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code :: python\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n import tensorflow_neuronx as tfnx\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n inputs = [input0]\n outputs = [dense0]\n model = tf.keras.Model(inputs=inputs, outputs=outputs)\n input0_tensor = tf.random.uniform([1, 3])\n model_neuron = tfnx.trace(model, input0_tensor)\n\n # a trn1.2xlarge has 2 neuron cores\n num_cores = 2\n multicore_model = tfn.auto_multicore(model_neuron, input0_tensor, num_cores=num_cores)\n multicore_model(input0_tensor)\n\nExample Python API Usage for TF2.x saved models:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code :: python\n\n from tensorflow.python import saved_model\n\n input0_tensor = tf.random.uniform([1, 3])\n num_cores = 4\n reload_model = saved_model.load(model_dir)\n multicore_model = tfn.auto_multicore(reload_model, input0_tensor, num_cores=num_cores)\n\n.. _tensorflow-ref-auto-replication-cli-api-neuronx:\n\nTensorFlow Neuron TF2.x (``tensorflow-neuronx TF2.x``) Auto Multicore Replication CLI (Beta)\n---------------------------------------------------------------------------------------------------------------\n\nThe Neuron auto multicore replication CLI enables modifying Tensorflow 2.x\ntraced saved models so that they can be automatically replicated across multiple cores. By performing\nthis call on Tensorflow Saved Models, we can support Tensorflow-Serving\nwithout significant modifications to the code.\n\nMethod\n^^^^^^\n\n``tf-neuron-auto-multicore MODEL_DIR --num_cores NUM_CORES --new_model_dir NEW_MODEL_DIR``\n\nArguments\n^^^^^^^^^\n\n- **MODEL_DIR:** The directory of a saved AWS-Neuron-optimized ``keras.Model``.\n- **NUM_CORES:** The desired number of cores where the model will be automatically\n replicated across\n- **NEW_MODEL_DIR:** The directory of where the AWS-Multicore-Neuron-optimized\n ``keras.Model`` will be saved\n\nExample CLI Usage for Tensorflow-Serving saved models:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code :: python\n\n tf-neuron-auto-multicore ./resnet --num_cores 8 --new_model_dir ./modified_resnet\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/tfneuronx-python-tracing-api.rst", "content": ".. _tfneuronx-ref-neuron-tracing-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuronx``) Tracing API\n====================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe Neuron tracing API enables tracing TensorFlow 2.x models for deployment\non trn1 and inf2 AWS machine learning accelerators.\n\nMethod\n------\n\n``tensorflow_neuronx.trace``\n\nDescription\n-----------\n\nTrace a ``keras.Model`` or a Python callable that can be decorated by\n``tf.function``, and return an AWS-Neuron-optimized ``keras.Model`` that\ncan execute on trn1 and inf2 AWS machine learning accelerators. Tracing is\nideal for ``keras.Model`` that accepts a list of ``tf.Tensor`` objects and\nreturns a list of ``tf.Tensor`` objects. It is expected that users will\nprovide example inputs, and the ``trace`` function will execute ``func``\nsymbolically and convert it to a ``keras.Model``.\n\nThe returned ``keras.Model`` will support inference only. Attributes or\nvariables held by the original function or ``keras.Model`` will be dropped.\n\nThe returned ``keras.Model`` can be exported as SavedModel and served using\nTensorFlow Serving. Please see the TensorFlow Serving documentation for more\ninformation about exporting to saved model and serving using TensorFlow\nServing.\n\nThe returned ``keras.Model`` has an ``.on_neuron_ratio`` attribute\nwhich shows the percentage of ops mapped to neuron hardware. This calculation\nignores PlaceholerOp, IdentityOp, ReadVariableOp and NoOp.\n\nOptions can be passed to Neuron compiler via the environment variable\n``NEURON_CC_FLAGS``. For example, the syntax\n``env NEURON_CC_FLAGS=\"--workdir ./artifacts\"`` directs the Neuron compiler to dump artifacts\nin the artifacts directory for debugging. See :ref:`neuron-compiler-cli-reference-guide` for more\ninformation about compiler options.\n\nArguments\n---------\n\n- **func:** The ``keras.Model`` or function to be traced.\n- **example_inputs:** A ``tf.Tensor`` or a tuple/list/dict of\n ``tf.Tensor`` objects for tracing the function. When ``example_inputs``\n is a ``tf.Tensor`` or a list of ``tf.Tensor`` objects, we expect\n ``func`` to have calling signature ``func(example_inputs)``. Otherwise,\n the expectation is that inference on ``func`` is done by calling\n ``func(*example_inputs)`` when ``example_inputs`` is a ``tuple``,\n or ``func(**example_inputs)`` when ``example_inputs`` is a ``dict``.\n The case where ``func`` accepts mixed positional and keyword arguments\n is currently unsupported.\n- **subgraph_builder_function:** (Optional) A callable with signature\n\n ``subgraph_builder_function(node : NodeDef) -> bool``\n (``NodeDef`` is defined in tensorflow/core/framework/node_def.proto)\n\n that is used as a call-back function to determine which part of\n the tensorflow GraphDef given by tracing ``func`` will be placed on\n Machine Learning Accelerators.\n\n If ``subgraph_builder_function`` is not provided, then ``trace`` will\n automatically place operations on Machine Learning Accelerators or\n on CPU to maximize the execution efficiency.\n\n If it is provided, and ``subgraph_builder_function(node)`` returns\n ``True``, and placing ``node`` on Machine Learning Accelerators\n will not cause deadlocks during execution, then ``trace`` will place\n ``node`` on Machine Learning Accelerators. If\n ``subgraph_builder_function(node)`` returns ``False``, then ``trace``\n will place ``node`` on CPU.\n\n.. _tensorflow-neuronx-special-flags:\n\nSpecial Flags\n-------------\n\nThese are flags that get passed directly to the Neuron tracing API\n(rather than the Neuron Compiler). The flags are still passed\nvia the environment variable ``NEURON_CC_FLAGS``.\n\n- **workdir:** example usage - ``NEURON_CC_FLAGS='--workdir ./artifacts'``\n will create a folder named artifacts in the current directory and\n save artifacts that can be used for debug.\n- **dynamic-batch-size:** example usage -\n ``NEURON_CC_FLAGS='--dynamic-batch-size'`` A flag to allow Neuron graphs to\n consume variable sized batches of data. Dynamic sizing is restricted to the\n 0th dimension of a tensor.\n- **extract-weights (Beta):** example usage - \n ``NEURON_CC_FLAGS='--extract-weights trn1.2xlarge'`` will reduce the compiled\n model's protobuf size by taking the weights out of the protobuf.\n Useful for compiling large models that would exceed the 2GB protobuf\n size limit. This feature is in beta. Model performance is not\n guaranteed and the flag does not work in combination with\n ``--neuroncore-pipeline-cores``, ``--dynamic-batch-size``, models with\n multiple NEFFs, and models that are 16GB or greater. \n Compiles models for different neuron instances depending on the instance type passed.\n Supports all trn1 and inf2 instance types except for trn1n.\n\nReturns\n-------\n\n- An AWS-Neuron-optimized ``keras.Model``.\n\n\nExample Usage\n-------------\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow_neuronx as tfnx\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n model = tf.keras.Model(inputs=[input0], outputs=[dense0])\n example_inputs = tf.random.uniform([1, 3])\n model_neuron = tfnx.trace(model, example_inputs) # trace\n # check to see how much of the model was compiled successfully\n print(model_neuron.on_neuron_ratio) \n\n model_dir = './model_neuron'\n model_neuron.save(model_dir)\n model_neuron_reloaded = tf.keras.models.load_model(model_dir)\n\n\nExample Usage with Manual Device Placement Using ``subgraph_builder_function``\n------------------------------------------------------------------------------\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow_neuronx as tfnx\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n reshape0 = tf.keras.layers.Reshape([1, 3])(dense0)\n output0 = tf.keras.layers.Dense(2)(reshape0)\n model = tf.keras.Model(inputs=[input0], outputs=[output0])\n example_inputs = tf.random.uniform([1, 3])\n\n def subgraph_builder_function(node):\n return node.op == 'MatMul'\n\n model_neuron = tfnx.trace(\n model, example_inputs,\n subgraph_builder_function=subgraph_builder_function,\n )\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/tfnx-analyze-model-api.rst", "content": ".. _tf-neuronx-ref-analyze-model-api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorFlow 2.x (``tensorflow-neuronx``) analyze_model API\n==========================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nMethod\n------\n\n``tensorflow_neuronx.analyze_model``\n\nDescription\n-----------\n\nAnalyzes a ``keras.Model`` or a Python callable that can be decorated by\n``tf.function`` for it's compatibility with Neuron. It displays supported \nvs. unsupported operators in the model as well as percentages and counts of \neach operator and returns a dictionary with operator statistics.\n\nArguments\n---------\n\n- **func:** The ``keras.Model`` or function to be analyzed.\n- **example_inputs:** A ``tf.Tensor`` or a tuple/list/dict of\n ``tf.Tensor`` objects for tracing the function. When ``example_inputs``\n is a ``tf.Tensor`` or a list of ``tf.Tensor`` objects, we expect\n ``func`` to have calling signature ``func(example_inputs)``. Otherwise,\n the expectation is that inference on ``func`` is done by calling\n ``func(*example_inputs)`` when ``example_inputs`` is a ``tuple``,\n or ``func(**example_inputs)`` when ``example_inputs`` is a ``dict``.\n The case where ``func`` accepts mixed positional and keyword arguments\n is currently unsupported.\n\nReturns\n-------\n\n- A results ``dict`` with these keys: ``'percent_supported'``, ``'supported_count'``, ``'total_count'``, ``'supported_operators'``, ``'unsupported_operators'``, ``'operators'``, ``'operator_count'``.\n\nExample Usage\n-------------\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow_neuron as tfnx\n\n input0 = tf.keras.layers.Input(3)\n dense0 = tf.keras.layers.Dense(3)(input0)\n model = tf.keras.Model(inputs=[input0], outputs=[dense0])\n example_inputs = tf.random.uniform([1, 3])\n results = tfnx.analyze_model(model, example_inputs)\n print(results)\n\n # expected output\n '''\n BiasAdd\n\tMatMul\n\t100.00% of all operations (2 of 2) are supported\n\t{'percent_supported': 100.0, 'supported_count': 2, 'total_count': 2, \n\t'supported_operators': {'BiasAdd', 'MatMul'}, 'unsupported_operators': [], \n\t'operators': ['BiasAdd', 'MatMul'], 'operator_count': {'MatMul': 1, 'BiasAdd': 1}}\n\t'''\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/tutorials/tutorial-tensorflowx-serving-NeuronRT-Visible-Cores.rst", "content": ".. _tensorflow-servingx-neuronrt-visible-cores:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUsing NEURON_RT_VISIBLE_CORES with TensorFlow Serving\n=====================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nTensorFlow serving allows customers to scale-up inference workloads\nacross a network. TensorFlow Neuron Serving uses the same API as normal\nTensorFlow Serving with two differences: (a) the saved model must be\ncompiled for neuron and (b) the entry point is a different binary\nnamed ``tensorflow_model_server_neuronx``. Follow the steps below \nto install the package using apt-get or dnf. This will be pre-installed in a future release.\n\nInstall TensorFlow Model Server and Serving API\n-----------------------------------------------\n\nFollow the steps in the TensorFlow NeuronX installation guide.\n\nThen ensure you install using either apt-get or dnf.\n\n.. code:: bash\n\n sudo apt-get install tensorflow-model-server-neuronx\n\nor\n\n.. code:: bash\n\n sudo dnf install tensorflow-model-server-neuronx\n\nAlso, you would need TensorFlow Serving API (use --no-deps to prevent\ninstallation of regular tensorflow).\n\n.. code:: bash\n\n pip install --no-deps tensorflow_serving_api\n\nFor the example image preprocessing using Keras preprocessing, the\nPython Imaging Library Pillow is required:\n\n.. code:: bash\n\n pip install pillow\n\nTo workaround h5py issue https://github.com/aws/aws-neuron-sdk/issues/220:\n\n.. code:: bash\n\n pip install \"h5py<3.0.0\"\n\n\nExport and Compile Saved Model\n------------------------------\n\nThe following example shows graph construction followed by the addition\nof Neuron compilation step before exporting to saved model.\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow_neuronx as tfnx\n import numpy as np\n\n tf.keras.backend.set_learning_phase(0)\n tf.keras.backend.set_image_data_format('channels_last')\n image_sizes = [224, 224]\n model = tf.keras.applications.ResNet50(weights='imagenet')\n example_inputs = tf.random.uniform([1, *image_sizes, 3], dtype=tf.float32)\n\n model_neuron = tfnx.trace(model, example_inputs)\n # run the model once to define the forward pass and allow for saving\n model_neuron(example_inputs)\n tf.keras.models.save_model(model_neuron, './resnet50_neuron/1')\n\n\n\nServing Saved Model\n-------------------\n\nUser can now serve the saved model with the\ntensorflow_model_server_neuron binary. To utilize multiple NeuronCores,\nit is recommended to launch multiple tensorflow model servers that\nlisten to the same gRPC port:\n\n.. code:: bash\n\n export NEURON_RT_VISIBLE_CORES=0 # important to set this environment variable before launching model servers\n tensorflow_model_server_neuron --model_name=resnet50_neuron \\\n --model_base_path=$(pwd)/resnet50_neuron/ --port=8500\n\n # then to run another server on a different neuron core open another\n # window and run this, except this time set NEURON_RT_VISIBLE_CORES=1\n # you can keep doing this up to the number of Neuron Cores on your machine\n\n export NEURON_RT_VISIBLE_CORES=1\n tensorflow_model_server_neuron --model_name=resnet50_neuron \\\n --model_base_path=$(pwd)/resnet50_neuron/ --port=8500\n\nThe compiled model is staged in neuron DRAM by the server to prepare\nfor inference.\n\nGenerate inference requests to the model server\n-----------------------------------------------\n\nNow run inferences via GRPC as shown in the following sample client\ncode:\n\n.. code:: python\n\n import numpy as np\n import grpc\n import tensorflow as tf\n from tensorflow.keras.preprocessing import image\n from tensorflow.keras.applications.resnet50 import preprocess_input\n from tensorflow_serving.apis import predict_pb2\n from tensorflow_serving.apis import prediction_service_pb2_grpc\n from tensorflow.keras.applications.resnet50 import decode_predictions\n\n tf.keras.backend.set_image_data_format('channels_last')\n\n if __name__ == '__main__':\n channel = grpc.insecure_channel('localhost:8500')\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n img_file = tf.keras.utils.get_file(\n \"./kitten_small.jpg\",\n \"https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg\")\n img = image.load_img(img_file, target_size=(224, 224))\n img_array = preprocess_input(image.img_to_array(img)[None, ...])\n request = predict_pb2.PredictRequest()\n request.model_spec.name = 'resnet50_neuron'\n request.inputs['input_1'].CopyFrom(\n tf.make_tensor_proto(img_array, shape=img_array.shape))\n result = stub.Predict(request)\n prediction = tf.make_ndarray(result.outputs['output_1'])\n print(decode_predictions(prediction))\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/tutorials/tutorials-tensorflow-neuronx.rst", "content": ".. _inference-tensorflow-neuronx-tutorials:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTutorials (``tensorflow-neuronx``)\n===================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n HuggingFace Roberta-Base \n /archive/tensorflow/tensorflow-neuronx/tutorials/tutorial-tensorflowx-serving-NeuronRT-Visible-Cores\n\n\n.. include:: /archive/tensorflow/tensorflow-neuronx/tutorials/tutorials-tensorflow-neuronx.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuronx/tutorials/tutorials-tensorflow-neuronx.txt", "content": "* HuggingFace Roberta-Base :ref:`[html]` :github:`[notebook] `\n* :ref:`tensorflow-servingx-neuronrt-visible-cores`\n\n.. note::\n To use Jupyter Notebook see:\n\n * :ref:`setup-jupyter-notebook-steps-troubleshooting`\n * :ref:`running-jupyter-notebook-as-script`" }, { "path": "archive/tensorflow/tensorflow-neuronx-inference.rst", "content": ".. _inference-tensorflow-neuronx:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInference on Inf2 & Trn1/Trn1n (``tensorflow-neuronx``)\n=======================================================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Tutorials \n API Reference Guide \n Misc \n\n\n.. include:: tensorflow-neuronx-inference.txt\n" }, { "path": "archive/tensorflow/tensorflow-neuronx-inference.txt", "content": ".. card:: Setup (``tensorflow-neuronx``)\n :class-body: sphinx-design-class-title-small\n\n See :doc:`TensorFlow NeuronX setup `.\n \n \n.. dropdown:: Tutorials (``tensorflow-neuronx``)\n :class-title: sphinx-design-class-title-med\n :animate: fade-in\n\n .. include:: /archive/tensorflow/tensorflow-neuronx/tutorials/tutorials-tensorflow-neuronx.txt\n\n\n.. dropdown:: API Reference Guide (``tensorflow-neuronx``)\n :class-title: sphinx-design-class-title-med\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /archive/tensorflow/tensorflow-neuronx/api-reference-guide.txt\n\n\n.. dropdown:: Misc (``tensorflow-neuronx``)\n :class-title: sphinx-design-class-title-med\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n \n .. include:: /archive/tensorflow/tensorflow-neuronx/misc-tensorflow-neuronx.txt" }, { "path": "archive/tensorflow/tensorflow-setup.rst", "content": ".. _tf-setup:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTensorflow Neuron Setup\n=======================\n\n.. warning::\n\n This document is archived. TensorFlow is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: tensorflow-setup.txt\n" }, { "path": "archive/tensorflow/tensorflow-setup.txt", "content": ".. card:: Tensorflow Neuron (``tensorflow-neuronx``) Setup for Inf2, Trn1/Trn1n Instances\n :class-body: sphinx-design-class-title-small\n\n See :doc:`TensorFlow NeuronX setup `.\n\n\n.. card:: Tensorflow Neuron (``tensorflow-neuron``) Setup for Inf1 Instances\n :class-body: sphinx-design-class-title-small\n\n See :doc:`TensorFlow Neuron setup `.\n" }, { "path": "archive/torch-neuron/additional-examples-inference-torch-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nAdditional Examples (``torch-neuron``)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n AWS Neuron Samples GitHub Repository \n\n\n\n.. include:: /archive/torch-neuron/additional-examples-inference-torch-neuron.txt\n" }, { "path": "archive/torch-neuron/additional-examples-inference-torch-neuron.txt", "content": "* `AWS Neuron Samples GitHub Repository `_\n" }, { "path": "archive/torch-neuron/api-compilation-python-api.rst", "content": ".. _torch_neuron_trace_api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nPyTorch-Neuron trace python API\n================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe PyTorch-Neuron trace Python API provides a method to generate\nPyTorch models for execution on Inferentia, which can be serialized as\nTorchScript. It is analogous to :func:`torch.jit.trace` function in PyTorch.\n\n.. py:function:: torch_neuron.trace(model, example_inputs, **kwargs)\n\n The :func:`torch_neuron.trace` method sends operations to\n the Neuron-Compiler (``neuron-cc``) for compilation and embeds compiled\n artifacts in a TorchScript graph.\n\n Compilation can be done on any EC2 machine with sufficient memory and\n compute resources. c5.4xlarge or larger is recommended.\n\n Options can be passed to Neuron compiler via the compile function. See\n :ref:`neuron-compiler-cli-reference`\n for more information about compiler options.\n\n This function partitions nodes into operations that are supported\n by Neuron and operations which are not. Operations which are not supported\n by Neuron are run on CPU. Graph partitioning can be controlled by the\n ``subgraph_builder_function``, ``minimum_segment_size``, and ``fallback``\n parameters (See below). By default all supported operations are compiled and\n run on Neuron.\n\n The compiled graph can be saved using the :func:`torch.jit.save` function and\n restored using :func:`torch.jit.load` function for inference on Inf1 instances.\n During inference, the previously compiled artifacts will be loaded into\n the Neuron Runtime for inference execution.\n\n *Required Arguments*\n\n :arg ~torch.nn.Module,callable model: The functions that that will be run with\n ``example_inputs`` arguments. The arguments and return types must compatible\n with :func:`torch.jit.trace`. When a :class:`~torch.nn.Module` is passed\n to :func:`torch_neuron.trace`, only the :func:`~torch.nn.Module.forward`\n method is run and traced.\n :arg tuple example_inputs: A tuple of example inputs that will be passed to\n the ``model`` while tracing. The resulting trace can be run with inputs\n of different types and shapes assuming the traced operations support\n those types and shapes. This parameter may also be a single\n :class:`torch.Tensor` in which case it is automatically wrapped in a\n ``tuple``.\n\n *Optional Keyword Arguments*\n\n :keyword list[str] compiler_args: List of strings representing\n ``neuron-cc`` compiler arguments. Note that these arguments apply to all\n subgraphs generated by allowlist partitioning. For example, use\n :code:`compiler_args=['--neuroncore-pipeline-cores', '4']` to set number\n of NeuronCores per subgraph to 4. See :ref:`neuron-compiler-cli-reference`\n for more information about compiler options.\n :keyword int compiler_timeout: Timeout in seconds for waiting\n ``neuron-cc`` to complete. Exceeding this timeout will cause a\n ``subprocess.TimeoutExpired`` exception.\n :keyword str compiler_workdir: Work directory used by\n ``neuron-cc``. Useful for debugging and/or inspecting ``neuron-cc``\n logs/IRs.\n :keyword callable subgraph_builder_function: A function which is evaluated\n on each node during graph partitioning. This takes in a torch graph\n operator node and returns a :class:`bool` value of whether\n it should be included in the fused Neuron graph or not. By default the\n partitioner selects all operators which are supported by Neuron.\n :keyword int minimum_segment_size: A parameter used during partitioning.\n This specifies the minimum number of graph nodes which should be compiled\n into a Neuron graph (default= :code:`2`). If the number of nodes is smaller\n than this size, the operations will run on CPU.\n :keyword float single_fusion_ratio_threshold: A parameter used during\n partitioning. During partitioning, if a single partition contains a\n fraction of operations greater than this threshold, only one graph\n partition will be compiled (default= :code:`0.6`). This is used to\n avoid compiling many small Neuron graphs. To force compilation of all\n graphs to Neuron (even when they are very small), a value of ``1.0``\n can be used.\n :keyword bool fallback: A function parameter to turn off graph partitioning.\n Indicates whether to attempt to fall back to CPU operations if an\n operation is not supported by Neuron. By default this is ``True``. If\n this is set to ``False`` and an operation is not supported by Neuron,\n this will fail compilation and raise an ``AttributeError``.\n :keyword bool dynamic_batch_size: A flag to allow Neuron graphs to consume\n variable sized batches of data. Dynamic sizing is restricted to the 0th\n dimension of a tensor.\n :keyword list optimizations: A list of :class:`~torch_neuron.Optimization`\n passes to apply to the model.\n :keyword bool separate_weights: A flag to enable compilation of models with \n over 1.9GB of constant parameters. By default this flag is ``False``. \n If this is set to ``True`` and the compiler version is not new enough \n to support the flag, this will raise an ``NotImplementedError``.\n :keyword \\*\\*kwargs: All other keyword arguments will be forwarded directly to\n :func:`torch.jit.trace`. This supports flags like ``strict=False``\n in order to allow dictionary outputs.\n\n :returns: The traced :class:`~torch.jit.ScriptModule` with embedded\n compiled neuron sub-graphs. Operations in this module will run on Neuron\n unless they are not supported by Neuron or manually partitioned to run\n on CPU.\n\n Note that in ``torch<1.8`` This would return a\n :class:`~torch.jit.ScriptFunction` if the input was function type.\n :rtype: ~torch.jit.ScriptModule, ~torch.jit.ScriptFunction\n\n\n.. py:class:: torch_neuron.Optimization\n\n A set of optimization passes that can be applied to the model.\n\n .. py:attribute:: FLOAT32_TO_FLOAT16\n\n A post-processing pass that converts all :attr:`torch.float32` tensors\n to :attr:`torch.float16` tensors. The advantage to this\n optimization pass is that input/output tensors will be type cast.\n This reduces the amount of data that will be copied to and from\n Inferentia hardware. The resulting traced model will accept both\n :attr:`torch.float32` and :attr:`torch.float16` inputs where the\n model used :attr:`torch.float32` inputs during tracing. It is only\n beneficial to enable this optimization if the throughput of a\n model is highly dependent upon data transfer speed. This optimization is\n not recommended if the final application will use :attr:`torch.float32`\n inputs since the :attr:`torch.float16` type cast will occur on CPU\n during inference.\n\n\nExample Usage\n-------------\n\nFunction Compilation\n~~~~~~~~~~~~~~~~~~~~\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n def foo(x, y):\n return 2 * x + y\n\n # Run `foo` with the provided inputs and record the tensor operations\n traced_foo = torch.neuron.trace(foo, (torch.rand(3), torch.rand(3)))\n\n # `traced_foo` can now be run with the TorchScript interpreter or saved\n # and loaded in a Python-free environment\n torch.jit.save(traced_foo, 'foo.pt')\n traced_foo = torch.jit.load('foo.pt')\n\nModule Compilation\n~~~~~~~~~~~~~~~~~~\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n import torch.nn as nn\n\n class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv = nn.Conv2d(1, 1, 3)\n\n def forward(self, x):\n return self.conv(x) + 1\n\n n = Net()\n n.eval()\n\n inputs = torch.rand(1, 1, 3, 3)\n\n # Trace a specific method and construct `ScriptModule` with\n # a single `forward` method\n neuron_forward = torch.neuron.trace(n.forward, inputs)\n\n # Trace a module (implicitly traces `forward`) and constructs a\n # `ScriptModule` with a single `forward` method\n neuron_net = torch.neuron.trace(n, inputs)\n\nPre-Trained Model Compilation\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nThe following is an example usage of the compilation Python API, with\ndefault compilation arguments, using a pretrained :class:`torch.nn.Module`:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 224, 224])\n model_neuron = torch.neuron.trace(model, image)\n\n\n.. _compiling-models-with-kwargs:\n\nCompiling models with torch.jit.trace kwargs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nThis example uses the :code:`strict=False` flag to compile a model with\ndictionary outputs. Similarly, any other keyword argument of\n:func:`torch.jit.trace` can be passed directly to\n:func:`torch_neuron.trace` so that it is passed to the underlying trace call.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n import torch.nn as nn\n\n class Model(nn.Module):\n def __init__(self):\n super(Model, self).__init__()\n self.conv = nn.Conv2d(1, 1, 3)\n\n def forward(self, x):\n return {'conv': self.conv(x) + 1}\n\n model = Model()\n model.eval()\n\n inputs = torch.rand(1, 1, 3, 3)\n\n # use the strict=False kwarg to compile a model with dictionary outputs\n # the model output format does not change\n model_neuron = torch.neuron.trace(model, inputs, strict=False)\n\n\nDynamic Batching\n~~~~~~~~~~~~~~~~\nThis example uses the optional :code:`dynamic_batch_size` option in order to\nsupport variable sized batches at inference time.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input of batch size 1\n image = torch.rand([1, 3, 224, 224])\n model_neuron = torch.neuron.trace(model, image, dynamic_batch_size=True)\n\n # Execute with a batch of 7 images\n batch = torch.rand([7, 3, 224, 224])\n results = model_neuron(batch)\n\n\nManual Partitioning\n~~~~~~~~~~~~~~~~~~~\nThe following example uses the optional :code:`subgraph_builder_function`\nparameter to ensure that only a specific convolution layer is compiled to\nNeuron. The remaining operations are executed on CPU.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n import torch.nn as nn\n\n class ExampleConvolutionLayer(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv = nn.Conv2d(1, 1, 3)\n\n def forward(self, x):\n return self.conv(x) + 1\n\n class Model(nn.Module):\n def __init__(self):\n super().__init__()\n self.layer = ExampleConvolutionLayer()\n\n def forward(self, x):\n return self.layer(x) * 100\n\n def subgraph_builder_function(node) -> bool:\n \"\"\"Select if the node will be included in the Neuron graph\"\"\"\n\n # Node names are tuples of Module names.\n if 'ExampleConvolutionLayer' in node.name:\n return True\n\n # Ignore all operations not in the example convolution layer\n return False\n\n model = Model()\n model.eval()\n\n inputs = torch.rand(1, 1, 3, 3)\n\n # Log output shows that `aten::_convolution` and `aten::add` are compiled\n # but `aten::mul` is not. This will seamlessly switch between Neuron/CPU\n # execution in a single graph.\n neuron_model = torch_neuron.trace(\n model,\n inputs,\n subgraph_builder_function=subgraph_builder_function\n )\n\n\nSeparate Weights\n~~~~~~~~~~~~~~~~\nThis example uses the optional :code:`separate_weights` option in order to\nsupport compilation of models greater than 1.9GB.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 224, 224])\n #the models' output format does not change\n model_neuron = torch.neuron.trace(model, image, separate_weights=True)\n" }, { "path": "archive/torch-neuron/api-core-placement.rst", "content": ".. _torch_core_placement_api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nPyTorch Neuron (``torch-neuron``) Core Placement API\n=====================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. automodule:: placement\n :module-name: torch_neuron.experimental\n :members:\n\n\n" }, { "path": "archive/torch-neuron/api-reference-guide-torch-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nAPI Reference Guide (``torch-neuron``)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n PyTorch Neuron trace Python API \n torch.neuron.DataParallel API \n /archive/torch-neuron/api-core-placement\n\n\n.. include:: /archive/torch-neuron/api-reference-guide-torch-neuron.txt\n" }, { "path": "archive/torch-neuron/api-reference-guide-torch-neuron.txt", "content": "* :ref:`PyTorch Neuron trace Python API `\n* :ref:`torch.neuron.DataParallel API `\n* :ref:`torch_core_placement_api`" }, { "path": "archive/torch-neuron/api-torch-neuron-dataparallel-api.rst", "content": ".. _api_torch_neuron_dataparallel_api:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\ntorch.neuron.DataParallel API\n=============================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe :func:`torch.neuron.DataParallel` Python API implements data parallelism on\n:class:`~torch.jit.ScriptModule` models created by the\n:ref:`torch_neuron_trace_api`.\nThis function is analogous to :class:`~torch.nn.DataParallel` in PyTorch.\nThe :ref:`torch-neuron-dataparallel-app-note` application note provides an\noverview of how :func:`torch.neuron.DataParallel` can be used to improve\nthe performance of inference workloads on Inferentia.\n\n.. py:function:: torch.neuron.DataParallel(model, device_ids=None, dim=0)\n\n Applies data parallelism by replicating the model on\n available NeuronCores and distributing data across the different\n NeuronCores for parallelized inference.\n\n By default, DataParallel will use all available NeuronCores\n allocated for the current process for parallelism. DataParallel will\n apply parallelism on ``dim=0`` if ``dim`` is not specified.\n\n DataParallel automatically enables\n :ref:`dynamic batching ` on\n eligible models if ``dim=0``. Dynamic batching can be dsiabled using\n :func:`torch.neuron.DataParallel.disable_dynamic_batching`.\n If dynamic batching is not enabled, the batch size at compilation-time must\n be equal to the batch size at inference-time divided by the number of\n NeuronCores being used. Specifically, the following must be true when\n dynamic batching is disabled:\n ``input.shape[dim] / len(device_ids) == compilation_input.shape[dim]``.\n DataParallel will throw a warning if dynamic batching cannot be enabled.\n\n DataParallel will try load all of a model’s NEFFs onto\n a single NeuronCore, only if all of the NEFFs can fit on a single\n NeuronCore. DataParallel does not currently support models that\n have been compiled with :ref:`neuroncore-pipeline`.\n\n :func:`torch.neuron.DataParallel` requires PyTorch >= 1.8.\n\n *Required Arguments*\n\n :arg ~torch.jit.ScriptModule model: Model created by the\n :ref:`torch_neuron_trace_api`\n to be parallelized.\n\n *Optional Arguments*\n\n :arg list device_ids: List of :obj:`int` or ``'nc:#'`` that specify the\n NeuronCores to use for parallelization (default: all NeuronCores).\n Refer to the :ref:`device_ids note ` for a description\n of how ``device_ids`` indexing works.\n :arg int dim: Dimension along which the input tensor is scattered across\n NeuronCores (default ``dim=0``).\n\n *Attributes*\n\n :arg int num_workers: Number of worker threads used for\n multithreaded inference (default: ``2 * number of NeuronCores``).\n :arg int split_size: Size of the input chunks\n (default: ``max(1, input.shape[dim] // number of NeuronCores)``).\n\n\n.. py:function:: torch.neuron.DataParallel.disable_dynamic_batching()\n\n Disables automatic dynamic batching on the DataParallel module. See\n :ref:`Dynamic batching disabled `\n for example of how DataParallel can be used with dynamic batching disabled.\n Use as follows:\n\n >>> model_parallel = torch.neuron.DataParallel(model_neuron)\n >>> model_parallel.disable_dynamic_batching()\n\n.. _device_ids_note:\n\n.. note::\n\n ``device_ids`` uses per-process NeuronCore granularity and zero-based\n indexing. Per-process granularity means that each Python process \"sees\"\n its own view of the world. Specifically, this means that ``device_ids``\n only \"sees\" the NeuronCores that are allocated for the current process.\n Zero-based indexing means that each Python process will index its\n allocated NeuronCores starting at 0, regardless of the \"global\" index of\n the NeuronCores. Zero-based indexing makes it possible to redeploy the exact\n same code unchanged in different process. This behavior is analogous to\n the ``device_ids`` argument in the PyTorch\n :class:`~torch.nn.DataParallel` function.\n\n As an example, assume DataParallel is run on an inf1.6xlarge, which\n contains four Inferentia chips each of which contains four NeuronCores:\n\n * If ``NEURON_RT_VISIBLE_CORES`` is not set, a single process can access\n all 16 NeuronCores. Thus specifying ``device_ids=[\"nc:0\"]`` will\n correspond to chip0:core0 and ``device_ids=[\"nc:14\"]`` will correspond\n to chip3:core2.\n\n * However, if two processes are launched where: process 1 has\n ``NEURON_RT_VISIBLE_CORES=0-6`` and process 2 has\n ``NEURON_RT_VISIBLE_CORES=7-15``, ``device_ids=[\"nc:14\"]``\n cannot be specified in either process. Instead, chip3:core2 can only be\n accessed in process 2. Additionally, chip3:core2 is specified in process 2\n with ``device_ids=[\"nc:7\"]``. Furthermore, in process 1,\n ``device_ids=[\"nc:0\"]`` would correspond to chip0:core0; in process 2\n ``device_ids=[\"nc:0\"]`` would correspond to chip1:core3.\n\n\nExamples\n--------\n\nThe following sections provide example usages of the\n:func:`torch.neuron.DataParallel` module.\n\nDefault usage\n^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-default.rst\n\nSpecifying NeuronCores\n^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-specify-ncs.rst\n\nDataParallel with dim != 0\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-dim-neq-zero.rst\n\nDynamic batching\n^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-dynamic-batching.rst\n\n.. _dataparallel_example_disable_dynamic_batching_api:\n\nDynamic batching disabled\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /archive/torch-neuron/torch-neuron-dataparallel-example-disable-dynamic-batching.rst\n\nFull tutorial with torch.neuron.DataParallel\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor an end-to-end tutorial that uses DataParallel, see the\n:ref:`PyTorch Resnet Tutorial `.\n" }, { "path": "archive/torch-neuron/developer-guide-torch-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nDeveloper Guide (``torch-neuron``)\n==================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n Running Inference on Variable Input Shapes with Bucketing \n Data Parallel Inference on PyTorch Neuron \n /archive/torch-neuron/guides/torch-lstm-support\n /archive/torch-neuron/guides/core-placement/torch-core-placement\n\n\n.. include:: /archive/torch-neuron/developer-guide-torch-neuron.txt\n" }, { "path": "archive/torch-neuron/developer-guide-torch-neuron.txt", "content": "* :ref:`Running Inference on Variable Input Shapes with Bucketing `\n* :ref:`Data Parallel Inference on PyTorch Neuron `\n* :ref:`torch_neuron_lstm_support`\n* :ref:`torch_neuron_core_placement_guide`" }, { "path": "archive/torch-neuron/guides/core-placement/torch-core-placement.rst", "content": ".. _torch_neuron_core_placement_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nPyTorch Neuron (``torch-neuron``) Core Placement\n================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThis programming guide describes the available techniques and APIs to be able\nto allocate NeuronCores to a process and place models onto specific NeuronCores.\nIn order of precedence, the current recommendation is to use the following\nplacement techniques:\n\n1. For most regular models, default core placement should be used in\n conjunction with ``NEURON_RT_NUM_CORES`` (:ref:`torch_placement_default`)\n2. For more specific core placement for NeuronCore Pipelined models, then\n ``NEURONCORE_GROUP_SIZES`` should be used (:ref:`torch_placement_ncg`).\n3. Finally, for even more granular control, then the beta\n explicit placement APIs may be used (:ref:`torch_placement_explicit`).\n\n.. contents:: Table of Contents\n :depth: 3\n\nThe following guide will assume a machine with 8 NeuronCores:\n\n- NeuronCores will use the notation ``nc0``, ``nc1``, etc.\n- NeuronCore Groups will use the notation ``ncg0``, ``ncg1`` etc.\n- Models will use the notation ``m0``, ``m1`` etc.\n\nNeuronCores, NeuronCore Groups, and model allocations will be displayed in\nthe following format:\n\n.. raw:: html\n :file: images/0-0-legend.svg\n\nNote that the actual cores that are visible to the process can be adjusted\naccording to the :ref:`nrt-configuration`.\n\nNeuronCore Pipeline\n-------------------\n\nA key concept to understand the intent behind certain core placement strategies\nis NeuronCore Pipelining (See :ref:`neuroncore-pipeline`). NeuronCore Pipelining\nallows a model to be automatically split into pieces and executed on different\nNeuronCores.\n\nFor most models only 1 NeuronCore will be required for execution. A model will\n**only** require more than one NeuronCore when using NeuronCore Pipeline.\nWhen model pipelining is enabled, the model is split between multiple\nNeuronCores and data is transferred between them. For example, if the compiler\nflag ``--neuroncore-pipeline-cores 4`` is used, this splits the model into\n4 pieces to be executed on 4 separate NeuronCores.\n\n.. _torch_placement_default:\n\nDefault Core Allocation & Placement\n-----------------------------------\n\nThe most basic requirement of an inference application is to be able to place a\nsingle model on a single NeuronCore. More complex applications may use multiple\nNeuronCores or even multiple processes each executing different models. The\nimportant thing to note about designing an inference application is that a\nsingle NeuronCore will always be allocated to a single process. *Processes do\nnot share NeuronCores*. Different configurations can be used to ensure that\nan application process has enough NeuronCores allocated to execute its model(s):\n\n- Default: A process will attempt to take ownership of **all NeuronCores**\n visible on the instance. This should be used when an instance is only running\n a single inference process since no other process will be allowed to take\n ownership of any NeuronCores.\n- ``NEURON_RT_NUM_CORES``: Specify the **number of NeuronCores** to allocate\n to the process. This places no restrictions on which NeuronCores will be used,\n however, the resulting NeuronCores will always be contiguous. This should be\n used in multi-process applications where each process should only use a subset\n of NeuronCores.\n- ``NEURON_RT_VISIBLE_CORES``: Specifies exactly **which NeuronCores** are\n allocated to the process by index. Similar to ``NEURON_RT_NUM_CORES``, this\n can be used in multi-process applications where each process should only use a\n subset of NeuronCores. This provides more fined-grained controls over the\n exact NeuronCores that are allocated to a given process.\n- ``NEURONCORE_GROUP_SIZES``: Specifies a number of **NeuronCore Groups** which\n are allocated to the process. This is described in more detail in the\n :ref:`torch_placement_ncg` section.\n\nSee the :ref:`nrt-configuration` for more environment variable details.\n\nExample: Default\n^^^^^^^^^^^^^^^^\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc0\n m1 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc1\n\n\n.. raw:: html\n :file: images/0-1-default-2.svg\n\nWith no environment configuration, the process will take ownership of all\nNeuronCores. In this example, only two of the NeuronCores are used by the\nprocess and the remaining are allocated but left idle.\n\n\nExample: ``NEURON_RT_NUM_CORES``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_NUM_CORES = '2'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc0\n m1 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc1\n\n.. raw:: html\n :file: images/0-2-default-rt-num-cores.svg\n\nSince there is no other process on the instance, only the first 2 NeuronCores\nwill be acquired by the process. Models load in a simple linear order to the\nleast used NeuronCores.\n\n\nExample: ``NEURON_RT_VISIBLE_CORES``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_VISIBLE_CORES = '4-5'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc4\n m1 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc5\n\n\n.. raw:: html\n :file: images/0-3-default-rt-visible-cores.svg\n\nUnlike ``NEURON_RT_NUM_CORES``, setting the visible NeuronCores allows the\nprocess to take control of a specific contiguous set. This allows an application\nto have a more fine-grained control of where models will be placed.\n\n\nExample: Overlapping Models\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_VISIBLE_CORES = '0-1'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc0\n m1 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc0-nc1\n m2 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc1\n\n.. raw:: html\n :file: images/0-4-default-overlap-model-2.svg\n\n.. raw:: html\n :file: images/0-4-default-overlap.svg\n\nThis shows how models may share NeuronCores but the default model placement\nwill attempt to evenly distribute NeuronCore usage rather than overlapping all\nmodels on a single NeuronCore.\n\n\nExample: Multiple Processes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_NUM_CORES = '2'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc0\n m1 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc1\n\n\nIn this example, if the script is run **twice**, the following allocations\nwill be made:\n\n.. raw:: html\n :file: images/0-5-default-multiprocess.svg\n\nNote that each process will take ownership of as many NeuronCores as is\nspecified by the ``NEURON_RT_NUM_CORES`` configuration.\n\n\n.. _torch_placement_ncg:\n\nNEURONCORE_GROUP_SIZES\n----------------------\n\n.. important::\n\n The use of explicit core placement should only be used when a specific\n performance goal is required. By default ``torch-neuron`` places models on\n the **least used** NeuronCores. This should be optimal for most\n applications.\n\n Secondly, ``NEURONCORE_GROUP_SIZES`` is being deprecated in a future\n release and should be avoided in favor of newer placement methods.\n Use ``NEURON_RT_NUM_CORES`` or ``NEURON_RT_VISIBLE_CORES`` with default\n placement if possible (See :ref:`torch_placement_default`)\n\n\nIn the current release of NeuronSDK, the most well-supported method of placing\nmodels onto specific NeuronCores is to use the ``NEURONCORE_GROUP_SIZES``\nenvironment variable. This will define a set of \"NeuronCore Groups\" for the\napplication process.\n\nNeuronCore Groups are *contiguous sets of NeuronCores* that are allocated to\na given process. Creating groups allows an application to ensure that a\nmodel has a defined set of NeuronCores that will always be allocated to it.\n\nNote that NeuronCore Groups *can* be used to allocate non-pipelined models\n(those requiring exactly 1 NeuronCore) to specific NeuronCores but this is\nnot the primary intended use. The intended use of NeuronCore Groups is to\nensure pipelined models (those requiring >1 NeuronCore) have exclusive access\nto a specific set of contiguous NeuronCores.\n\nIn the cases where models are being used *without* NeuronCore Pipeline, the\ngeneral recommendation is to use default placement\n(See :ref:`torch_placement_default`).\n\nThe following section demonstrates how ``NEURONCORE_GROUP_SIZES`` can be used\nand the issues that may arise.\n\nExample: Single NeuronCore Group\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn the example where one model requires 4 NeuronCores, the correct environment\nconfiguration would be:\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURONCORE_GROUP_SIZES = '4'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-4-neuron-pipeline-cores.pt') # Loads to nc0-nc3\n\n\n.. raw:: html\n :file: images/1-ncg-4.svg\n\nThis is the most basic usage of a NeuronCore Group. The environment setup\ncauses the process to take control of 4 NeuronCores and then the script loads\na model compiled with a NeuronCore Pipeline size of 4 to the first group.\n\n\nExample: Multiple NeuronCore Groups\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWith more complicated configurations, the intended use of\n``NEURONCORE_GROUP_SIZES`` is to create 1 Group per model with the correct size\nto ensure that the models are placed on the intended NeuronCores. Similarly, the\nenvironment would need to be configured to create a NeuronCore Group for each\nmodel:\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURONCORE_GROUP_SIZES = '3,4,1'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-3-neuron-pipeline-cores.pt') # Loads to nc0-nc2\n m1 = torch.jit.load('model-with-4-neuron-pipeline-cores.pt') # Loads to nc3-nc6\n m2 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc7\n\n\n\n\n.. raw:: html\n :file: images/2-ncg-3-4-1.svg\n\n\nIssue: Overlapping Models with Differing Model Sizes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen multiple models are loaded to a single NeuronCore Group, this can cause\nunintended inefficiencies. A single model is only intended to span a single\nNeuronCore Group. Applications with many models of varying sizes can be\nrestricted by NeuronCore Group configurations since the most optimal model\nlayout may require more fine-grained controls.\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURONCORE_GROUP_SIZES = '2,2'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc0-nc1\n m1 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc2-nc3\n m2 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc0\n m3 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc2\n m4 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc0\n\n\n.. raw:: html\n :file: images/3-models-m4-0-warning.svg\n\n.. raw:: html\n :file: images/3-models-m2-0-m3-2.svg\n\n.. raw:: html\n :file: images/3-ncg-2-2.svg\n\n\nHere the ``NEURONCORE_GROUP_SIZES`` does not generate an optimal layout\nbecause placement strictly follows the layout of NeuronCore Groups. A\npotentially more optimal layout would be to place ``m4`` onto ``nc1``. In this\ncase, since a pipelined model will not be able to have exclusive access to a set\nof NeuronCores, the default NeuronCore placement (no NeuronCore Groups\nspecified) would more evenly distribute the models.\n\nAlso note here that this is an example of where the order of model loads\naffects which model is assigned to which NeuronCore Group. If the order of the\nload statements is changed, models may be assigned to different NeuronCore\nGroups.\n\n\nIssue: Incompatible Model Sizes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAnother problem occurs when attempting to place a model which does not evenly\nfit into a single group:\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURONCORE_GROUP_SIZES = '2,2'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n m0 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc0-nc1\n m1 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc2-nc3\n m2 = torch.jit.load('model-with-3-neuron-pipeline-cores.pt') # Loads to nc0-nc2\n\n\n.. raw:: html\n :file: images/4-models-m2-0-2-warning.svg\n\n.. raw:: html\n :file: images/3-ncg-2-2.svg\n\n\nThe model will be placed *across* NeuronCore Groups since there is no obvious\ngroup to assign the model to according to the environment variable\nconfiguration. Depending on the individual model and application requirements,\nthe placement here may not be optimal.\n\n\nIssue: Multiple Model Copies\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIt is common in inference serving applications to use multiple replicas of a\nsingle model across different NeuronCores. This allows the hardware to be fully\nutilized to maximize throughput. In this scenario, when using NeuronCore\nGroups, the only way to replicate a model on multiple NeuronCores is to create a\n*new model* object. In the example below, 4 models loads are performed to place\na model in each NeuronCore Group.\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURONCORE_GROUP_SIZES = '2,2,2,2'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n models = list()\n for _ in range(4):\n model = torch.jit.load('model-with-2-neuron-pipeline-cores.pt')\n models.append(model)\n\n\n.. raw:: html\n :file: images/3-ncg-2-2-2-2-copies.svg\n\n\nThe largest consequence of this type of model allocation is that the application\ncode is responsible for routing inference requests to models. There are a\nvariety of ways to implement the inference switching but in all cases routing\nlogic needs to be implemented in the application code.\n\n\nIssue Summary\n^^^^^^^^^^^^^\n\nThe use of ``NEURONCORE_GROUP_SIZES`` has the following problems:\n\n- **Variable Sized Models**: Models which require crossing NeuronCore Group\n boundaries may be placed poorly. This means group configuration limits the\n size of which models can be loaded.\n- **Model Load Order**: Models are loaded to NeuronCore Groups greedily. This\n means that the order of model loads can potentially negatively affect\n application performance by causing unintentional overlap.\n- **Implicit Placement**: NeuronCore Groups cannot be explicitly chosen in the\n application code.\n- **Manual Replication**: When loading multiple copies of a model to different\n NeuronCore Groups, this requires that multiple model handles are used.\n\n\n.. _torch_placement_explicit:\n\nExplicit Core Placement\n-------------------------------------\n\nTo address the limitations of ``NEURONCORE_GROUP_SIZES``, a new set of APIs has\nbeen added which allows specific NeuronCores to be chosen by the application\ncode. These can be found in the :ref:`torch_neuron_core_placement_api` documentation.\n\n\nExample: Manual Core Selection\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe most direct usage of the placement APIs is to manually select the\nstart NeuronCore that each model is loaded to. This will automatically use as\nmany NeuronCores as is necessary for that model (1 for most models, >1 for\nNeuronCore Pipelines models).\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_NUM_CORES = '4'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n # NOTE: Order of loads does NOT matter\n\n with torch_neuron.experimental.neuron_cores_context(2):\n m1 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc2-nc3\n\n with torch_neuron.experimental.neuron_cores_context(0):\n m2 = torch.jit.load('model-with-3-neuron-pipeline-cores.pt') # Loads to nc0-nc2\n\n with torch_neuron.experimental.neuron_cores_context(0):\n m0 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads to nc0-nc1\n\n with torch_neuron.experimental.neuron_cores_context(3):\n m3 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads to nc3\n\n\n.. raw:: html\n :file: images/5-models-m2-0-2-m3-3.svg\n\n.. raw:: html\n :file: images/5-placement.svg\n\n\nNote that this directly solves the ``NEURONCORE_GROUP_SIZES`` issues of:\n\n- **Variable Sized Models**: Now since models are directly placed on the\n NeuronCores requested by the application, there is no disconnect\n between the model sizes and NeuronCore Group sizes.\n- **Model Load Order**: Since the NeuronCores are explicitly selected, there is\n no need to be careful about the order in which models are loaded since they\n can be placed deterministically regardless of the load order.\n- **Implicit Placement**: Similarly, explicit placement means there is no chance\n that a model will end up being allocated to an incorrect NeuronCore Group.\n\n\nExample: Automatic Multicore\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing explicit core placement it is possible to replicate a model to multiple\nNeuronCores simultaneously. This means that a single model object within python\ncan utilize all available NeuronCores (or NeuronCores allocated to the process).\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_NUM_CORES = '8'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n with torch_neuron.experimental.multicore_context():\n m0 = torch.jit.load('model-with-1-neuron-pipeline-cores.pt') # Loads replications to nc0-nc7\n\n\n.. raw:: html\n :file: images/6-multicore.svg\n\n\nThis addresses the last ``NEURONCORE_GROUP_SIZES`` issue of:\n\n- **Manual Replication**: Since models can be automatically replicated to\n multiple NeuronCores, this means that applications no longer need to implement\n routing logic and perform multiple loads.\n\nThis API has a secondary benefit that the exact same loading logic can be used\non an ``inf1.xlarge`` or an ``inf1.6xlarge``. In either case, it will use all\nof the NeuronCores that are visible to the process. This means that no special\nlogic needs to be coded for different instance types.\n\n\nExample: Explicit Replication\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nReplication is also possible with the\n:func:`~torch_neuron.experimental.neuron_cores_context` API. The number of\nreplications is chosen by ``replications = floor(nc_count / cores_per_model)``.\n\n\n**Environment Setup**:\n\n.. code-block:: bash\n\n export NEURON_RT_NUM_CORES = '8'\n\n**Python Script**:\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n with torch_neuron.experimental.neuron_cores_context(start_nc=2, nc_count=4):\n m0 = torch.jit.load('model-with-2-neuron-pipeline-cores.pt') # Loads replications to nc2-nc5\n\n\n.. raw:: html\n :file: images/7-replication.svg\n" }, { "path": "archive/torch-neuron/guides/torch-lstm-support.rst", "content": ".. _torch_neuron_lstm_support:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nDeveloper Guide - PyTorch Neuron (``torch-neuron``) |LSTM| Support\n==================================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nThe `torch-neuron` package can support |LSTM| operations and yield\nhigh performance on both fixed-length and variable-length sequences. Most\nnetwork configurations can be supported, with the exception of those that\nrequire |PackedSequence| usage outside of |LSTM| or |pad_packed_sequence|\noperations. Neuron must guarantee that the shapes can remain fixed throughout\nthe network.\n\nThe following sections describe which scenarios can and cannot be supported.\n\nSupported Usage\n---------------\n\nFixed-Length Sequences\n~~~~~~~~~~~~~~~~~~~~~~\n\nIn normal usage of an |LSTM|, the inputs and outputs are expected to be a fixed\nsize sequence length. This is the most basic usage of an |LSTM| but may not be\napplicable to applications where the input sequence length may vary.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n class Network(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.lstm = torch.nn.LSTM(input_size=3, hidden_size=7)\n\n def forward(self, inputs):\n output, (ht, ct) = self.lstm(inputs)\n return output, (ht, ct)\n\n # Example Inputs\n seq_len, batch_size, input_size = 5, 2, 3\n inputs = torch.rand(seq_len, batch_size, input_size)\n\n # Trace\n torch_neuron.trace(Network(), (inputs,))\n\n\nPacked Input, Padded Output, *Pre-Sorted* Inputs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA common usage of an |LSTM| is when the input sequence sizes vary according\nto an input sequence lengths (such as tokens).\n\nFor example, the following sentences could result in two different\nsequence lengths after tokenization:\n\n.. code-block:: python\n\n # Input\n text = [\n 'Hello, sailor',\n 'Example',\n ]\n\n # ... Tokenization ...\n\n # Result\n tokens = [\n [101, 7592, 1010, 11803, 102],\n [101, 2742, 102, 0, 0],\n ]\n lengths = [5, 3]\n\nBecause the lengths are different, the final |LSTM| state will be dependent upon\nthe lengths of each sequence in the batch. Torch provides a way to deal with\nthese types of sequences by densely packing batches into a |PackedSequence|. The\nmost common way this is constructed is by using the |pack_padded_sequence|\nutility function prior to feeding inputs into the |LSTM|.\n\nPacking the above sequences would result in the following data and batch\nsize tensors.\n\n.. code-block:: python\n\n data = [101, 101, 7592, 2742, 1010, 102, 11803, 102]\n batch_sizes = [2, 2, 2, 1, 1]\n\n\nIn addition to correctly computing final |LSTM| state, using a packed\nsequence instead of a padded sequence also improves model performance on CPU.\nOn Neuron, where computation is fixed to the maximum length ahead of time,\n**this is does not improve performance**.\n\nWhen an |LSTM| is processing a |PackedSequence|, it must do so in a descending\nsorted length order. To ensure that sequences are sorted, |pack_padded_sequence|\nprovides an ``enforce_sorted`` flag. When ``enforce_sorted`` is ``True``, the\ninput is *already expected* to contain sequences sorted by length in a\ndecreasing order along the batch dimension. Note that this must be enforced in\nthe application-level code but is only relevant when batch size > 1.\n\nThe following network can compile successfully because the input and output\nto the network are guaranteed to be a fixed shape. The input shape is expected\nto be a padded tensor and the output tensor is expected to be padded to the\nmaximum sequence length using the |pad_packed_sequence| function call:\n\n.. code-block:: python\n :emphasize-lines: 14\n\n import torch\n import torch_neuron\n\n class Network(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.lstm = torch.nn.LSTM(input_size=3, hidden_size=7)\n\n def forward(self, inputs, lengths):\n packed_input = torch.nn.utils.rnn.pack_padded_sequence(\n inputs,\n lengths=lengths,\n enforce_sorted=True,\n )\n packed_result, (ht, ct) = self.lstm(packed_input)\n padded_result, _ = torch.nn.utils.rnn.pad_packed_sequence(packed_result)\n return padded_result, ht, ct\n\n # Example Inputs\n seq_len, batch_size, input_size = 5, 2, 3\n inputs = torch.rand(seq_len, batch_size, input_size)\n lengths = torch.tensor([seq_len] * batch_size)\n\n # Trace\n torch_neuron.trace(Network(), (inputs, lengths))\n\n\nPacked Input, Padded Output, *Unsorted* Inputs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen ``enforce_sorted`` is ``False``, the input will be sorted unconditionally.\nThis causes some CPU overhead on Neuron because unsupported operators will be\ninserted into the graph such as ``aten::sort`` and ``aten::scatter_``. The\n``aten::lstm`` operation can still be supported, but it will be less efficient\nthan when ``enforce_sorted`` is ``True``.\n\nThe following code is able to be traced, but results in the sorting\noperations running on CPU. This is not problematic in this case because the\n``aten::sort`` and ``aten::scatter_`` are executed on CPU at the very beginning\nof the graph just prior to Neuron execution.\n\nLike the previous example, the call to |pad_packed_sequence| ensures that the\noutput is a fixed-shape based on the maximum sequence length.\n\n.. code-block:: python\n :emphasize-lines: 14\n\n import torch\n import torch_neuron\n\n class Network(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.lstm = torch.nn.LSTM(input_size=3, hidden_size=7)\n\n def forward(self, inputs, lengths):\n packed_input = torch.nn.utils.rnn.pack_padded_sequence(\n inputs,\n lengths=lengths,\n enforce_sorted=False,\n )\n packed_result, (ht, ct) = self.lstm(packed_input)\n padded_result, _ = torch.nn.utils.rnn.pad_packed_sequence(packed_result)\n return padded_result, ht, ct\n\n # Example Inputs\n seq_len, batch_size, input_size = 5, 2, 3\n inputs = torch.rand(seq_len, batch_size, input_size)\n lengths = torch.tensor([seq_len] * batch_size)\n\n # Trace\n trace = torch_neuron.trace(Network(), (inputs, lengths))\n\n\nPacked Inputs, Final Hidden & Cell State Only\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen **only** the final |LSTM| hidden & cell state is used, it does not\nmatter if the inputs are packed or unpacked since these state\ntensors will not vary in size.\n\n.. code-block:: python\n :emphasize-lines: 16,17\n\n import torch\n import torch_neuron\n\n class Network(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.lstm = torch.nn.LSTM(input_size=3, hidden_size=7)\n\n def forward(self, inputs, lengths):\n packed_input = torch.nn.utils.rnn.pack_padded_sequence(\n inputs,\n lengths=lengths,\n enforce_sorted=True,\n )\n packed_output, (ht, ct) = self.lstm(packed_input)\n return ht, ct\n\n # Example Inputs\n seq_len, batch_size, input_size = 5, 2, 3\n inputs = torch.rand(seq_len, batch_size, input_size)\n lengths = torch.tensor([seq_len] * batch_size)\n\n # Trace\n trace = torch_neuron.trace(Network(), (inputs, lengths))\n\nNote that when the ``packed_output`` is unused, it does not need to be passed\nto the |pad_packed_sequence| to enable the |LSTM| to be compiled.\n\nUnsupported Usage\n-----------------\n\nNeuron does not support the use of a |PackedSequence| outside of the |LSTM|\noperation and the |pad_packed_sequence| operation. This is because the shape of\na |PackedSequence| can vary depending on the input data. This is incompatible\nwith the Neuron restriction that all tensor sizes must be known at compilation\ntime. When a |PackedSequence| is used only by an |LSTM| or |pad_packed_sequence|\noperation, Neuron *can guarantee* the size of the intermediary tensors by\npadding on behalf of the application.\n\nThis means that If the |PackedSequence| is either used by a different operation\nor returned from the network this would result in all of the |LSTM| operations to\nbe executed on CPU or the network compilation will fail.\n\n\n|PackedSequence| Returned\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe following is unsupported because the |PackedSequence| result of the |LSTM|\nis returned by the network:\n\n.. code-block:: python\n :emphasize-lines: 14\n\n class Network(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.lstm = torch.nn.LSTM(input_size=3, hidden_size=7)\n\n def forward(self, inputs, lengths):\n packed_input = torch.nn.utils.rnn.pack_padded_sequence(\n inputs,\n lengths=lengths,\n enforce_sorted=False,\n )\n packed_result, (ht, ct) = self.lstm(packed_input)\n return packed_result.data, ht, ct\n\n\n**Behavior**: In this case, compilation fails and the following warning is\ngenerated:\n\n.. code-block:: text\n\n Operator \"aten::lstm\" consuming a PackedSequence input can only be supported when its corresponding PackedSequence output is unused or unpacked using \"aten::_pad_packed_input\". Found usage by \"prim::Return\"\n\n\n**Resolution**: To avoid this error, the ``packed_result`` should be padded\nprior to being returned from the network by using |pad_packed_sequence|\n\n\nInvalid |PackedSequence| Usage\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe following is unsupported because the |PackedSequence| result of the |LSTM|\nis used by a non-LSTM operator:\n\n.. code-block:: python\n :emphasize-lines: 14\n\n class Network(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.lstm = torch.nn.LSTM(input_size=3, hidden_size=7)\n\n def forward(self, inputs, lengths):\n packed_input = torch.nn.utils.rnn.pack_padded_sequence(\n inputs,\n lengths=lengths,\n enforce_sorted=False,\n )\n packed_result, (ht, ct) = self.lstm(packed_input)\n return torch.max(packed_result.data)\n\n**Behavior**: In this case, compilation fails and the following warning is\ngenerated:\n\n.. code-block:: text\n\n Operator \"aten::lstm\" consuming a PackedSequence input can only be supported when its corresponding PackedSequence output is unused or unpacked using \"aten::_pad_packed_input\". Found usage by \"aten::max\"\n\n**Resolution**: To avoid this error, the ``packed_result`` should be padded\nprior to being used in the :func:`~torch.max` from the network by\nusing |pad_packed_sequence|.\n\n\n.. |LSTM| replace:: :class:`~torch.nn.LSTM`\n.. |PackedSequence| replace:: :class:`~torch.nn.utils.rnn.PackedSequence`\n.. |pack_padded_sequence| replace:: :func:`~torch.nn.utils.rnn.pack_padded_sequence`\n.. |pad_packed_sequence| replace:: :func:`~torch.nn.utils.rnn.pad_packed_sequence`\n" }, { "path": "archive/torch-neuron/index.rst", "content": ".. _torch-neuron-main:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nPyTorch Neuron (torch-neuron) — Archived\n==========================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer actively developed.\n For new workloads, use TorchNeuron Native or torch-neuronx.\n See :doc:`/frameworks/torch/index` for current PyTorch support.\n\nPyTorch Neuron (``torch-neuron``) was the original PyTorch integration for AWS Inferentia (Inf1) instances.\nThis package supported inference workloads on NeuronCores v1 architecture.\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nAPI Reference\n-------------\n\n.. toctree::\n :maxdepth: 1\n\n api-reference-guide-torch-neuron\n api-compilation-python-api\n api-core-placement\n api-torch-neuron-dataparallel-api\n\nDeveloper Guide\n---------------\n\n.. toctree::\n :maxdepth: 1\n\n developer-guide-torch-neuron\n troubleshooting-guide\n\nTutorials\n---------\n\n.. toctree::\n :maxdepth: 1\n\n tutorials/tutorials-inference-torch-neuron\n\nSetup\n-----\n\n.. toctree::\n :maxdepth: 1\n\n setup/pytorch-install\n setup/pytorch-update\n\nMisc\n----\n\n.. toctree::\n :maxdepth: 1\n\n additional-examples-inference-torch-neuron\n misc-inference-torch-neuron\n" }, { "path": "archive/torch-neuron/inference-torch-neuron.rst", "content": ".. _inference-torch-neuron:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-13\n\nInference with ``torch-neuron`` (Inf1)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer actively developed.\n For new workloads, use TorchNeuron Native or torch-neuronx.\n See :doc:`/frameworks/torch/index` for current PyTorch support.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Tutorials \n Additional Examples \n API Reference Guide \n Developer Guide \n Misc \n\n\n.. card:: Setup (``torch-neuron``)\n :link: setup-torch-neuron\n :link-type: ref\n :class-body: sphinx-design-class-title-small\n\n\n.. dropdown:: Tutorials (``torch-neuron``)\n\t:class-title: sphinx-design-class-title-small\n\t:animate: fade-in\n\t:name: torch-neuronx-training-tutorials\n\n\t.. include:: /archive/torch-neuron/tutorials/tutorials-inference-torch-neuron.txt\n\n\n.. dropdown:: Additional Examples (``torch-neuron``)\n\t:class-title: sphinx-design-class-title-small\n\t:class-body: sphinx-design-class-body-small\n\t:animate: fade-in\n\t\n\t.. include:: /archive/torch-neuron/additional-examples-inference-torch-neuron.txt\n\n\n.. dropdown:: API Reference Guide (``torch-neuron``)\n\t:class-title: sphinx-design-class-title-small\n\t:animate: fade-in\n\n\t.. include:: /archive/torch-neuron/api-reference-guide-torch-neuron.txt\n\n\n.. dropdown:: Developer Guide (``torch-neuron``)\n\t:class-title: sphinx-design-class-title-small\n\t:animate: fade-in\n\n\t.. include:: /archive/torch-neuron/developer-guide-torch-neuron.txt\n\n.. dropdown:: Misc (``torch-neuron``)\n\t:class-title: sphinx-design-class-title-small\n\t:animate: fade-in\n\n\t* :ref:`neuron-cc-ops-pytorch`\n\t* :ref:`pytorch-neuron-inference-troubleshooting`\n\t* :ref:`pytorch-neuron-rn`\n" }, { "path": "archive/torch-neuron/misc-inference-torch-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nMisc (``torch-neuron``)\n=======================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n\t:maxdepth: 1\n\t:hidden:\n\n\t/release-notes/archive/neuron-cc/neuron-cc-ops/neuron-cc-ops-pytorch\n\t/archive/torch-neuron/troubleshooting-guide\n\t/release-notes/components/pytorch\n\n\n.. include:: /archive/torch-neuron/misc-inference-torch-neuron.txt\n" }, { "path": "archive/torch-neuron/misc-inference-torch-neuron.txt", "content": "* :ref:`neuron-cc-ops-pytorch`\n* :ref:`pytorch-neuron-inference-troubleshooting`\n* :ref:`pytorch-neuron-rn`" }, { "path": "archive/torch-neuron/placement.py", "content": "\"\"\"\n\n\n.. warning::\n\n The following functionality is beta and **will not be supported** in\n future releases of the Neuron SDK. This module serves only as a preview for\n future functionality. In future releases, equivalent functionality may\n be moved directly to the :code:`torch_neuron` module and will no longer be\n available in the :code:`torch_neuron.experimental` module.\n\nFunctions which enable placement of :class:`torch.jit.ScriptModule` to specific\nNeuronCores. Two sets of functions are provided which can be used\ninterchangeably but have different performance characteristics and advantages:\n\n- The :func:`~torch_neuron.experimental.multicore_context` &\n :func:`~torch_neuron.experimental.neuron_cores_context` functions are context\n managers that allow a model to be placed on a given NeuronCore at\n :func:`torch.jit.load` time. These functions are the most efficient way of\n loading a model since the model is loaded directly to a NeuronCore. The\n alternative functions described below require that a model is unloaded from\n one core and then reloaded to another.\n- The :func:`~torch_neuron.experimental.set_multicore` &\n :func:`~torch_neuron.experimental.set_neuron_cores` functions allow a model\n that has already been loaded to a NeuronCore to be moved to a different\n NeuronCore. This functionality is less efficient than directly loading a model\n to a NeuronCore within a context manager but allows device placement to be\n fully dynamic at runtime. This is analogous to the :meth:`torch.nn.Module.to`\n function for device placement.\n\n.. important::\n\n A prerequisite to enable placement functionality is that\n the loaded :class:`torch.jit.ScriptModule` has already been compiled with\n the :func:`torch_neuron.trace` API. Attempting to place a regular\n :class:`torch.nn.Module` onto a NeuronCore prior to compilation will do\n nothing.\n\"\"\"\nimport contextlib\n\n\ndef set_neuron_cores(trace: 'torch.jit.ScriptModule', start_nc: int=-1, nc_count: int=-1):\n \"\"\"\n Set the NeuronCore start/count for all Neuron subgraphs in a torch Module.\n\n This will unload the model from an existing NeuronCore if it is already\n loaded.\n\n *Requires Torch 1.8+*\n\n Arguments:\n trace: A torch module which contains one or more Neuron subgraphs.\n start_nc: The starting NeuronCore index where the Module is placed. The\n value ``-1`` automatically loads to the optimal NeuronCore (least\n used). Note that this index is always relative to NeuronCores\n visible to this process.\n nc_count: The number of NeuronCores to use. The value ``-1`` will load\n a model to exactly the number of cores required by that model (1 for\n most models, >1 when using NeuronCore Pipeline). If ``nc_count``\n is greater than the number of NeuronCores required by the\n model, the model will be replicated across multiple\n NeuronCores. ``(replications = floor(nc_count / cores_per_model))``\n\n Raises:\n RuntimeError: If the Neuron runtime cannot be initialized.\n ValueError: If the ``nc_count`` is an invalid number of NeuronCores.\n\n Examples:\n\n *Single Load*: Move a model to the first visible NeuronCore after\n loading.\n\n >>> model = torch.jit.load('example_neuron_model.pt')\n >>> torch_neuron.experimental.set_neuron_cores(model, start_nc=0, nc_count=1)\n >>> model(example) # Executes on NeuronCore 0\n >>> model(example) # Executes on NeuronCore 0\n >>> model(example) # Executes on NeuronCore 0\n\n *Multiple Core Replication*: Replicate a model to 2 NeuronCores after\n loading. This allows a single :class:`torch.jit.ScriptModule` to\n use multiple NeuronCores by running round-robin executions.\n\n >>> model = torch.jit.load('example_neuron_model.pt')\n >>> torch_neuron.experimental.set_neuron_cores(model, start_nc=2, nc_count=2)\n >>> model(example) # Executes on NeuronCore 2\n >>> model(example) # Executes on NeuronCore 3\n >>> model(example) # Executes on NeuronCore 2\n\n *Multiple Model Load*: Move and pin 2 models to separate NeuronCores.\n This causes each :class:`torch.jit.ScriptModule` to always execute on\n a specific NeuronCore.\n\n >>> model1 = torch.jit.load('example_neuron_model.pt')\n >>> torch_neuron.experimental.set_neuron_cores(model1, start_nc=2)\n >>> model2 = torch.jit.load('example_neuron_model.pt')\n >>> torch_neuron.experimental.set_neuron_cores(model2, start_nc=0)\n >>> model1(example) # Executes on NeuronCore 2\n >>> model1(example) # Executes on NeuronCore 2\n >>> model2(example) # Executes on NeuronCore 0\n >>> model2(example) # Executes on NeuronCore 0\n \"\"\"\n\n\ndef set_multicore(trace: 'torch.jit.ScriptModule'):\n \"\"\"\n Loads all Neuron subgraphs in a torch Module to all visible NeuronCores.\n\n This loads each Neuron subgraph within a :class:`torch.jit.ScriptModule`\n to multiple NeuronCores without requiring multiple calls to\n :func:`torch.jit.load`. This allows a single\n :class:`torch.jit.ScriptModule` to use multiple NeuronCores for\n concurrent threadsafe inferences. Executions use a round-robin strategy\n to distribute across NeuronCores.\n\n This will unload the model from an existing NeuronCore if it is already\n loaded.\n\n *Requires Torch 1.8+*\n\n Arguments:\n trace: A torch module which contains one or more Neuron subgraphs.\n\n Raises:\n RuntimeError: If the Neuron runtime cannot be initialized.\n\n Examples:\n\n *Multiple Core Replication*: Move a model across all visible\n NeuronCores after loading. This allows a single\n :class:`torch.jit.ScriptModule` to use all NeuronCores by\n running round-robin executions.\n\n >>> model = torch.jit.load('example_neuron_model.pt')\n >>> torch_neuron.experimental.set_multicore(model)\n >>> model(example) # Executes on NeuronCore 0\n >>> model(example) # Executes on NeuronCore 1\n >>> model(example) # Executes on NeuronCore 2\n \"\"\"\n\n\n@contextlib.contextmanager\ndef neuron_cores_context(start_nc: int=-1, nc_count: int=-1):\n \"\"\"\n A context which sets the NeuronCore start/count for all Neuron subgraphs.\n\n Any calls to :func:`torch.jit.load` will cause any underlying Neuron\n subgraphs to load to the specified NeuronCores within this context.\n This context manager only needs to be used during the model load.\n After loading, inferences do not need to occur in this context in order\n to use the correct NeuronCores.\n\n Note that this context is *not* threadsafe. Using multiple core placement\n contexts from multiple threads may not correctly place models.\n\n Arguments:\n start_nc: The starting NeuronCore index where the Module is placed. The\n value ``-1`` automatically loads to the optimal NeuronCore (least\n used). Note that this index is always relative to NeuronCores\n visible to this process.\n nc_count: The number of NeuronCores to use. The value ``-1`` will load\n a model to exactly the number of cores required by that model (1 for\n most models, >1 when using NeuronCore Pipeline). If ``nc_count``\n is greater than the number of NeuronCores required by the\n model, the model will be replicated across multiple\n NeuronCores. ``(replications = floor(nc_count / cores_per_model))``\n\n Raises:\n RuntimeError: If the Neuron runtime cannot be initialized.\n ValueError: If the ``nc_count`` is an invalid number of NeuronCores.\n\n Examples:\n\n *Single Load*: Directly load a model from disk to the first visible\n NeuronCore.\n\n >>> with torch_neuron.experimental.neuron_cores_context(start_nc=0, nc_count=1):\n >>> model = torch.jit.load('example_neuron_model.pt')\n >>> model(example) # Executes on NeuronCore 0\n >>> model(example) # Executes on NeuronCore 0\n >>> model(example) # Executes on NeuronCore 0\n\n *Multiple Core Replication*: Directly load a model from disk to 2\n NeuronCores. This allows a single :class:`torch.jit.ScriptModule` to\n use multiple NeuronCores by running round-robin executions.\n\n >>> with torch_neuron.experimental.neuron_cores_context(start_nc=2, nc_count=2):\n >>> model = torch.jit.load('example_neuron_model.pt')\n >>> model(example) # Executes on NeuronCore 2\n >>> model(example) # Executes on NeuronCore 3\n >>> model(example) # Executes on NeuronCore 2\n\n *Multiple Model Load*: Directly load 2 models from disk and pin them to\n separate NeuronCores. This causes each :class:`torch.jit.ScriptModule`\n to always execute on a specific NeuronCore.\n\n >>> with torch_neuron.experimental.neuron_cores_context(start_nc=2):\n >>> model1 = torch.jit.load('example_neuron_model.pt')\n >>> with torch_neuron.experimental.neuron_cores_context(start_nc=0):\n >>> model2 = torch.jit.load('example_neuron_model.pt')\n >>> model1(example) # Executes on NeuronCore 2\n >>> model1(example) # Executes on NeuronCore 2\n >>> model2(example) # Executes on NeuronCore 0\n >>> model2(example) # Executes on NeuronCore 0\n \"\"\"\n\n\n@contextlib.contextmanager\ndef multicore_context():\n \"\"\"\n A context which loads all Neuron subgraphs to all visible NeuronCores.\n\n This loads each Neuron subgraph within a :class:`torch.jit.ScriptModule`\n to multiple NeuronCores without requiring multiple calls to\n :func:`torch.jit.load`. This allows a single\n :class:`torch.jit.ScriptModule` to use multiple NeuronCores for\n concurrent threadsafe inferences. Executions use a round-robin strategy\n to distribute across NeuronCores.\n\n Any calls to :func:`torch.jit.load` will cause any underlying Neuron\n subgraphs to load to the specified NeuronCores within this context.\n This context manager only needs to be used during the model load.\n After loading, inferences do not need to occur in this context in order\n to use the correct NeuronCores.\n\n Note that this context is *not* threadsafe. Using multiple core placement\n contexts from multiple threads may not correctly place models.\n\n Raises:\n RuntimeError: If the Neuron runtime cannot be initialized.\n\n Examples:\n\n *Multiple Core Replication*: Directly load a model to all visible\n NeuronCores. This allows a single :class:`torch.jit.ScriptModule`\n to use all NeuronCores by running round-robin executions.\n\n >>> with torch_neuron.experimental.multicore_context():\n >>> model = torch.jit.load('example_neuron_model.pt')\n >>> model(example) # Executes on NeuronCore 0\n >>> model(example) # Executes on NeuronCore 1\n >>> model(example) # Executes on NeuronCore 2\n \"\"\"\n" }, { "path": "archive/torch-neuron/setup/index.rst", "content": ".. _setup-torch-neuron-archived:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nSetup Guide for Inf1\n====================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n.. toctree::\n\t:maxdepth: 1\n\n\tFresh install \n\tUpdate to latest release \n\tInstall previous releases \n\t/archive/torch-neuron/setup/pytorch-install-cxx11\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.14.2-pytorch-install.rst", "content": ".. _install-neuron-1.14.2-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.14.2)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents::\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.14.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.14.2\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.14.2 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.15.0-pytorch-install.rst", "content": ".. _install-neuron-1.15.0-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.15.0)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.0 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.15.1-pytorch-install.rst", "content": ".. _install-neuron-1.15.1-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.15.1)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.1 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.15.2-pytorch-install.rst", "content": ".. _install-neuron-1.15.2-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.15.2)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.15.2 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.16.1-pytorch-install.rst", "content": ".. _install-neuron-1.16.1-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.16.1)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.1\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.1 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.16.2-pytorch-install.rst", "content": ".. _install-neuron-1.16.2-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.16.2)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.2\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.2\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.2\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.2 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.16.3-pytorch-install.rst", "content": ".. _install-neuron-1.16.3-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.16.3)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.16.3 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.17.2-pytorch-install.rst", "content": ".. _install-neuron-1.17.2-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.17.2)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.17.2 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.18.0-pytorch-install.rst", "content": ".. _install-neuron-1.18.0-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.18.0)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.10.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.7.1\n\n\n .. tab-item:: PyTorch 1.5.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.18.0 --framework-version=pytorch-1.5.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-1.19.0-pytorch-install.rst", "content": ".. _install-neuron-1.19.0-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 1.19.0)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=1.19.0 --framework-version=pytorch-1.7.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-2.3.0-pytorch-install.rst", "content": ".. _install-neuron-2.3.0-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 2.3.0)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.3.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.3.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.3.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.3.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.3.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.3.0\n\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.3.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.3.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.3.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.3.0 --framework-version=pytorch-1.7.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-2.4.0-pytorch-install.rst", "content": ".. _install-neuron-2.4.0-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 2.4.0)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.4.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.4.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.4.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.4.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.4.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.4.0\n\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.4.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.4.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.4.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.4.0 --framework-version=pytorch-1.7.1\n\n" }, { "path": "archive/torch-neuron/setup/prev-releases/neuron-2.5.0-pytorch-install.rst", "content": ".. _install-neuron-2.5.0-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (Neuron 2.5.0)\n======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.12.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.5.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.5.0 \n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.5.0 \n \n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=develop --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n\n\n \n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.12.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 \n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 \n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.5.0 \n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.5.0 \n\n \n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=compile --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.12.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 \n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 \n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.5.0 \n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.5.0 \n \n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.11.0\n\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.10.2\n\n\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.9.1\n\n\n .. tab-item:: PyTorch 1.8.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.8.1\n\n\n\n .. tab-item:: PyTorch 1.7.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux AMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=non-dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Ubuntu DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=ubuntu --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n .. tab-item:: Amazon Linux DLAMI\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/neuronsetuphelper.py --file src/helperscripts/neuron-releases-manifest.json --install pytorch --mode=deploy --ami=dlami --os=amazonlinux --neuron-version=2.5.0 --framework-version=pytorch-1.7.1\n\n" }, { "path": "archive/torch-neuron/setup/pytorch-install-cxx11.rst", "content": ".. _pytorch-install-cxx11:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall with support for cxx11 ABI\n==================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. warning::\n\n The intended user of this guide is using a custom built version of\n ``torch`` or compiling a non-python application which must be built using\n the cxx11 ABI.\n\n *Most applications do not require this specialized distribution.*\n\n For regular installation instructions see: :ref:`Fresh install `\n\nThe standard ``torch-neuron`` packages (which are normally installed according\nto the :ref:`Fresh install ` guide) are compiled with\nthe pre-cxx11 ABI and linked against the pre-cxx11 ``libtorch``. These\ncompilation options ensure that the ``torch-neuron`` ABI matches the *publicly*\nreleased version of the ``torch`` package that is installed from the default\nPyPI index.\n\nTo support applications with specific ABI requirements, Neuron distributes\npackages which are linked against the cxx11 version of\n``libtorch``. These ``torch-neuron`` packages are built using the\n``-D_GLIBCXX_USE_CXX11_ABI=1`` compilation flag.\n\nThe only difference between these packages and the standard packages\nis the torch plugin library contained within the package. This is the\n``libtorchneuron.so`` library located in the ``torch_neuron/lib/`` package\ndirectory. All other libraries and python files within the packages are\nidentical. This means that these cxx11-compatible packages are drop-in\nreplacements in environments that are incompatible with the standard releases of\n``torch-neuron``. Behavior is identical whether compiling models or executing\ninferences.\n\nInstallation\n^^^^^^^^^^^^\n\nAll versions of the library are available to download from the following pip\nindex:\n\n::\n\n https://pip.repos.neuron.amazonaws.com/cxx11\n\n\nTo install a wheel, it is recommended to use the ``--no-deps`` flag since\nversions of ``torch`` compiled using the cxx11 ABI are not distributed on this\nindex.\n\n::\n\n pip install --extra-index-url=https://pip.repos.neuron.amazonaws.com/cxx11 torch-neuron --no-deps\n\n\nSpecific versions of ``torch-neuron`` with cxx11 ABI support can be installed\njust like standard versions of ``torch-neuron``.\n\n::\n\n pip install --extra-index-url=https://pip.repos.neuron.amazonaws.com/cxx11 \"torch-neuron>=1.8\" --no-deps\n pip install --extra-index-url=https://pip.repos.neuron.amazonaws.com/cxx11 \"torch-neuron==1.9.1\" --no-deps\n pip install --extra-index-url=https://pip.repos.neuron.amazonaws.com/cxx11 \"torch-neuron<1.10\" --no-deps\n\n.. important::\n\n This pip index does not include a distribution of ``torch`` compiled with\n the new cxx11 ABI. The intent of this index is *only* to provide Neuron SDK\n wheels.\n\n The version of ``torch`` that is distributed on the default PyPI index is\n compiled with the old pre-cxx11 ABI.\n\n If a cxx11 ``torch-neuron`` package is installed *with* dependencies\n using the *default* PyPI index, then the installed version of ``torch`` will\n be using the pre-cxx11 ABI and ``torch-neuron`` will be using the cxx11\n ABI. This ABI mismatch will lead to errors in both python usage and at link\n time for non-python applications.\n\nFAQ\n^^^\n\nWhen should I use a cxx11 torch-neuron wheel?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nDistributions compiled with the new cxx11 ABI should only be used in the\nfollowing cases:\n\n 1. You have built your own version of ``torch`` which uses the new cxx11 ABI and\n need a corresponding version of ``torch-neuron`` that is compatible.\n 2. You are compiling an application against a ``libtorch``\n which uses the cxx11 ABI and would like to include\n ``libtorchneuron.so`` as well. Torch distributes these cxx11 ``libtorch``\n libraries with a ``libtorch-cxx11`` prefix.\n\n Example:\n\n ::\n\n https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.10.2%2Bcpu.zip\n\n\nCan I download a library/header zip file similar to the torch distribution?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCurrently ``torch-neuron`` does not distribute a bundled library ``.zip`` with\nonly library/header files.\n\nThe recommended alternative when compiling ``libtorchneuron.so`` into a\nnon-python application is to install the ``torch-neuron`` wheel using ``pip``\naccording to the installation instructions. Then use the ``libtorchneuron.so``\nlibrary from within the python ``site-packages`` directory.\n\nA second alternative to isolate the package contents from a python environment\nis to download the wheel and unpack the contents:\n\n.. code:: bash\n\n pip download --extra-index-url=https://pip.repos.neuron.amazonaws.com/cxx11 torch-neuron --no-deps\n wheel unpack torch_neuron-*.whl\n\nIf the exact version of the ``torch-neuron`` package is known and no\npython/pip is available in the build environment, an alternative to is fetch the\npackage file directly and ``unzip`` the wheel:\n\n.. code::\n\n wget https://pip.repos.neuron.amazonaws.com/cxx11/torch-neuron/torch_neuron--py3-none-any.whl\n unzip torch_neuron--py3-none-any.whl\n\n\n.. _pytorch-cxx11-versioning:\n\nHow can I know which ABI torch-neuron is using?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPackages which use the pre-cxx11 ABI have no local identifier and use the\nfollowing version scheme:\n\n::\n\n .\n\nPackages which use the cxx11 ABI have a ``+cxx11`` local identifier and use\nfollowing version scheme:\n\n::\n\n .+cxx11\n\n\nThis allows the ABI to be validated in the by inspecting the local identifier\n(or version suffix).\n\nExample:\n::\n\n 1.8.1.0.0.0.0+cxx11\n 1.9.1.0.0.0.0+cxx11\n 1.10.2.0.0.0.0+cxx11\n\n\nHow can I know which ABI torch is using?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe ``torch`` python package provides an API at the that allows you to check if\nthe underlying ``libtorch`` was compiled with the cxx11 ABI:\n\n.. code:: python\n\n import torch\n torch.compiled_with_cxx11_abi() # True/False\n\nCurrently ``torch-neuron`` does not have an equivalent API. If the cxx11 ABI was\nused, it will be visible in the version string (See :ref:`pytorch-cxx11-versioning`).\n\n\nTroubleshooting\n^^^^^^^^^^^^^^^\n\nWhat python errors could I see if I mix ABI versions?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUsing a version of ``torch`` compiled with the cxx11 ABI will trigger an error\nin the python interpreter when importing a version of ``torch-neuron`` using\nthe old (pre-cxx11) ABI from the standard index. This will manifest as an\nerror when the ``import torch_neuron`` statement is executed.\n\n::\n\n Traceback (most recent call last):\n File \"/python3.7/site-packages/torch_neuron/__init__.py\", line 64, in \n _register_extension()\n File \"/python3.7/site-packages/torch_neuron/__init__.py\", line 60, in _register_extension\n torch.ops.load_library(neuron_op_filename)\n File \"/python3.7/site-packages/torch/_ops.py\", line 110, in load_library\n ctypes.CDLL(path)\n File \"/python3.7/ctypes/__init__.py\", line 364, in __init__\n self._handle = _dlopen(self._name, mode)\n OSError: /python3.7/site-packages/torch_neuron/lib/libtorchneuron.so: undefined symbol: _ZN5torch6detail10class_baseC2ERKSsS3_SsRKSt9type_infoS6_\n\n\nSimilarly if using the standard pre-cxx11 version of ``torch`` with the cxx11\nversion of ``torch-neuron`` will also cause an error upon import.\n\n::\n\n Traceback (most recent call last):\n File \"/python3.7/site-packages/torch_neuron/__init__.py\", line 79, in \n _register_extension()\n File \"/python3.7/site-packages/torch_neuron/__init__.py\", line 75, in _register_extension\n torch.ops.load_library(neuron_op_filename)\n File \"/python3.7/site-packages/torch/_ops.py\", line 110, in load_library\n ctypes.CDLL(path)\n File \"/python3.7/ctypes/__init__.py\", line 364, in __init__\n self._handle = _dlopen(self._name, mode)\n OSError: /python3.7/site-packages/torch_neuron/lib/libtorchneuron.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE\n\n\nIn either of these cases, the remedy is to ensure that the ABI of the ``torch``\ndistribution matches the ABI of the ``torch-neuron`` distribution.\n\nWhat compiler/linking errors could I see if I mix ABI versions?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf you link an application which uses the old (pre-cxx11) ABI\n``libtorchneuron.so`` with a cxx11 version of ``torch``, this will trigger a\nlink error.\n\n::\n\n libtorchneuron.so: undefined reference to `torch::detail::class_base::class_base(std::string const&, std::string const&, std::string, std::type_info const&, std::type_info const&)'\n libtorchneuron.so: undefined reference to `c10::Error::Error(c10::SourceLocation, std::string)'\n libtorchneuron.so: undefined reference to `c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::string const&)'\n libtorchneuron.so: undefined reference to `c10::ClassType::getMethod(std::string const&) const'\n libtorchneuron.so: undefined reference to `c10::ivalue::ConstantString::create(std::string)'\n libtorchneuron.so: undefined reference to `c10::DeviceTypeName(c10::DeviceType, bool)'\n libtorchneuron.so: undefined reference to `torch::jit::parseSchema(std::string const&)'\n libtorchneuron.so: undefined reference to `unsigned short caffe2::TypeMeta::_typeMetaData()'\n libtorchneuron.so: undefined reference to `c10::Warning::warn(c10::SourceLocation const&, std::string const&, bool)'\n libtorchneuron.so: undefined reference to `torch::jit::parseSchemaOrName(std::string const&)'\n libtorchneuron.so: undefined reference to `c10::Symbol::fromQualString(std::string const&)'\n libtorchneuron.so: undefined reference to `c10::Error::Error(std::string, std::string, void const*)'\n libtorchneuron.so: undefined reference to `c10::detail::infer_schema::make_function_schema(std::string&&, std::string&&, c10::ArrayRef, c10::ArrayRef)'\n libtorchneuron.so: undefined reference to `c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&)'\n libtorchneuron.so: undefined reference to `torch::jit::canonicalSchemaString(c10::FunctionSchema const&)'\n\n\nSimilarly, an error will also occur in the opposite scenario where the\ncxx11 ``libtorchneuron.so`` library is used with the pre-cxx11 ``libtorch``:\n\n::\n\n libtorchneuron.so: undefined reference to `c10::ivalue::ConstantString::create(std::__cxx11::basic_string, std::allocator >)'\n libtorchneuron.so: undefined reference to `torch::jit::parseSchemaOrName(std::__cxx11::basic_string, std::allocator > const&)'\n libtorchneuron.so: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >)'\n libtorchneuron.so: undefined reference to `c10::Error::Error(std::__cxx11::basic_string, std::allocator >, std::__cxx11::basic_string, std::allocator >, void const*)'\n libtorchneuron.so: undefined reference to `torch::jit::canonicalSchemaString[abi:cxx11](c10::FunctionSchema const&)'\n libtorchneuron.so: undefined reference to `torch::detail::class_base::class_base(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >, std::type_info const&, std::type_info const&)'\n libtorchneuron.so: undefined reference to `c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&)'\n libtorchneuron.so: undefined reference to `c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&)'\n libtorchneuron.so: undefined reference to `c10::detail::infer_schema::make_function_schema(std::__cxx11::basic_string, std::allocator >&&, std::__cxx11::basic_string, std::allocator >&&, c10::ArrayRef, c10::ArrayRef)'\n libtorchneuron.so: undefined reference to `torch::jit::parseSchema(std::__cxx11::basic_string, std::allocator > const&)'\n libtorchneuron.so: undefined reference to `c10::DeviceTypeName[abi:cxx11](c10::DeviceType, bool)'\n libtorchneuron.so: undefined reference to `c10::Symbol::fromQualString(std::__cxx11::basic_string, std::allocator > const&)'\n libtorchneuron.so: undefined reference to `unsigned short caffe2::TypeMeta::_typeMetaData, std::allocator > >()'\n libtorchneuron.so: undefined reference to `c10::ClassType::getMethod(std::__cxx11::basic_string, std::allocator > const&) const'\n libtorchneuron.so: undefined reference to `c10::Warning::warn(c10::SourceLocation const&, std::__cxx11::basic_string, std::allocator > const&, bool)'\n\n\nIn either of these cases, the remedy is to ensure that the ABI of the\n``libtorch`` distribution matches the ABI of the ``libtorchneuron.so``\ndistribution.\n\nThe ``torch`` ABI must match the ``torch-neuron`` ABI or an error will occur.\n" }, { "path": "archive/torch-neuron/setup/pytorch-install-prev-al2.rst", "content": ".. _pytorch-neuron-install-prev-al2:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall Previous PyTorch Neuron Releases for Amazon Linux (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n\n\nThis section will assist you in installing previous Neuron releases.\n\n.. tab-set::\n\n\n .. tab-item:: Neuron 2.18.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.18.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.17.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.17.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.16.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.16.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-install-prev-al2023.rst", "content": "\n.. _pytorch-neuron-install-prev-al2023:\n\n.. Install previous PyTorch Neuron releases for Amazon Linux 2023 - archived\n\nUse the tabs below to install a specific previous Neuron SDK release. Select the Neuron version you need.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-install-prev-u20.rst", "content": "\n.. _pytorch-neuron-install-prev-u20:\n\n.. Install previous PyTorch Neuron releases for Ubuntu 20.04 - archived\n\nUse the tabs below to install a specific previous Neuron SDK release. Select the Neuron version you need.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-install-prev-u22.rst", "content": "\n.. _pytorch-neuron-install-prev-u22:\n\n.. Install previous PyTorch Neuron releases for Ubuntu 22.04 - archived\n\nUse the tabs below to install a specific previous Neuron SDK release. Select the Neuron version you need.\n\n.. tab-set::\n\n .. tab-item:: Neuron 2.21.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.21.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.20.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.20.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Neuron 2.19.0\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --neuron-version=2.19.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-install-prev.rst", "content": ".. _install-prev-neuron-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall previous PyTorch Neuron releases (``torch-neuron``)\n============================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. toctree::\n :maxdepth: 1\n\n Neuron 2.5.0 \n Neuron 2.4.0 \n Neuron 2.3.0 \n Neuron 1.19.0 \n Neuron 1.18.0 \n Neuron 1.17.2 \n Neuron 1.16.3 \n Neuron 1.16.2 \n Neuron 1.16.1 \n Neuron 1.15.2 \n Neuron 1.15.1 \n Neuron 1.15.0 \n Neuron 1.14.2 \n\n" }, { "path": "archive/torch-neuron/setup/pytorch-install.rst", "content": ".. _install-neuron-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nInstall PyTorch Neuron (``torch-neuron``)\n=========================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.12.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.12.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.12.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.10.2 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.10.2 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.9.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.9.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.12.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.12.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.12.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.10.2 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.10.2 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.9.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.9.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.12.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.12.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.12.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.11.0\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.10.2\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.10.2 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.10.2 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: PyTorch 1.9.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.9.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.9.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-update-al2-dlami.rst", "content": ".. _pytorch-neuron-al2-update:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=dlami-framework\n" }, { "path": "archive/torch-neuron/setup/pytorch-update-al2023.rst", "content": "\n.. _pytorch-neuron-al2023-update:\n\n.. Update PyTorch Neuron (torch-neuron) on Amazon Linux 2023 - archived\n\nIf you already have a previous Neuron release installed, select the PyTorch version tab below to get the update commands.\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-update-u20-dlami.rst", "content": "\n.. _pytorch-neuron-u20-update:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest PyTorch Neuron (``torch-neuron``)\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nIf you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=dlami-framework\n" }, { "path": "archive/torch-neuron/setup/pytorch-update-u20.rst", "content": "\n.. _pytorch-neuron-u20-update:\n\n.. Update PyTorch Neuron (torch-neuron) on Ubuntu 20.04 - archived\n\nIf you already have a previous Neuron release installed, select the PyTorch version tab below to get the update commands.\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-update-u22.rst", "content": "\n.. _pytorch-neuron-u22-update:\n\n.. Update PyTorch Neuron (torch-neuron) on Ubuntu 22.04 - archived\n\nIf you already have a previous Neuron release installed, select the PyTorch version tab below to get the update commands.\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. include:: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/setup/pytorch-update.rst", "content": ".. _update-neuron-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUpdate to latest PyTorch Neuron (``torch-neuron``)\n==================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. include:: /setup/install-templates/inf1/note-setup-cntr.rst\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\n\nDevelop on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/develop_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. tab-set::\n .. tab-item:: PyTorch 1.13.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\nCompile on compute instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/compile_mode.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=compile --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n\n\nDeploy on AWS ML accelerator instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. include:: /setup/install-templates/inf1/deploy_mode.rst\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n\n.. tab-set::\n\n .. tab-item:: PyTorch 1.13.1\n\n .. tab-set::\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=non-dlami\n\n .. tab-item:: Amazon Linux 2 DLAMI Base\n\n .. include :: /setup/install-templates/inf1/note-setup-general.rst\n\n .. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --mode=deploy --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=non-dlami\n" }, { "path": "archive/torch-neuron/torch-neuron-dataparallel-example-default.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\nThe default DataParallel use mode will replicate the model\non all available NeuronCores in the current process. The inputs will be split\non ``dim=0``.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 224, 224])\n model_neuron = torch.neuron.trace(model, image)\n\n # Create the DataParallel module\n model_parallel = torch.neuron.DataParallel(model_neuron)\n\n # Create a batched input\n batch_size = 5\n image_batched = torch.rand([batch_size, 3, 224, 224])\n\n # Run inference with a batched input\n output = model_parallel(image_batched)\n" }, { "path": "archive/torch-neuron/torch-neuron-dataparallel-example-dim-neq-zero.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\nIn this example we run DataParallel inference using four NeuronCores and\n``dim = 2``. Because ``dim != 0``, dynamic batching is not enabled.\nConsequently, the DataParallel inference-time batch size must be four times the\ncompile-time batch size. DataParallel will generate a warning that dynamic\nbatching is disabled because ``dim != 0``.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n\n # Create an example model\n class Model(torch.nn.Module):\n def __init__(self):\n super().__init__()\n self.conv = torch.nn.Conv2d(3, 3, 3)\n\n def forward(self, x):\n return self.conv(x) + 1\n\n model = Model()\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 8, 8])\n model_neuron = torch.neuron.trace(model, image)\n\n # Create the DataParallel module using 4 NeuronCores and dim = 2\n model_parallel = torch.neuron.DataParallel(model_neuron, device_ids=[0, 1, 2, 3], dim=2)\n\n # Create a batched input\n # Note that image_batched.shape[dim] / len(device_ids) == image.shape[dim]\n batch_size = 4 * 8\n image_batched = torch.rand([1, 3, batch_size, 8])\n\n # Run inference with a batched input\n output = model_parallel(image_batched)\n" }, { "path": "archive/torch-neuron/torch-neuron-dataparallel-example-disable-dynamic-batching.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\nIn the following example, we use\n:func:`torch.neuron.DataParallel.disable_dynamic_batching` to disable dynamic\nbatching. We provide an example of a batch size that will not work when dynamic\nbatching is disabled as well as an example of a batch size that does work when\ndynamic batching is disabled.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 224, 224])\n model_neuron = torch.neuron.trace(model, image)\n\n # Create the DataParallel module and use 4 NeuronCores\n model_parallel = torch.neuron.DataParallel(model_neuron, device_ids=[0, 1, 2, 3], dim=0)\n\n # Disable dynamic batching\n model_parallel.disable_dynamic_batching()\n\n # Create a batched input (this won't work)\n batch_size = 8\n image_batched = torch.rand([batch_size, 3, 224, 224])\n\n # This will fail because dynamic batching is disabled and\n # image_batched.shape[dim] / len(device_ids) != image.shape[dim]\n # output = model_parallel(image_batched)\n\n # Create a batched input (this will work)\n batch_size = 4\n image_batched = torch.rand([batch_size, 3, 224, 224])\n\n # This will work because\n # image_batched.shape[dim] / len(device_ids) == image.shape[dim]\n output = model_parallel(image_batched)\n" }, { "path": "archive/torch-neuron/torch-neuron-dataparallel-example-dynamic-batching.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\nIn the following example, we use the :func:`torch.neuron.DataParallel` module\nto run inference using several different batch sizes without recompiling the\nNeuron model.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 224, 224])\n model_neuron = torch.neuron.trace(model, image)\n\n # Create the DataParallel module\n model_parallel = torch.neuron.DataParallel(model_neuron)\n\n # Create batched inputs and run inference on the same model\n batch_sizes = [2, 3, 4, 5, 6]\n for batch_size in batch_sizes:\n image_batched = torch.rand([batch_size, 3, 224, 224])\n\n # Run inference with a batched input\n output = model_parallel(image_batched)\n" }, { "path": "archive/torch-neuron/torch-neuron-dataparallel-example-specify-ncs.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\nThe following example uses the ``device_ids`` argument to use the first three\nNeuronCores for DataParallel inference.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n from torchvision import models\n\n # Load the model and set it to evaluation mode\n model = models.resnet50(pretrained=True)\n model.eval()\n\n # Compile with an example input\n image = torch.rand([1, 3, 224, 224])\n model_neuron = torch.neuron.trace(model, image)\n\n # Create the DataParallel module, run on the first three NeuronCores\n # Equivalent to model_parallel = torch.neuron.DataParallel(model_neuron, device_ids=[0, 1, 2])\n model_parallel = torch.neuron.DataParallel(model_neuron, device_ids=['nc:0', 'nc:1', 'nc:2'])\n\n # Create a batched input\n batch_size = 5\n image_batched = torch.rand([batch_size, 3, 224, 224])\n\n # Run inference with a batched input\n output = model_parallel(image_batched)\n" }, { "path": "archive/torch-neuron/troubleshooting-guide.rst", "content": ".. _pytorch-neuron-inference-troubleshooting:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTroubleshooting Guide for PyTorch Neuron (``torch-neuron``)\n===========================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\nPatching PyTorch version 1.13 for CVEs\n--------------------------------------\n\nPyTorch version 1.13 has the following CVEs:\n- CVE-2025-32434\n- CVE-2024-31580\n- CVE-2024-31583\n\nTo patch PyTorch version 1.13, run the following on a CPU instance with Ubuntu 22 AMI (it takes 30 minutes on a c5.4xlarge):\n\n::\n\n git clone --recursive https://github.com/pytorch/pytorch -b v1.13.1\n cd pytorch\n git cherry-pick b5c3a17c2c207ebefcb85043f0cf94be9b2fef81\n git cherry-pick 9c7071b0e324f9fb68ab881283d6b8d388a4bcd2\n wget https://github.com/user-attachments/files/22013116/patch_v113.txt\n git apply patch_v113.txt\n\nTo build the pip wheel, see `build steps `_. A condensed version is provided below.\n\nInstall Miniconda by following `installation steps `_ and run the following commands:\n\n::\n\n source ~/miniconda3/bin/activate\n conda create --name conda_py39 python=3.9\n conda activate conda_py39\n conda install astunparse numpy==1.19.5 ninja pyyaml setuptools cmake cffi typing_extensions future six requests dataclasses\n conda install mkl mkl-include# CUDA only: Add LAPACK support for the GPU if needed\n conda install -c pytorch magma-cuda110 # or the magma-cuda* that matches your CUDA version from https://anaconda.org/pytorch/repo\n sudo apt install cmake g++\n\n export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-\"$(dirname $(which conda))/../\"}\n PYTORCH_BUILD_VERSION=1.13.2 PYTORCH_BUILD_NUMBER=1 python setup.py bdist_wheel\n # the PyTorch pip wheel will be in dist directory\n\n\nGeneral Torch-Neuron issues\n---------------------------\n\nIf you see an error about \"Unknown builtin op: neuron::forward_1\" like below, please ensure that import line \"import torch_neuron\" (to register the Neuron custom operation) is in the inference script before using torch.jit.load.\n\n::\n\n Unknown builtin op: neuron::forward_1.\n Could not find any similar ops to neuron::forward_1. This op may not exist or may not be currently supported in TorchScript.\n\n\nTorchVision related issues\n--------------------------\n\nIf you encounter an error like below, it is because latest torchvision\nversion >= 0.7 is not compatible with Torch-Neuron 1.5.1. Please\ndowngrade torchvision to version 0.6.1:\n\n::\n\n E AttributeError: module 'torch.jit' has no attribute '_script_if_tracing' \n\n\n2GB protobuf limit related issues\n---------------------------------\n\nIf you encounter an error like below, it is because the model size is larger than 2GB.\nTo compile such large models, use the :ref:`separate_weights=True ` flag. Note,\nensure that you have the latest version of compiler installed to support this flag.\nYou can upgrade neuron-cc using \n:code:`python3 -m pip install neuron-cc[tensorflow] -U --force --extra-index-url=https://pip.repos.neuron.amazonaws.com`\n\n::\n\n E google.protobuf.message.DecodeError: Error parsing message with type 'tensorflow.GraphDef'\n\n\n\n\ntorch.jit.trace issues\n----------------------\nThe :doc:`Trace API `\nuses the PyTorch :func:`torch.jit.trace` function to generate\n:class:`~torch.jit.ScriptModule` models for execution on Inferentia. Due to that,\nto execute your PyTorch model on Inferentia it must be torch-jit-traceable,\notherwise you need to make sure your model is torch-jit-traceable. You can try\nmodifying your underlying PyTorch model code to make it traceable. If it's not\npossible to change your model code, you can :ref:`write a wrapper around your\nmodel ` that makes it torch-jit-traceable to\ncompile it for Inferentia.\n\nPlease visit :func:`torch.jit.trace` to review the properties that a model must\nhave to be torch-jit-traceable. The PyTorch-Neuron trace API\n:func:`torch_neuron.trace` accepts :code:`**kwargs` for :func:`torch.jit.trace`.\nFor example, you can use the :code:`strict=False` flag to\n:ref:`compile models with dictionary outputs `.\n\n\n.. _wrapping-non-traceable-models:\n\nCompiling models with outputs that are not torch-jit-traceable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nTo enable compilation of models with non torch-jit-traceable outputs, you can\nuse a technique that involves writing a wrapper that converts the model's\noutput into a form that is torch-jit-traceable. You can then compile the\nwrapped model for Inferentia using :func:`torch_neuron.trace`.\n\n\nThe following example uses a wrapper to compile a model with non\ntorch-jit-traceable outputs. This model cannot be compiled for Inferentia in\nits current form because it outputs a list of tuples and tensors, which is not\ntorch-jit-traceable.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n import torch.nn as nn\n\n class Model(nn.Module):\n def __init__(self):\n super(Model, self).__init__()\n self.conv = nn.Conv2d(1, 1, 3)\n\n def forward(self, x):\n a = self.conv(x) + 1\n b = self.conv(x) + 2\n c = self.conv(x) + 3\n # An output that is a list of tuples and tensors is not torch-traceable\n return [(a, b), c]\n\n model = Model()\n model.eval()\n\n inputs = torch.rand(1, 1, 3, 3)\n\n # Try to compile the model\n model_neuron = torch.neuron.trace(model, inputs) # ERROR: This cannot be traced, we must change the output format\n\n\nTo compile this model for Inferentia, we can write a wrapper around the model\nto convert its outputs into a tuple of tensors, which is torch-jit-traceable.\n\n.. code-block:: python\n\n class NeuronCompatibilityWrapper(nn.Module):\n def __init__(self):\n super(NeuronCompatibilityWrapper, self).__init__()\n self.model = Model()\n\n def forward(self, x):\n out = self.model(x)\n # An output that is a tuple of tuples and tensors is torch-jit-traceable\n return tuple(out)\n\nNow, we can successfully compile the model for Inferentia using the\n:code:`NeuronCompatibilityWrapper` wrapper as follows:\n\n.. code-block:: python\n\n model = NeuronCompatibilityWrapper()\n model.eval()\n\n # Compile the traceable wrapped model\n model_neuron = torch.neuron.trace(model, inputs)\n\nIf the model's outputs must be in the original form, a second wrapper can be\nused to transform the outputs after compilation for Inferentia. The following\nexample uses the :code:`OutputFormatWrapper` wrapper to convert the compiled\nmodel's output back into the original form of a list of tuples and tensors.\n\n.. code-block:: python\n\n class OutputFormatWrapper(nn.Module):\n def __init__(self):\n super(OutputFormatWrapper, self).__init__()\n self.traceable_model = NeuronCompatibilityWrapper()\n\n def forward(self, x):\n out = self.traceable_model(x)\n # Return the output in the original format of Model()\n return list(out)\n\n model = OutputFormatWrapper()\n model.eval()\n\n # Compile the traceable wrapped model\n model.traceable_model = torch.neuron.trace(model.traceable_model, inputs)\n\n\nCompiling a submodule in a model that is not torch-jit-traceable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe following example shows how to compile a submodule that is part of a non\ntorch-jit-traceable model. In this example, the top-level model :code:`Outer`\nuses a dynamic flag, which is not torch-jit-traceable. However, the\nsubmodule :code:`Inner` is torch-jit-traceable and can be compiled for\nInferentia.\n\n.. code-block:: python\n\n import torch\n import torch_neuron\n import torch.nn as nn\n\n class Inner(nn.Module) :\n def __init__(self):\n super().__init__()\n self.conv = nn.Conv2d(1, 1, 3)\n\n def forward(self, x):\n return self.conv(x) + 1\n\n\n class Outer(nn.Module):\n def __init__(self):\n super().__init__()\n self.inner = Inner()\n\n def forward(self, x, add_offset: bool = False):\n base = self.inner(x)\n if add_offset:\n return base + 1\n return base\n\n model = Outer()\n inputs = torch.rand(1, 1, 3, 3)\n\n # Compile the traceable wrapped submodule\n model.inner = torch.neuron.trace(model.inner, inputs)\n\n # TorchScript the model for serialization\n script = torch.jit.script(model)\n torch.jit.save(script, 'model.pt')\n\n loaded = torch.jit.load('model.pt')\n\nAlternatively, for usage scenarios in which the model configuration is static\nduring inference, the dynamic flags can be hardcoded in a wrapper to make\nthe model torch-jit-traceable and enable compiling the entire model for Inferentia.\nIn this example, we assume the :code:`add_offset` flag is always\n:code:`True` during inference, so we can hardcode this conditional path in the\n:code:`Static` wrapper to remove the dynmaic behavior and compile the entire\nmodel for Inferentia.\n\n.. code-block:: python\n\n class Static(nn.Module):\n def __init__(self):\n super().__init__()\n self.outer = Outer()\n\n def forward(self, x):\n # hardcode `add_offset=True`\n output = self.outer(x, add_offset=True)\n return output\n\n model = Static()\n\n # We can now compile the entire model because `add_offset=True` is hardcoded in the Static wrapper\n model_neuron = torch.neuron.trace(model, inputs)\n" }, { "path": "archive/torch-neuron/tutorials/neuroncore_pipeline_pytorch.rst", "content": ".. _pytorch-tutorials-neuroncore-pipeline-pytorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUsing NeuronCore Pipeline with PyTorch Tutorial\n================================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\n\nOverview\n--------\n\nIn this tutorial we will benchmark latency of a Hugging Face Transformers model deployed in model pipeline paralle mode using the NeuronCore Pipeline feature. We will compare the results with the usual data parallel (multi-worker) deployment. We compile a pretrained BERT base model and run the benchmarking locally.\n\nTo enable faster enviroment setup, We will run both compilation and deployment (inference) on an single inf1.6xlarge instance. You can take similar steps to recreate the benchmark on other instance sizes, such as inf1.xlarge.\n\nIf you already have an Inf1 instance environment ready, this tutorial is availabe as a Jupyter notebook at :pytorch-neuron-src:`neuroncore_pipeline_pytorch.ipynb ` and instructions can be viewed at: \n\n.. toctree::\n :maxdepth: 1\n\n /src/examples/pytorch/pipeline_tutorial/neuroncore_pipeline_pytorch.ipynb\n\nInstructions of how to setup the environment and run the tutorial are available in the next sections.\n\n.. _pytorch-neuroncore-pipeline-pytorch-env-setup:\n\nSetup The Environment \n---------------------\n\nLaunch an Inf1 instance by following the below steps, please make sure to choose an inf1.6xlarge instance.\n\n.. include:: /setup/install-templates/inf1/launch-inf1-dlami.rst\n\n\n.. _pytorch-neuroncore-pipeline-pytorch-run-tutorial:\n\nRun The Tutorial\n----------------\n\nAfter connecting to the instance from the terminal, clone the Neuron Github repository to the EC2 instance and then change the working directory to the tutorial directory:\n\n.. code::\n\n git clone https://github.com/aws/aws-neuron-sdk.git\n cd aws-neuron-sdk/src/examples/pytorch\n \n\n\nThe Jupyter notebook is available as a file with the name :pytorch-neuron-src:`neuroncore_pipeline_pytorch.ipynb `, you can either run the Jupyter notebook from a browser or run it as a script from terminal:\n\n\n* **Running tutorial from browser**\n\n * First setup and launch the Jupyter notebook on your local browser by following instructions at :ref:`Running Jupyter Notebook Browser`\n * Open the Jupyter notebook from the menu and follow the instructions\n \n \nYou can also view the Jupyter notebook at:\n\n.. toctree::\n :maxdepth: 1\n\n /src/examples/pytorch/pipeline_tutorial/neuroncore_pipeline_pytorch.ipynb\n\n\n.. _pytorch-neuroncore-pipeline-pytorch-cleanup-instances:\n\nClean up your instance/s\n------------------------\n\nAfter you've finished with the instance/s that you created for this tutorial, you should clean up by terminating the instance/s, please follow instructions at `Clean up your instance `_.\n" }, { "path": "archive/torch-neuron/tutorials/pytorch-tutorial-setup.rst", "content": ".. _pytorch-tutorial-setup:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nPyTorch Tutorial Setup\n======================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n#. Launch an Inf1.6xlarge Instance:\n .. include:: /setup/install-templates/inf1/launch-inf1-dlami.rst\n\n#. Set up a development environment:\n * Enable or install PyTorch-Neuron: :ref:`install-neuron-pytorch`.\n \n\n#. Run tutorial in Jupyter notebook:\n * Follow instruction at :ref:`Setup Jupyter notebook ` to:\n \n #. Start the Jupyter Notebook on the instance\n #. Run the Jupyter Notebook from your local browser\n\n * Connect to the instance from the terminal, clone the Neuron Github repository to the Inf1 instance and then change the working directory to the tutorial directory:\n\n .. code::\n\n git clone https://github.com/aws/aws-neuron-sdk.git\n cd aws-neuron-sdk/src/examples/pytorch\n\n * Locate the tutorial notebook file (.ipynb file) under ``aws-neuron-sdk/src/examples/pytorch``\n * From your local browser, open the tutorial notebook from the menu and follow the instructions.\n\n \n" }, { "path": "archive/torch-neuron/tutorials/transformers-marianmt.rst", "content": ".. _pytorch-tutorials-marianmt:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nPyTorch HuggingFace MarianMT Tutorial\n=====================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nOverview\n--------\n\nIn this tutorial you will compile and deploy the `HuggingFace MarianMT `_ model for sequence-to-seqeunce language translation on an Inf1 instance.\n\nTo enable faster environment setup, you will run the tutorial on an inf1.6xlarge instance to enable both compilation and deployment (inference) on the same instance.\n\nIn a production environment we encourage you to try different instance sizes to optimize to your specific deployment needs.\n\nIf you have already launched an Inf1 instance and have Neuron pytorch DLAMI environment ready, tutorial is available as a Jupyter notebook at :pytorch-neuron-src:`transformers-marianmt.ipynb ` and instructions can be viewed at:\n\n.. toctree::\n :maxdepth: 1\n\n /src/examples/pytorch/transformers-marianmt.ipynb\n\nInstructions of how to setup Neuron pytorch environment and run the tutorial as a Jupyter notebook are available in the next sections.\n\n.. _pytorch-marianmt-env-setup:\n\nSetup The Environment\n---------------------\n\nLaunch an Inf1 instance by following the below steps, please make sure to choose an inf1.6xlarge instance.\n\n.. include:: /setup/install-templates/inf1/launch-inf1-dlami.rst\n\n\n.. _pytorch-marianmt-run-tutorial:\n\nRun The Tutorial\n----------------\n\nAfter connecting to the instance from the terminal, clone the Neuron Github repository to the EC2 instance and then change the working directory to the tutorial directory:\n\n.. code::\n\n git clone https://github.com/aws/aws-neuron-sdk.git\n cd aws-neuron-sdk/src/examples/pytorch\n\nThe Jupyter notebook is available as a file with the name :pytorch-neuron-src:`transformers-marianmt.ipynb ` that you can run from browser:\n\n* **Running tutorial from browser**\n\n * First setup and launch the Jupyter notebook on your local browser by following instructions at :ref:`Running Jupyter Notebook Browser`\n * Open the Jupyter notebook from the menu and follow the instructions\n\nYou can also view the Jupyter notebook at:\n\n.. toctree::\n :maxdepth: 1\n\n /src/examples/pytorch/transformers-marianmt.ipynb\n\n\n.. _marianmt-cleanup-instances:\n\nClean up your instance/s\n------------------------\n\nAfter you've finished with the instance/s that you created for this tutorial, you should clean up by terminating the instance/s, please follow instructions at `Clean up your instance `_.\n" }, { "path": "archive/torch-neuron/tutorials/tutorial-libtorch.rst", "content": ".. _pytorch-tutorials-libtorch:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nLibTorch C++ Tutorial\n=========================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nThis tutorial demonstrates the use of `LibTorch `_ with Neuron, the SDK for Amazon Inf1, Inf2 and Trn1 instances. By the end of this tutorial, you will understand how to write a native C++ application that performs inference on EC2 Inf1, Inf2 and Trn1 instances. We will use an inf1.6xlarge and a pretrained BERT-Base model to determine if one sentence is a paraphrase of another.\n\nVerify that this tutorial is running in a virtual environement that was set up according to the `Torch-Neuronx Installation Guide ` or `Torch-Neuron Installation Guide `\n\nNotes\n-----\n\nThe tutorial has been tested on Inf1, Inf2 and Trn1 instances on ubuntu instances.\n\n\nRun the tutorial\n----------------\n\nThis tutorial is self contained. It produces similar output to :ref:`[html] ` :pytorch-neuron-src:`[notebook] `.\n\nNote: The tutorial will use about 8.5 GB of disk space. Ensure you have sufficient space before beginning.\n\nRight-click and copy :download:`this link address to the tutorial archive`.\n\n.. code:: bash\n\n wget \n tar xvf libtorch_demo.tar.gz\n\nYour directory tree should now look like this:\n\n::\n\n libtorch_demo\n ├── bert_neuronx\n │ ├── compile.py\n │ └── detect_instance.py\n ├── clean.sh\n ├── core_count\n │ ├── build.sh\n │ └── main.cpp\n ├── example_app\n │ ├── build.sh\n │ ├── core_count.hpp\n │ ├── example_app.cpp\n │ ├── README.txt\n │ ├── utils.cpp\n │ └── utils.hpp\n ├── neuron.patch\n ├── run_tests.sh\n ├── setup.sh\n ├── tokenizer.json\n └── tokenizers_binding\n ├── build_python.sh\n ├── build.sh\n ├── remote_rust_tokenizer.h\n ├── run_python.sh\n ├── run.sh\n ├── tokenizer.json\n ├── tokenizer_test\n ├── tokenizer_test.cpp\n └── tokenizer_test.py\n\nThis tutorial uses the `HuggingFace Tokenizers `_ library implemented in Rust.\nInstall Cargo, the package manager for the Rust programming language.\n\n\n +----------------------------------+----------------------------------+\n | Ubuntu | Amazon Linux 2023 |\n +----------------------------------+----------------------------------+\n | .. code-block:: bash | .. code-block:: bash |\n | | |\n | sudo apt install -y cargo | sudo dnf install -y cargo |\n +----------------------------------+----------------------------------+\n\n\nRun the setup script to download additional depdendencies and build the app. (This may take a few minutes to complete.)\n\n.. literalinclude:: tutorial_source_instructions/run_libtorch.sh\n :language: bash\n :lines: 6-7\n\n::\n\n ...\n + PATH_NEURON_LIB=/opt/aws/neuron/lib/\n + g++ utils.cpp example_app.cpp -o ../example-app -O2 -D_GLIBCXX_USE_CXX11_ABI=0 -I../libtorch/include -L../tokenizers_binding/lib -L/opt/aws/neuron/lib/ -L../libtorch/lib -Wl,-rpath,libtorch/lib -Wl,-rpath,tokenizers_binding/lib -Wl,-rpath,/opt/aws/neuron/lib/ -ltokenizers -ltorchneuron -ltorch_cpu -lc10 -lpthread -lnrt\n ~/libtorch_demo\n Successfully completed setup\n\n.. _libtorch-benchmark:\n\nBenchmark\n---------\n\nThe setup script should have compiled and saved a PyTorch model compiled for neuron (bert_neuron_b6.pt). Run the provided sanity tests to ensure everything is working properly.\n\n.. literalinclude:: tutorial_source_instructions/run_libtorch.sh\n :language: bash\n :lines: 10\n\n::\n\n Running tokenization sanity checks.\n\n None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n Tokenizing: 100%|██████████████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 15021.69it/s]\n Python took 0.67 seconds.\n Sanity check passed.\n Begin 10000 timed tests.\n ..........\n End timed tests.\n C++ took 0.226 seconds.\n\n Tokenization sanity checks passed.\n Running end-to-end sanity check.\n\n The company HuggingFace is based in New York City\n HuggingFace's headquarters are situated in Manhattan\n not paraphrase: 10%\n paraphrase: 90%\n\n The company HuggingFace is based in New York City\n Apples are especially bad for your health\n not paraphrase: 94%\n paraphrase: 6%\n\n Sanity check passed.\n\nFinally, run the example app directly to benchmark the BERT model.\n\n.. note::\n\n You can safely ignore the warning about ``None of PyTorch, Tensorflow >= 2.0, ...``. This occurs because the test runs in a small virtual environment that doesn't require the full frameworks.\n\n.. literalinclude:: tutorial_source_instructions/run_libtorch.sh\n :language: bash\n :lines: 13\n\n::\n\n Getting ready................\n Benchmarking................\n Completed 32000 operations in 43 seconds => 4465.12 pairs / second\n \n ====================\n Summary information:\n ====================\n Batch size = 6\n Num neuron cores = 16\n Num runs per neuron core = 2000\n\n**Congratulations!** By now you should have successfully built and used a native C++ application with LibTorch.\n\nTroubleshooting\n---------------\n\n* In the event of SIGBUS errors you may have insufficient disk space for the creation of temporary model files at runtime. Consider clearing space or mounting additional disk storage.\n* In the event of a neuron runtime failure, confirm that the Neuron kernel module is loaded using ``sudo modprobe neuron``.\n\n.. _libtorch-cleanup:\n\n\n" }, { "path": "archive/torch-neuron/tutorials/tutorial-torchserve.rst", "content": ".. _pytorch-tutorials-torchserve:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nBERT TorchServe Tutorial\n========================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nThis tutorial demonstrates the use of `TorchServe `_ with Neuron, the SDK for Amazon Inf1 instances. By the end of this tutorial, you will understand how TorchServe can be used to serve a model backed by EC2 Inf1 instances. We will use a pretrained BERT-Base model to determine if one sentence is a paraphrase of another.\n\nVerify that this tutorial is running in a virtual environement that was set up according to the `Torch-Neuronx Installation Guide ` or `Torch-Neuron Installation Guide `\n\n.. _torchserve-compile:\n\n\nRun the tutorial\n----------------\n\nOpen a terminal, log into your remote instance, and activate a Pytorch virtual environment setup (see the :ref:`Pytorch Installation Guide `). To complete this tutorial, you will need a compiled BERT model. If you have already completed the HuggingFace Pretrained BERT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] ` then you already have the necessary file. Otherwise, you can setup your environment as shown below and then run :download:`trace_bert_neuron.py ` to obtain a traced BERT model.\n\n\nYou should now have a compiled ``bert_neuron_b6.pt`` file, which is required going forward.\n\nOpen a shell on the instance you prepared earlier, create a new directory named ``torchserve``. Copy your compiled model from the previous tutorial into this new directory.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 4-6\n\n::\n\n bert_neuron_b6.pt\n\nPrepare a new Python virtual environment with the necessary Neuron and TorchServe components. Use a virtual environment to keep (most of) the various tutorial components isolated from the rest of the system in a controlled way.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 8\n\nInstall the system requirements for TorchServe.\n\n.. tab-set::\n\n .. tab-item:: Amazon Linux 2023 DLAMI Base\n\n .. code-block:: bash\n\n sudo dnf install jq java-11-amazon-corretto-headless\n sudo alternatives --config java\n sudo alternatives --config javac\n\n .. tab-item:: Ubuntu 20 DLAMI Base\n\n .. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 10\n\n\n.. code:: bash\n\n java -version\n\n::\n\n openjdk version \"11.0.17\" 2022-10-18\n OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu218.04)\n OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu218.04, mixed mode, sharing)\n\n.. code:: bash\n\n javac -version\n\n::\n\n javac 11.0.17\n\nVerify that TorchServe is now available.\n\n.. code:: bash\n\n torchserve --version\n\n::\n\n TorchServe Version is 0.7.0\n\n\n.. _torchserve-setup:\n\nSetup TorchServe\n----------------\n\nDuring this tutorial you will need to download a few files onto your instance. The simplest way to accomplish this is to paste the download links provided above each file into a ``wget`` command. (We don't provide the links directly because they are subject to change.) For example, right-click and copy the download link for ``config.json`` shown below.\n\n.. literalinclude:: /src/examples/pytorch/torchserve/config.json\n :language: JSON\n :caption: :download:`config.json `\n\n\nNow execute the following in your shell:\n\n.. code:: bash\n\n wget \n ls\n\n::\n\n bert_neuron_b6.pt config.json\n\nDownload the `custom handler script `_ that will eventually respond to inference requests.\n\n.. literalinclude:: /src/examples/pytorch/torchserve/handler_bert.py\n :language: python\n :caption: :download:`handler_bert.py `\n :linenos:\n\nNext, we need to associate the handler script with the compiled model using ``torch-model-archiver``. Run the following commands in your terminal:\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 12-16\n\n.. note::\n\n If you modify your model or a dependency, you will need to rerun the archiver command with the ``-f`` flag appended to update the archive.\n\nThe result of the above will be a ``mar`` file inside the ``model_store`` directory.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 18\n\n::\n\n bert-max_length128-batch_size6.mar\n\nThis file is essentially an archive associated with a fixed version of your model along with its dependencies (e.g. the handler code).\n\n.. note::\n\n The version specified in the ``torch-model-archiver`` command can be appended to REST API requests to access a specific version of your model. For example, if your model was hosted locally on port 8080 and named \"bert\", the latest version of your model would be available at ``http://localhost:8080/predictions/bert``, while version 1.0 would be accessible at ``http://localhost:8080/predictions/bert/1.0``. We will see how to perform inference using this API in Step 6.\n\nCreate a `custom config `_ file to set some parameters. This file will be used to configure the server at launch when we run ``torchserve --start``.\n\n.. literalinclude:: /src/examples/pytorch/torchserve/torchserve.config\n :language: properties\n :caption: :download:`torchserve.config `\n\n.. note::\n\n This will cause TorchServe to bind on all interfaces. For security in real-world applications, you’ll probably want to use port 8443 and `enable SSL `_.\n\n\n.. _torchserve-run:\n\nRun TorchServe\n--------------\n\nIt's time to start the server. Typically we'd want to launch this in a separate console, but for this demo we’ll just redirect output to a file.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 20\n\nVerify that the server seems to have started okay.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 22\n\n::\n\n {\n \"status\": \"Healthy\"\n }\n\n.. note::\n\n If you get an error when trying to ping the server, you may have tried before the server was fully launched. Check ``torchserve.log`` for details.\n\nUse the Management API to instruct TorchServe to load our model.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 24-26\n\n::\n\n {\n \"status\": \"Model \\\"bert-max_length128-batch_size6\\\" Version: 1.0 registered with 4 initial workers\"\n }\n\n.. note::\n\n Any additional attempts to configure the model after the initial curl request will cause the server to return a 409 error. You’ll need to stop/start/configure the server to realize any changes.\n\nThe ``MAX_BATCH_DELAY`` is a timeout value that determines how long to wait before processing a partial batch. This is why the handler code needs to check the batch dimension and potentially add padding. TorchServe will instantiate the number of model handlers indicated by ``INITIAL_WORKERS``, so this value controls how many models we will load onto Inferentia in parallel. This tutorial was performed on an inf1.xlarge instance (one Inferentia chip), so there are four NeuronCores available. If you want to control worker scaling more dynamically, `see the docs `_.\n\n.. warning::\n If you attempt to load more models than NeuronCores available, one of two things will occur. Either the extra models will fit in device memory but performance will suffer, or you will encounter an error on your initial inference. You shouldn't set ``INITIAL_WORKERS`` above the number of NeuronCores. However, you may want to use fewer cores if you are using the :ref:`neuroncore-pipeline` feature.\n\nIt looks like everything is running successfully at this point, so it's time for an inference.\n\nCreate the ``infer_bert.py`` file below on your instance.\n\n.. literalinclude:: /src/examples/pytorch/torchserve/infer_bert.py\n :language: python\n :caption: :download:`infer_bert.py `\n :linenos:\n\nThis script will send a ``batch_size`` number of requests to our model. In this example, we are using a model that estimates the probability that one sentence is a paraphrase of another. The script sends positive examples in the first half of the batch and negative examples in the second half.\n\nExecute the script in your terminal.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 28\n\n::\n\n 1 ['paraphrase']\n 3 ['not paraphrase']\n 4 ['not paraphrase']\n 0 ['paraphrase']\n 5 ['not paraphrase']\n 2 ['paraphrase']\n\nWe can see that the first three threads (0, 1, 2) all report ``paraphrase``, as expected. If we instead modify the script to send an incomplete batch and then wait for the timeout to expire, the excess padding results will be discarded.\n\n\n.. _torchserve-benchmark:\n\nBenchmark TorchServe\n--------------------\n\nWe've seen how to perform a single batched inference, but how many inferences can we process per second? A separate upcoming tutorial will document performance tuning to maximize throughput. In the meantime, we can still perform a simple naïve stress test. The code below will spawn 64 worker threads, with each thread repeatedly sending a full batch of data to process. A separate thread will periodically print throughput and latency measurements.\n\n.. literalinclude:: /src/examples/pytorch/torchserve/benchmark_bert.py\n :language: python\n :caption: :download:`benchmark_bert.py `\n :linenos:\n\nRun the benchmarking script.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 30\n\n::\n\n pid 28523: current throughput 0.0, latency p50=0.000 p90=0.000\n pid 28523: current throughput 617.7, latency p50=0.092 p90=0.156\n pid 28523: current throughput 697.3, latency p50=0.082 p90=0.154\n pid 28523: current throughput 702.8, latency p50=0.081 p90=0.149\n pid 28523: current throughput 699.1, latency p50=0.085 p90=0.147\n pid 28523: current throughput 703.8, latency p50=0.083 p90=0.148\n pid 28523: current throughput 699.3, latency p50=0.083 p90=0.148\n ...\n\n**Congratulations!** By now you should have successfully served a batched model over TorchServe.\n\nYou can now shutdown torchserve.\n\n.. literalinclude:: tutorial_source_instructions/run_torchserve_u20.sh\n :language: bash\n :lines: 32\n\n\n" }, { "path": "archive/torch-neuron/tutorials/tutorial_source_instructions/run_libtorch.sh", "content": "#!/bin/bash\nset -eExuo\n#Run the setup script\ncd aws-neuron-sdk/src/examples/pytorch\nsudo apt install -y cargo \ncd libtorch_demo\nchmod +x setup.sh && ./setup.sh\n\n#Run sanity checks\n./run_tests.sh bert_neuron_b6.pt\n\n#Benchmark\n./example-app bert_neuron_b6.pt" }, { "path": "archive/torch-neuron/tutorials/tutorial_source_instructions/run_torchserve_u20.sh", "content": "#!/bin/bash\nset -eExuo\ncd aws-neuron-sdk/src/examples/pytorch\ncd torchserve\npython trace_bert_neuronx.py\nls\n\npip install transformers==4.52.* torchserve==0.7.0 torch-model-archiver==0.7.0 captum==0.6.0\n\nsudo apt install openjdk-11-jdk -y\n\nmkdir model_store\nMAX_LENGTH=$(jq '.max_length' config.json)\nBATCH_SIZE=$(jq '.batch_size' config.json)\nMODEL_NAME=bert-max_length$MAX_LENGTH-batch_size$BATCH_SIZE\ntorch-model-archiver --model-name \"$MODEL_NAME\" --version 1.0 --serialized-file ./bert_neuron_b6.pt --handler \"./handler_bert_neuronx.py\" --extra-files \"./config.json\" --export-path model_store\n\nls model_store\n\ntorchserve --start --ncs --model-store model_store --ts-config torchserve.config 2>&1 >torchserve.log\nsleep 10\ncurl http://127.0.0.1:8080/ping\n\nMAX_BATCH_DELAY=5000 # ms timeout before a partial batch is processed\nINITIAL_WORKERS=2 # Number from table above\ncurl -X POST \"http://localhost:8081/models?url=$MODEL_NAME.mar&batch_size=$BATCH_SIZE&initial_workers=$INITIAL_WORKERS&max_batch_delay=$MAX_BATCH_DELAY\"\n\npython infer_bert.py\n\npython benchmark_bert.py\n\ntorchserve --stop\n" }, { "path": "archive/torch-neuron/tutorials/tutorials-inference-torch-neuron.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nTutorials for Inference with torch-neuron (Inf1)\n====================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n Computer Vision Tutorials \n Natural Language Processing (NLP) Tutorials \n Utilizing Neuron Capabilities Tutorials \n\n\n\n.. include:: /archive/torch-neuron/tutorials/tutorials-inference-torch-neuron.txt\n" }, { "path": "archive/torch-neuron/tutorials/tutorials-inference-torch-neuron.txt", "content": ".. tab-set::\n\n .. tab-item:: Computer Vision Tutorials\n\n\n * ResNet-50 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * PyTorch YOLOv4 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n\n .. tab-item:: Natural Language Processing (NLP) Tutorials\n\n\n * HuggingFace pretrained BERT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * HuggingFace pretrained BERT tutorial with shared weights :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * Bring your own HuggingFace pretrained BERT container to Sagemaker Tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n * LibTorch C++ tutorial :ref:`[html] `\n * TorchServe tutorial :ref:`[html] `\n * HuggingFace MarianMT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n\n .. tab-item:: Utilizing Neuron Capabilities Tutorials\n\n\n * BERT TorchServe tutorial :ref:`[html] `\n * NeuronCore Pipeline tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n\n.. note::\n\n To use Jupyter Notebook see:\n\n * :ref:`setup-jupyter-notebook-steps-troubleshooting`\n * :ref:`running-jupyter-notebook-as-script`\n" }, { "path": "archive/torch-neuron/tutorials/tutorials-torch-neuron-computervision.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nComputer Vision Tutorials (``torch-neuron``)\n============================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n\n* ResNet-50 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n* PyTorch YOLOv4 tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n\n\n" }, { "path": "archive/torch-neuron/tutorials/tutorials-torch-neuron-nlp.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nNatural Language Processing (NLP) Tutorials (``torch-neuron``)\n==============================================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n* HuggingFace pretrained BERT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n* HuggingFace pretrained BERT tutorial with shared weights :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n* Bring your own HuggingFace pretrained BERT container to Sagemaker Tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n* LibTorch C++ tutorial :ref:`[html] `\n* TorchServe tutorial :ref:`[html] `\n* HuggingFace MarianMT tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n\n /src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.ipynb\n /src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert_shared_weights.ipynb\n /src/examples/pytorch/byoc_sm_bert_tutorial/sagemaker_container_neuron.ipynb\n tutorial-libtorch\n tutorial-torchserve\n transformers-marianmt\n" }, { "path": "archive/torch-neuron/tutorials/tutorials-utilizing-neuron-capabilities.rst", "content": ".. meta::\n :noindex:\n :nofollow:\n :description: This content is archived and no longer maintained.\n :date-modified: 2026-03-11\n\nUtilizing Neuron Capabilities Tutorials\n=======================================\n\n.. warning::\n\n This document is archived. torch-neuron (Inf1) is no longer officially supported\n by the AWS Neuron SDK. It is provided for reference only. For current\n framework support, see :doc:`/frameworks/index`.\n\n\n* BERT TorchServe tutorial :ref:`[html] `\n* NeuronCore Pipeline tutorial :ref:`[html] ` :pytorch-neuron-src:`[notebook] `\n\n\n.. toctree::\n\t:hidden:\n\n\ttutorial-torchserve\n\t/src/examples/pytorch/pipeline_tutorial/neuroncore_pipeline_pytorch.ipynb\n" }, { "path": "archive/transformers-neuronx/api-reference-guide.rst", "content": "" }, { "path": "archive/transformers-neuronx/api-reference-guide.txt", "content": "" }, { "path": "archive/transformers-neuronx/developer-guide.rst", "content": ".. _tn_developer_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\n\nTransformers Neuron Developer Guide (``transformers-neuronx``)\n==============================================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /archive/transformers-neuronx/transformers-neuronx-developer-guide\n /archive/transformers-neuronx/transformers-neuronx-developer-guide-for-continuous-batching\n\n\n.. include:: /libraries/transformers-neuronx/developer-guide.txt\n" }, { "path": "archive/transformers-neuronx/developer-guide.txt", "content": "* :ref:`transformers_neuronx_developer_guide`" }, { "path": "archive/transformers-neuronx/index.rst", "content": ".. _transformers_neuronx_archive_readme:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\n\nTransformers NeuronX (``transformers-neuronx``)\n==============================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Setup \n Developer Guide \n Tutorials \n Misc \n\n\n.. include:: /archive/transformers-neuronx/transformers-neuronx.txt\n" }, { "path": "archive/transformers-neuronx/setup/index.rst", "content": ".. _transformers-neuronx-setup:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\nTransformers NeuronX Setup (``transformers-neuronx``)\n=====================================================\n\nIf you already have setup your environment to run PyTorch NeuronX, you just need to install Transformers NeuronX library using\nthe following instruction.\n\n.. code-block::\n\n pip install transformers-neuronx --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n\nIf you are starting from scratch, Neuron Multi Framework DLAMI is recommended as it comes pre-installed with Transformers NeuronX virtual environment.\nYou can refer to the :ref:`instructions to launch a Neuron instance using Multi Framework DLAMI `\n\n\n" }, { "path": "archive/transformers-neuronx/transformers-neuronx-api-reference.rst", "content": "" }, { "path": "archive/transformers-neuronx/transformers-neuronx-developer-guide-for-continuous-batching.rst", "content": ".. _transformers_neuronx_developer_guide_for_cb:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\nTransformers NeuronX (``transformers-neuronx``) Developer Guide for Continuous Batching\n=======================================================================================\n\nTransformers NeuronX is integrated with vLLM to enable continuous batching for high-throughput \nLLM serving and inference. This guide aims to help users get started with continuous batching for\nTransformers NeuronX and vLLM by providing:\n\n- :ref:`Transformers NeuronX ` An overview of Transformers NeuronX.\n- :ref:`cb-overview` The continuous batching procedure implemented by Transformers NeuronX and vLLM.\n- :ref:`cb-install` Installation and usage instructions for Transformers NeuronX and vLLM.\n- :ref:`cb-release-221-features` A showcase of new features in Transformers NeuronX and vLLM.\n- :ref:`cb-faq`\n\n.. _cb-tnx-overview:\n\nTransformers NeuronX (``transformers-neuronx``)\n-----------------------------------------------\n\nTransformers NeuronX for Trn1 and Inf2 is a software package that enables\nPyTorch users to perform large language model (LLM) :ref:`performant inference ` on\nsecond-generation Neuron hardware (See: :ref:`NeuronCore-v2 `).\nThe :ref:`Neuron performance page ` lists expected inference performance for commonly used Large Language Models.\n\n.. _cb-overview:\n\nContinuous Batching with Transformers NeuronX and vLLM\n------------------------------------------------------\n\nTransformers NeuronX implements the following operational flow with vLLM for continuous batching support:\n\n1. Context encode multiple prompts using virtual dynamic batching.\n2. Decode all sequences simultaneously until a sequence generates an EOS token.\n3. Evict the finished sequence and insert a new prompt encoding.\n4. Resume the decoding process, repeating steps 2 and 3 until all sequences are decoded.\n\n.. _cb-supported-model-architectures:\n\nSupported Model Architectures\n-----------------------------\n\nTransformers NeuronX supports continuous batching for models compatible with the following Hugging Face classes:\n\n- ``LlamaForCausalLM``\n- ``MistralForCausalLM``\n\n.. _cb-install:\n\nInstall vLLM and Get Started with Offline Inference\n---------------------------------------------------\n\nNeuron maintains a fork of vLLM (v0.6.2) that contains the necessary changes to support inference with Transformers NeuronX.\nNeuron is working with the vLLM community to upstream these changes to make them available in a future version.\n\nInstall vLLM\n^^^^^^^^^^^^\n\nFirst install ``neuronx-cc`` and the ``transformers-neuronx`` packages. Then install the vLLM fork from source:\n\n.. code-block:: bash\n\n git clone -b v0.6.x-neuron https://github.com/aws-neuron/upstreaming-to-vllm.git\n cd upstreaming-to-vllm\n pip install -r requirements-neuron.txt\n VLLM_TARGET_DEVICE=\"neuron\" && pip install -e .\n\n.. note::\n\n Please note the vLLM ``pip`` package from PyPI is not compatible with Neuron. To work with Neuron, install vLLM using the source as outlined above.\n\n.. note::\n\n The current supported version of Pytorch for Neuron installs ``triton`` version ``2.1.0``. This is incompatible with ``vllm >= 0.5.3``. You may see an error ``cannot import name 'default_dump_dir...``. To work around this, run ``pip install --upgrade triton==3.0.0`` after installing the vLLM wheel.\n\nIf Neuron packages are detected correctly in the installation process, ``vllm-0.1.dev2830+g22c56ee.neuron216`` will be installed (The ``neuron`` version depends on the installed\n``neuronx-cc`` version).\n\nRun Offline Batched Inference with Transformers NeuronX and vLLM\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn the following example we demonstrate how to perform continuous batching with a Llama model.\n\n.. note::\n\n Since Llama models are gated, please accept the Llama Community License Agreement and request access to the model.\n Then use a Hugging Face user access token to download the model.\n\n.. code-block:: python\n\n from vllm import LLM, SamplingParams\n \n # Sample prompts.\n prompts = [\n \"Hello, my name is\",\n \"The president of the United States is\",\n \"The capital of France is\",\n \"The future of AI is\",\n ]\n # Create a sampling params object.\n sampling_params = SamplingParams(temperature=0.8, top_p=0.95)\n\n # Create an LLM.\n llm = LLM(\n model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n max_num_seqs=8,\n # The max_model_len and block_size arguments are required to be same as max sequence length,\n # when targeting neuron device. Currently, this is a known limitation in continuous batching\n # support in transformers-neuronx.\n max_model_len=128,\n block_size=128,\n # The device can be automatically detected when AWS Neuron SDK is installed.\n # The device argument can be either unspecified for automated detection, or explicitly assigned.\n device=\"neuron\",\n tensor_parallel_size=2)\n\n # Generate texts from the prompts. The output is a list of RequestOutput objects\n # that contain the prompt, generated text, and other information.\n outputs = llm.generate(prompts, sampling_params)\n # Print the outputs.\n for output in outputs:\n prompt = output.prompt\n generated_text = output.outputs[0].text\n print(f\"Prompt: {prompt!r}, Generated text: {generated_text!r}\")\n\nRun the API Server\n^^^^^^^^^^^^^^^^^^\nTo run the OpenAI-compatible API server in vLLM, run either command below:\n\n.. code-block:: bash\n\n vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --tensor-parallel-size 32 --max-num-seqs 4 --max-model-len 2048 --block-size 8\n\n.. code-block:: bash\n\n python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --tensor-parallel-size 32 --max-num-seqs 4 --max-model-len 2048 --block-size 8\n\n.. _cb-release-221-features:\n\nNew Features in Neuron Release 2.21\n-----------------------------------\n\nNeuron's vLLM integration with Transformers NeuronX is tested using a public fork of vLLM v0.6.2.\nNew features and enhancements introduced in this fork will be described below.\nNeuron's intent is to upstream these features to vLLM as soon as possible after release.\nPrior to upstreaming, these features can be accessed in the AWS Neuron GitHub\nrepository https://github.com/aws-neuron/upstreaming-to-vllm/tree/v0.6.x-neuron.\n\n**Neuron Release 2.21 Features for the v0.6.2 vLLM Neuron Fork**\n\n- :ref:`Sequence bucketing ` configuration for context encoding and token generation.\n- :ref:`Granular NeuronConfig control ` in vLLM entrypoints.\n- Inference support for :ref:`speculative decoding `.\n- Inference support for :ref:`EAGLE speculative decoding `.\n\n**Neuron Release 2.20 Features**\n\n- Multi-node inference support for larger models. Example scripts are included in `vLLM `_ .\n- Direct loading of Hugging Face-compatible checkpoints without creation of a ``-split`` directory.\n\n.. _cb-sequence-bucketing:\n\nSequence Bucketing\n^^^^^^^^^^^^^^^^^^\nTo configure buckets, set the following environment variables. Refer to the `developer guide `_\nfor details on how to configure the values. These environment variables need to be set before starting the vLLM server or instantiating the ``LLM`` object.\n\n- ``NEURON_CONTEXT_LENGTH_BUCKETS``: Bucket sizes for context encoding.\n- ``NEURON_TOKEN_GEN_BUCKETS``: Bucket sizes for token generation.\n\nFor example: ``export NEURON_CONTEXT_LENGTH_BUCKETS=\"128,512,1024\"``\n\n\n.. _cb-neuron-config-override:\n\nNeuronConfig Override\n^^^^^^^^^^^^^^^^^^^^^\nThe default ``NeuronConfig`` in vLLM uses the latest optimizations from the Neuron SDK. However, you can override the default values or add a new configuration from the `developer guide `_ by setting the ``override_neuron_config`` parameter while creating the ``LLM`` object.\n\n.. code-block:: python\n\n llm = LLM(\n model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n max_num_seqs=8,\n max_model_len=128,\n block_size=128\n device=\"neuron\",\n tensor_parallel_size=32,\n #Override or update the NeuronConfig\n override_neuron_config={\"shard_over_sequence\":True})\n\nWhile standing up the API server, set the ``override-neuron-config`` argument. For example:\n\n.. code-block:: bash\n\n python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --tensor-parallel-size 32 --max-num-seqs 4 --max-model-len 2048 --block-size 8 --override-neuron-config {\\\"shard_over_sequence\\\":\\\"True\\\"}\n\n\n.. _cb-quantization:\n\nQuantization\n^^^^^^^^^^^^\nTo use `int8 weight storage `_ ,\nset the environment variable ``NEURON_QUANT_DTYPE`` to ``s8``.\n\n\n.. _cb-speculative-decoding:\n\nSpeculative Decoding\n^^^^^^^^^^^^^^^^^^^^\nSpeculative decoding is a token generation optimization technique that\nuses a small draft model to generate ``K`` tokens autoregressively and a\nlarger target model to determine which draft tokens to accept, all in a combined forward pass.\nFor more information on speculative decoding, please see `[Leviathan, 2023] `_ and `[Chen et al., 2023] `_.\n\nSpeculative decoding is now available for inference with Transformers NeuronX and vLLM:\n\n.. code-block:: python\n\n from vllm import LLM, SamplingParams\n\n # Sample prompts.\n prompts = [\n \"Hello, my name is\",\n \"The president of the United States is\",\n \"The capital of France is\",\n \"The future of AI is\",\n ]\n # Create a sampling params object.\n sampling_params = SamplingParams(temperature=0.8, top_p=0.95)\n\n # Create an LLM.\n llm = LLM(\n model=\"meta-llama/Meta-Llama-3.1-70B-Instruct\",\n speculative_model=\"meta-llama/Llama-3.2-1B-Instruct\",\n # The max_model_len, speculative_max_model_len, and block_size arguments are required to be same as max sequence length,\n # when targeting neuron device. Currently, this is a known limitation in continuous batching\n # support in transformers-neuronx.\n max_model_len=128,\n block_size=128,\n speculative_max_model_len=128,\n dtype=\"bfloat16\",\n max_num_seqs=4,\n num_speculative_tokens=4,\n # The device can be automatically detected when AWS Neuron SDK is installed.\n # The device argument can be either unspecified for automated detection, or explicitly assigned.\n device=\"neuron\",\n tensor_parallel_size=32,\n use_v2_block_manager=True,\n )\n\n outputs = llm.generate(prompts, sampling_params)\n # Print the outputs.\n for output in outputs:\n prompt = output.prompt\n generated_text = output.outputs[0].text\n print(f\"Prompt: {prompt!r}, Generated text: {generated_text!r}\")\n\n.. note::\n\n Please ensure that the selected target and draft model are from the same model family. For example, if the target model is an instruction-tuned Llama model,\n the draft model must also be a lower-capacity instruction-tuned Llama model.\n\n.. _cb-eagle-speculative-decoding:\n\nEAGLE Speculative Decoding\n^^^^^^^^^^^^^^^^^^^^^^^^^^\nExtrapolation Algorithm for Greater Language-model Efficiency (EAGLE) extends the speculative decoding\ntechnique described above by:\n\n- Utilizing a specially trained EAGLE draft model that predicts feature outputs through an Autoregression Head and next token outputs through an LM Head.\n- Reducing sampling uncertainty by using the next autoregressively sampled token and a current feature map as draft model inputs.\n\nFor more information on EAGLE, please see `[Li et al., 2024] `_\n\nEAGLE speculative decoding can be applied without changes to the speculative decoding code sample above. Transformers NeuronX and vLLM will recognize\na draft model as an EAGLE draft when ``is_eagle: True`` is set in the model's Hugging Face ``config.json`` file.\n\n\n.. _cb-faq:\n\nFrequently Asked Questions\n--------------------------\n\n**Is PagedAttention supported in the vLLM integration?**\n\nNo, PagedAttention is not currently supported. It will be supported in a future Neuron release.\n" }, { "path": "archive/transformers-neuronx/transformers-neuronx-developer-guide.rst", "content": ".. _transformers_neuronx_developer_guide:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\nTransformers NeuronX (``transformers-neuronx``) Developer Guide\n================================================================\n\nTransformers NeuronX for Trn1 and Inf2 is a software package that enables\nPyTorch users to perform large language model (LLM) :ref:`performant inference ` on\nsecond-generation Neuron hardware (See: :ref:`NeuronCore-v2 `).The :ref:`Neuron performance page ` lists expected inference performance for commonly used Large Language Models.\n\n\nIntroduction\n------------\n\nThe `Transformers NeuronX repository `_\ncontains the source code of the AWS Neuron Transformers integration project.\nAs it stands now, it mainly serves the purpose of\nrunning transformer decoder inference (autoregressive sampling)\nworkflows on the Neuron platform.\n\nNote: This project is **actively** in development. The Neuron team is\nstill heavily modifying the Neuron optimized module classes. The\nfunctionality provided in this repository will not maintain long-term\nAPI stability until version >= 1.0.0. For applications willing to reuse\ncode from this repository, we recommend treating the Neuron optimized\nmodule implementations as samples, and pin the version of the main\nlibrary package ``torch-neuronx`` to avoid breaking interface changes as\nnew features are developed.\n\n\n\nCheckpoint compatibility with HuggingFace Transformers\n------------------------------------------------------\n\n``transformers-neuronx`` is checkpoint-compatible with HuggingFace\nTransformers. While the Neuron team reimplemented some HuggingFace\nTransformers models from scratch for the purpose of maximizing the\nexecution efficiency of transformer decoders on Neuron, the\nimplementations are done with maximizing compatibility in mind, meaning\none can train transformer decoder models, say GPT2, using the standard\nHuggingFace Transformers library, and then construct an\ninference-optimized decoder model using transformers-neuronx's\n``GPT2ForSampling`` class. If training was done with other libraries\nsuch as MegatronLM, then it is still possible to convert the obtained\ncheckpoint to the standard HuggingFace Transformers checkpoint format,\nand then move on to transformers-neuronx's optimized decoder\nimplementations.\n\n\nNeuron optimized transformer decoders implemented in XLA High Level Operations (HLO)\n------------------------------------------------------------------------------------\n\nDue to the stateful nature of the autoregressive sampling computation,\nan efficient implementation of autoregressive sampling using the Neuron\nSDK requires rewriting the model forward function into a pure-function\ncomputation running on fixed-shape tensors. Furthermore, we want the\npure-function computation be implemented in a compiled language so that\nthe Neuron compiler can perform extensive code analysis and\noptimization. We chose XLA High Level Operations (HLO) as the compiled\nlanguage for implementing Neuron optimized transformer decoder classes.\nThe source code of these classes contains Python functions written in a\nsyntax called \"PyHLO\", name of a Neuron internal tool for\nwriting/compiling the HLO language in Python. As an example, a \"language\nmodel head\" implemented in PyHLO may look like the following.\n\n::\n\n class LmHeadHlo:\n\n ...\n\n def lm_head(self, scribe):\n dtype = self.dtype\n hidden_size = self.hidden_size\n n_active_tokens = self.n_active_tokens\n batch_size = self.batch_size\n vocab_size = self.vocab_size\n hidden = dtype[hidden_size, n_active_tokens, batch_size].Parameter(parameter_number=0)\n weight = dtype[hidden_size, vocab_size].Parameter(parameter_number=1)\n rhs_size = n_active_tokens * batch_size\n hidden = dtype[hidden_size, rhs_size].Reshape(hidden)\n dot_dims = dict(lhs_contracting_dimensions=[0], rhs_contracting_dimensions=[0])\n logits = dtype[vocab_size, rhs_size].Dot(weight, hidden, dot_dimension_numbers=dot_dims)\n return dtype[vocab_size, n_active_tokens, batch_size].Reshape(logits)\n\n ...\n\nThe ``transformers_neuronx.compiler.compile_py_func`` function can\nconvert the Python ``lm_head`` function into ``HloModuleProto``, a valid\ninput format for the ``neuronx-cc`` compiler.\n\n\nTensor-parallelism support\n--------------------------\n\nFor transformer decoders used in large language models,\ntensor-parallelism is necessary as it provides a way to shard the\nmodels' large weight matrices onto multiple NeuronCores, and having\nNeuronCores working on the same matrix multiply operation\ncollaboratively. transformers-neuronx's tensor-parallelism support makes\nheavy use of collective operations such as all-reduce, which is\nsupported natively by the Neuron runtime.\n\nThere are some principles for setting tensor-parallelism degree (number\nof NeuronCores participating in sharded matrix multiply operations) for\nNeuron-optimized transformer decoder models.\n\n1. The number of attention heads needs to be divisible by the\n tensor-parallelism degree.\n2. The total data size of model weights and key-value caches needs to be\n smaller than 16 GB times the tensor-parallelism degree.\n3. Currently, the Neuron runtime supports tensor-parallelism degrees 1,\n 2, 8, and 32 on Trn1 and supports tensor-parallelism degrees 1, 2, 4,\n 8, and 24 on Inf2.\n\nSome examples:\n\n1. ``facebook/opt-13b`` has 40 attention heads, and when running at\n batch size 1 and float16 precision the model requires ~29 GB memory,\n therefore a ``trn1.2xlarge`` with 32 GB device memory is sufficient.\n2. ``facebook/opt-30b`` has 56 attention heads, and at batch size 1 and\n float16 precision the model requires ~66 GB memory, therefore it can\n run on 8 NeuronCores on one ``trn1.32xlarge`` using 128 GB device\n memory.\n3. ``gpt2-xl`` has 25 attention heads and requires ~4 GB memory at\n bfloat16 precision. It runs without tensor-parallelism only.\n\n\nFeatures\n--------\n\n\nCompile-time Configurations\n---------------------------\n\nTransformers Neuron models support a variety of compile-time configurations\nthat can be used to tune model performance. All models support the following\nconfigurations:\n\n- ``batch_size``: The batch size to compile a model for. Once the batch size has\n been set, this is the only size that is supported at inference time. Neuron\n uses ahead-of-time compilation to achieve high performance which requires\n that the compiled artifact shapes must be known at compilation time.\n- ``n_positions``: The maximum number of positions (or sequence length) to allow\n during generation. This parameter directly controls the width of the KV\n cache. This parameter should be set to the maximum expected sequence length\n for the end application.\n- ``tp_degree``: This parameter controls the number of tensor parallel shards to\n split the model into. Each shard will execute on a separate NeuronCore. To\n minimize latency, it is recommended to set the tensor parallelism to be\n equal to the number of NeuronCores that are available on an instance.\n- ``amp``: This allows a models weights and compute to be cast to a different\n type. The options are; ``'bf16'``, ``'f16'``, or ``'f32'``. For\n models trained in ``float32``, the 16-bit mixed precision options\n (``'bf16'``, ``'f16'``) generally provide sufficient accuracy while\n significantly improving performance.\n- ``context_length_estimate``: This parameter controls the maximum sequence\n length of the prompt/context handling compute graph. This parameter is\n not supported in ``GPTNeoXForSampling`` and ``GPTJForSampling``.\n\n.. code-block:: python\n\n from transformers_neuronx import NeuronAutoModelForCausalLM\n\n model = NeuronAutoModelForCausalLM.from_pretrained(\n 'gpt2', # Uses the GPT2 checkpoint from https://huggingface.co/gpt2\n batch_size=1, # Allow inference with batch size 1 inputs\n n_positions=128, # Allow a maximum size of 128 prompt & output tokens\n tp_degree=2, # Shard the model weights & compute across 2 NeuronCores\n amp='f16', # Downcast the weights & compute to float16\n context_length_estimate=64, # Build an optimized context encoding network for a maximum prompt size of 64\n )\n model.to_neuron() # Load/compile the model\n\n\n\nCheckpoint support and automatic model selection\n------------------------------------------------\n\n*New in release 2.18*\n\nTransformers Neuron now supports a greater variety of checkpoints including\nolder pytorch binary checkpoints and newer `safetensors`_ checkpoints. For\nimproved load speed and reduced host memory consumption, it is recommended to\nalways use ``safetensors`` by default. Both regular and sharded variants of\ncheckpoints are supported. It is no longer recommended to use the\n``save_pretrained_split`` function which was used in older Transformers Neuron\nexamples.\n\nIn addition to supporting standard checkpoint formats, Transformers Neuron\nprovides an AutoModel class ``NeuronAutoModelForCausalLM`` which can be\nused to load the correct model without explicitly importing the\narchitecture-specific class.\n\n.. _safetensors: https://github.com/huggingface/safetensors\n\n.. code-block:: python\n\n from transformers_neuronx import NeuronAutoModelForCausalLM\n\n # Loads: https://huggingface.co/bigscience/bloom-560m\n bloom = NeuronAutoModelForCausalLM.from_pretrained('bigscience/bloom-560m')\n bloom.to_neuron()\n\n # Loads: https://huggingface.co/openlm-research/open_llama_3b_v2\n llama = NeuronAutoModelForCausalLM.from_pretrained('openlm-research/open_llama_3b_v2')\n llama.to_neuron()\n\n # This is equivalent to the following:\n from transformers_neuronx import BloomForSampling\n model = BloomForSampling.from_pretrained('bigscience/bloom-560m')\n model.to_neuron()\n\n from transformers_neuronx import LlamaForSampling\n llama = LlamaForSampling.from_pretrained('openlm-research/open_llama_3b_v2')\n llama.to_neuron()\n\n\n.. note::\n\n Advanced features of huggingface hub access are not supported. This\n includes private repositories which require access tokens and branches.\n\n In order to support more advanced repository downloads, please download the\n model to a local directory and load it from there.\n\n\n\nHugging Face generate() API support\n-----------------------------------\n\nTransformers Neuron models support the Hugging Face `generate() `__\nAPI via the ``HuggingFaceGenerationModelAdapter`` adapter class. In the following example we\ndemonstrate how to run sampling with temperature using the ``GPT2`` model:\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer, AutoConfig\n from transformers_neuronx import GPT2ForSamplingWithContextBroadcasting, HuggingFaceGenerationModelAdapter\n\n # Create and compile the Neuron model\n model = GPT2ForSamplingWithContextBroadcasting.from_pretrained('gpt2')\n model.to_neuron()\n\n # Use the `HuggingFaceGenerationModelAdapter` to access the generate API\n config = AutoConfig.from_pretrained('gpt2')\n model = HuggingFaceGenerationModelAdapter(config, model)\n\n # Get a tokenizer and example input\n tokenizer = AutoTokenizer.from_pretrained('gpt2')\n tokenizer.pad_token_id = tokenizer.eos_token_id\n tokenizer.padding_side = 'left'\n text = \"Hello, I'm a language model,\"\n encoded_input = tokenizer(text, return_tensors='pt', padding=True)\n\n # Run inference using temperature\n with torch.inference_mode():\n model.reset_generation()\n generated_sequence = model.generate(\n input_ids=encoded_input.input_ids,\n attention_mask=encoded_input.attention_mask,\n do_sample=True,\n max_length=256,\n temperature=0.7,\n )\n\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n\nNote: As the Hugging Face generation API can expand the input's batch dimension\nbased on different generation configurations, we need to compile the neuron\nmodel with different compile batch_size compared to the run time batch_size\n(batch dimension of inputs to generation API).\n- if ``do_sample=True``, ``compile_batch_size = runtime_batch_size x num_return_sequences x beam_size``\n- otherwise, ``compile_batch_size = runtime_batch_size x num_return_sequences``\n\n\n\nNeuron Persistent Cache\n------------------------\n\nThe Neuron Persistent Cache is now enabled for Transformers Neuron by default.\nModel artifacts which have been compiled once will be cached and reused on\nsuccessive runs when possible. Model artifacts will only be reused when\ncompiling with the same compiler version (neuronx-cc), model configurations,\nand compiler flags. It also includes other features (i.e. using an S3 bucket as\nthe cache backend). For more detailed information, see the\n:ref:`Persistent cache documentation `\n\n\n.. _int8_weight_storage_support:\n\n\nint8 weight storage support\n---------------------------\n\nTransformers Neuron supports int8 weight storage for the ``GPT2`` model class.\nint8 weight storage can be used to reduce memory bandwidth usage to improve\nmodel performance. int8 weight storage support for additional model classes\nwill be added in an upcoming release. In the following example we demonstrate\nhow to apply int8 weight storage to the ``GPT2`` model via the\n``QuantizationConfig`` and ``NeuronConfig`` configs:\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer\n from transformers_neuronx import GPT2ForSamplingWithContextBroadcasting, NeuronConfig, QuantizationConfig\n\n # Set the weight storage config use int8 quantization and bf16 dequantization\n neuron_config = NeuronConfig(\n quant=QuantizationConfig(quant_dtype='s8', dequant_dtype='bf16'),\n )\n\n # Create and compile the Neuron model\n model = GPT2ForSamplingWithContextBroadcasting.from_pretrained(\n 'gpt2',\n amp='bf16', # NOTE: When using quantization, amp type must match dequant type\n neuron_config=neuron_config\n )\n model.to_neuron()\n\n # Get a tokenizer and example input\n tokenizer = AutoTokenizer.from_pretrained('gpt2')\n text = \"Hello, I'm a language model,\"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model.sample(encoded_input.input_ids, sequence_length=256, start_ids=None)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n\n\nParallel Input Prompt Context Encoding\n--------------------------------------\n\nTransformers Neuron supports parallel input prompt context encoding for the ``GPT2``\nmodel class. Parallel context encoding can be used to significantly reduce\nthe latency of the input prompt context encoding before the autoregressive\ndecoder token generation loop. Parallel context encoding support for additional\nmodel classes will be added in an upcoming release.\n\nThe ``GPT2ForSamplingWithContextBroadcasting`` class has a ``context_length_estimate``\nvariable that determines the number of input prompt tokens that will be processed in\nparallel. For optimal results, this should be set to a power of 2 that is\nclosest to the most frequently seen input prompt length.\nIn the following example we demonstrate how to apply parallel context encoding\nto the ``GPT2`` model via the ``GPT2ForSamplingWithContextBroadcasting`` class.\nIn this example, we set the ``context_length_estimate`` to be 128, which is\nthe closest power of 2 the length of the input prompt (97 tokens).\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer\n from transformers_neuronx import GPT2ForSamplingWithContextBroadcasting\n\n # Create and compile the Neuron model\n model = GPT2ForSamplingWithContextBroadcasting.from_pretrained(\n 'gpt2',\n context_length_estimate=256 # Create an optimized network which handles prompts up to 256 tokens\n )\n model.to_neuron()\n\n # Get a tokenizer and example input\n tokenizer = AutoTokenizer.from_pretrained('gpt2')\n text = \"Hello, I'm a generative AI language model. Generative AI is a type of AI that can create new content and ideas, including conversations, stories, images, videos, and music. It is powered by large models that are pre-trained on vast amounts of data and commonly referred to as foundation models (FMs). With generative AI on AWS, you can reinvent your applications, create entirely new customer experiences, drive unprecedented levels of productivity, and transform your business. \"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model.sample(encoded_input.input_ids, sequence_length=256)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n\n\nThe ``GPT2ForSamplingWithContextBroadcasting`` class can also process\nan input prompt that has a different batch size from the batch size of the\nautoregressive decoder output. For example, an input prompt with batch size = 1 can\nbe used to produce an output of batch size = 5 to generate multiple suggestions\nfor the same input prompt. The input prompt batch size can be specified using\nthe ``prompt_batch_size`` argument and the autoregressive decoder output batch\nsize can be specified using the ``batch_size`` argument. In the following example\nwe demonstrate how to apply parallel context encoding to the ``GPT2`` model\nto generate 5 outputs for a single input.\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer\n from transformers_neuronx import GPT2ForSamplingWithContextBroadcasting\n\n # Create and compile the Neuron model\n model = GPT2ForSamplingWithContextBroadcasting.from_pretrained(\n 'gpt2',\n prompt_batch_size=1, # This allows prompt and output batch to vary\n batch_size=5,\n context_length_estimate=256\n )\n model.to_neuron()\n\n # Get a tokenizer and example input\n tokenizer = AutoTokenizer.from_pretrained('gpt2')\n text = \"Hello, I'm a generative AI language model. Generative AI is a type of AI that can create new content and ideas, including conversations, stories, images, videos, and music. It is powered by large models that are pre-trained on vast amounts of data and commonly referred to as foundation models (FMs). With generative AI on AWS, you can reinvent your applications, create entirely new customer experiences, drive unprecedented levels of productivity, and transform your business. \"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model.sample(encoded_input.input_ids, sequence_length=256)\n\n for i, output in enumerate(generated_sequence):\n print('-' * 50)\n print(f'Batch {i} output:')\n print(tokenizer.decode(output))\n\n\nSerialization support\n---------------------\n\nTransformers NeuronX supports model serialization (model saving and loading) for\nall models except the ``GPTJForSampling`` and ``GPTNeoXForSampling``` model\nclasses. In the following example we demonstrate how to save and load\nthe compiled artifacts for the ``GPT2`` model:\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer\n from transformers_neuronx import GPT2ForSamplingWithContextBroadcasting\n\n # Create and compile the Neuron model\n model = GPT2ForSamplingWithContextBroadcasting.from_pretrained('gpt2')\n model.to_neuron()\n\n # Save the compiled Neuron model\n model.save('gpt2-compiled-artifacts')\n\n # Load the Neuron model\n model = GPT2ForSamplingWithContextBroadcasting.from_pretrained('gpt2')\n # Load the compiled Neuron artifacts\n model.load('gpt2-compiled-artifacts')\n # Since prior artifacts are loaded, this skips compilation\n model.to_neuron()\n\n # Get a tokenizer and example input\n tokenizer = AutoTokenizer.from_pretrained('gpt2')\n text = \"Hello, I'm a language model,\"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model.sample(encoded_input.input_ids, sequence_length=256, start_ids=None)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\nTransformers NeuronX also supports the serialization of presharded weights.\nThis reduces future model load time by saving a transformed and sharded\nset of weights as a new safetensors checkpoint. When this checkpoint is loaded,\nsharding and transformations normally done by Transformers NeuronX will be skipped,\nreducing model load time significantly. The saving of presharded weights is only\navailable when ``on_device_embedding`` is true. In the following example we\ndemonstrate how to save and load presharded weights along with compiled artifacts on a Llama model:\n\n.. code-block:: python\n\n from transformers_neuronx import LlamaForSampling\n from transformers_neuronx import NeuronConfig\n from transformers import AutoTokenizer\n\n neuron_config = NeuronConfig(on_device_embedding=True)\n\n # Create and compile the Neuron model\n model_neuron = LlamaForSampling.from_pretrained('openlm-research/open_llama_3b', batch_size=1, tp_degree=8, n_positions=128, neuron_config=neuron_config)\n model_neuron.to_neuron()\n\n # save the presharded weights and compiled artifacts to a directory\n model_neuron.save('llama-artifacts', sharded_weights=True)\n\n del model_neuron\n\n # use the presharded checkpoint to reduce model load time\n model_neuron_presharded = LlamaForSampling.from_pretrained('llama-artifacts', batch_size=1, tp_degree=8, n_positions=128, neuron_config=neuron_config)\n\n # load in the compiled artifcats to skip compilation\n model_neuron_presharded.load('llama-artifacts')\n model_neuron_presharded.to_neuron()\n\nCPU Compilation Support\n-----------------------\n\nTransformers NeuronX now supports compilation on CPU. \nCPU compilation is compatible with model serialization and presharding weights, \nand is available for all models except the GPTJForSampling and GPTNeoXForSampling \nmodel classes. To compile on CPU, the initial call to to_neuron() is replaced with \ncpu_compile(). In the following example we demonstrate how to compile on CPU for \nthe LLaMA model:\n\n.. code-block:: python\n\n from transformers_neuronx import LlamaForSampling\n from transformers_neuronx import NeuronConfig\n from transformers import AutoTokenizer\n\n neuron_config = NeuronConfig(on_device_embedding=True)\n\n # Create and compile the model on CPU\n model_neuron = LlamaForSampling.from_pretrained('openlm-research/open_llama_3b', batch_size=1, tp_degree=8, n_positions=128, neuron_config=neuron_config)\n model_neuron.cpu_compile() # instead of model_neuron.to_neuron()\n\n # save the weights and compiled artifacts to a directory\n model_neuron.save('llama-artifacts')\n\nTo use the saved artifacts generated by CPU compilation on a Neuron device: \n\n.. code-block:: python\n \n from transformers_neuronx import LlamaForSampling\n from transformers_neuronx import NeuronConfig\n from transformers import AutoTokenizer\n\n neuron_config = NeuronConfig(on_device_embedding=True)\n\n # use the presharded checkpoint to reduce model load time\n model_neuron_presharded = LlamaForSampling.from_pretrained('llama-artifacts', batch_size=1, tp_degree=8, n_positions=128, neuron_config=neuron_config)\n\n # load in the compiled artifacts to skip compilation\n model_neuron_presharded.load('llama-artifacts')\n\n # now, use CPU compiled artifacts to run the model\n model_neuron_presharded.to_neuron()\n\n\nCompilation worker count support\n--------------------------------\n\nTransformers-neuronx supports providing compilation worker count for all models. This setting controls how many workers will execute HLO graph compilation tasks in parallel. A lower setting reduces CPU memory utilization when compiling a model, but increases the compilation time. This setting is useful to prevent out of CPU memory errors when compiling large models. By default, the number of workers used is equal to the total HLO graphs required for compilation. Compilation worker count integrates with both CPU compilation flow using ``cpu_compile()`` and neuron device compilation flow using ``to_neuron()``.\nTo set the compilation worker count, use the ``compilation_worker_count`` argument in ``NeuronConfig``. The following sample shows how to compile the graphs one by one.\n\n.. code-block:: python\n\n neuron_config = NeuronConfig(compilation_worker_count=1)\n\n\nGrouped-query attention (GQA) support [Beta]\n---------------------------------------------\n\nTransformers Neuron supports grouped-query attention (GQA) models for\n``Llama`` and ``Mistral`` model classes.\nThere are multiple sharding strategies for K/V cache, in order to satisfy different constraints.\n\n- ``GQA.SHARD_OVER_HEADS`` distributes K/V caches along head dimension. This can be only used when K/V heads is multiple of tensor-parallelism degree. This is the default configuration.\n- ``GQA.SHARD_OVER_BATCH`` distributes K/V caches along batch dimension. This can be only used when batch size is multiple of tensor-parallelism degree. This can be useful for large-batch inference.\n- ``GQA.REPLICATED_HEADS`` replicates K/V heads. This can be used when neither batch size nor K/V heads can be divisible by tensor-parallelism degree. This can be useful for low-latency small-batch inference.\n- ``GQA.ALL_GATHER_HEADS`` evenly splits the K/V heads across all NeuronCores. This is optimized for large-batch inference of GQA model without replication.\n\n.. _mistral_gqa_code_sample:\n\nIn the following example we demonstrate how to configure these distributed inference strategies and\nperform inference with the ``Mistral`` model:\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer\n from transformers_neuronx import MistralForSampling, GQA, NeuronConfig\n\n # Set sharding strategy for GQA to be shard over heads\n neuron_config = NeuronConfig(\n group_query_attention=GQA.SHARD_OVER_HEADS\n )\n\n # Create and compile the Neuron model\n model_neuron = MistralForSampling.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2', amp='bf16', neuron_config=neuron_config)\n model_neuron.to_neuron()\n\n # Get a tokenizer and exaple input\n tokenizer = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2')\n text = \"[INST] What is your favourite condiment? [/INST]\"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model_neuron.sample(encoded_input.input_ids, sequence_length=256, start_ids=None)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n\n\nRepeated Ngram Filtering\n------------------------\n\nRepeated Ngram Filtering reduces redundant ngram phrases within the generated text. It uses the same API as `HuggingFace API for NoRepeatedNGram `__. Set the parameter no_repeat_ngram_size to the size of ngram phrases to be filtered and pass it to the sampling function as in the example ``model.sample(inputs_ids, no_repeat_ngram_size=3)``\n\n\nOn-device sampling support [Beta]\n--------------------------------------\n\nTransformers-neuronx supports on-device sampling for all models except Mixtral models. The features\ncan be enabled by setting ``on_device_generation`` in ``NeuronConfig`` to an instance of ``GenerationConfig``.\n\nIn the following example, we demonstrate how to use on-device generation for a ``Llama`` model using\n``top_k``, ``top_p``, ``top_p_min_tokens`` and ``temperature``.\n\n\nTop-K on-device sampling support [Beta]\n---------------------------------------\nTransformers Neuron supports Top-K Sampling on-device for all models except Mixtral models.\nIn the following example, we demonstrate how to use on-device Top-K for the ``Llama`` model via\nthe ``GenerationConfig`` and ``NeuronConfig`` configs.\n\n.. code-block:: python\n\n import torch\n from transformers_neuronx import LlamaForSampling\n from transformers_neuronx.config import NeuronConfig, GenerationConfig\n from transformers import AutoTokenizer\n\n neuron_config = NeuronConfig(\n on_device_generation=GenerationConfig(max_length=128, top_k=10, top_p=0.9, top_p_min_tokens=1, temperature=0.9, do_sample=True)\n )\n\n # Create and compile the Neuron model\n model_neuron = LlamaForSampling.from_pretrained('openlm-research/open_llama_3b', batch_size=1, tp_degree=8, n_positions=128, neuron_config=neuron_config)\n model_neuron.to_neuron()\n\n # Get a tokenizer and exaple input\n tokenizer = AutoTokenizer.from_pretrained('openlm-research/open_llama_3b')\n text = \"Hello, I'm a language model,\"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model_neuron.sample(encoded_input.input_ids, sequence_length=128, top_k=10)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n\nBy default, transformers-neuronx uses the same, fixed sampling parameters for all sequences across all invocations\nof the model when on-device generation is enabled. It is possible to provide new sampling parameters per\nmodel invocation by enabling the ``dynamic`` feature in the ``GenerationConfig``. It is also possible to provide\ndifferent sampling parameters for each sequence in the batch by using the ``per_batch_line`` feature.\nWhen using this feature, it is recommended to limit the number of tokens that are considered during\nsampling across all sequences by setting ``global_top_k`` to a reasonably low number e.g. 250 to prevent\npoor performance when computing ``top_p`` tokens over a large vocabulary without any prior filtering. When using\n``per_batch_line``, ``top_k``, ``top_p``, ``top_p_min_tokens`` and ``temperature`` accept lists with value per\nsequence in the batch.\n\n\nIn the following example, we demonstrate how to use the ``dynamic`` and ``per_batch_line`` features together.\n\n.. code-block:: python\n\n import torch\n from transformers_neuronx import LlamaForSampling\n from transformers_neuronx.config import NeuronConfig, GenerationConfig\n from transformers import AutoTokenizer\n\n batch_size = 2\n generation_config = GenerationConfig(\n max_length=128, dynamic=True, per_batch_line=True, do_sample=True,\n top_k=[1] * batch_size,\n top_p=[1.0] * batch_size,\n top_p_min_tokens=[1] * batch_size,\n temperature=[1.0] * batch_size,\n global_top_k=256\n )\n\n neuron_config = NeuronConfig(\n on_device_generation=generation_config\n )\n\n # Create and compile the Neuron model\n model_neuron = LlamaForSampling.from_pretrained('openlm-research/open_llama_3b', batch_size=2, tp_degree=8, n_positions=128, neuron_config=neuron_config)\n model_neuron.to_neuron()\n\n # Get a tokenizer and exaple input\n tokenizer = AutoTokenizer.from_pretrained('openlm-research/open_llama_3b')\n tokenizer.pad_token = tokenizer.eos_token\n text = [\"Hello, I'm a language model,\", \"Hello, I'm also a language model,\"]\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Run inference\n with torch.inference_mode():\n generated_sequence = model_neuron.sample(encoded_input.input_ids, sequence_length=128)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n # Use different settings for each sequence in the batch\n # Supported because we use `generation_config.per_batch_line = True`\n generation_config.top_k = [1, 20]\n generation_config.top_p = [1.0, 0.9]\n generation_config.top_p_min_tokens = [1, 1]\n generation_config.temperature = [1.0, 0.9]\n\n # Update the generation configuration dynamically\n # Supported because we use `generation_config.dynamic = True`\n model_neuron.update_generation_config(generation_config)\n\n generated_sequence = model_neuron.sample(encoded_input.input_ids, sequence_length=128)\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\n\n\nRunning inference with multiple models\n--------------------------------------\n\nMultiple transformers-neuronx models can be loaded at the same time as long\nas the total number of consumed NeuronCores is less than or equal to the total\nnumber of NeuronCores on the instance. For example, three tp-degree=8 models can be\nloaded and run in parallel on an inf2.48xlarge which has 24 NeuronCores. The\n``NEURON_RT_NUM_CORES`` and ``NEURON_RT_VISIBLE_CORES`` environment variables\ncan be used to allocate the necessary number of NeuronCores to each process\nto run multiple transformers-neuronx models in parallel. See the\n:ref:`torch_neuronx_core_placement_guide` section for additional information\nabout how to use these environment variables.\n\nIt is important to notice that when multiple models are used on a single instance,\nthe number of threads should be reduced to avoid race condition on host side.\nAssume the neuron instance (i.e. trn1) has 192 CPU cores.\nIf one of the models keeps all CPU cores busy, there would be significant performance\ndegradation in the rest of models. As a result, the number of threads for each model\nshould be limited to part of available cores. To do this, ``OMP_NUM_THREADS`` environment\nvariable can be set. For example, if there are 192 CPU cores available and four tp-degree=8\nmodels are used, one can export OMP_NUM_THREADS=48 to avoid race condition.\n\n\nStreamer\n----------------------------\n\nLLMs generate tokens in auto-regressive loop. A model.sample call waits till\nthe end of full sequence generation before returning the generated response.\nIt is possible to output an output token as soon as it is generated. To do this,\na streamer object can be used. Streamer is an object which has 2 methods: put and end.\nThere are several predefined streamer in transformers library such as TextIteratorStreamer.\nThe following example shows how to define a streamer and use it in transformers-neuronx:\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer\n from transformers_neuronx import MistralForSampling, GQA\n\n import transformers\n from time import time\n\n # Create a custom streamer inherited from transformers.generation.streamers.BaseStreamer\n class CustomStreamer(transformers.generation.streamers.BaseStreamer):\n def __init__(self) -> None:\n self.reset()\n\n def reset(self):\n self.token_latencies = []\n self.iter = 0\n self.now = time()\n\n def put(self, tokens):\n now = time()\n token_latency = now - self.now\n print(f\"Iteration {self.iter:4d}: Latency [s] {token_latency:6.3f} -- Token {tokens}\")\n self.now = now\n self.iter += 1\n self.token_latencies.append(token_latency)\n\n\n def end(self):\n print(\"First 10 token latencies:\", self.token_latencies[:10])\n\n\n # Create and compile the Neuron model\n model_neuron = MistralForSampling.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2', amp='bf16')\n model_neuron.to_neuron()\n\n # Get a tokenizer and exaple input\n tokenizer = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2')\n text = \"[INST] What is your favourite condiment? [/INST]\"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n streamer = CustomStreamer()\n # Run inference\n with torch.inference_mode():\n generated_sequence = model_neuron.sample(encoded_input.input_ids, sequence_length=256, start_ids=None, streamer=streamer)\n\n\nStopping Criteria\n------------------\nWe can define custom stopping criteria to stop autoregressive loop. For example, if\nwe want to limit autoregressive loop after 0.5s, we can define and use stopping criteria\nclass as follows:\n\n\n.. code-block:: python\n\n import torch\n import transformers\n from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer\n from transformers_neuronx import MistralForSampling, GQA, NeuronConfig\n from transformers_neuronx.stopping_criteria import StoppingCriteria, StoppingCriteriaList\n\n from time import time\n from typing import List, Optional, Callable\n\n\n class MaxTimeCriteria(StoppingCriteria):\n \"\"\"\n This class can be used to stop generation whenever the full generation exceeds some amount of time. By default, the\n time will start being counted when you initialize this function. You can override this by passing an\n `initial_time`.\n\n Args:\n max_time (`float`):\n The maximum allowed time in seconds for the generation.\n initial_time (`float`, *optional*, defaults to `time()`):\n The start of the generation allowed time.\n \"\"\"\n\n def __init__(self, max_time: float, initial_timestamp: Optional[float] = None):\n self.max_time = max_time\n self.initial_timestamp = time() if initial_timestamp is None else initial_timestamp\n\n def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n dt = time() - self.initial_timestamp\n end_condition = dt > self.max_time\n if end_condition:\n print(\"Stopping!\")\n return end_condition\n\n # Create a streamer. This can be a custom streamer too inherited from transformers.generation.streamers.BaseStreamer\n class CustomStreamer(transformers.generation.streamers.BaseStreamer):\n def __init__(self) -> None:\n self.reset()\n\n def reset(self):\n self.token_latencies = []\n self.iter = 0\n self.now = time()\n\n def put(self, tokens):\n now = time()\n token_latency = now - self.now\n print(f\"Iteration {self.iter:4d}: Latency [s] {token_latency:6.3f} -- Token {tokens}\")\n self.now = now\n self.iter += 1\n self.token_latencies.append(token_latency)\n\n\n def end(self):\n pass\n\n # Create and compile the Neuron model\n model_neuron = MistralForSampling.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2', amp='bf16')\n model_neuron.to_neuron()\n\n # Get a tokenizer and exaple input\n tokenizer = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2')\n text = \"[INST] What is your favourite condiment? [/INST]\"\n encoded_input = tokenizer(text, return_tensors='pt')\n\n # Add stopping criteria to stop after 0.5 seconds\n stopping_criteria_list= StoppingCriteriaList([MaxTimeCriteria(0.5)])\n streamer = CustomStreamer()\n\n # Run inference\n with torch.inference_mode():\n model_neuron.sample(input_ids=encoded_input.input_ids, sequence_length=256, stopping_criteria_list=stopping_criteria_list, streamer=streamer)\n\n\nSpeculative sampling [Beta]\n---------------------------\n\nTransformers Neuron supports speculative sampling for the ``Llama`` and ``GPT2``\nmodel classes. In speculative sampling, we use use a smaller draft model to speculate future tokens.\nThese are then sent to the larger target model, which accepts or rejects these tokens.\nFor more detailed information, see the original proposal by\nDeepMind titled `Accelerating Large Language Model Decoding with Speculative Sampling `__.\nOur implementation for speculative sampling is lossless. In addition to standalone draft models,\nwe also support `Eagle draft models `__.\nCurrently we only support Eagle v1.\n\nIn the following example, we demonstrate how to perform speculative sampling using the ``Llama`` model.\nIn this example, we are performing multinomial sampmling.\n\n.. code-block:: python\n\n import torch\n from transformers import LlamaTokenizer\n from transformers_neuronx import NeuronAutoModelForCausalLM, NeuronConfig, GenerationConfig\n from transformers_neuronx.fused_speculation import FusedSpeculativeDecoder\n\n # Specify path to draft and target\n draft = '/home/ubuntu/Llama-2-7b-chat-hf'\n target = '/home/ubuntu/Llama-2-70b-chat-hf'\n\n # Specify generation parameters\n gen_kwargs = {\n \"top_k\": 50,\n \"top_p\": 0.9,\n \"do_sample\": True,\n \"temperature\": 0.7,\n }\n\n # Load draft model\n draft_neuron_model = NeuronAutoModelForCausalLM.from_pretrained(\n draft, \n n_positions=1024,\n batch_size=1, \n tp_degree=32, \n amp='bf16', \n neuron_config=NeuronConfig(\n padding_side=\"right\",\n attention_layout=Layout.BSH,\n collectives_layout=\"BSH\",\n on_device_embedding=True,\n on_device_generation=GenerationConfig(**gen_kwargs),\n ),\n )\n draft_neuron_model.to_neuron()\n # Load target model\n target_neuron_model = NeuronAutoModelForCausalLM.from_pretrained(\n target, \n n_positions=1024,\n batch_size=1, \n tp_degree=32, \n amp='bf16',\n neuron_config=NeuronConfig(\n padding_side=\"right\",\n attention_layout=Layout.BSH,\n collectives_layout=\"BSH\",\n on_device_embedding=True,\n on_device_generation=GenerationConfig(**gen_kwargs),\n ),\n )\n target_neuron_model.to_neuron()\n \n # Compile the speculative sampling model\n # Here we set sepculation length to be 4\n fsd = FusedSpeculativeDecoder(\n draft_neuron_model, \n target_neuron_model, \n 4,\n )\n fsd.to_neuron()\n\n # Initialize tokenizer and text prompt\n tokenizer = LlamaTokenizer.from_pretrained(target)\n prompt = \"Hello, I'm a generative AI language model.\"\n inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n # Call speculative sampling on given input\n response = fsd.sample(\n input_ids=inputs.input_ids,\n attention_mask=inputs.attention_mask,\n sequence_length=30,\n )\n\n # Decode the response\n generated_text = tokenizer.decode(response[0])\n print(f\"\\nDecoded tokens: {generated_text}\")\n\nThe following sample shows how to enable EAGLE speculation.\nTo get the EAGLE draft model to work, manually copy the LM head \nweights from the target model to the draft model. Additionally, \nyou need to rename the keys in the draft model's ``state_dict`` \nto match those in the target model.\n\n.. code-block:: python\n\n import torch\n from transformers import LlamaTokenizer\n from transformers_neuronx import NeuronAutoModelForCausalLM, NeuronConfig, GenerationConfig\n from transformers_neuronx.fused_speculation import FusedSpeculativeDecoder\n\n # Specify path to draft and target\n # The Eagle draft model can be downloaded from Eagle website\n draft = '/home/ubuntu/EAGLE-llama2-chat-70B'\n target = '/home/ubuntu/Llama-2-70b-chat-hf'\n\n # Specify generation parameters\n gen_kwargs = {\n \"top_k\": 50,\n \"top_p\": 0.9,\n \"do_sample\": True,\n \"temperature\": 0.7,\n }\n\n # Load draft model\n draft_neuron_model = NeuronAutoModelForCausalLM.from_pretrained(\n draft, \n n_positions=1024,\n batch_size=1, \n tp_degree=32, \n amp='bf16', \n neuron_config=NeuronConfig(\n is_eagle_draft=True,\n has_pre_attention_norm=False,\n # Need the above two configs for Eagle\n padding_side=\"right\",\n attention_layout=Layout.BSH,\n collectives_layout=\"BSH\",\n on_device_embedding=True,\n on_device_generation=GenerationConfig(**gen_kwargs),\n ),\n )\n draft_neuron_model.to_neuron()\n # Load target model\n target_neuron_model = NeuronAutoModelForCausalLM.from_pretrained(\n target, \n n_positions=1024,\n batch_size=1, \n tp_degree=32, \n amp='bf16',\n neuron_config=NeuronConfig(\n is_eagle_target=True,\n # Need the above config for Eagle\n padding_side=\"right\",\n attention_layout=Layout.BSH,\n collectives_layout=\"BSH\",\n on_device_embedding=True,\n on_device_generation=GenerationConfig(**gen_kwargs),\n ),\n )\n target_neuron_model.to_neuron()\n \n # Compile the speculative sampling model\n # Here we set sepculation length to be 4\n fsd = FusedSpeculativeDecoder(\n draft_neuron_model, \n target_neuron_model, \n 4,\n )\n fsd.to_neuron()\n\n # The rest are the same\n\n\nQKV Weight Fusion\n--------------------------------------\n\nConcatenating a model's query, key and value weight matrices often achieves better performance because larger matrices allow\nfor more efficient data movement and compute. QKV weight fusion can be enabled by setting ``fuse_qkv=True`` in the ``NeuronConfig``:\n\n.. code-block:: python\n\n neuron_config = NeuronConfig(fuse_qkv=True)\n\n\nAttention Layout\n--------------------------------------\n\nThe intermediate tensor layouts in a model's attention layer can impact the\ncompiler's optimization opportunities and thus can impact a model's performance.\nUsing ``(batch, sequence, hidden)`` (or ``BSH``) layout for attention often\nachieves better performance since it can enable better overlapping of compute\nwith collectives and can reduce transposes. We intend to enable ``BSH``\nattention by default in a future release. For now, ``BSH`` attention layout can\nbe enabled by setting ``attention_layout=\"BSH\"`` in the ``NeuronConfig``:\n\n.. code-block:: python\n\n neuron_config = NeuronConfig(attention_layout=\"BSH\")\n\n\nBucketing\n------------------\nLLM inference is a generate process that can produce variable length sequences.\nThis poses a problem since the Neuron compiler produces executables which expect statically shaped inputs and outputs.\nTo make LLM work with different shapes, transformers_neuronx generates buckets\nand applies padding wherever it is required.\n\nThere are at least two set of buckets for each LLM inference that can be set by user:\n1) Context encoding (pre-fill) buckets and 2) output token generation buckets.\n\n\n**Token generation buckets**\n\nIn token generation, tokens are generated iteratively.\nAt each token position, transformer need to attend to the previous tokens only.\nBut in the naive implementation with static shapes, one may attend to all KV-cache (full sequence length).\nTo solve this problem, we use token generation buckets.\nToken generation buckets determine the attention lengths.\nFor instance, if the max sequence length is 1024 tokens and current token\nis at position 120, there is no need to attend to all 1024 tokens in the current step.\nWe can use token generation buckets to attend to different portions of KV-cache.\nBy default, token generation buckets which are powers of 2 starting from 128\ntokens are used (i.e. 128, 256, 512, up to sequence length). In the example above,\nbucket 128 would be used for position 120 which would reduce the wasted compute significantly.\nUser can change these buckets by setting a list for ``n_positions`` (see example below).\nOtherwise, if a number is given for ``n_positions`` (sequence length), instead of a list,\nthen the powers of 2 buckets starting from 128 will be used.\nThe last bucket would be ``n_positions`` (sequence length), even if it is not a power of 2.\n\n**Context encoding buckets**\n\nThe prompt tokens can be processed in parallel.\nAs a result, we need to set the bucket sizes for different estimated length of\ninput prompts. We can specify these context bucket sizes using the ``context_length_estimate`` argument.\nIn general, it is better to have all the bucket to be multiples of 256 tokens.\nBut adding too many buckets would increase device memory consumption and add extra latency\nfor bucket switching.\nUsually, the powers of 2 starting from 128 tokens are used for\ncontext encoding buckets. If the total sequence length (``n_positions``) is beyond 2048\ntokens, it is desirable to add extra buckets with multiple of 512 or 1024 tokens.\nIt is not recommended to add buckets of multiples of 256 tokens or smaller for context buckets beyond 2k to avoid bucket switching latency.\nAt runtime, the smallest bucket which fits the input context will be used.\nBy default, the context encoding buckets set to half of output-token buckets.\nAdding extra context buckets would reduce the wasted compute and improves performance.\nHowever, the extra executables would reduce memory space since executables require device memory space.\n\nNotice that the default output token generation buckets work well for wide range\nof applications. However, ideal context encoding buckets depends on the specific use case.\nFor instance, if all the requests have a context length of about 1500 +/- 500 tokens,\nadding more buckets closer to 1500 might help context encoding time.\nIn this example, adding buckets of 1024, 1280, 1536, 1792, 2048 tokens (distance of 256 tokens) could help.\nMoreover, the largest context encoding bucket should be larger than the largest context length.\nOtherwise, the performance would degrade significantly.\n\n\nTo set context encoding and token generation buckets manually:\n\n.. code-block:: python\n\n context_length_estimate = [1024, 1280, 1536, 1792, 2048] # The best context estimate depends on the use case\n n_positions = [128, 256, 512, 1024, 2048, 3072] # Usually default buckets are appropriate\n\n model = NeuronAutoModelForCausalLM.from_pretrained(\n 'gpt2',\n batch_size=1,\n n_positions=n_positions,\n tp_degree=2,\n amp='f16',\n context_length_estimate=context_length_estimate,\n )\n\n\n\n\nMulti-node inference support (TP/PP)\n---------------------------------------\n\nPrerequisite: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup-trn1-multi-node-execution.html\n\nWhen models are too large to fit on single node, Transformers NeuronX multi-node inference (tensor parallel and pipeline parallel) can be used to shard model weights across multiple Neuron instances (only supported on Trn1 and Trn1n). Single node inference code can easily be extended to multi-node inference.\n\nNote that Transformers Neuronx currently doesn't support multi-node Tensor Parallel and Pipeline Parallel at same time, when Pipeline Parallel is used, the Tensor Parallel has to be within a node (TP<=32 on Trn1/Trn1n).\n\nIn the below sections, we first outline the sample code for single node execution and then provide instructions to migrate the code to use multi-node tensor parallel or multi-node pipeline parallel. To start with, the code below is for single node script, running llama2-3b model with tensor parallel degree as 32.\n\n.. code-block:: python\n\n import torch\n from transformers import AutoTokenizer, AutoConfig\n from transformers_neuronx import LlamaForSampling, HuggingFaceGenerationModelAdapter\n\n # Create and compile the Neuron model\n model = LlamaForSampling.from_pretrained(\"openlm-research/open_llama_3b\", tp_degree=32)\n model.to_neuron()\n\n # Use the `HuggingFaceGenerationModelAdapter` to access the generate API\n config = AutoConfig.from_pretrained(\"openlm-research/open_llama_3b\")\n model = HuggingFaceGenerationModelAdapter(config, model)\n\n # Get a tokenizer and example input\n tokenizer = AutoTokenizer.from_pretrained(\"openlm-research/open_llama_3b\")\n tokenizer.pad_token_id = tokenizer.eos_token_id\n tokenizer.padding_side = 'left'\n text = \"Hello, I'm a language model,\"\n encoded_input = tokenizer(text, return_tensors='pt', padding=True)\n\n\n\n # Run inference using temperature\n with torch.inference_mode():\n model.reset_generation()\n generated_sequence = model.generate(\n input_ids=encoded_input.input_ids,\n attention_mask=encoded_input.attention_mask,\n do_sample=True,\n max_length=256,\n temperature=0.7,\n )\n\n print([tokenizer.decode(tok) for tok in generated_sequence])\n\ncommand line:\n\n.. code-block:: bash\n\n python3 multi_node_dev_example.py\n\n**Multi-Node Tensor Parallel**\n\nCompared to single node tensor parallel, multi-node tensor parallel shards the model weights in the same way but having mores cores across nodes. In the meantime, it requires each node’s ``model.forward()`` receives the exact same input, otherwise there would be unexpected behaviors (runtime failure, wrong output).\n\nConfigurations (environment variables to be configured on each node):\n\n- ``NEURON_RT_ROOT_COMM_ID``: the master node's ``:``\n- ``NEURON_RANK_ID``: rank of the node, 0 means master node\n- ``NEURON_LOCAL_TP``: the local tensor parallel degree on each node\n\nexample:\n\nChange the single node script to use ``tp=64`` (2 node). Set the ``torch.manual_seed`` to ensure the sampling loop running on each node will sample same token as next input.\n\n\nNode 1 command line:\n\n.. code-block:: bash\n\n NEURON_RT_ROOT_COMM_ID=10.1.201.64:63423 NEURON_RANK_ID=0 NEURON_LOCAL_TP=32 python3 multi_node_dev_example.py\n\nNode 2 command line (same as Node 1 but set ``NEURON_RANK_ID`` as 1):\n\n.. code-block:: bash\n\n NEURON_RT_ROOT_COMM_ID=10.1.201.64:63423 NEURON_RANK_ID=1 NEURON_LOCAL_TP=32 python3 multi_node_dev_example.py\n\nYou can also refer to `Tutorial `__ to run lama 3.1 405b multinode 16k tutorial with multi-node tensor parallel.\n\n**Multi-Node Pipeline Parallel**\n\nWhile having the weight tensor sharded as tensor pararallel, one can utilize pipeline parallel to partition the layers across different node, the intermediate tensor (hidden) will be transferred from one pipeline stage (nodes) to the next pipeline stage (nodes). The final output will be sent from last pipeline stage back to first pipeline stage.\n\nCompared to multi-node tensor parallel, for non-zero rank, the ``model.forward`` in pipeline parallel will fallback to while loop and block on the input broadcasting from master.\n\nConfigurations (environment variables to be configured on each node):\n\n- ``NEURON_RT_ROOT_COMM_ID``: the master node's ``:``\n- ``CPU_COMM_ID``: similar to NEURON_RT_ROOT_COMM_ID , but need to set with different port\n- ``NEURON_RANK_ID``: rank of the node, 0 means master node\n- ``NEURON_PP_STAGES``: number of pipeline stages (nodes)\n\nexample:\n\nKeep the original single node script with tp=32.\n\nNode 1 command line:\n\n.. code-block:: bash\n\n NEURON_PP_STAGES=2 CPU_COMM_ID=10.1.201.64:8989 NEURON_RT_ROOT_COMM_ID=10.1.201.64:63423 NEURON_RANK_ID=0 python3 multi_node_dev_example.py\n\nNode 2 command line (same as Node 1 but set ``NEURON_RANK_ID`` as 1):\n\n.. code-block:: bash\n\n NEURON_PP_STAGES=2 CPU_COMM_ID=10.1.201.64:8989 NEURON_RT_ROOT_COMM_ID=10.1.201.64:63423 NEURON_RANK_ID=1 python3 multi_node_dev_example.py\n\n\nLong Sequence length support up to 128k\n---------------------------------------\n**Flash Attention**\n\nWith the integration of FlashAttention kernel, developers can use longer sequence lengths for LLAMA models. The Flash Attention kernel is automatically used when the input sequence length is greater than 8k without any additional configuration. Refer to `Tutorial `__ for usage of 32k sequence length on a variation of LLAMA3-8B Model.\n\n**Flash Decoding**\n\nFlash Decoding (FD) is a technique that significantly speeds up attention during inference, especially for long-context\ntasks in large language models (LLMs) with GQA.\n\n.. image:: ./flash_decoding.gif\n :alt: Flash Decoding\n :width: 800px\n :align: center\n\nWith integration of FD, developers can achieve faster inference with larger sequence\nand batch size by reducing the KV cache replication.\nRefer to `Tutorial `__ on flash decoding usage for 128k sequence length sampling. Flash decoding\ncan be enabled by setting the flag `shard_over_sequence=True` in `NeuronConfig`\n\n.. code-block:: python\n\n neuron_config = NeuronConfig(shard_over_sequence=True)\n\n\nNote that you can skip the first Allgather introduced by flash decoding at the cost of duplicate Q weights, this is only recommended for relatively small models (i.e. 3B, 8B) and large batch size.\n\n.. code-block:: python\n\n neuron_config = NeuronConfig(shard_over_sequence=True, duplicate_q_weight_sos=True)\n\n**Known limitations and FAQs**\n\n- Flash decoding is expected to have performance degradation (PTL) for smaller sequence and batch sizes. We recommend flash decoding when **batch-size x sequence length > 16k**\n- Flash decoding support is not enabled for the following features\n\n - Speculative Decoding\n - Multi Head Attention (MHA) models\n\n\n" }, { "path": "archive/transformers-neuronx/transformers-neuronx-misc.rst", "content": ".. _transformers-neuronx-misc:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\nMisc (``transformers-neuronx``)\n===============================\n\n\n" }, { "path": "archive/transformers-neuronx/transformers-neuronx-misc.txt", "content": "* :ref:`transformers-neuronx-rn`" }, { "path": "archive/transformers-neuronx/transformers-neuronx-tutorials.rst", "content": ".. _transformers_neuronx_tutorials:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This topic is currently archived and not maintained. It is provided for reference only.\n\nTransformers NeuronX Tutorials \n===============================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1 \n Hugging Face facebook/opt-13b autoregressive sampling on Inf2 & Trn1 \n Hugging Face facebook/opt-30b autoregressive sampling on Inf2 & Trn1 \n Hugging Face facebook/opt-66b autoregressive sampling on Inf2 \n\n\n.. include:: /libraries/transformers-neuronx/transformers-neuronx-tutorials.txt\n" }, { "path": "archive/transformers-neuronx/transformers-neuronx-tutorials.txt", "content": "* `Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1 `_\n* `Hugging Face facebook/opt-13b autoregressive sampling on Inf2 & Trn1 `_\n* `Hugging Face facebook/opt-30b autoregressive sampling on Inf2 & Trn1 `_\n* `Hugging Face facebook/opt-66b autoregressive sampling on Inf2 `_\n\n\n" }, { "path": "archive/transformers-neuronx/transformers-neuronx.txt", "content": ".. dropdown:: Setup (``transformers-neuronx``)\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /libraries/transformers-neuronx/setup/index.rst\n\n\n.. dropdown:: Developer Guide (``transformers-neuronx``)\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /libraries/transformers-neuronx/developer-guide.txt\n\n\n.. dropdown:: Tutorials (``transformers-neuronx``)\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /libraries/transformers-neuronx/transformers-neuronx-tutorials.txt\n\n\n.. dropdown:: Misc (``transformers-neuronx``)\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /libraries/transformers-neuronx/transformers-neuronx-misc.txt\n" }, { "path": "archive/tutorials/finetune_t5.rst", "content": ".. _torch-hf-t5-finetune:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nFine-tune T5 model on Trn1\n================================\n\n.. note:: \n This page was archived on 7/31/2025.\n\n\nIn this tutorial, we show how to fine-tune a Hugging Face (HF) T5 model \nusing HF trainer API. This example fine-tunes a `T5 model for\na text-summarization `__ task on CNN/DailyMail dataset.\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n.. include:: /frameworks/torch/torch-neuronx/tutorials/note-performance.txt\n\nSetup and compilation\n---------------------\n\nBefore running the tutorial please follow the installation instructions at:\n\n:ref:`Install PyTorch Neuron on Trn1 `\n\nPlease set the storage of instance to *512GB* or more if you also want to run through the BERT pretraining and GPT pretraining tutorials.\n\nFor all the commands below, make sure you are in the virtual environment that you have created above before you run the commands:\n\n.. code:: shell\n\n source ~/aws_neuron_venv_pytorch/bin/activate\n\nFirst we install a recent version of HF transformers, scikit-learn and evaluate packages in our environment as well as download the source matching the installed version. In this example, we chose version 4.26.0 and the text summarization example from HF transformers source:\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_setup_code.sh\n :language: shell\n :lines: 5-9\n\nSingle-worker training\n----------------------\n\nWe will run text-summarization fine-tuning task following the example in\nREADME.md located in the path\n`~/transformers/examples/pytorch/summarization.`\n\nWe use full BF16 casting using `XLA_USE_BF16=1` to enable best\nperformance. First, paste the following script into your terminal to\ncreate a “run.sh” file and change it to executable:\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_single_worker_training_code.sh\n :language: shell\n :lines: 7-46\n\nWe optionally precompile the model and training script using\n`neuron\\_parallel\\_compile `__ to warm up the persistent graph cache (Neuron\nCache) such that the actual run has fewer compilations (faster run\ntime):\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_single_worker_training_code.sh\n :language: shell\n :lines: 49\n\nNote: For these auto-regressive models, do not run the\n``predict_with_generate`` method when doing the precompile step. This is\nbecause the ``neuron_parallel_compile`` utility will run the training\nscript in graph extraction mode and no actual execution of the graph\nwill be done. Hence, the outputs at each step are invalid. Since the\nauto-regressive generation at each step is dependent on output of\nprevious step, the generate step would fail since the outputs from\nprevious steps are invalid.\n\nPrecompilation is optional and only needs to be done once unless\nhyperparameters such as batch size are modified. After the optional\nprecompilation, the actual run will be faster with minimal additional\ncompilations.\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_single_worker_training_code.sh\n :language: shell\n :lines: 51\n\nIf precompilation was not done, the first execution of ./run.sh will be\nslower due to serial compilations. Rerunning the same script a second\ntime would show quicker execution as the compiled graphs will be already\ncached in persistent cache.\n\nRunning the above script will run the T5-small fine-tuning on a single\nprocess.\n\n**Note:** As you may have noticed, we are not running the\n``predict_with_generate`` as part of training. This is because,\n``predict_with_generate`` requires auto-regressive sampling where the\ninputs to the decoder are created by appending outputs of previous\nsteps. This causes the inputs to the decoder to change shape and thereby\nresulting in a new graph. In other words, the current ``generate`` api\nprovided by HF transformers leads to repeated compilations. We are working on\nbuilding a Neuron friendly version of ``generate`` api and it will be\nmade available as part of future release. This will enable us to run\n``predict_with_generate`` as part of training script.\n\nAs a workaround, we can run the ``predict_with_generate`` on CPU after\nthe model is trained. Once training is completed, a trained checkpoint\nwould be saved. We can load the trained model and run the\n``predict_with_generate`` to compute the final accuracy.\n\nTo do so, in run_summarization.py, add the following before ``transformers`` get imported.\nThis can be done by adding the below lines before all the ``imports``:\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_single_worker_training_code.sh\n :language: python\n :lines: 55-59\n\nYou can now run the following and it should run the predict method on CPU device.\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_single_worker_training_code.sh\n :language: shell\n :lines: 67-78\n\nNote: To run on CPU, we need to make sure that NEURON\\_NUM\\_DEVICES is\nset to 0. This will make sure no xla\\_devices are created and the\ntrainer would use the default device (CPU).\n\n.. _multi_worker_training:\n\nMulti-worker Training\n---------------------\n\nThe above script will run one worker on one NeuronCore. To run on\nmultiple cores, first add these lines to top of run\\_summarization.py to disable\nDistributed Data Parallel (DDP) when using torchrun (see Known issues\nand limitations section below):\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_modify_run_summarization_code.sh\n :language: python\n :lines: 8-10\n\nThen launch the run\\_summarization.py script with torchrun using\n--nproc\\_per\\_node=N option to specify the number of workers (N=2 for\ntrn1.2xlarge, and N=2, 8, or 32 for trn1.32xlarge). The following\nexample runs 2 workers. Paste the following script into your terminal to\ncreate a “run\\_2w.sh” file and change it to executable:\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_multi_worker_training_code.sh\n :language: shell\n :lines: 7-46\n\nAgain, we optionally precompile the model and training script using\nneuron\\_parallel\\_compile to warm up the persistent graph cache (Neuron\nCache), ignoring the results from this precompile run as it is only for\nextracting and compiling the XLA graphs:\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_multi_worker_training_code.sh\n :language: python\n :lines: 49\n\nPrecompilation is optional and only needs to be done once unless\nhyperparameters such as batch size are modified. After the optional\nprecompilation, the actual run will be faster with minimal additional\ncompilations.\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_multi_worker_training_code.sh\n :language: python\n :lines: 51\n\nDuring run, you will notice that the “Total train batch size” is now\n8 and the “Total optimization steps” is now half the number for one\nworker training. Also, if you open ``neuron-top`` in a separate terminal, \nyou should see 2 cores been utilized.\n\nTo train T5-large model, you can set the ``model_name_or_path`` argument to ``t5-large``.\nPlease note, currently running ``t5-large`` on trn1-2xl machine can result in ``HOST OOM`` during \ncompilation. Hence, it is recommended that you run a ``t5-large`` model training on a trn1-32xl machine.\n\nOn a trn1-32xl machine, you can create a run_32w.sh on the terminal using the following commands:\n\n.. literalinclude:: tutorial_source_code/t5_finetuning/t5_finetuning_32_worker_training_code.sh\n :language: shell\n :lines: 7-46\n\nYou can now follow the same steps as listed above. This script would run a t5-large model by launching a training script \nusing 32 data-parallel workers.\n\n\n.. _t5_known_issues:\n\nKnown issues and limitations\n----------------------------\n\nThe following are currently known issues:\n\n- Long compilation times: this can be alleviated with\n ``neuron_parallel_compile`` tool to extract graphs from a short trial run and\n compile them in parallel ahead of the actual run, as shown above.\n- T5-Large compilation causing processes to get killed on trn1-2xl: It is recommended \n to ``t5-large`` model training on a trn1-32xl machine, as it avoids CPU OOM and also provides \n faster training by making use of 32 data-parallel workers.\n" }, { "path": "archive/tutorials/finetuning_llama2_7b_ptl.rst", "content": ".. _llama2_7b_tp_zero1_ptl_finetune_tutorial:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nFine-tuning Llama2 7B with tensor parallelism and ZeRO-1 optimizer using Neuron PyTorch-Lightning\n=================================================================================================\n\nThis tutorial shows how to fine-tune Llama2 7B with tensor parallelism and ZeRO-1 using Neuron PyTorch-Lightning APIs. For pre-training information and additional context, see the Llama2 7B Tutorial\nand :ref:`Neuron PT-Lightning Developer Guide `. \n\n\nSetting up the environment\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor this experiment, we will use AWS ParallelCluster with at least four trn1.32xlarge compute nodes.\nTo set up a cluster and prepare it for use, see `Train your model on ParallelCluster `__.\nTo set up the packages on the head node of the cluster, see\n:ref:`Install PyTorch Neuron on Trn1 `.\n\nInstall the ``neuronx-distributed`` package inside the virtual environment using the following command:\n\n.. code:: ipython3\n\n python -m pip install neuronx_distributed --extra-index-url https://pip.repos.neuron.amazonaws.com\n\nNext, download the scripts for fine-tuning.\n\n\n1. Create a directory to hold the experiments.\n\n.. code:: ipython3\n\n mkdir -p ~/examples/tp_zero1_llama2_7b_hf_finetune_ptl\n cd ~/examples/tp_zero1_llama2_7b_hf_finetune_ptl\n\n2. Download training scripts for the experiments.\n\n.. code:: ipython3\n\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/lightning/data_module.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/lightning/module_llama.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/lightning/tp_zero1_llama2_7b_hf_finetune_ptl.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/lightning/tp_zero1_llama2_7b_hf_finetune_ptl.sh\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/lightning/finetune_config/config.json\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/lr.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/modeling_llama_nxd.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/requirements.txt\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/requirements_ptl.txt\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/training_utils.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/main/examples/training/llama/convert_checkpoints.py\n\n3. Install the additional requirements and give the right permissions to the shell script.\n\n.. code:: ipython3\n\n python3 -m pip install -r requirements.txt\n python3 -m pip install -r requirements_ptl.txt # Currently we're supporting Lightning version 2.4.0\n python3 -m pip install optimum-neuron==0.0.18 nltk # Additional dependencies for evaluation\n python3 -m pip install --no-warn-conflicts transformers==4.32.1 # Ping transformers version 4.32.1\n chmod +x tp_zero1_llama2_7b_hf_finetune_ptl.sh\n\nDownload the Llama2-7B pre-trained checkpoint from HuggingFace.\n\n\n1. Create a Python script ``get_model.py`` with the following lines: \n\n.. code:: ipython3\n\n import torch\n from transformers.models.llama.modeling_llama import LlamaForCausalLM\n model = LlamaForCausalLM.from_pretrained(\"NousResearch/Llama-2-7b-hf\")\n torch.save(model.state_dict(), \"llama-7b-hf-pretrained.pt\")\n\n2. Run the download script and conversion script to pull and convert the checkpoint, note that conversion scripts requires high memory so need to login to a compute node to do so:\n\n.. code:: ipython3\n\n ssh compute1-dy-training-0-1\n source ~/aws_neuron_venv_pytorch/bin/activate\n cd ~/examples/tp_zero1_llama2_7b_hf_finetune_ptl\n python3 get_model.py\n python3 convert_checkpoints.py --tp_size 8 --convert_from_full_model --config config.json --input_dir llama-7b-hf-pretrained.pt --output_dir llama7B-pretrained/pretrained_weight\n\n3. (Optional) If you are loading checkpoint from different directory, set the checkpoint path by adding the following flag to ``tp_zero1_llama2_7b_hf_finetune_ptl.sh``:\n\n * ``--pretrained_ckpt``.\n\n This provides direction to the pre-trained checkpoint to be loaded.\n\nThen, set the dataset for the fine-tuning job. In this example, we will use Dolly, which is an open source dataset\nof instruction-following records on categories outlined in the InstructGPT paper, including brainstorming, classification,\nclosed QA, generation, information extraction, open QA, and summarization.\n\n.. code-block:: json\n\n {\n \"instruction\": \"Alice's parents have three daughters: Amy, Jessy, and what's the name of the third daughter?\",\n \n \"context\": \"\",\n \n \"response\": \"The name of the third daughter is Alice\"\n }\n\nConfigure the following flags in ``tp_zero1_llama2_7b_hf_finetune_ptl.sh``:\n\n.. code:: ipython3\n\n --data_dir \"databricks/databricks-dolly-15k\" \\\n --task \"open_qa\"\n\nAt this point, you are all set to start fine-tuning.\n\nRunning fine-tuning\n^^^^^^^^^^^^^^^^^^^\n\nBy this step, the cluster is all set up for running experiments. \nBefore running training, first pre-compile the graphs using the :ref:`neuron_parallel_compile `.\nRun the command below:\n\n.. code:: ipython3\n\n sbatch --exclusive \\\n --nodes 1 \\\n --wrap=\"srun neuron_parallel_compile bash $(pwd)/tp_zero1_llama2_7b_hf_finetune_ptl.sh\"\n\nThis script uses a tensor-parallel size of 8.\nThis automatically sets the zero-1 sharding degree to 4 (32 workers / tensor_parallel_size). \n\n`Note`: You can use any number of nodes in this case by adjusting the number of nodes in the above \nSlurm command accordingly. Also, the number of nodes used in the parallel_compile command should be same as the number used in the actual \ntraining run. This is because, as the number of nodes change, the data-parallel degree changes too. This \nresults in more workers participating in operations like `gradient all-reduce`, which results in new graphs getting \ncreated. \n\nAfter the graphs are compiled, you can run training and observe how the loss goes down.\nBefore the actual fine-tune started, we need to prepare the dataset\n\n.. code:: ipython3\n\n python3 -c \"import nltk; nltk.download('punkt')\" \n\nTo run the training, run the above command without ``neuron_parallel_compile``:\n\n.. code:: ipython3\n\n sbatch --exclusive \\\n --nodes 1 \\\n --wrap=\"srun bash $(pwd)/tp_zero1_llama2_7b_hf_finetune_ptl.sh\"\n\nAt the end of fine-tuning, run evaluation once with a test data split by generating sentences and calculating ROUGE scores.\nThe final evaluation results and ROUGE score are then printed in your terminal.\n\n\nCheckpointing\n^^^^^^^^^^^^^^\n\nTo enable checkpoint saving, add the following flags to ``tp_zero1_llama2_7b_hf_finetune_ptl.sh``:\n\n* ``--save_checkpoint`` Enables checkpoint saving.\n* ``--checkpoint_freq`` Number of steps to save a checkpoint.\n* ``--checkpoint_dir`` Direction to save the checkpoint.\n* ``--num_kept_checkpoint`` Number of checkpoints to save. Older checkpoint are deleted manually. Set to -1 to keep all saved checkpoints.\n* ``--save_load_xser`` Loads with torch_xla serialization to reduce time saving. We recommend enabling xser for significantly faster save and load times. Note that if the checkpoint is saved with xser, it can only be loaded with xser, and vice versa. \n\nTo enable checkpoint loading, add the following flags to ``tp_zero1_llama2_7b_hf_finetune_ptl.sh``:\n\n* ``--resume_ckpt`` Resumes the checkpoint process.\n* ``--load_step`` The step to retrieve the checkpoint from.\n* ``--checkpoint_dir`` Direction to load the checkpoint from.\n* ``--save_load_xser`` Loads with torch_xla serialization to reduce time saving. We recommend enabling xser for significantly faster save and load times. Note that if the checkpoint is saved with xser, it can only be loaded with xser, and vice versa. \n" }, { "path": "archive/tutorials/gpt3_neuronx_nemo_megatron_pretraining.rst", "content": ".. _gpt3_neuronx_nemo_megatron_pretraining:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nLaunch a GPT-3 pretraining job using neuronx-nemo-megatron\n==========================================================\n\nArchived tutorials for gpt3 pretraining using neuronx-nemo-megatron\n * `Launch a GPT-3 23B pretraining job using neuronx-nemo-megatron `_\n * `Launch a GPT-3 46B pretraining job using neuronx-nemo-megatron `_\n * `Launch a GPT-3 175B pretraining job using neuronx-nemo-megatron `_" }, { "path": "archive/tutorials/megatron_gpt_pretraining.rst", "content": ".. _megatron_gpt_pretraining:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nMegatron GPT Pretraining\n========================\n\n.. note:: \n This page was archived on 7/31/2025.\n\nIn this example, we will compile and train a Megatron GPT model on a single instance or\non multiple instances using ParallelCluster with the NxD Training library.\nThe example has the following main sections:\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nSetting up the environment\n--------------------------\n\nParallelCluster Setup\n^^^^^^^^^^^^^^^^^^^^^\n\nIn this example, we will use 8 instances with ParallelCluster,\nplease follow the instructions here to create a cluster:\n`Train your model on ParallelCluster\n`_\n\nParallelCluster automates the creation of trn1 clusters,\nand provides the SLURM job management system for scheduling and managing distributed training jobs.\nPlease note that the home directory on your ParallelCluster\nhead node will be shared with all of the worker nodes via NFS.\n\nInstall Dependencies\n^^^^^^^^^^^^^^^^^^^^\n\nOnce you have launched a trn1 instance or ParallelCluster,\nplease follow this guide on how to install the latest Neuron packages:\n`PyTorch Neuron Setup Guide\n`_.\n\nNext, we will need to install NxD Training and its dependencies.\nPlease see the following installation guide for installing NxD Training:\n:ref:`NxDT Installation Guide `\n\n\nDownload the dataset\n--------------------\n\nThis tutorial makes use of a preprocessed Wikipedia dataset that is stored in S3.\nThe dataset can be downloaded to your cluster or instance by running\nthe following commands on the head node or your trn1 instance:\n\n.. code-block:: bash\n\n export DATA_DIR=~/examples_datasets/gpt2\n mkdir -p ${DATA_DIR} && cd ${DATA_DIR}\n wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json\n wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt\n aws s3 cp s3://neuron-s3/training_datasets/gpt/wikipedia/my-gpt2_text_document.bin . --no-sign-request\n aws s3 cp s3://neuron-s3/training_datasets/gpt/wikipedia/my-gpt2_text_document.idx . --no-sign-request\n aws s3 cp s3://neuron-s3/training_datasets/gpt/wikipedia/license.txt . --no-sign-request\n\n\n\nPre-compile the model\n---------------------\n\nBy default, PyTorch Neuron uses a just in time (JIT) compilation flow that sequentially\ncompiles all of the neural network compute graphs as they are encountered during a training job.\nThe compiled graphs are cached in a local compiler cache so that subsequent training jobs\ncan leverage the compiled graphs and avoid compilation\n(so long as the graph signatures and Neuron version have not changed).\n\nAn alternative to the JIT flow is to use the included ``neuron_parallel_compile``\ncommand to perform ahead of time (AOT) compilation. In the AOT compilation flow,\nthe compute graphs are first identified and extracted during a short simulated training run,\nand the extracted graphs are then compiled and cached using parallel compilation,\nwhich is considerably faster than the JIT flow.\n\nFirst, clone the open-source ``neuronx-distributed-training`` library\n\n.. code:: ipython3\n\n git clone https://github.com/aws-neuron/neuronx-distributed-training\n cd neuronx-distributed-training/examples\n\nNow, ensure that you are using the proper config file in the ``conf/`` directory.\nIn the ``train.sh`` file, ensure that the ``CONF_FILE`` variable is properly\nset to the config for the model you want to use. In our case,\nit will be ``megatron_gpt_config``. The default config here is a 6.7B parameter model,\nbut users can also add their own ``conf/*.yaml`` files and run different configs and\nhyperparameters if desired. Please see :ref:`Config Overview `\nfor examples and usage for the ``.yaml`` config files.\n\nNext, run the following commands to launch an AOT pre-compilation job on your instance:\n\n.. code-block:: bash\n\n export COMPILE=1\n ./train.sh\n\nThe compile output and logs will be shown directly in the terminal\nand you will see a message similar to this:\n\n.. code-block:: bash\n\n 2024-08-11 23:04:08.000738: INFO ||PARALLEL_COMPILE||: Total graphs: 22\n 2024-08-11 23:04:08.000738: INFO ||PARALLEL_COMPILE||: Total successful compilations: 22\n 2024-08-11 23:04:08.000738: INFO ||PARALLEL_COMPILE||: Total failed compilations: 0\n\nThen, you know your compilation has successfully completed.\n\n.. note::\n The number of graphs will differ based on package versions, models, and other factors.\n This is just an example.\n\nIf you are using ParallelCluster, then you will need to update the ``conf/megatron_gpt_config.yaml``\nwith\n\n.. code-block:: yaml\n\n num_nodes: 8\n\nThen to run the compile job:\n\n.. code-block:: bash\n\n export COMPILE=1\n sbatch --exclusive \\\n --nodes 8 \\\n --cpus-per-task 128 \\\n --wrap=\"srun ./train.sh\"\n\nOnce you have launched the precompilation job, run the squeue command to view the\nSLURM job queue on your cluster. If you have not recently run a job on your cluster,\nit may take 4-5 minutes for the requested trn1.32xlarge nodes to be launched and initialized.\nOnce the job is running, squeue should show output similar to the following:\n\n.. code-block:: bash\n\n JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)\n 10 compute1 wrap ubuntu R 5:11 8 compute1-dy-queue1-i1-[0-7]\n\nYou can view the output of the precompilation job by examining the file named\n``slurm-ZZ.out``,\nwhere ZZ represents the JOBID of your job in the squeue output above.\n\n.. code-block:: bash\n\n tail -f slurm-10.out\n\nOnce the precompilation job is complete, just like the above output\nyou should see a message similar to the following in the logs:\n\n.. code-block:: bash\n\n 2024-08-11 23:04:08.000738: INFO ||PARALLEL_COMPILE||: Total graphs: 22\n 2024-08-11 23:04:08.000738: INFO ||PARALLEL_COMPILE||: Total successful compilations: 22\n 2024-08-11 23:04:08.000738: INFO ||PARALLEL_COMPILE||: Total failed compilations: 0\n\nAt this point, you can press ``CTRL-C`` to exit the tail command.\n\nTraining the model\n------------------\n\nThe pre-training job is launched almost exactly the same as the compile job.\nWe now turn off the ``COMPILE`` environment variable and\nrun the same training script to start pre-training.\n\nOn a single instance:\n\n.. code-block:: bash\n\n export COMPILE=0\n ./train.sh\n\nIf you are using ParallelCluster:\n\n.. code-block:: bash\n\n export COMPILE=0\n sbatch --exclusive \\\n --nodes 8 \\\n --cpus-per-task 128 \\\n --wrap=\"srun ./train.sh\"\n\nAs outlined above, you can again use the ``squeue`` command to view the job queue,\nand also monitor the job in the same way with the ``tail`` command to see the training logs.\nOnce the model is loaded onto the Trainium accelerators and training has commenced,\nyou will begin to see output indicating the job progress:\n\nExample:\n\n.. code-block:: bash\n\n Epoch 0: 0%| | 189/301501 [59:12<1573:03:24, 18.79s/it, loss=7.75, v_num=3-16, reduced_train_loss=7.560, global_step=188.0, consumed_samples=24064.0]\n Epoch 0: 0%| | 190/301501 [59:30<1572:41:13, 18.79s/it, loss=7.74, v_num=3-16, reduced_train_loss=7.560, global_step=189.0, consumed_samples=24192.0]\n Epoch 0: 0%| | 191/301501 [59:48<1572:21:28, 18.79s/it, loss=7.73, v_num=3-16, reduced_train_loss=7.910, global_step=190.0, consumed_samples=24320.0]\n\nMonitoring Training\n-------------------\n\nTensorboard monitoring\n^^^^^^^^^^^^^^^^^^^^^^\n\nIn addition to the text-based job monitoring described in the previous section,\nyou can also use standard tools such as TensorBoard to monitor training job progress.\nTo view an ongoing training job in TensorBoard, you first need to identify the\nexperiment directory associated with your ongoing job.\nThis will typically be the most recently created directory under\n``~/neuronx-distributed-training/examples/nemo_experiments/megatron_gpt/``.\nOnce you have identifed the directory, cd into it, and then launch TensorBoard:\n\n.. code-block:: bash\n\n cd ~/neuronx-distributed-training/examples/nemo_experiments/megatron_gpt/\n tensorboard --logdir ./\n\nWith TensorBoard running, you can then view the TensorBoard dashboard by browsing to\n``http://localhost:6006`` on your local machine. If you cannot access TensorBoard at this address,\nplease make sure that you have port-forwarded TCP port 6006 when SSH'ing into the head node,\n\n.. code-block:: bash\n\n ssh -i YOUR_KEY.pem ubuntu@HEAD_NODE_IP_ADDRESS -L 6006:127.0.0.1:6006\n\nneuron-top / neuron-monitor / neuron-ls\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe `neuron-top `_\ntool can be used to view useful information about NeuronCore utilization, vCPU and RAM utilization,\nand loaded graphs on a per-node basis. To use neuron-top during on ongoing training job,\nfirst SSH into one of your compute nodes from the head node (if using ParallelCluster), and then run ``neuron-top``:\n\n.. code-block:: bash\n\n ssh compute1-dy-queue1-i1-1 # to determine which compute nodes are in use, run the squeue command\n neuron-top\n\nSimilarly, once you are logged into one of the active compute nodes,\nyou can also use other Neuron tools such as\n`neuron-monitor `_\nand `neuron-ls `_\nto capture performance and utilization statistics and to understand NeuronCore allocation.\n\nTroubleshooting Guide\n---------------------\n\nFor issues with NxD Training, please see:\n:ref:`NxD Training Known Issues `\n\nFor ParallelCluster issues see:\n`AWS ParallelCluster Troubleshooting `_\n" }, { "path": "archive/tutorials/multinode-training-model-profiling.rst", "content": ".. meta::\n :description: Learn how to use Neuron Explorer to analyze performance during multi-node training on AWS Trainium instances with SLURM job scheduling\n :date-modified: 12/02/2025\n\nProfiling Multi-Node Training Jobs with Neuron Explorer\n========================================================\n\nThis tutorial demonstrates how to use Neuron Explorer to analyze performance during multi-node training on AWS Trainium instances. We will run a scaled-down version of the :doc:`NxD Training Llama3 8B tutorial ` across 2 nodes, capture performance traces, and visualize them using Perfetto. we will run training across 2 nodes with reduced steps and layers so that compilation and profiling complete quickly.\n\nPrerequisites\n-------------\n\n* Access to a multi-node Trainium cluster (4 nodes in this example)\n* Neuron SDK installed and configured along with :doc:`NxD Training library installation `\n* Review of the :doc:`NxD Training Llama3 8B tutorial `\n* Familiarity with SLURM job scheduling\n\nSetup and Configuration\n-----------------------\n\nStep 1: Initial Setup\n~~~~~~~~~~~~~~~~~~~~~~\n\nA. Download the dataset script:\n\n.. code-block:: bash\n\n # Download get_dataset.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/llama/get_dataset.py\n\nB. Create a directory for dataset and get the corresponding config file -\n\n.. code-block:: bash\n\n mkdir ~/examples_datasets/ && cd ~/examples_datasets/\n\n # Download config.json \n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/llama/tp_zero1_llama_hf_pretrain/8B_config_llama3/config.json ~/\n\nC. Get the tokenizer using the following code snippet -\n\n.. code-block:: python\n\n # tokenizer.py\n from huggingface_hub import login\n from transformers import AutoTokenizer\n\n login(token='YourHuggingFaceToken')\n\n tokenizer = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B')\n\n tokenizer.save_pretrained(\".\")\n\n.. code-block:: bash\n\n python3 tokenizer.py\n\nD. Run the get_dataset.py -\n\n.. code-block:: bash\n\n python3 ~/get_dataset.py --llama-version 3\n\nE. Clone neuronx-distributed-training git repo\n\n.. code-block:: bash\n\n cd ~\n git clone https://github.com/aws-neuron/neuronx-distributed-training.git\n cd ~/neuronx-distributed-training/examples\n\nStep 2: Modify the Configuration Files\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUpdate the training configuration to minimize runtime while still generating useful profiling data:\n\n1. In ``hf_llama3_8B_config.yaml``, make the following changes:\n\n.. code-block:: yaml\n\n max_steps: 5 # Run only 5 steps for faster turnaround\n num_layers: 2 # Reduce model depth to 2 layers\n num_nodes: 2 # Run only 2 nodes\n global_batch_size: 32 # Set a relatively smaller GBS to avoid large trace volume\n\nThese changes ensure the job compiles and runs quickly while still exercising the profiler.\n\n2. In ``train.sh``, set the configuration file name:\n\n.. code-block:: bash\n\n CONF_FILE=hf_llama3_8B_config\n\nThis ensures the job runs with your modified config.\n\nStep 3: Compile the Model\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBefore training, the model must be compiled into Neuron Executable Files (NEFFs). To do this:\n\n.. code-block:: bash\n\n export COMPILE=1 \n export CONF_FILE=hf_llama3_8B_config\n\n sbatch --exclusive \\\n --nodes=2 \\\n --cpus-per-task=128 \\\n --wrap=\"srun ./train.sh\"\n\n* ``COMPILE=1`` tells the script to run in compile-only mode.\n* ``--nodes=2`` requests 2 Trainium nodes for compilation.\n* ``srun ./train.sh`` launches the job via Slurm across the allocated nodes.\n\n.. note::\n The first compilation may take some time depending on the model size. Once compiled, NEFFs are cached for reuse in later training runs.\n\nStep 4: Run the Training Job with Profiling Enabled\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNow that compilation is done, we can run the training job while enabling Neuron Explorer:\n\n.. code-block:: bash\n\n export COMPILE=0\n export CONF_FILE=hf_llama3_8B_config\n\n NEURON_RT_INSPECT_DEVICE_PROFILE=1 NEURON_RT_INSPECT_ENABLE=1 \\\n NEURON_RT_INSPECT_OUTPUT_DIR=./output \\\n sbatch --exclusive \\\n --nodes=2 \\\n --cpus-per-task=128 \\\n --wrap=\"srun ./train.sh\"\n\nHere's what's happening:\n\n* ``COMPILE=0``: Use precompiled NEFFs instead of recompiling.\n* ``NEURON_RT_INSPECT_ENABLE=1``: Turns on runtime inspection for profiling.\n* ``NEURON_RT_INSPECT_OUTPUT_DIR=./output``: All profiler logs will be saved into the ``./output`` directory.\n* Slurm runs the job across 2 nodes with 128 CPUs per task.\n\nAt the end of this step, you should see an output directory containing runtime inspection logs from each node.\n\nStep 5: Generate a Perfetto Profile\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNeuron Explorer produces raw trace data. To visualize it, convert the logs into a Perfetto compatible trace file:\n\n1. Run the Neuron Explorer CLI:\n\n.. code-block:: bash\n\n neuron-profile view -d ./output --output-format perfetto\n\nThis command consolidates the logs and generates a Perfetto compatible trace file.\n\nStep 6: Visualize in Perfetto\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1. Download the generated trace file to your local machine.\n2. Open the Perfetto UI.\n3. Drag and drop the trace file into the browser window.\n\nYou'll now see a timeline view of your training job, including kernel execution, operator scheduling, and activity across NeuronCores. This visualization helps you identify compute vs. memory bottlenecks, idle time, and overall efficiency of the training job.\n\nStep 7: Understanding the System Level Profile\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOnce the profile is loaded in Perfetto, you'll see both nodes (2 in our case) along with their workers, listed on the left-hand side as process IDs (PIDs). Each worker captures the same trace, so expanding any one of them will give you the information you need. The key runtime event to focus on is the Neuron Runtime API call named ``nc_exec_running``. This API is responsible for executing a Neuron Executable File (NEFF) on the NeuronCores.\n\nIf you hover over or click on one of these calls, Perfetto will display details about which NEFF is being executed. While you may see other runtime API calls, our primary interest is in ``nc_exec_running`` since it directly represents the model execution on Neuron hardware.\n\n.. image:: /tools/profiler/images/multinode-training-1.png\n\nIn the example trace shown, the calls to ``nc_exec_running`` appear back-to-back with no significant delays in between. This indicates that, at a system level, the runtime is efficiently dispatching work to NeuronCores. The ``model_name`` field in the arguments section will display the name of the NEFF which is being used in the corresponding ``nc_exec_running``.\n\nStep 8: Linking to device level profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSince we are able to see the NEFF name from ``nc_exec_running`` api call, we will now see how to visualize the profile for that NEFF. This effectively means how the model performance on a given Neuron core looks like. For this, on your trainium cluster, navigate to your compile cache directory (If you are following this tutorial it could be set as ``compiler_cache_url`` in config.yaml file). Navigate to the directory and search for the respective module directory based on the name, and you will see artifacts in that directory as shown below -\n\n.. code-block:: text\n\n ├── compile_flags.json\n ├── model.done\n ├── model.hlo_module.pb\n └── model.neff" }, { "path": "archive/tutorials/nxd-source-code/gpt_neox_tp_zero1/gpt_neox_20b.sh", "content": "#!/bin/bash\nset -eExuo\n\ncd ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain/\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/common/adamw_fp32_optim_params.py ./\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/common/get_dataset.py ./\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/common/requirements.txt ./\npython3 -m pip install -r requirements.txt\n\npython3 get_dataset.py\n\nPATH=$PATH:/opt/slurm/bin/\n\nsbatch --exclusive \\\n--nodes 4 \\\n--cpus-per-task 128 \\\n--wrap=\"srun neuron_parallel_compile bash $(pwd)/tp_dp_gpt_neox_20b_hf_pretrain.sh\"\n\nsbatch --exclusive \\\n--nodes 4 \\\n--cpus-per-task 128 \\\n--wrap=\"srun bash $(pwd)/tp_dp_gpt_neox_20b_hf_pretrain.sh\"\n" }, { "path": "archive/tutorials/nxd-source-code/gpt_neox_tp_zero1/gpt_neox_6_9b.sh", "content": "#!/bin/bash\nset -eExuo\n\ncd ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_6.9b_hf_pretrain/\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/common/adamw_fp32_optim_params.py ./\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/common/get_dataset.py ./\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/common/requirements.txt ./\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain/modeling_gpt_neox_nxd.py ./\nln -sf ~/neuronx-distributed/examples/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain/utils.py ./\npython3 -m pip install -r requirements.txt\n\npython3 get_dataset.py\n\nPATH=$PATH:/opt/slurm/bin/\n\nsbatch --exclusive \\\n--nodes 4 \\\n--wrap=\"srun neuron_parallel_compile bash $(pwd)/tp_dp_gpt_neox_6.9b_hf_pretrain.sh\"\n\nsbatch --exclusive \\\n--nodes 4 \\\n--wrap=\"srun bash $(pwd)/tp_dp_gpt_neox_6.9b_hf_pretrain.sh\"\n" }, { "path": "archive/tutorials/nxd-source-code/llama_tp_pp_ptl/llama_2_13b.sh", "content": "#!/bin/bash\nset -eExuo\n\ncd ~/neuronx-distributed/examples/training/llama/lightning\nchmod +x run_llama_13b_tp_pp_ptl.sh\nmkdir 13B_config\ncp ~/neuronx-distributed/examples/training/llama/tp_pp_llama_hf_pretrain/13B_config_llama2/config.json ./13B_config\n\n\nsudo rm -rf /home/ubuntu/.cache/\npip install --upgrade filelock\n\npython3 get_dataset.py --llama-version 2\n\nPATH=$PATH:/opt/slurm/bin/\n\nsbatch --exclusive \\\n--nodes 32 \\\n--cpus-per-task 128 \\\n--wrap=\"srun neuron_parallel_compile bash $(pwd)/run_llama_13b_tp_pp_ptl.sh\"\n\nsbatch --exclusive \\\n--nodes 32 \\\n--cpus-per-task 128 \\\n--wrap=\"srun bash $(pwd)/run_llama_13b_tp_pp_ptl.sh\"\n" }, { "path": "archive/tutorials/nxd-source-code/llama_tp_pp_ptl/llama_2_70b.sh", "content": "#!/bin/bash\nset -eExuo\n\ncd ~/neuronx-distributed/examples/training/llama/lightning\nchmod +x run_llama_70b_tp_pp_ptl.sh\nmkdir 70B_config\ncp ~/neuronx-distributed/examples/training/llama/tp_pp_llama_hf_pretrain/70B_config_llama2/config.json ./70B_config\n\n\nsudo rm -rf /home/ubuntu/.cache/\npip install --upgrade filelock\n\npython3 get_dataset.py --llama-version 2\n\nPATH=$PATH:/opt/slurm/bin/\n\nsbatch --exclusive \\\n--nodes 32 \\\n--cpus-per-task 128 \\\n--wrap=\"srun neuron_parallel_compile bash $(pwd)/run_llama_70b_tp_pp_ptl.sh\"\n\nsbatch --exclusive \\\n--nodes 32 \\\n--cpus-per-task 128 \\\n--wrap=\"srun bash $(pwd)/run_llama_70b_tp_pp_ptl.sh\"\n" }, { "path": "archive/tutorials/nxd-source-code/llama_tp_pp_ptl/llama_2_7b.sh", "content": "#!/bin/bash\nset -eExuo\n\ncd ~/neuronx-distributed/examples/training/llama/tp_zero1_llama_hf_pretrain\nchmod +x tp_zero1_llama2_7B_hf_pretrain.sh\nln -sf 7B_config_llama2/config.json ./\n\nsudo rm -rf /home/ubuntu/.cache/\npip install --upgrade filelock\n\npython3 get_dataset.py --llama-version 2\n\nPATH=$PATH:/opt/slurm/bin/\n\nsbatch --exclusive \\\n--nodes 4 \\\n--cpus-per-task 128 \\\n--wrap=\"srun neuron_parallel_compile bash $(pwd)/tp_zero1_llama2_7B_hf_pretrain.sh\"\n\nsbatch --exclusive \\\n--nodes 4 \\\n--cpus-per-task 128 \\\n--wrap=\"srun bash $(pwd)/tp_zero1_llama2_7B_hf_pretrain.sh\"\n" }, { "path": "archive/tutorials/nxd-source-code/llama_tp_pp_ptl/llama_tp_pp_ptl_setup.sh", "content": "#!/bin/bash\nset -eExuo\n\ncd ~/neuronx-distributed/examples/training/llama/lightning\nln -sf ~/neuronx-distributed/examples/training/llama/get_dataset.py ./\nln -sf ~/neuronx-distributed/examples/training/llama/lr.py ./\nln -sf ~/neuronx-distributed/examples/training/llama/modeling_llama_nxd.py ./\nln -sf ~/neuronx-distributed/examples/training/llama/requirements.txt ./\nln -sf ~/neuronx-distributed/examples/training/llama/requirements_ptl.txt ./\nln -sf ~/neuronx-distributed/examples/training/llama/training_utils.py ./\n\npython3 -m pip install -r requirements.txt\npython3 -m pip install -r requirements_ptl.txt # Currently we're supporting Lightning version 2.1.0" }, { "path": "archive/tutorials/ssd300_demo/requirements.txt", "content": "numpy>1.18.5\ntensorflow_neuron==1.15.5.2.8.9.0\nneuron_cc==1.13.5.0\ntensorflow-serving-api==1.15.0\ntorch>=1.0,<2.0\ntorchvision<1.0\nmatplotlib<4.0\nCython<0.29\npycocotools==2.0.1\ntensorflow-serving-api==1.15.0\n" }, { "path": "archive/tutorials/ssd300_demo/ssd300_demo.rst", "content": ".. _tensorflow-ssd300:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nRunning SSD300 with AWS Neuron\n==============================\n\n.. note:: \n This page was archived on 7/31/2025.\n\n*Update 11/16: The model checkpoint\nlink*\\ https://api.ngc.nvidia.com/v2/models/nvidia/ssdpyt_fp32/versions/1/files/nvidia_ssdpyt_fp32_20190225.pt\\ *is\ncurrently broken and the AWS Neuron team is working on providing an\nalternative source.*\n\n\nThis demo shows a Neuron compatible SSD300 implementation that is\nfunctionally equivalent to open source SSD300 model. This demo uses\nTensorFlow-Neuron, PyTorch SSD300 model and checkpoint\n(https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/) and also\nshows the performance achieved by the Inf1 instance.\n\nTable of Contents\n-----------------\n\n1. Launch EC2 instance and update AWS Neuron SDK software\n2. Generating Neuron compatible SSD300 TensorFlow SavedModel\n\n - Convert open source PyTorch SSD300 model and checkpoint into\n Neuron compatible SSD300 TensorFlow SavedModel\n\n3. Evaluate the generated SSD300 TensorFlow SavedModel for both accuracy\n and performance\n\n - Running threaded inference through the COCO 2017 validation\n dataset\n\nLaunch EC2 instances and update tensorflow-neuron and neuron-cc\n---------------------------------------------------------------\n\nFor this demo, launch one inf1.xlarge EC2 instance. We recommend using\nthe latest Ubuntu 18 Deep Learning AMI (DLAMI).\n\nPlease configure your ubuntu16/ubuntu18/yum repo following the steps in\nthe :ref:`install-neuron-tensorflow` in order to install\n``tensorflow-model-server-neuron``.\n\nGenerating Neuron compatible SSD300 TensorFlow SavedModel\n---------------------------------------------------------\n\nFirst connect to your inf1.xlarge instance\n\nCompile open source PyTorch SSD300 model and checkpoint into Neuron compatible SSD300 TensorFlow SavedModel\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn the same directory ssd300_demo, run the following:\n\n1. Create venv and install dependencies\n\n.. code:: bash\n\n sudo apt update\n sudo apt install g++ python3-dev python3-venv unzip\n sudo apt install tensorflow-model-server-neuron\n python3 -m venv env\n source ./env/bin/activate\n pip install pip setuptools --upgrade\n pip install -r ./requirements.txt --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n2. Clone NVIDIA's DeepLearningExamples repo that contains PyTorch\n SSD300.\n\n.. code:: bash\n\n git clone https://github.com/NVIDIA/DeepLearningExamples.git\n cd DeepLearningExamples\n git checkout a644350589f9abc91b203f73e686a50f5d6f3e96\n cd ..\n\n3. Download PyTorch SSD300 checkpoint file.\n\n.. code:: bash\n\n curl -LO https://api.ngc.nvidia.com/v2/models/nvidia/ssdpyt_fp32/versions/1/files/nvidia_ssdpyt_fp32_20190225.pt\n\n4. Download COCO 2017 validation set and annotations.\n\n.. code:: bash\n\n curl -LO http://images.cocodataset.org/zips/val2017.zip\n unzip ./val2017.zip\n curl -LO http://images.cocodataset.org/annotations/annotations_trainval2017.zip\n unzip ./annotations_trainval2017.zip\n\n5. Convert PyTorch SSD300 model and checkpoint into a Neuron-compatible\n TensorFlow SavedModel.\n\n.. code:: bash\n\n python ssd300_model.py --torch_checkpoint=./nvidia_ssdpyt_fp32_20190225.pt --output_saved_model=./ssd300_tf_neuron/1\n\nThis converts PyTorch SSD300 model and checkpoint to a Neuron-compatible\nTensorFlow SavedModel using tensorflow-neuron and neuron-cc. The\ncompilation output is stored in ``./ssd300_tf_neuron``.\n\n6. Launch the ``tensorflow-model-server-neuron`` gRPC server at default\n port 8500 in the background.\n\n.. code:: bash\n\n tensorflow_model_server_neuron --model_base_path=$(pwd)/ssd300_tf_neuron &\n\n7. In client, evaluate the Neuron-compatible TensorFlow SavedModel for\n both accuracy and performance. Note that this client by default\n assumes a ``tensorflow-model-server-neuron`` listening at\n ``localhost:8500``. On inf1.xlarge, the expected throughput is 100\n images/second once the server is fully warmed up, and the expected\n mean average precision (mAP) is 0.253.\n\n.. code:: bash\n\n python ssd300_evaluation_client.py --val2017=./val2017 --instances_val2017_json=./annotations/instances_val2017.json\n\n8. After running the demo, please cleanup resources allocated in Neuron\n runtime by gracefully killing the ``tensorflow_model_server_neuron``\n process, e. g.,\n\n.. code:: bash\n\n killall tensorflow_model_server_neuron\n" }, { "path": "archive/tutorials/ssd300_demo/ssd300_detection.py", "content": "import argparse\nimport json\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport numpy as np\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport tensorflow as tf\nimport tensorflow.neuron as tfn\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--image', required=True, help='Path to image that is to be detected. Support jpeg and png format.')\n parser.add_argument('--image_with_detections', required=True, help='Path to save image after detection (with bounding boxes drawn). Png format.')\n parser.add_argument('--saved_model', required=True, help='TensorFlow SSD300 SavedModel')\n parser.add_argument('--score_threshold', type=float, default=0.15, help='Minimum required score for drawing a bounding box')\n parser.add_argument('--instances_val2017_json', default=None, help='Json file that contains labeling information')\n parser.add_argument('--save_results', default=None)\n parser.add_argument('--disable_version_check', action='store_true')\n args = parser.parse_args()\n if not args.disable_version_check:\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version < LooseVersion('1.15.0.1.0.1333.0'):\n raise RuntimeError(\n 'tensorflow-neuron version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U tensorflow-neuron --index-url=https://pip.repos.neuron.amazonaws.com\"'.format(tfn_version))\n\n with open(args.image, 'rb') as f:\n img_jpg_bytes = f.read()\n model_feed_dict = {'batch_image': [img_jpg_bytes]}\n\n predictor = tf.contrib.predictor.from_saved_model(args.saved_model)\n results = predictor(model_feed_dict)\n if args.save_results is not None:\n np.savez(args.save_results, **results)\n boxes_np = results['boxes']\n scores_np = results['scores']\n classes_np = results['classes']\n\n if args.instances_val2017_json is not None:\n with open(args.instances_val2017_json) as f:\n annotate_json = json.load(f)\n label_info = {idx+1: cat['name'] for idx, cat in enumerate(annotate_json['categories'])}\n\n plt.switch_backend('agg')\n fig, ax = plt.subplots(1)\n ax.imshow(Image.open(args.image).convert('RGB'))\n\n wanted = scores_np[0] > args.score_threshold\n for xywh, label_no_bg in zip(boxes_np[0][wanted], classes_np[0][wanted]):\n rect = patches.Rectangle((xywh[0], xywh[1]), xywh[2], xywh[3], linewidth=1, edgecolor='g', facecolor='none')\n ax.add_patch(rect)\n rx, ry = rect.get_xy()\n rx = rx + rect.get_width() / 2.0\n if args.instances_val2017_json is not None:\n ax.annotate(label_info[label_no_bg + 1], (rx, ry), color='w', backgroundcolor='g', fontsize=10,\n ha='center', va='center', bbox=dict(boxstyle='square,pad=0.01', fc='g', ec='none', alpha=0.5))\n plt.savefig(args.image_with_detections)\n plt.close(fig)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "archive/tutorials/ssd300_demo/ssd300_evaluation.py", "content": "import argparse\nimport os\nimport json\nimport glob\nfrom concurrent import futures\nimport time\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow.neuron as tfn\nfrom pycocotools.cocoeval import COCOeval\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.coco import COCO\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import dboxes300_coco\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import SSDTransformer\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import COCODetection\n\n\ndef get_val_dataset(val_annotate, val_coco_root):\n dboxes = dboxes300_coco()\n val_trans = SSDTransformer(dboxes, (300, 300), val=True)\n val_coco = COCODetection(val_coco_root, val_annotate, val_trans)\n return val_coco\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--saved_model', required=True, help='TensorFlow SSD300 SavedModel')\n parser.add_argument('--val2017', required=True, help='Path to COCO 2017 validation dataset')\n parser.add_argument('--instances_val2017_json', required=True, help='Json file that contains labeling information')\n parser.add_argument('--num_sessions', type=int, default=1, help='Number of tensorflow sessions')\n parser.add_argument('--num_threads', type=int, default=4, help='Number of threads')\n parser.add_argument('--throughput_interval', type=int, default=10, help='Interval for counting throughput')\n parser.add_argument('--save_results', default=None)\n parser.add_argument('--disable_version_check', action='store_true')\n args = parser.parse_args()\n if not args.disable_version_check:\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version < LooseVersion('1.15.0.1.0.1333.0'):\n raise RuntimeError(\n 'tensorflow-neuron version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U tensorflow-neuron --index-url=https://pip.repos.neuron.amazonaws.com\"'.format(tfn_version))\n predictor_list = [tf.contrib.predictor.from_saved_model(args.saved_model) for _ in range(args.num_sessions)]\n\n val_dataset = get_val_dataset(args.instances_val2017_json, args.val2017)\n inv_map = {v: k for k, v in val_dataset.label_map.items()}\n model_feed_dict_list = []\n for img_id in val_dataset.img_keys:\n img_path = os.path.join(args.val2017, val_dataset.images[img_id][0])\n with open(img_path, 'rb') as f:\n img_jpg_bytes = f.read()\n model_feed_dict_list.append({'batch_image': [img_jpg_bytes]})\n\n latency_list = []\n throughput_list = []\n def predict(pred, model_feed_dict):\n start = time.time()\n result = pred(model_feed_dict)\n latency_list.append(time.time() - start)\n return result\n\n def performance():\n last_num_infer = len(latency_list)\n while len(latency_list) < len(model_feed_dict_list):\n current_num_infer = len(latency_list)\n throughput = (current_num_infer - last_num_infer) / args.throughput_interval\n throughput_list.append(throughput)\n p50 = 0.0\n p90 = 0.0\n if latency_list:\n p50 = np.percentile(latency_list, 50)\n p90 = np.percentile(latency_list, 90)\n print('pid {}: current throughput {}, latency p50={:.3f} p90={:.3f}'.format(os.getpid(), throughput, p50, p90))\n last_num_infer = current_num_infer\n time.sleep(args.throughput_interval)\n\n executor = futures.ThreadPoolExecutor(max_workers=(args.num_sessions*args.num_threads)+1)\n performance_future = executor.submit(performance)\n eval_futures = []\n for idx, model_feed_dict in enumerate(model_feed_dict_list):\n eval_fut = executor.submit(predict, predictor_list[idx%len(predictor_list)], model_feed_dict)\n eval_futures.append(eval_fut)\n waited_results = []\n for idx, eval_fut in enumerate(eval_futures):\n if idx % 100 == 0:\n print('evaluating image {}/{}'.format(idx, len(eval_futures)))\n waited_results.append(eval_fut.result())\n eval_results = []\n for idx, (img_id, results) in enumerate(zip(val_dataset.img_keys, waited_results)):\n boxes = results['boxes']\n for box, label, prob in zip(results['boxes'][0], results['classes'][0], results['scores'][0]):\n res = [img_id, box[0], box[1], box[2], box[3], prob, inv_map[label+1]] # +1 to account for background\n eval_results.append(res)\n performance_future.result()\n\n coco_gt = COCO(annotation_file=args.instances_val2017_json)\n coco_dt = coco_gt.loadRes(np.array(eval_results).astype(np.float32))\n coco_eval = COCOeval(coco_gt, coco_dt, iouType='bbox')\n coco_eval.evaluate()\n coco_eval.accumulate()\n coco_eval.summarize()\n if args.save_results is not None:\n np.save(args.save_results, coco_eval.stats)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "archive/tutorials/ssd300_demo/ssd300_evaluation_client.py", "content": "import argparse\nimport os\nimport json\nimport glob\nfrom concurrent import futures\nimport time\nimport subprocess\nfrom distutils.version import LooseVersion\nimport numpy as np\nimport tensorflow as tf\nimport grpc\nfrom tensorflow_serving.apis import predict_pb2\nfrom tensorflow_serving.apis import prediction_service_pb2_grpc\nfrom pycocotools.cocoeval import COCOeval\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.coco import COCO\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import dboxes300_coco\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import SSDTransformer\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import COCODetection\n\n\ndef get_val_dataset(val_annotate, val_coco_root):\n dboxes = dboxes300_coco()\n val_trans = SSDTransformer(dboxes, (300, 300), val=True)\n val_coco = COCODetection(val_coco_root, val_annotate, val_trans)\n return val_coco\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--server_address', default='localhost:8500', help='tensorflow-model-server-neuron grpc address')\n parser.add_argument('--model_name', default='default', help='Serving model name')\n parser.add_argument('--val2017', required=True, help='Path to COCO 2017 validation dataset')\n parser.add_argument('--instances_val2017_json', required=True, help='Json file that contains labeling information')\n parser.add_argument('--num_threads', type=int, default=4, help='Number of threads')\n parser.add_argument('--throughput_interval', type=int, default=10, help='Interval for counting throughput')\n parser.add_argument('--save_results', default=None)\n args = parser.parse_args()\n\n channel = grpc.insecure_channel(args.server_address)\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n\n val_dataset = get_val_dataset(args.instances_val2017_json, args.val2017)\n inv_map = {v: k for k, v in val_dataset.label_map.items()}\n request_list = []\n for img_id in val_dataset.img_keys:\n img_path = os.path.join(args.val2017, val_dataset.images[img_id][0])\n with open(img_path, 'rb') as f:\n img_jpg_bytes = f.read()\n data = np.array([img_jpg_bytes], dtype=object)\n data = tf.contrib.util.make_tensor_proto(data, shape=data.shape)\n request = predict_pb2.PredictRequest()\n request.model_spec.name = args.model_name\n request.inputs['batch_image'].CopyFrom(data)\n request_list.append(request)\n\n latency_list = []\n throughput_list = []\n def predict(request):\n start = time.time()\n result = stub.Predict(request).outputs\n latency_list.append(time.time() - start)\n return result\n\n def performance():\n last_num_infer = len(latency_list)\n while len(latency_list) < len(request_list):\n current_num_infer = len(latency_list)\n throughput = (current_num_infer - last_num_infer) / args.throughput_interval\n throughput_list.append(throughput)\n p50 = 0.0\n p90 = 0.0\n if latency_list:\n p50 = np.percentile(latency_list, 50)\n p90 = np.percentile(latency_list, 90)\n print('pid {}: current throughput {}, latency p50={:.3f} p90={:.3f}'.format(os.getpid(), throughput, p50, p90))\n last_num_infer = current_num_infer\n time.sleep(args.throughput_interval)\n\n executor = futures.ThreadPoolExecutor(max_workers=args.num_threads+1)\n performance_future = executor.submit(performance)\n eval_futures = []\n for idx, request in enumerate(request_list):\n eval_fut = executor.submit(predict, request)\n eval_futures.append(eval_fut)\n waited_results = []\n for idx, eval_fut in enumerate(eval_futures):\n if idx % 100 == 0:\n print('evaluating image {}/{}'.format(idx, len(eval_futures)))\n waited_results.append(eval_fut.result())\n eval_results = []\n for idx, (img_id, results) in enumerate(zip(val_dataset.img_keys, waited_results)):\n results = {key: tf.make_ndarray(value) for key, value in results.items()}\n boxes = results['boxes']\n for box, label, prob in zip(results['boxes'][0], results['classes'][0], results['scores'][0]):\n res = [img_id, box[0], box[1], box[2], box[3], prob, inv_map[label+1]] # +1 to account for background\n eval_results.append(res)\n performance_future.result()\n\n coco_gt = COCO(annotation_file=args.instances_val2017_json)\n coco_dt = coco_gt.loadRes(np.array(eval_results).astype(np.float32))\n coco_eval = COCOeval(coco_gt, coco_dt, iouType='bbox')\n coco_eval.evaluate()\n coco_eval.accumulate()\n coco_eval.summarize()\n if args.save_results is not None:\n np.save(args.save_results, coco_eval.stats)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "archive/tutorials/ssd300_demo/ssd300_model.py", "content": "import sys\nimport os\nimport argparse\nimport time\nimport itertools\nfrom functools import partial\nfrom collections import Counter\nimport json\nimport shutil\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.core.framework import attr_value_pb2\nimport tensorflow.neuron as tfn\nimport torch\n\n\ndef decode_jpeg_resize(input_tensor, image_size):\n # decode jpeg\n tensor = tf.image.decode_png(input_tensor, channels=3)\n\n # resize\n decoded_shape = tf.shape(tensor)\n tensor = tf.cast(tensor, tf.float32)\n decoded_shape_hw = decoded_shape[0:2]\n decoded_shape_hw_float32 = tf.cast(decoded_shape_hw, tf.float32)\n tensor = tf.image.resize(tensor, image_size)\n\n # normalize\n tensor -= np.array([0.485, 0.456, 0.406]).astype(np.float32) * 255.0\n return tensor, decoded_shape_hw_float32[::-1]\n\n\ndef preprocessor(input_tensor, image_size):\n with tf.name_scope('Preprocessor'):\n tensor, bbox_scale_hw = tf.map_fn(\n partial(decode_jpeg_resize, image_size=image_size), input_tensor,\n dtype=(tf.float32, tf.float32), back_prop=False, parallel_iterations=16)\n return tensor, bbox_scale_hw\n\n\ndef tf_Conv2d(input_tensor, module, first_conv=False):\n np_dtype = input_tensor.dtype.as_numpy_dtype\n kernel_np = module.weight.detach().numpy().transpose([2, 3, 1, 0])\n if first_conv:\n kernel_np /= (np.array([0.229, 0.224, 0.225]).astype(np.float32) * 255.0)[:, np.newaxis]\n kernel = tf.constant(kernel_np.astype(np_dtype))\n if any(module.padding):\n pad_h, pad_w = module.padding\n padding = [[0, 0], [pad_h, pad_h], [pad_w, pad_w], [0, 0]]\n input_tensor = tf.pad(input_tensor, padding)\n stride_h, stride_w = module.stride\n tensor = tf.nn.conv2d(input_tensor, kernel, strides=[1, stride_h, stride_w, 1], padding='VALID')\n if module.bias is not None:\n bias = tf.constant(module.bias.detach().numpy().astype(np_dtype))\n tensor = tf.nn.bias_add(tensor, bias)\n return tensor\n\ndef tf_BatchNorm2d(input_tensor, module):\n def _norm_np(ts):\n return ts.astype(input_tensor.dtype.as_numpy_dtype)\n mean = _norm_np(module.running_mean.detach().numpy())\n offset = _norm_np(module.bias.detach().numpy())\n inv_std = np.sqrt(module.running_var.detach().numpy() + module.eps)\n scale_inv_std = _norm_np(module.weight.detach().numpy() / inv_std)\n return scale_inv_std * (input_tensor - mean) + offset\n\ndef tf_MaxPool2d(input_tensor, module):\n pad = module.padding\n tensor = tf.pad(input_tensor, [[0, 0], [pad, pad], [pad, pad], [0, 0]])\n return tf.nn.max_pool2d(tensor, ksize=module.kernel_size, strides=module.stride, padding='VALID')\n\ndef tf_Bottleneck(input_tensor, module):\n tensor = tf_Conv2d(input_tensor, module.conv1)\n tensor = tf_BatchNorm2d(tensor, module.bn1)\n tensor = tf.nn.relu(tensor)\n tensor = tf_Conv2d(tensor, module.conv2)\n tensor = tf_BatchNorm2d(tensor, module.bn2)\n tensor = tf.nn.relu(tensor)\n tensor = tf_Conv2d(tensor, module.conv3)\n tensor = tf_BatchNorm2d(tensor, module.bn3)\n if module.downsample is not None:\n input_tensor = tf_Conv2d(input_tensor, module.downsample[0])\n input_tensor = tf_BatchNorm2d(input_tensor, module.downsample[1])\n return tf.nn.relu(input_tensor + tensor)\n\ndef tf_SequentialBottleneck(tensor, seq, resnet):\n with tf.name_scope('{}.Sequential'.format(seq)):\n for idx, module in enumerate(resnet[seq]):\n with tf.name_scope('{}.BasicBlock'.format(idx)):\n tensor = tf_Bottleneck(tensor, module)\n return tensor\n\ndef tf_bbox_view(detection_feed, modules, ndim):\n results = []\n for idx, (tensor, mod) in enumerate(zip(detection_feed, modules)):\n with tf.name_scope('branch{}'.format(idx)):\n tensor = tf_Conv2d(tensor, mod)\n tensor = tf.transpose(tensor, [0, 3, 1, 2])\n tensor = tf.cast(tensor, tf.float32)\n\n shape = tensor.shape.as_list()\n batch_size = -1 if shape[0] is None else shape[0]\n new_shape = [batch_size, ndim, np.prod(shape[1:]) // ndim]\n results.append(tf.reshape(tensor, new_shape))\n tensor = tf.concat(results, axis=-1)\n return tensor\n\n\ndef tf_feature_extractor(input_tensor, resnet):\n with tf.name_scope('FeatureExtractor'):\n with tf.name_scope('0.Conv2d'):\n tensor = tf_Conv2d(input_tensor, resnet[0], first_conv=True)\n with tf.name_scope('1.BatchNorm2d'):\n tensor = tf_BatchNorm2d(tensor, resnet[1])\n with tf.name_scope('2.ReLU'):\n tensor = tf.nn.relu(tensor)\n with tf.name_scope('3.MaxPool2d'):\n tensor = tf_MaxPool2d(tensor, resnet[3])\n tensor = tf_SequentialBottleneck(tensor, 4, resnet)\n tensor = tf_SequentialBottleneck(tensor, 5, resnet)\n tensor = tf_SequentialBottleneck(tensor, 6, resnet)\n tensor = tf.cast(tensor, tf.float16)\n return tensor\n\n\ndef tf_box_predictor(tensor, ssd300_torch):\n with tf.name_scope('BoxPredictor'):\n detection_feed = [tensor]\n for idx, block in enumerate(ssd300_torch.additional_blocks):\n with tf.name_scope('{}.Sequential'.format(idx)):\n tensor = tf_Conv2d(tensor, block[0])\n tensor = tf_BatchNorm2d(tensor, block[1])\n tensor = tf.nn.relu(tensor)\n tensor = tf_Conv2d(tensor, block[3])\n tensor = tf_BatchNorm2d(tensor, block[4])\n tensor = tf.nn.relu(tensor)\n detection_feed.append(tensor)\n with tf.name_scope('Boxes'):\n loc = tf_bbox_view(detection_feed, ssd300_torch.loc, ndim=4)\n with tf.name_scope('Probabilities'):\n conf = tf_bbox_view(detection_feed, ssd300_torch.conf, ndim=ssd300_torch.label_num)\n return loc, conf\n\n\n@tfn.fuse(batch_size=1, dynamic_batch_size=True)\ndef tf_ssd300(input_tensor, ssd300_torch):\n with tf.name_scope('SSD300'):\n tensor = tf_feature_extractor(input_tensor, ssd300_torch.feature_extractor.feature_extractor)\n loc, conf = tf_box_predictor(tensor, ssd300_torch)\n return loc, conf\n\n\ndef scale_back_batch(bboxes_in, scores_in, scale_xy, scale_wh, dboxes_xywh):\n \"\"\"\n Do scale and transform from xywh to ltrb\n suppose input Nx4xnum_bbox Nxlabel_numxnum_bbox\n \"\"\"\n with tf.name_scope('ScaleBackBatch'):\n bboxes_in = tf.transpose(bboxes_in, [0, 2, 1])\n scores_in = tf.transpose(scores_in, [0, 2, 1])\n\n bboxes_xy = bboxes_in[:, :, :2]\n bboxes_wh = bboxes_in[:, :, 2:]\n bboxes_xy *= scale_xy\n bboxes_wh *= scale_wh\n\n bboxes_xy = bboxes_xy * dboxes_xywh[:, :, 2:] + dboxes_xywh[:, :, :2]\n bboxes_wh = tf.exp(bboxes_wh) * dboxes_xywh[:, :, 2:]\n\n bboxes_wh_half = 0.5 * bboxes_wh\n bboxes_lt = bboxes_xy - bboxes_wh_half\n bboxes_rb = bboxes_xy + bboxes_wh_half\n\n bboxes_in = tf.concat([bboxes_lt, bboxes_rb], axis=-1)\n\n return bboxes_in, tf.nn.softmax(scores_in, axis=-1)\n\ndef select_nms_outputs(input_tensors):\n boxes_xywh, scores, classes, valid_detections = input_tensors\n return boxes_xywh[:valid_detections], scores[:valid_detections], classes[:valid_detections]\n\ndef postprocessor(ploc_ts, plabel_ts, bbox_scale_hw_ts, scale_xy, scale_wh, dboxes_xywh):\n with tf.name_scope('Postprocessor'):\n ploc_ts = tf.cast(ploc_ts, tf.float32)\n plabel_ts = tf.cast(plabel_ts, tf.float32)\n bboxes_ts, probs_ts = scale_back_batch(ploc_ts, plabel_ts, scale_xy, scale_wh, dboxes_xywh)\n bboxes_ts = bboxes_ts[:, :, tf.newaxis, :]\n probs_ts = probs_ts[:, :, 1:]\n nms_outputs = tf.image.combined_non_max_suppression(\n bboxes_ts,\n probs_ts,\n max_output_size_per_class=200,\n max_total_size=200,\n iou_threshold=0.5,\n score_threshold=0.05,\n pad_per_class=False,\n clip_boxes=False,\n name='CombinedNonMaxSuppression',\n )\n nmsed_boxes_x0y0x1y1, nmsed_scores, nmsed_classes, valid_detections = nms_outputs\n nmsed_boxes_x0y0 = nmsed_boxes_x0y0x1y1[..., :2]\n nmsed_boxes_x1y1 = nmsed_boxes_x0y0x1y1[..., 2:]\n bbox_scale_hw_ts = bbox_scale_hw_ts[:, tf.newaxis, :]\n nmsed_boxes_xy = nmsed_boxes_x0y0 * bbox_scale_hw_ts\n nmsed_boxes_wh = (nmsed_boxes_x1y1 - nmsed_boxes_x0y0) * bbox_scale_hw_ts\n nmsed_boxes_xywh = tf.concat([nmsed_boxes_xy, nmsed_boxes_wh], axis=-1)\n nmsed_boxes_xywh, nmsed_scores, nmsed_classes = tf.map_fn(\n select_nms_outputs, (nmsed_boxes_xywh, nmsed_scores, nmsed_classes, valid_detections),\n dtype=(tf.float32, tf.float32, tf.float32), back_prop=False, parallel_iterations=16)\n return nmsed_boxes_xywh, nmsed_scores, nmsed_classes\n\n\nclass DefaultBoxes(object):\n\n def __init__(self, fig_size, feat_size, steps, scales, aspect_ratios,\n scale_xy=0.1, scale_wh=0.2):\n\n self.feat_size = feat_size\n self.fig_size = fig_size\n\n self.scale_xy_ = scale_xy\n self.scale_wh_ = scale_wh\n\n # According to https://github.com/weiliu89/caffe\n # Calculation method slightly different from paper\n self.steps = steps\n self.scales = scales\n\n fk = fig_size/np.array(steps)\n self.aspect_ratios = aspect_ratios\n\n self.default_boxes = []\n # size of feature and number of feature\n for idx, sfeat in enumerate(self.feat_size):\n\n sk1 = scales[idx]/fig_size\n sk2 = scales[idx+1]/fig_size\n sk3 = np.sqrt(sk1*sk2)\n all_sizes = [(sk1, sk1), (sk3, sk3)]\n\n for alpha in aspect_ratios[idx]:\n w, h = sk1*np.sqrt(alpha), sk1/np.sqrt(alpha)\n all_sizes.append((w, h))\n all_sizes.append((h, w))\n for w, h in all_sizes:\n for i, j in itertools.product(range(sfeat), repeat=2):\n cx, cy = (j+0.5)/fk[idx], (i+0.5)/fk[idx]\n self.default_boxes.append((cx, cy, w, h))\n\n self.dboxes = np.array(self.default_boxes)\n self.dboxes = self.dboxes.clip(min=0, max=1)\n # For IoU calculation\n self.dboxes_ltrb = self.dboxes.copy()\n self.dboxes_ltrb[:, 0] = self.dboxes[:, 0] - 0.5 * self.dboxes[:, 2]\n self.dboxes_ltrb[:, 1] = self.dboxes[:, 1] - 0.5 * self.dboxes[:, 3]\n self.dboxes_ltrb[:, 2] = self.dboxes[:, 0] + 0.5 * self.dboxes[:, 2]\n self.dboxes_ltrb[:, 3] = self.dboxes[:, 1] + 0.5 * self.dboxes[:, 3]\n\n @property\n def scale_xy(self):\n return self.scale_xy_\n\n @property\n def scale_wh(self):\n return self.scale_wh_\n\n def __call__(self, order=\"ltrb\"):\n if order == \"ltrb\": return self.dboxes_ltrb\n if order == \"xywh\": return self.dboxes\n\n\ndef dboxes300_coco():\n figsize = 300\n feat_size = [38, 19, 10, 5, 3, 1]\n steps = [8, 16, 32, 64, 100, 300]\n # use the scales here: https://github.com/amdegroot/ssd.pytorch/blob/master/data/config.py\n scales = [21, 45, 99, 153, 207, 261, 315]\n aspect_ratios = [[2], [2, 3], [2, 3], [2, 3], [2], [2]]\n dboxes = DefaultBoxes(figsize, feat_size, steps, scales, aspect_ratios)\n return dboxes\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--torch_checkpoint', required=True, help='Path to PyTorch SSD300 model checkpoint')\n parser.add_argument('--output_saved_model', required=True, help='Output TensorFlow SavedModel that runs on Inferentia')\n parser.add_argument('--disable_version_check', action='store_true')\n args = parser.parse_args()\n if os.path.exists(args.output_saved_model):\n raise OSError('SavedModel dir {} already exists'.format(args.output_saved_model))\n\n if not args.disable_version_check:\n neuroncc_version = LooseVersion(pkg_resources.get_distribution('neuron-cc').version)\n if neuroncc_version < LooseVersion('1.0.18000'):\n raise RuntimeError(\n 'neuron-cc version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U neuron-cc --index-url=https://pip.repos.neuron.amazonaws.com\"'.format(neuroncc_version))\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version < LooseVersion('1.15.3.1.0.1900.0'):\n raise RuntimeError(\n 'tensorflow-neuron version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U tensorflow-neuron --index-url=https://pip.repos.neuron.amazonaws.com\"'.format(tfn_version))\n\n sys.path.append(os.getcwd())\n from DeepLearningExamples.PyTorch.Detection.SSD.src import model as torch_ssd300_model\n ssd300_torch = torch_ssd300_model.SSD300()\n ckpt = torch.load(args.torch_checkpoint, map_location=torch.device('cpu'))\n ssd300_torch.load_state_dict(ckpt['model'])\n ssd300_torch.eval()\n\n input_tensor = tf.placeholder(tf.string, [None])\n image_tensor, bbox_scale_hw_tensor = preprocessor(input_tensor, [300, 300])\n\n dboxes = dboxes300_coco()\n dboxes_xywh = dboxes(order=\"xywh\")[np.newaxis, ...]\n\n ploc_tensor, plabel_tensor = tf_ssd300(image_tensor, ssd300_torch)\n boxes_tensor, scores_tensor, classes_tensor = postprocessor(\n ploc_tensor, plabel_tensor, bbox_scale_hw_tensor, dboxes.scale_xy, dboxes.scale_wh, dboxes_xywh)\n outputs = {\n 'boxes': boxes_tensor,\n 'scores': scores_tensor,\n 'classes': classes_tensor,\n }\n\n sess = tf.Session()\n try:\n sess.run(outputs)\n except:\n pass\n\n for op in sess.graph.get_operations():\n if op.type == 'NeuronOp':\n if not op.get_attr('executable'):\n raise AttributeError(\n 'Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly '\n '(\"pip install neuron-cc --force --index-url=https://pip.repos.neuron.amazonaws.com\" '\n 'to force reinstall neuron-cc).')\n model_config = op.node_def.attr['model_config'].list\n if model_config.i:\n model_config.i[0] = 1\n else:\n model_config.i.extend([1, 1, 1, 10])\n op._set_attr('model_config', attr_value_pb2.AttrValue(list=model_config))\n tf.saved_model.simple_save(sess, args.output_saved_model, {'batch_image': input_tensor}, outputs)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "archive/tutorials/training-gpt-neox-20b.rst", "content": ".. _gpt_neox_20b_tp_zero1_tutorial:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently unsupported and not maintained. It is provided for reference only.\n\nTraining GPT-NeoX 20B with Tensor Parallelism and ZeRO-1 Optimizer \n=========================================================================================\n\nIn this section, we showcase to pretrain a GPT-NeoX 20B model by using the sequence parallel optimization\nof tensor parallelism in the ``neuronx-distributed`` package. Please refer to the `Neuron Samples repository `__ to view the files in this tutorial.\n\nThis GPT-NeoX 20B tutorial differs from the :ref:`GPT-NeoX 6.9B tutorial` in the following ways:\n\n* sequence parallel optimization has been applied\n* parallel cross entropy has been applied\n* the model size has been increased from 6.9B to 20B\n* the TP degree has been increased from 8 to 32\n\nSetting up environment is same as the :ref:`GPT-NeoX 6.9B tutorial`.\n\n**Let’s download the scripts for pretraining:**\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_20b.sh\n :language: shell\n :lines: 4-8\n\nNext let’s download and pre-process the dataset:\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_20b.sh\n :language: shell\n :lines: 10\n\nAt this point, you are all set to start training.\n\n**Running training**\n\nWe first pre-compile the graphs using the ``neuron_parallel_compile``.\nLet’s run the command below:\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_20b.sh\n :language: shell\n :lines: 14-17\n\nThis script uses a tensor-parallel size of 32.\nThis will automatically set the zero-1 sharding degree to 4 (4 * 32 workers / tensor_parallel_size).\nOnce the graphs are compiled we can now run training and observe our loss goes down.\nTo run the training, we just the above command but without ``neuron_parallel_compile``.\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_20b.sh\n :language: shell\n :lines: 19-22\n\n\n**Sequence Parallel**\n\nWe made the following model level modifications to enable sequence parallel:\n\n* turn on ``sequence_parallel_enabled`` of ``ColumnParallelLinear`` and ``RowParallelLinear``\n in ``GPTNeoXAttention`` and ``GPTNeoXMLP``;\n* replace torch ``LayerNorm`` in ``GPTNeoXLayer`` and ``GPTNeoXModel`` with neuronx-distributed ``LayerNorm``\n with ``sequence_parallel_enabled``\n turned on;\n* dimension transposition of intermediate states in the forward function of ``GPTNeoXAttention``.\n* dimension transposition and collective communication of intermediate states in the forward function of ``GPTNeoXModel``.\n\nIn the training training script level, we enable:\n\n* all-reduce sequence parallel gradients at the gradient accumulation boundary.\n\nPlease check `modeling_gpt_neox_nxd.py `__ and `tp_dp_gpt_neox_20b_hf_pretrain.py `__ for details.\n\n\n**Parallel Cross Entropy**\n\nTo enable parallel cross entropy, we made the following model level modeifincations:\n\n* replace the ``CrossEntropyLoss`` with neuronx-distributed ``parallel_cross_entropy`` in the forward\n function of ``GPTNeoXForCausalLM``.\n* use ``ColumnParallelLinear`` for the ``embed_out`` layer in ``GPTNeoXForCausalLM``.\n\nPlease check ``modeling_gpt_neox_nxd.py`` for details.\n" }, { "path": "archive/tutorials/training-gpt-neox.rst", "content": ".. _gpt_neox_tp_zero1_tutorial:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This documentation for the AWS Neuron SDK is currently unsupported and not maintained. It is provided for reference only.\n\nTraining GPT-NeoX 6.9B with Tensor Parallelism and ZeRO-1 Optimizer\n=========================================================================================\n\nIn this section, we showcase to pretrain a GPT-NeoX 6.9B model by using tensor parallelism\nand zero-1 optimizer in the ``neuronx-distributed`` package. Please refer to the `Neuron Samples repository `__ to view the files in this tutorial.\n\n**Setting up environment:**\n \n\nFor this experiment, we will use a ParallelCluster with at least four trn1-32xl compute nodes.\n`Train your model on ParallelCluster `__\nintroduces how to setup and use a ParallelCluster.\nWe need first to create and activate a python virtual env on the head node of the ParallelCluster.\nNext follow the instructions mentioned here:\n:ref:`Install PyTorch Neuron on Trn1 ` to install neuron python packages.\n\nWe also need to install and clone the ``neuronx-distributed`` package using the following command:\n\n.. code:: ipython3\n\n python -m pip install neuronx_distributed --extra-index-url https://pip.repos.neuron.amazonaws.com\n git clone git@github.com:aws-neuron/neuronx-distributed.git\n\nLet’s download the scripts for pretraining.\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_6_9b.sh\n :language: shell\n :lines: 4-10\n\nNext let’s download and pre-process the dataset:\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_6_9b.sh\n :language: shell\n :lines: 12\n\nAt this point, you are all set to start training.\n\n**Running training**\n \n\nWe first pre-compile the graphs using the ``neuron_parallel_compile``.\nLet’s run the command below:\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_6_9b.sh\n :language: shell\n :lines: 16-18\n\nThis script uses a tensor-parallel size of 8.\nThis will automatically set the zero-1 sharding degree to 16 (4 * 32 workers / tensor_parallel_size).\nOnce the graphs are compiled we can now run training and observe our loss goes down.\nTo run the training, we just the above command but without ``neuron_parallel_compile``.\n\n.. literalinclude:: nxd-source-code/gpt_neox_tp_zero1/gpt_neox_6_9b.sh\n :language: shell\n :lines: 20-22\n\n**ZeRO-1 Optimizer**\n \n\nThe training script uses ZeRO-1 optimizer, where the optimizer states are partitioned across\nthe ranks so that each rank updates only its partition.\nBelow shows the code snippet of using ZeRO-1 optimizer in training script:\n\n.. code:: ipython3\n\n from neuronx_distributed.optimizer import NeuronZero1Optimizer\n\n optimizer = NeuronZero1Optimizer(\n optimizer_grouped_parameters,\n AdamW_FP32OptimParams,\n lr=flags.lr,\n pin_layout=False,\n sharding_groups=parallel_state.get_data_parallel_group(as_list=True),\n grad_norm_groups=parallel_state.get_tensor_model_parallel_group(as_list=True),\n )\n" }, { "path": "archive/tutorials/training_codegen25_7b.rst", "content": ".. _codegen25_7b_tp_zero1_tutorial:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nTraining CodeGen2.5 7B with Tensor Parallelism and ZeRO-1 Optimizer \n==============================================================================================\n\nIn this tutorial, we showcase how to pretrain a CodeGen2.5 7B model for program synthesis. Since Codegen2.5's architecture is identical to the one of Llama2, you may want to take a look at our `Llama2 tutorial `__ first.\n\nAfter setting up the environment and installing ``neuronx-distributed``, we need to download a data set containing source code (in this case Java code) and then preprocess and tokenize it to match the code-infill format (more about this below). Use the following commands to download the required files. Note, that we reuse our llama2 training files.\n\n.. code:: bash\n\n mkdir -p ~/examples/tp_zero1_codegen25_7b_hf_pretrain\n cd ~/examples/tp_zero1_codegen25_7b_hf_pretrain\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/llama/modeling_llama_nxd.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/llama/tp_zero1_llama_hf_pretrain/tp_zero1_llama_hf_pretrain.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/llama/tp_zero1_llama_hf_pretrain/logger.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/codegen25/tp_zero1_codegen25_7b_hf_pretrain.sh\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/codegen25/get_dataset_infill.py\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/codegen25/get_dataset_infill.sh\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/codegen25/requirements.txt\n chmod +x tp_zero1_codegen25_7b_hf_pretrain.sh\n chmod +x get_dataset_infill.sh\n python3 -m pip install -r requirements.txt\n\nData Preprocessing and Tokenization\n------------------------------------\n\nTo tokenize the data, we will use the CodeGen2.5 tokenizer from the HuggingFace repository. Download it by cloning the repository.\n\n.. code:: bash\n\n cd ~/examples\n git clone https://huggingface.co/Salesforce/codegen25-7b-mono\n cd codegen25-7b-mono\n rm config.json # Need to use our config.json for some Trainium-specific settings\n wget https://raw.githubusercontent.com/aws-neuron/neuronx-distributed/master/examples/training/codegen25/config.json\n cd ..\n\nThis tutorial makes use of a clean JAVA subset of the TheStack corpus and we preprocess it to fit the infill-format.\nThe infill format samples a random number of spans and formats the input the following way:\n\n.. code:: Python\n\n def count_words(filename: str) -> Dict[str, int]:\n \"\"\"Count the number of occurrences of each word in the file.\"\"\"\n with open(filename, 'r') as f:\n word_counts = {}\n for line in f:\n if word in word_counts:\n for word in line.split():\n word_counts[word] += 1\n else:\n word_counts[word] = 1\n return word_counts\n\nbecomes \n\n.. code:: Python\n\n def count_words(filename: str) -> Dict[str, int]:\n \"\"\"Count the number of occurrences of each word in the file.\"\"\"\n with open(filename, 'r') as f:\n in word_counts:\n for word in line.split():\n word_counts[word] += 1\n else:\n word_counts[word] = 1\n return word_counts<|endoftext|>\n word_counts = {}\n for line in f:\n if word \n\nFor each span, we introduce two ```` tokens. One signals the model that a span is missing at this position, and one (at the end of the code) which is followed by the original code span. Lastly, each span is suffixed with an end of mask (````) token. \nYou can preprocess and tokenize the dataset by running:\n\n.. code:: bash\n\n cd ~/examples/tp_zero1_codegen25_7b_hf_pretrain\n ./get_dataset_infill.sh\n\nThis will preprocess and store the data in your home directory at ``~/example_datasets/bigcode-stack-java_tokenized_infill``.\n\nStarting Training\n-----------------\nAt this point, you are all set to start training.\n\nPer default, we use a tensor parallel degree of 8, a global batch size of 256, and train for 10k steps. Feel free to change these settings in the ``tp_zero1_codegen25_7b_hf_pretrain.sh`` script.\n\nWe first pre-compile the graphs using the ``neuron_parallel_compile``. Let’s run the command below:\n\n.. code:: Python\n\n sbatch --exclusive \\\n --nodes 1 \\\n --wrap=\"srun neuron_parallel_compile bash $(pwd)/tp_zero1_codegen25_7b_hf_pretrain.sh\"\n\nOnce the graphs are compiled we can run training and observe our loss going down. \nTo do so, we run the same command omitting ``neuron_parallel_compile``.\n\n.. code:: Python\n\n sbatch --exclusive \\\n --nodes 1 \\\n --wrap=\"srun bash $(pwd)/tp_zero1_codegen25_7b_hf_pretrain.sh\"\n\n\nHappy training!\n" }, { "path": "archive/tutorials/training_llama2_tp_pp_ptl.rst", "content": ".. _llama2_tp_pp_ptl_tutorial:\n\n.. meta::\n :noindex:\n :nofollow:\n :description: This tutorial for the AWS Neuron SDK is currently archived and not maintained. It is provided for reference only.\n\nTraining Llama-2-7B/13B/70B using Tensor Parallelism and Pipeline Parallelism with Neuron PyTorch-Lightning\n============================================================================================================\n\nIn this section, we showcase to pretrain a Llama2 7B/13B/70B with Tensor Parallelism and Pipeline Parallel using Neuron PyTorch-Lightning APIs, please refer to the Llama2 13B/70B Tutorial\nand the Neuron PT-Lightning Developer Guide for more context.\n\n\nSetting up environment:\n^^^^^^^^^^^^^^^^^^^^^^^\n \nFor this experiment, we will use AWS ParallelCluster with at least four trn1.32xlarge compute nodes(at least 32 nodes are needed for 13B/70B model size).\n`Train your model on ParallelCluster `__\nintroduces how to setup and use a ParallelCluster.\nTo setup the packages on the headnode of the ParallelCluster, follow the instructions mentioned here:\n:ref:`Install PyTorch Neuron on Trn1 `.\n\nWe also need to install the ``neuronx-distributed`` package inside the virtual env using the following command:\n\n.. code:: ipython3\n\n python -m pip install neuronx_distributed --extra-index-url https://pip.repos.neuron.amazonaws.com\n git clone git@github.com:aws-neuron/neuronx-distributed.git\n\nLet’s download the scripts for pretraining:\n\n\n1. Navigate to a directory to hold our experiments\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_tp_pp_ptl_setup.sh\n :language: shell\n :lines: 4\n\n2. Link the training scripts for our experiments\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_tp_pp_ptl_setup.sh\n :language: shell\n :lines: 5-10\n\nIf you want to pre-train Llama 7B, you would need to run the following steps -\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_7b.sh\n :language: shell\n :lines: 5-8\n\nIf you want to pre-train Llama 13B, you would need to run the following steps -\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_13b.sh\n :language: shell\n :lines: 5-8\n\nIf you want to pre-train Llama 70B, you would need to run the following steps -\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_70b.sh\n :language: shell\n :lines: 5-8\n\n3. Installing the additional requirements and giving the right permissions to our shell script\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_tp_pp_ptl_setup.sh\n :language: shell\n :lines: 12-13\n\n\nNext, we tokenize our dataset. \n``Note``: To tokenize the data, we must request the tokenizer from `HuggingFace` and `Meta` by following \nthe instructions at the following link: `HuggingFace Llama 2 7B Model `__ .\nUse of the Llama 2 model is governed by the Meta license. In order to download the model weights and tokenizer, please \nvisit the above website and accept their License before requesting access. After access has been granted, \nyou may use the download scripts provided by Meta to download the model weights and tokenizer to your cluster.\n\nOnce you have downloaded the tokenizer and model weights, you can copy the ``tokenizer.model`` to the ``~/examples/llama2_lightning`` directory.\n\nNext let’s download and pre-process the dataset:\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_7b.sh\n :language: shell\n :lines: 13\n\n``Note``: In case you see an error of the following form when downloading data: ``huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/ubuntu/examples/llama2_lightning'. Use `repo_type` argument if needed.`` \nThis could be because of a stale cache. Try deleting the cache using: \n\n.. code:: ipython3\n\n sudo rm -rf /home/ubuntu/.cache/\n\n\nAt this point, you are all set to start training.\n\nTraining Llama2-7B with Tensor Parallelism\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nBy this step, the ParallelCluster is all setup for running experiments. \nBefore we run training, we first pre-compile the graphs using the :ref:`neuron_parallel_compile `.\nLet’s run the command below:\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_7b.sh\n :language: shell\n :lines: 17-20\n\nThis script uses a tensor-parallel size of 8.\nThis will automatically set the zero-1 sharding degree to 16 (4 * 32 workers / tensor_parallel_size). \n\n``Note``: You can use any number of nodes in this case, would just need to adjust the number of nodes in the above \nslurm command accordingly. Also, the number of nodes used in parallel_compile command should be same as the actual \ntraining run. This is because, as the number of nodes change, the data-parallel degree would change too. This would \nresult in more workers participating in operations like `gradient all-reduce` which would result in new graphs getting \ncreated. \n\nOnce the graphs are compiled we can now run training and observe our loss goes down.\nTo run the training, we just run the above command but without ``neuron_parallel_compile``.\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_7b.sh\n :language: shell\n :lines: 22-25\n\nTraining Llama2-13B/70B with Tensor Parallelism and Pipeline Parallelism\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nHere we use ``Llama70B`` as an example. To run 13B, simply change the script from ``run_llama_70b_tp_pp.sh`` to ``run_llama_13B_tp_pp.sh``\nBefore we run training, we first pre-compile the graphs using the :ref:`neuron_parallel_compile `.\nLet’s run the command below:\n\nPre-compiling\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_70b.sh\n :language: shell\n :lines: 17-20\n\nThis script uses a tensor-parallel size of 8, pipeline-parallel size of 8\nTo run the training, we just use the above command but without ``neuron_parallel_compile``.\n\n.. literalinclude:: nxd-source-code/llama_tp_pp_ptl/llama_2_7b.sh\n :language: shell\n :lines: 22-25\n\n\nCheckpointing:\n^^^^^^^^^^^^^^\n\nTo enable checkpoint saving, add following flags to ``run_llama_7b_tp_ptl.sh``/ ``run_llama_13b_tp_pp.sh`` / ``run_llama_70B_tp_pp.sh``:\n* ``--save_checkpoint`` Add this flag to enable checkpoint saving\n* ``--checkpoint_freq`` Number of steps to save a checkpoint\n* ``--checkpoint_dir`` Direction to save the checkpoint \n* ``--num_kept_checkpoint`` Number of checkpoints to save, older checkpoint will be deleted manually, set to -1 to keep all saved checkpoints\n* ``--save_load_xser`` load with torch xla serialization to reduce time saving, it's recommended to enable xser for significantly faster save/load. Note that if the chekpoint is saved with xser, it can only be loaded with xser, vice versa. \n\nTo enable checkpoint loading, add following flags to ``run_llama_7b_tp_ptl.sh``/ ``run_llama_13b_tp_pp.sh`` / ``run_llama_70B_tp_pp.sh``:\n* ``--resume_ckpt`` \n* ``--load_step`` Step to retrieve checkpoint from\n* ``--checkpoint_dir`` Direction to load the checkpoint from\n* ``--save_load_xser`` load with torch xla serialization to reduce time saving, it's recommended to enable xser for significantly faster save/load. Note that if the chekpoint is saved with xser, it can only be loaded with xser, vice versa. \n" }, { "path": "archive/tutorials/tutorial_source_code/t5_finetuning/t5_finetuning_32_worker_training_code.sh", "content": "#!/bin/bash\nset -eExuo pipefail\n\ncd ~/transformers/examples/pytorch/summarization\n\n# Create run 32 worker script\ntee run_32w.sh > /dev/null < /dev/null < /dev/null < /dev/null <> temp_run_summarization.py\nmv temp_run_summarization.py run_summarization.py\nchmod +x run_summarization.py\n\n# Run run summarization to predict without generate\nNEURON_NUM_DEVICES=0 python3 ./run_summarization.py \\\n --model_name_or_path \\\n --dataset_name cnn_dailymail \\\n --dataset_config \"3.0.0\" \\\n --do_predict \\\n --predict_with_generate \\\n --source_prefix \"summarize: \" \\\n --per_device_eval_batch_size 4 \\\n --max_source_length 512 \\\n --pad_to_max_length \\\n --no_cuda \\\n --output_dir /tmp/tst-summarization |& tee log_run" }, { "path": "archive/tutorials/tutorial_source_code/t5_finetuning/t5_modify_run_summarization_code.sh", "content": "#!/bin/bash\nset -eExuo pipefail\n\ncd ~/transformers/examples/pytorch/summarization\n\n# Insert code into run summarization to disable DDP for torchrun\ntee temp_run_summarization.py > /dev/null <> temp_run_summarization.py\nmv temp_run_summarization.py run_summarization.py\nchmod +x run_summarization.py" }, { "path": "audit-report.md", "content": "# Frameworks Audit Report\n\n## Orphaned Pages\n\n| File Path | Type | Reason | Action |\n|---|---|---|---|\n| frameworks/mxnet-neuron/container-sm-hosting-devflow.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/dlc-then-ec2-devflow.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/dlc-then-ecs-devflow.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/env-setup.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/refman.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/rn.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/setup/mxnet-install-prev.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/setup/mxnet-update-al2.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/setup/mxnet-update-u22.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/tutorials/bert_mxnet/index.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/mxnet-neuron/tutorials/index.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/inference.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/container-sm-hosting-devflow.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/dlc-then-k8s-devflow.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/env-setup.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/refman.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/rn.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/setup/index.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-al2.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-update-al2.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/tf1_faq.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/tutorials/yolo_v4_demo/code.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/tutorials/yolo_v4_demo/yolo_v4_demo.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-update.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuronx/tensorflow-neuron-quickstart.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuronx/tensorflow-neuron-supported-operators.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuronx/tutorials/inference/tensorflow-neuronx-serving-tutorial.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/inference.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuron/env-setup.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuron/setup/pytorch-update-al2.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuron/tutorials/index.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/note-setup-general.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.3.0-pytorch-install.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.4.0-pytorch-install.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.5.0-pytorch-install.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.6.0-pytorch-install.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/pytorch-install-prev.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/pytorch-update.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/setup-inference.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/setup/setup-training.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/tutorials/inference/index.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/torch-neuronx/tutorials/training/index.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/torch/training.rst | .rst | Not in any toctree or cross-reference | Delete |\n| frameworks/tensorflow/tensorflow-neuron/tutorials/bert_demo/uncased_L-24_H-1024_A-16.vocab.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/dropdown-neuron-setup.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/tab-inference-torch-neuronx.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/tab-training-torch-neuronx.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/torch-neuronx/api-reference-guide/inference/inference-api-guide-torch-neuronx.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/torch-neuronx/api-reference-guide/training/index.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/torch-neuronx/programming-guide/inference/index.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/torch-neuronx/programming-guide/training/index.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n| frameworks/torch/torch-neuronx/setup/install-templates/pytorch-dev-install.txt | .txt (include fragment) | Not referenced by any .. include:: directive | Delete |\n\n## Stale Pages\n\n| File Path | Staleness Indicators | Recommendation |\n|---|---|---|\n| frameworks/mxnet-neuron/misc-mxnet-neuron.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/mxnet-neuron/misc-mxnet-neuron.txt | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/mxnet-neuron/rn.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/mxnet-neuron/setup/mxnet-install.rst | Amazon Linux 2 | Will be archived |\n| frameworks/mxnet-neuron/setup/mxnet-update.rst | Amazon Linux 2 | Will be archived |\n| frameworks/mxnet-neuron/tutorials/bert_mxnet/index.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/api-compilation-python-api.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/refman.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/rn.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install.rst | Amazon Linux 2 | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-update.rst | Amazon Linux 2 | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/tf1_faq.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/tf2_faq.rst | Ubuntu 18.04 | Will be archived |\n| frameworks/tensorflow/tensorflow-neuron/tutorials/bert_demo/bert_demo.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/tensorflow/tensorflow-neuronx/setup/prev-releases/neuronx-2.8.0-tensorflow-install.rst | Amazon Linux 2 | Will be archived |\n| frameworks/tensorflow/tensorflow-neuronx/setup/prev-releases/neuronx-2.9.0-tensorflow-install.rst | Amazon Linux 2 | Will be archived |\n| frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-neuronx-install.rst | Amazon Linux 2 | Will be archived |\n| frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-update.rst | Amazon Linux 2 | Will be archived |\n| frameworks/torch/dropdown-neuron-setup.txt | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/guide-torch-neuron-vs-torch-neuronx-inference.rst | References deprecated neuron-cc compiler | Update or archive |\n| frameworks/torch/inference-torch-neuron.txt | References deprecated neuron-cc compiler | Update or archive |\n| frameworks/torch/torch-neuron/api-compilation-python-api.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/torch/torch-neuron/misc-inference-torch-neuron.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/torch/torch-neuron/misc-inference-torch-neuron.txt | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/torch/torch-neuron/setup/pytorch-install.rst | Amazon Linux 2 | Will be archived |\n| frameworks/torch/torch-neuron/setup/pytorch-update.rst | Amazon Linux 2 | Will be archived |\n| frameworks/torch/torch-neuron/troubleshooting-guide.rst | References deprecated neuron-cc compiler | Will be archived |\n| frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-trace.rst | References deprecated neuron-cc compiler | Update or archive |\n| frameworks/torch/torch-neuronx/programming-guide/inference/core-placement.rst | References deprecated neuron-cc compiler | Update or archive |\n| frameworks/torch/torch-neuronx/setup-trn1-multi-node-execution.rst | Ubuntu 20.04 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.4.0-pytorch-install.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.6.0-pytorch-install.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.7.0-pytorch-install.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.8.0-pytorch-install.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.9.0-pytorch-install.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/pytorch-install.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/setup/pytorch-update.rst | Amazon Linux 2; torch-neuron setup/update with unsupported OS: Amazon Linux 2 | Update or archive |\n| frameworks/torch/torch-neuronx/training-troubleshooting.rst | Ubuntu 18.04; torch-neuron setup/update with unsupported OS: Ubuntu 18.04 | Update or archive |\n" }, { "path": "build.sh", "content": "#!/bin/bash\n# build.sh - Docker + uv workflow for Neuron docs\n\nset -e\nIMAGE_NAME=\"neuron-docs\"\n\ncase \"${1:-build}\" in\n build)\n docker build -t \"$IMAGE_NAME\" .\n ;;\n html)\n docker run --rm -v \"$(pwd):/docs\" \"$IMAGE_NAME\" -c \"sphinx-build -b html . _build/html -j auto\"\n ;;\n shell)\n docker run --rm -it -v \"$(pwd):/docs\" \"$IMAGE_NAME\"\n ;;\n clean)\n rm -rf _build\n ;;\n *)\n echo \"Usage: $0 {build|html|shell|clean}\"\n exit 1\n ;;\nesac\n" }, { "path": "compiler/error-codes/EARG001.rst", "content": ".. _error-code-earg001:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EARG001.\n\nNCC_EARG001\n===========\n\n**Error message**: This error occurs when you attempt to use a Logical Neuron Core (LNC) configuration that is not supported by the target Neuron architecture.\n\nFor example, a trn1 instance running the following code will run into this error:\n\n.. code-block:: python\n\n traced_model = torch_neuronx.trace(\n model,\n input,\n compiler_args=['--lnc', '2'] # ERROR: lnc=2 not supported on trn1\n )\n\nOn trn1, only lnc=1 is supported.\n\nPhysical Neuron Core:\n\n- Actual hardware compute unit on the chip\n\n- Has dedicated compute resources, memory, etc.\n\nLogical Neuron Core:\n\n- Software abstraction grouping multiple physical cores\n\n- Controlled via the NEURON_LOGICAL_NC_CONFIG environment variable or the --lnc flag (when using neuronx-cc directly)\n\nFor more information: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/device-memory.html#logical-neuron-cores\n" }, { "path": "compiler/error-codes/EBIR023.rst", "content": ".. _error-code-ebir023:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EBIR023.\n\nNCC_EBIR023\n===========\n\n**Error message**: MLP kernel intermediate size exceeds the maximum supported value of 4096.\n\nConsider tiling large intermediate tensors in your kernel to stay within the supported limit, or increase tensor parallelism to shard the intermediate dimension across more cores.\n" }, { "path": "compiler/error-codes/EBVF030.rst", "content": ".. _error-code-ebvf030:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EBVF030.\n\nNCC_EBVF030\n===========\n\n**Error message**: The number of instructions generated exceeds the limit.\n\nConsider applying model parallelism as partitioning the model will help break large computational graphs into smaller subgraphs.\n\nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api_guide.html#api-guide\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/EHCA005.rst", "content": ".. _error-code-ehca005:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EHCA005.\n\nNCC_EHCA005\n===========\n\n**Error message**: The compiler encountered a custom call instruction with a target name that is not recognized.\n\nThe Neuron compiler currently recognizes the following custom call targets:\n\n - AwsNeuronErf\n - AwsNeuronGelu\n - AwsNeuronGeluApprxTanh\n - AwsNeuronGeluBackward\n - AwsNeuronSilu\n - AwsNeuronSiluBackward\n - AwsNeuronRmsNorm\n - AwsNeuronSoftmax\n - AwsNeuronSoftmaxBackward\n - AwsNeuronCollectiveMatmul\n - AwsNeuronIntMatmult\n - AwsNeuronArgMax\n - AwsNeuronArgMin\n - AwsNeuronTopK\n - AwsNeuronDropoutMaskV1\n - AwsNeuronCustomNativeKernel\n - AwsNeuronCustomOp\n - AwsNeuronDevicePrint\n - ResizeNearest\n - ResizeBilinear\n - ResizeNearestGrad\n - AwsNeuronLNCShardingConstraint\n - AwsNeuronTransferWithStaticRing\n - AwsNeuronModuleMarkerStart-Forward\n - AwsNeuronModuleMarkerStart-Backward\n - AwsNeuronModuleMarkerEnd-Forward\n - AwsNeuronModuleMarkerEnd-Backward\n - NeuronBoundaryMarker-Start\n - NeuronBoundaryMarker-End\n\nErroneous code example:\n\n.. code-block:: python\n\n def lowering(ctx, x_val):\n result_type = ir.RankedTensorType(x_val.type)\n # This target name will not be recognized by HandleCustomCall\n return hlo.CustomCallOp(\n [result_type],\n [x_val],\n call_target_name=\"UNRECOGNIZED_TARGET\",\n has_side_effect=ir.BoolAttr.get(False),\n ).results\n\nUse a supported custom call target:\n\n.. code-block:: python\n\n def lowering(ctx, x_val):\n result_type = ir.RankedTensorType(x_val.type)\n return hlo.CustomCallOp(\n [result_type],\n [x_val],\n call_target_name=\"AwsNeuronSilu\",\n has_side_effect=ir.BoolAttr.get(False),\n backend_config=ir.StringAttr.get(\"\"),\n api_version=ir.IntegerAttr.get(ir.IntegerType.get_signless(32), 2),\n ).results\n" }, { "path": "compiler/error-codes/EOOM001.rst", "content": ".. _error-code-eoom001:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EOOM001.\n\nNCC_EOOM001\n===========\n\n**Error message**: The combined memory needed for the model tensors exceeds the high-bandwidth memory limit.\n\nThe memory usage consists of:\n\n- I/O tensors: Input and output activation tensors\n- Internal allocations: Scratchpad memory for intermediate computations\n- SBUF spills: Data that cannot fit in on-chip SBUF memory and must spill to HBM\n\nThere are several ways to potentially fix this issue.\n\n1. Simply reduce the batch/tensor size if possible\n2. Utilize pipeline/tensor parallelism via neuronx-distributed\n\nShort snippet of tensor parallelism:\n\n.. code-block:: python\n\n class ParallelSelfAttention(transformers.models.bert.modeling_bert.BertSelfAttention):\n def __init__(self, config, position_embedding_type=None):\n super().__init__(config, position_embedding_type)\n self.query = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.key = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.value = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n # Since we shard the number of attention heads across tensor parallel\n # ranks, each rank would have a subset of heads, hence, we update\n # the num_attention_heads here.\n tp_size = parallel_state.get_tensor_parallel_size()\n self.num_attention_heads = self.num_attention_heads // tp_size\n self.all_head_size = self.all_head_size // tp_size\n\nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/activation_memory_reduction.html\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/EOOM002.rst", "content": ".. _error-code-eoom002:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EOOM002.\n\nNCC_EOOM002\n===========\n\n**Error message**: The combined memory needed for the model tensors exceeds the high-bandwidth memory limit.\n\nThe memory usage consists of:\n\n- I/O tensors: Input and output activation tensors\n- Internal allocations: Scratchpad memory for intermediate computations\n- SBUF spills: Data that cannot fit in on-chip SBUF memory and must spill to HBM\n\nThere are several ways to potentially fix this issue.\n\n1. Simply reduce the batch/tensor size if possible\n2. Utilize pipeline/tensor parallelism via neuronx-distributed\n\nShort snippet of tensor parallelism:\n\n.. code-block:: python\n\n class ParallelSelfAttention(transformers.models.bert.modeling_bert.BertSelfAttention):\n def __init__(self, config, position_embedding_type=None):\n super().__init__(config, position_embedding_type)\n self.query = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.key = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.value = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n # Since we shard the number of attention heads across tensor parallel\n # ranks, each rank would have a subset of heads, hence, we update\n # the num_attention_heads here.\n tp_size = parallel_state.get_tensor_parallel_size()\n self.num_attention_heads = self.num_attention_heads // tp_size\n self.all_head_size = self.all_head_size // tp_size\n\nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/activation_memory_reduction.html\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/ESFH002.rst", "content": ".. _error-code-esfh002:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error ESFH002.\n\nNCC_ESFH002\n===========\n\n**Error message**: The compiler encountered a unsigned 64-bit integer constant with a value that cannot be safely converted to 32-bit representation. \n\nThe Neuron hardware operates on 32-bit or narrower data types and attempts to convert 64-bit integers to 32-bit. 64-bit constants that exceed the 32-bit range and cannot be safely converted will fail compilation. Try to use uint32 for constants when possible and restructure code to avoid large constants.\n\nErroneous code example:\n\n.. code-block:: python\n\n @jax.jit\n def foo():\n # direct uint64 constant in arithmetic operation\n x = jnp.array([1, 2, 3], dtype=jnp.uint64)\n # large constant that exceeds uint32 max\n large_constant = jnp.uint64(5_000_000_000)\n return x + large_constant\n\nUse uint32 for constants when possible:\n\n.. code-block:: python\n\n @jax.jit\n def test():\n x = jnp.array([1, 2, 3], dtype=jnp.uint32)\n large_constant = jnp.uint32(5_000_000_000)\n return x + large_constant" }, { "path": "compiler/error-codes/ESPP004.rst", "content": ".. _error-code-espp004:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error ESPP004.\n\nNCC_ESPP004\n===========\n\n**Error message**: The compiler encountered a data type that is not supported for code generation.\n\nErroneous code example:\n\n.. code-block:: python\n\n import numpy as np\n import jax.numpy as jnp\n import jax\n from jax._src import dtypes\n from jax._src.lax import lax as lax_internal\n\n # float4_e2m1fn type not supported\n dtype = np.dtype(dtypes.float4_e2m1fn)\n val = lax_internal._convert_element_type(0, dtype, weak_type=False)\n\nUse a supported data type:\n\n.. code-block:: python\n\n import numpy as np\n import jax.numpy as jnp\n import jax\n from jax._src import dtypes\n from jax._src.lax import lax as lax_internal\n\n # float4_e2m1fn type not supported\n dtype = jnp.bfloat16\n val = lax_internal._convert_element_type(0, dtype, weak_type=False)\n\nMore information on supported data types https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/data-types.html\n" }, { "path": "compiler/error-codes/ESPP047.rst", "content": ".. _error-code-espp047:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error ESPP047.\n\nNCC_ESPP047\n===========\n\n**Error message**: The compiler found usage of an unsupported 8-bit floating-point data type.\n\nErroneous code example:\n\n.. code-block:: python\n\n class Model(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear1 = nn.Linear(10, 20)\n self.linear2 = nn.Linear(20, 10)\n\n def forward(self, x):\n x = self.linear1(x)\n x = torch.relu(x)\n x = self.linear2(x)\n return x\n\n # Unsupported 8-bit floating-point data type being used here\n input_tensor = torch.randn(1, 10).to(torch.float8_e4m3fn)\n\n\nTo fix this error:\n\n.. code-block:: python\n\n class Model(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear1 = nn.Linear(10, 20)\n self.linear2 = nn.Linear(20, 10)\n\n def forward(self, x):\n x = self.linear1(x)\n x = torch.relu(x)\n x = self.linear2(x)\n return x\n\n input_tensor = torch.randn(1, 10).to(torch.float8_e4m3fn)\n # Convert to a supported type\n input_tensor = input_tensor.to(torch.float16)\n" }, { "path": "compiler/error-codes/EUOC002.rst", "content": ".. _error-code-euoc002:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EUOC002.\n\nNCC_EUOC002\n===========\n\n**Error message**: An unsupported operator was used.\n\nTry using alternative operators from the full list of supported operators via `neuronx-cc list-operators --framework XLA` to workaround the limitation.\n\nBefore:\n\n.. code-block:: python\n\n class Model(torch.nn.Module):\n def forward(self, A, b):\n return torch.triangular_solve(b, A)\n\nPossible workaround:\n\n.. code-block:: python\n\n class Model(torch.nn.Module):\n def forward(self, A, b):\n # Although slower than triangular_solve, this is mathematically equivalent\n A_inv = torch.inverse(A)\n return A_inv @ b\n" }, { "path": "compiler/error-codes/EVRF001.rst", "content": ".. _error-code-evrf001:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF001.\n\nNCC_EVRF001\n===========\n\n**Error message**: An unsupported operator was used.\n\nTry using alternative operators from the full list of supported operators via `neuronx-cc list-operators --framework XLA` to workaround the limitation.\n\nBefore:\n\n.. code-block:: python\n\n class Model(torch.nn.Module):\n def forward(self, A, b):\n return torch.triangular_solve(b, A)\n\n\nPossible workaround:\n\n.. code-block:: python\n\n class Model(torch.nn.Module):\n def forward(self, A, b):\n # Although slower than triangular_solve, this is mathematically equivalent\n A_inv = torch.inverse(A)\n return A_inv @ b\n" }, { "path": "compiler/error-codes/EVRF004.rst", "content": ".. _error-code-evrf004:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF004.\n\nNCC_EVRF004\n===========\n\n**Error message**: Complex data types are not supported on the Neuron device.\n\nYou cannot use complex data types (such as ``complex64``, ``complex128``, and others) on the Neuron device directly. \n\nOne fix is to offload complex operations to CPU, like so:\n\n.. code-block:: python\n\n x = torch.tensor([1+2j, 3+4j], dtype=torch.complex64).to('cpu')\n\n.. note::\n\n Since data transfer between CPU and device is expensive, this is best used when complex operations are rare.\n\nYou can also address this error by manually emulating complex tensors using real and imaginary parts:\n\n.. code-block:: python\n\n real = x.real\n imag = x.imag\n ...\n # (a + bi) * (c + di)\n real_out = a_real * b_real - a_imag * b_imag\n imag_out = a_real * b_imag + a_imag * b_real\n" }, { "path": "compiler/error-codes/EVRF005.rst", "content": ".. _error-code-evrf005:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF005.\n\nNCC_EVRF005\n===========\n\n**Error message**: The compiler found usage of F8E4M3FNUZ, F8E4M3B11FNUZ, or F8E5M2FNUZ data type which is not supported.\n\nErroneous code example:\n\n.. code-block:: python\n\n class Model(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear1 = nn.Linear(10, 20)\n self.linear2 = nn.Linear(20, 10)\n def forward(self, x):\n x = self.linear1(x)\n x = torch.relu(x)\n x = self.linear2(x)\n return x\n input_tensor = torch.randn(1, 10).to(torch.float8_e4m3fnuz)\n\nTo fix this error:\n\n.. code-block:: python\n \n class Model(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear1 = nn.Linear(10, 20)\n self.linear2 = nn.Linear(20, 10)\n def forward(self, x):\n x = self.linear1(x)\n x = torch.relu(x)\n x = self.linear2(x)\n return x\n input_tensor = torch.randn(1, 10).to(torch.float8_e4m3fnuz)\n # Convert to a supported type\n input_tensor = input_tensor.to(torch.float16)\n\n* More information on supported data types: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/data-types.html\n" }, { "path": "compiler/error-codes/EVRF006.rst", "content": ".. _error-code-evrf006:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF006.\n\nNCC_EVRF006\n===========\n\nThe compiler encountered a RNGBitGenerator operation using a random number generation algorithm other than RNG_DEFAULT.\n-----------------------------------------------------------------------------------------------------------------------\n\nEnsure that you are using standard JAX/PyTorch random APIs and not explicity specifying an RNG algorithm.\n" }, { "path": "compiler/error-codes/EVRF007.rst", "content": ".. _error-code-evrf007:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF007.\n\nNCC_EVRF007\n===========\n\n**Error message**: The number of instructions generated exceeds the limit.\n\nConsider applying model parallelism as partitioning the model will help break large computational graphs into smaller subgraphs.\n\nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api_guide.html#api-guide\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/EVRF009.rst", "content": ".. _error-code-evrf009:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF009.\n\nNCC_EVRF009\n===========\n\n**Error message**: The combined memory needed for the model's activation tensors exceeds the high-bandwidth memory limit.\n\nThere are several ways to potentially fix this issue.\n\n1. Simply reduce the batch/tensor size if possible\n2. Utilize pipeline/tensor parallelism via neuronx-distributed\n\nShort snippet of tensor parallelism:\n\n.. code-block:: python\n\n class ParallelSelfAttention(transformers.models.bert.modeling_bert.BertSelfAttention):\n def __init__(self, config, position_embedding_type=None):\n super().__init__(config, position_embedding_type)\n\n self.query = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.key = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.value = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n # Since we shard the number of attention heads across tensor parallel\n # ranks, each rank would have a subset of heads, hence, we update\n # the num_attention_heads here.\n tp_size = parallel_state.get_tensor_parallel_size()\n self.num_attention_heads = self.num_attention_heads // tp_size\n self.all_head_size = self.all_head_size // tp_size\n\n\nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/activation_memory_reduction.html\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/EVRF010.rst", "content": ".. _error-code-evrf010:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF010.\n\nNCC_EVRF010\n===========\n\n**Error message**: The compiler encountered simultaneous use of input and kernel dilation, which is not supported.\n\nErroneous code example:\n\n.. code-block:: python\n\n x = jnp.ones((1, 4, 4, 1), dtype=jnp.float32)\n kernel = jnp.ones((3, 3, 1, 1), dtype=jnp.float32)\n\n result = lax.conv_general_dilated(\n x,\n kernel,\n window_strides=(1, 1),\n padding=((2, 2), (2, 2)),\n lhs_dilation=(2, 2), # input dilation\n rhs_dilation=(2, 2), # kernel dilation\n dimension_numbers=('NHWC', 'HWIO', 'NHWC')\n )\n\n\nIf possible, use only only input or kernel dilation:\n\n.. code-block:: python\n\n x = jnp.ones((1, 4, 4, 1), dtype=jnp.float32)\n kernel = jnp.ones((3, 3, 1, 1), dtype=jnp.float32)\n\n result = lax.conv_general_dilated(\n x,\n kernel,\n window_strides=(1, 1),\n padding=((2, 2), (2, 2)),\n lhs_dilation=(1, 1), # no input dilation\n rhs_dilation=(2, 2),\n dimension_numbers=('NHWC', 'HWIO', 'NHWC')\n )\n\nOr apply dilation manually and apply convolution to the remainder.\n" }, { "path": "compiler/error-codes/EVRF011.rst", "content": ".. _error-code-evrf011:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF011.\n\nNCC_EVRF011\n===========\n\n**Error message**: The compiler encountered strided convolution combined with dilated input, which is not supported.\n\nErroneous code example:\n\n.. code-block:: python\n\n x = jnp.ones((1, 4, 4, 1), dtype=jnp.float32)\n kernel = jnp.ones((3, 3, 1, 1), dtype=jnp.float32)\n\n result = lax.conv_general_dilated(\n x,\n kernel,\n window_strides=(2, 2), # strided convolution\n padding=((2, 2), (2, 2)),\n lhs_dilation=(2, 2), # and dilated input\n rhs_dilation=(1, 1),\n dimension_numbers=('NHWC', 'HWIO', 'NHWC')\n )\n\n\nIf possible, remove stride or input dilation:\n\n.. code-block:: python\n\n x = jnp.ones((1, 4, 4, 1), dtype=jnp.float32)\n kernel = jnp.ones((3, 3, 1, 1), dtype=jnp.float32)\n\n result = lax.conv_general_dilated(\n x, kernel,\n window_strides=(2, 2), \n padding=((2, 2), (2, 2)),\n lhs_dilation=(1, 1), # remove input dilation\n rhs_dilation=(1, 1),\n dimension_numbers=('NHWC', 'HWIO', 'NHWC')\n )\n\n\nOr apply upsampling and downsampling separately.\n" }, { "path": "compiler/error-codes/EVRF013.rst", "content": ".. _error-code-evrf013:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF013.\n\nNCC_EVRF013\n===========\n\n**Error message**: TopK does not support int32 or int64 input tensors.\n\nErroneous code example:\n\n.. code-block:: python\n\n def forward(self, x):\n # assume x is an integer tensor\n # error: cannot call TopK on integer dtypes\n k = 5\n values, indices = torch.topk(x, k=k, dim=-1)\n return values, indices\n\n\nTo fix this error, you can cast your tensor to a supported floating point dtype.\n\n.. code-block:: python\n\n def forward(self, x):\n x = x.float()\n k = 5\n values, indices = torch.topk(x, k=k, dim=-1)\n return values, indices\n" }, { "path": "compiler/error-codes/EVRF015.rst", "content": ".. _error-code-evrf015:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF015.\n\nNCC_EVRF015\n===========\n\n**Error message**: The compiler encountered a custom call instruction with a target name that is not recognized.\n\nThe Neuron compiler currently recognizes the following custom call targets:\n\n - AwsNeuronErf\n - AwsNeuronGelu\n - AwsNeuronGeluApprxTanh\n - AwsNeuronGeluBackward\n - AwsNeuronSilu\n - AwsNeuronSiluBackward\n - AwsNeuronRmsNorm\n - AwsNeuronSoftmax\n - AwsNeuronSoftmaxBackward\n - AwsNeuronCollectiveMatmul\n - AwsNeuronIntMatmult\n - AwsNeuronArgMax\n - AwsNeuronArgMin\n - AwsNeuronTopK\n - AwsNeuronDropoutMaskV1\n - AwsNeuronCustomNativeKernel\n - AwsNeuronCustomOp\n - AwsNeuronDevicePrint\n - ResizeNearest\n - ResizeBilinear\n - ResizeNearestGrad\n - AwsNeuronLNCShardingConstraint\n - AwsNeuronTransferWithStaticRing\n - AwsNeuronModuleMarkerStart-Forward\n - AwsNeuronModuleMarkerStart-Backward\n - AwsNeuronModuleMarkerEnd-Forward\n - AwsNeuronModuleMarkerEnd-Backward\n - NeuronBoundaryMarker-Start\n - NeuronBoundaryMarker-End\n\nErroneous code example:\n\n.. code-block:: python\n\n def lowering(ctx, x_val):\n result_type = ir.RankedTensorType(x_val.type)\n # This target name will not be recognized by HandleCustomCall\n return hlo.CustomCallOp(\n [result_type],\n [x_val],\n call_target_name=\"UNRECOGNIZED_TARGET\",\n has_side_effect=ir.BoolAttr.get(False),\n ).results\n\n\nUse a supported custom call target:\n\n.. code-block:: python\n\n def lowering(ctx, x_val):\n result_type = ir.RankedTensorType(x_val.type)\n return hlo.CustomCallOp(\n [result_type],\n [x_val],\n call_target_name=\"AwsNeuronSilu\",\n has_side_effect=ir.BoolAttr.get(False),\n backend_config=ir.StringAttr.get(\"\"),\n api_version=ir.IntegerAttr.get(ir.IntegerType.get_signless(32), 2),\n ).results\n\n" }, { "path": "compiler/error-codes/EVRF016.rst", "content": ".. _error-code-evr016:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVR016.\n\nNCC_EVRF016\n===========\n\nThe NCC_EVRF016 error is raised when the Neuron compiler detects that you are trying to use an integer or boolean type with one of the restricted reduction functions.\n\n**Error message**: The scatter-reduce operation cannot perform reduction logic if the data being scattered or the destination tensor is using an integer or boolean data type.\n\nThe hardware instructions used on the Neuron device for these specific scatter-and-reduce functions are optimized for and limited to floating-point arithmetic. When the compiler detects that you are trying to use an integer or boolean type with one of the restricted reduction functions, it stops the compilation process to prevent a hardware crash or incorrect calculation.\n\n**Example of the error**\n\nThe following example shows the **NCC\\_EVRF016** error because the :code:`input_tensor` is defined using an integer data type (:code:`torch.int32`) while being used with a reduction function (:code:`reduce='sum'`) in the :code:`scatter_reduce_` operation.\n\n.. code-block:: python\n\n def forward(self, input_tensor, indices_tensor, src_tensor):\n output = input_tensor.clone()\n \n output.scatter_reduce_(\n dim=1,\n index=indices_tensor,\n src=src_tensor,\n reduce='sum',\n )\n return output\n\n # ERROR: using integer dtype with scatter-reduce\n input_tensor = torch.zeros(BATCH_SIZE, DIM_SIZE, dtype=torch.int32)\n ...\n\n**How to fix**\n\nTo fix this error, you must cast your input and source tensors to a floating-point data type (e.g., torch.float32 or torch.bfloat16).\n\n.. code-block:: python\n\n def forward(self, input_tensor, indices_tensor, src_tensor):\n output = input_tensor.clone()\n \n output.scatter_reduce_(\n dim=1,\n index=indices_tensor, \n src=src_tensor, \n reduce='sum',\n )\n return output\n\n # FIXED: changed to float32\n # now works with scatter-reduce\n input_tensor = torch.zeros(BATCH_SIZE, DIM_SIZE, dtype=torch.float32)\n ...\n" }, { "path": "compiler/error-codes/EVRF017.rst", "content": ".. _error-code-evrf017:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF017.\n\nNCC_EVRF017\n===========\n\n**Error message**: The compiler encountered a reduce-window operation with base dilation (input dilation) greater than 1, which is not supported.\n\nErroneous code example:\n\n.. code-block:: python\n\n result = lax.reduce_window(\n x, -jnp.inf, lax.max,\n window_dimensions=(1, 1, 1, 1),\n window_strides=(1, 1, 1, 1),\n padding='VALID',\n base_dilation=(1, 2, 1, 1) # ERROR: applying base dilation of 2 in dimension 1\n )\n\nIf possible, change base dilation to be all 1s:\n\n.. code-block:: python\n\n result = lax.reduce_window(\n x, -jnp.inf, lax.max,\n window_dimensions=(1, 1, 1, 1),\n window_strides=(1, 1, 1, 1),\n padding='VALID',\n base_dilation=(1, 1, 1, 1) # FIXED: all values are 1 (no dilation)\n )\n\nOr consider manual dilation if necessary.\n" }, { "path": "compiler/error-codes/EVRF018.rst", "content": ".. _error-code-evrf018:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF018.\n\nNCC_EVRF018\n===========\n\n**Error message**: The compiler encountered a reduce-window operation with window dilation greater than 1, which is not supported.\n\nErroneous code example:\n\n.. code-block:: python\n\n result = lax.reduce_window(\n jnp.ones((1, 4, 4, 1)), -jnp.inf, lax.max,\n window_dimensions=(1, 2, 2, 1),\n window_strides=(1, 1, 1, 1),\n padding='VALID',\n window_dilation=(1, 2, 2, 1) # 2 is greater than 1\n )\n\n\nIf possible, remove window_dilation or change values to be all 1s:\n\n.. code-block:: python\n\n result = lax.reduce_window(\n jnp.ones((1, 4, 4, 1)), -jnp.inf, lax.max,\n window_dimensions=(1, 2, 2, 1),\n window_strides=(1, 1, 1, 1),\n padding='VALID',\n window_dilation=(1, 1, 1, 1) \n )\n\nOr consider manual dilation if necessary.\n" }, { "path": "compiler/error-codes/EVRF019.rst", "content": ".. _error-code-evrf019:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF019.\n\nNCC_EVRF019\n===========\n\n**Error message**: The compiler encountered a reduce-window operation with more or less than 2 operands. Support for reduce_window is available for exactly one input tensor and one initial value for reduction.\n\nErroneous code example:\n\n.. code-block:: python\n\n # reduce-window operation with more or less than 2 operands is not supported\n # 4 operands are being provided instead of 2\n lax.reduce_window(\n (x, x), # ERROR: a tuple of two input tensors\n (-jnp.inf, jnp.inf), # ERROR: a tuple of two initial values\n lambda a, b: (jnp.maximum(a[0], b[0]), jnp.minimum(a[1], b[1])),\n window_dimensions=(1, 2, 2, 1),\n window_strides=(1, 2, 2, 1),\n padding='VALID'\n )\n\n\nIf possible, split multi-operand reduce_window with multiple single-operand reduce_window operations.\n\n.. code-block:: python\n\n # For max pooling\n # 2 operands are correctly being provided\n max_pool = lax.reduce_window(\n x, # FIXED: a single input tensor\n -jnp.inf, # FIXED: a single initial value\n lax.max,\n window_dimensions=(1, 2, 2, 1),\n window_strides=(1, 2, 2, 1),\n padding='VALID'\n )\n \n # For min pooling\n # 2 operands are correctly being provided\n min_pool = lax.reduce_window(\n x, # FIXED: a single input tensor\n jnp.inf, # FIXED: a single initial value\n lax.min,\n window_dimensions=(1, 2, 2, 1),\n window_strides=(1, 2, 2, 1),\n padding='VALID'\n )\n" }, { "path": "compiler/error-codes/EVRF022.rst", "content": ".. _error-code-evrf022:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF022.\n\nNCC_EVRF022\n===========\n\n**Error message**: Shift-right-arithmetic operation on non 32-bit inputs is not supported. Cast the first argument's data type to be S32, U32, or F32.\n\nErroneous code example:\n\n.. code-block:: python\n\n def forward(self, input, other):\n return torch.bitwise_right_shift(input, other)\n \n # This will be the first argument and must be 32-bit\n input = torch.tensor([16, 32, 64], dtype=torch.int16)\n # The second argument can be non 32-bit\n other = torch.tensor([1, 2, 3], dtype=torch.int16)\n\n\nTo fix this error:\n\n.. code-block:: python\n\n def forward(self, input, other):\n return torch.bitwise_right_shift(input, other)\n\n # Correctly setting the first argument to be 32-bit\n input = torch.tensor([16, 32, 64], dtype=torch.int32)\n other = torch.tensor([1, 2, 3], dtype=torch.int16)\n" }, { "path": "compiler/error-codes/EVRF031.rst", "content": ".. _error-code-evrf031:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EVRF031.\n\nNCC_EVRF031\n===========\n\n**Error message**: The compiler encountered a scatter out-of-bounds error. The indices created via iota instruction contain values that are beyond the size of the operand dimension.\n\nErroneous code example:\n\n.. code-block:: python\n\n # size 3 in dimension 0\n operand = jnp.zeros((3, 4), dtype=jnp.float32)\n\n # iota generates indices [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n indices = lax.iota(jnp.int32, 10) # ERROR: size 10 > operand dimension 3\n indices = indices.reshape(10, 1)\n\n updates = jnp.ones((10, 4), dtype=jnp.float32) # ERROR: 10 updates but operand only has 3 rows\n\n result = lax.scatter(\n operand,\n indices, # ERROR: index values in [0, 10) but operand dimension only allows indices in [0, 3)\n updates,\n lax.ScatterDimensionNumbers(\n update_window_dims=(1,),\n inserted_window_dims=(0,),\n scatter_dims_to_operand_dims=(0,)\n )\n )\n\n\nEnsure that the iota size matches the operand dimension size:\n\n.. code-block:: python\n\n N = 3\n D = 4\n operand = jnp.zeros((N, D), dtype=jnp.float32)\n\n # FIXED: match iota size to operand dimension\n indices = lax.iota(jnp.int32, N) # size N is same as operand dimension\n indices = indices.reshape(N, 1)\n\n # FIXED: updates size matches operand dimension\n updates = jnp.ones((N, D), dtype=jnp.float32)\n\n result = lax.scatter(\n operand,\n indices, # FIXED: indices now in valid range [0, 3)\n updates,\n lax.ScatterDimensionNumbers(\n update_window_dims=(1,),\n inserted_window_dims=(0,),\n scatter_dims_to_operand_dims=(0,)\n )\n )\n" }, { "path": "compiler/error-codes/EXSP001.rst", "content": ".. _error-code-exsp001:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EXSP001.\n\nNCC_EXSP001\n===========\n\nThe combined memory needed for the model's activation tensors exceeds the high-bandwidth memory limit. \n------------------------------------------------------------------------------------------------------\n\nThere are several ways to potentially fix this issue.\n\n1. Simply reduce the batch/tensor size if possible\n2. Utilize pipeline/tensor parallelism via neuronx-distributed\n\nShort snippet of tensor parallelism:\n\n.. code-block:: python\n\n class ParallelSelfAttention(transformers.models.bert.modeling_bert.BertSelfAttention):\n def __init__(self, config, position_embedding_type=None):\n super().__init__(config, position_embedding_type)\n\n self.query = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.key = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n self.value = ColumnParallelLinear(config.hidden_size,\n self.all_head_size,\n gather_output=False)\n # Since we shard the number of attention heads across tensor parallel\n # ranks, each rank would have a subset of heads, hence, we update\n # the num_attention_heads here.\n tp_size = parallel_state.get_tensor_parallel_size()\n self.num_attention_heads = self.num_attention_heads // tp_size\n self.all_head_size = self.all_head_size // tp_size\n\n\nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/activation_memory_reduction.html\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/EXTP004.rst", "content": ".. _error-code-extp004:\n\n.. meta::\n :description: AWS Neuron SDK Graph Compiler error code documentation for error EXTP004.\n\nNCC_EXTP004\n===========\n\n**Error message**: The number of instructions generated exceeds the limit.\n\nConsider applying model parallelism as partitioning the model will help break large computational graphs into smaller subgraphs.\n \nFor more information: \n\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api_guide.html#api-guide\n- https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n" }, { "path": "compiler/error-codes/index.rst", "content": ".. meta::\n :description: \"Neuron Compiler error code documentation home.\"\n :date-modified: 12/02/2025\n\n.. _ncc-errors-home:\n\nNeuron Compiler Error Codes\n============================\n\nThis page lists the error codes you can encounter while developing with the Neuron Compiler. For more details on any individual error, click the link for that error code in the table below.\n\n.. list-table::\n :header-rows: 1\n\n * - Error Code\n - Error Message\n - Recommendation\n * - :ref:`NCC_EARG001 `\n - Unsupported Logical Neuron Core (LNC) configuration.\n - You attempted to use a Logical Neuron Core configuration that is not supported by the target Neuron architecture.\n * - :ref:`NCC_EBIR023 `\n - MLP kernel intermediate size exceeds the maximum supported value of 4096.\n - Consider tiling large intermediate tensors in your kernel to stay within the supported limit, or increase tensor parallelism to shard the intermediate dimension across more cores.\n * - :ref:`NCC_EBVF030 `\n - The number of instructions generated exceeds the limit.\n - Consider applying model parallelism as partitioning the model will help break large computational graphs into smaller subgraphs.\n * - :ref:`NCC_EHCA005 `\n - The compiler encountered a custom call instruction with a target name that is not recognized.\n - Use a supported custom call target from the list of recognized targets.\n * - :ref:`NCC_EOOM001 `\n - The combined memory needed for the model's activation tensors exceeds the high-bandwidth memory limit.\n - You may need to reduce batch/tensor size or utilize pipeline/tensor parallelism via neuronx-distributed.\n * - :ref:`NCC_EOOM002 `\n - The combined memory needed for the model's activation tensors exceeds the high-bandwidth memory limit.\n - You may need to reduce batch/tensor size or utilize pipeline/tensor parallelism via neuronx-distributed.\n * - :ref:`NCC_ESFH002 `\n - The compiler encountered a unsigned 64-bit integer constant with a value that cannot be safely converted to 32-bit representation.\n - Try to use uint32 for constants when possible and restructure code to avoid large constants.\n * - :ref:`NCC_ESPP004 `\n - The compiler encountered a data type that is not supported for code generation.\n - Use a supported data type as listed in the Neuron documentation.\n * - :ref:`NCC_ESPP047 `\n - Unsupported 8-bit floating-point data type.\n - The compiler found usage of an unsupported 8-bit floating-point data type. Convert to a supported type like torch.float16.\n * - :ref:`NCC_EUOC002 `\n - An unsupported operator was used.\n - Try using alternative operators from the full list of supported operators via `neuronx-cc list-operators --framework XLA` to workaround the limitation.\n * - :ref:`NCC_EVRF001 `\n - An unsupported operator was used.\n - Try using alternative operators from the full list of supported operators to workaround the limitation.\n * - :ref:`NCC_EVRF004 `\n - Complex data types are not supported on the Neuron device.\n - You cannot use complex data types (such as ``complex64``, ``complex128``, and others) on the Neuron device directly.\n * - :ref:`NCC_EVRF005 `\n - Unsupported F8E4M3FNUZ, F8E4M3B11FNUZ, or F8E5M2FNUZ data type.\n - The compiler found usage of unsupported 8-bit floating-point data types. Convert to a supported type like torch.float16.\n * - :ref:`NCC_EVRF006 `\n - The compiler encountered a RNGBitGenerator operation using a random number generation algorithm other than RNG_DEFAULT.\n - Ensure that you are using standard JAX/PyTorch random APIs and not explicity specifying an RNG algorithm.\n * - :ref:`NCC_EVRF007 `\n - The number of instructions generated exceeds the limit.\n - Consider applying model parallelism as partitioning the model will help break large computational graphs into smaller subgraphs.\n * - :ref:`NCC_EVRF009 `\n - The combined memory needed for the model's activation tensors exceeds the high-bandwidth memory limit.\n - You may need to reduce batch/tensor size or utilize pipeline/tensor parallelism via neuronx-distributed.\n * - :ref:`NCC_EVRF010 `\n - The compiler encountered simultaneous use of input and kernel dilation, which is not supported.\n - If possible, use only input or kernel dilation, not both simultaneously.\n * - :ref:`NCC_EVRF011 `\n - The compiler encountered strided convolution combined with dilated input, which is not supported.\n - If possible, remove stride or input dilation, or apply upsampling and downsampling separately.\n * - :ref:`NCC_EVRF013 `\n - TopK does not support integer input tensors (int32, int64).\n - The TopK operation cannot be performed on integer data types.\n * - :ref:`NCC_EVRF015 `\n - The compiler encountered a custom call instruction with a target name that is not recognized.\n - Use a supported custom call target from the list of recognized targets.\n * - :ref:`NCC_EVRF016 `\n - The scatter-reduce operation cannot perform reduction logic if the data being scattered or the destination tensor is using an integer or boolean data type.\n - Cast your input and source tensors to a floating-point data type (e.g., torch.float32 or torch.bfloat16).\n * - :ref:`NCC_EVRF017 `\n - Reduce-window operation with base dilation greater than 1 is not supported.\n - Change base dilation to be all 1s or consider manual dilation if necessary.\n * - :ref:`NCC_EVRF018 `\n - Reduce-window operation with window dilation greater than 1 is not supported.\n - Remove window_dilation or change values to be all 1s, or consider manual dilation if necessary.\n * - :ref:`NCC_EVRF019 `\n - The compiler encountered a reduce-window operation with more or less than 2 operands.\n - If possible, split multi-operand reduce_window with multiple single-operand reduce_window operations.\n * - :ref:`NCC_EVRF022 `\n - Shift-right-arithmetic operation on non 32-bit inputs is not supported. Cast the first argument's data type to be S32, U32, or F32.\n - You need to use 32-bit data types for shift operations. Cast inputs to int32, uint32, or float32.\n - Reduce batch/tensor size or utilize tensor parallelism via neuronx-distributed.\n * - :ref:`NCC_EVRF031 `\n - The compiler encountered a scatter out-of-bounds error.\n - Ensure that the iota size matches the operand dimension size.\n * - :ref:`NCC_EXSP001 `\n - The combined memory needed for the model's activation tensors exceeds the high-bandwidth memory limit.\n - You may need to reduce batch/tensor size or utilize pipeline/tensor parallelism via neuronx-distributed.\n * - :ref:`NCC_EXTP004 `\n - The number of instructions generated exceeds the limit.\n - Consider applying model parallelism as partitioning the model will help break large computational graphs into smaller subgraphs.\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n\n EARG001\n EBIR023\n EBVF030\n EHCA005\n EOOM001\n EOOM002\n ESFH002\n ESPP004\n ESPP047\n EUOC002\n EVRF001\n EVRF004\n EVRF005\n EVRF006\n EVRF007\n EVRF009\n EVRF010\n EVRF011\n EVRF013\n EVRF015\n EVRF016\n EVRF017\n EVRF018\n EVRF019\n EVRF022\n EVRF031\n EXSP001\n EXTP004\n" }, { "path": "compiler/index.rst", "content": ".. _neuron_cc:\n\nNeuron Graph Compiler\n======================\n\nThe Neuron Graph Compiler is a sophisticated compilation system that transforms Machine Learning models from various frameworks (TensorFlow, MXNet, PyTorch, XLA HLO) into highly optimized code for AWS Neuron accelerators. It performs deep analysis of model structure, applies hardware-specific optimizations, and generates executable code tailored for maximum performance on Neuron hardware.\n\nThe Neuron compiler is available in two versions to support different AWS ML accelerator architectures:\n \n* **neuronx-cc**: The newer XLA-based compiler supporting NeuronCores v2 architecture (Trn1, Inf2, Trn1n, Trn2). This compiler leverages the XLA (Accelerated Linear Algebra) framework to provide advanced optimizations for modern ML workloads.\n* **neuron-cc**: The TVM-based compiler supporting NeuronCores v1 architecture (Inf1). This compiler uses the TVM (Tensor Virtual Machine) framework as its foundation.\n\nKey capabilities of the Neuron Graph Compiler include:\n\n* **Performance optimization**: Intelligently converts FP32 operations to more efficient formats (BF16/FP16/TF32/FP8) with configurable precision-performance tradeoffs. By default, the compiler automatically casts FP32 matrix multiplication operations to BF16 for optimal performance while maintaining accuracy.\n\n* **Model-specific optimizations**: Provides specialized optimizations for different model architectures:\n * **Generic**: Applies general optimizations suitable for all model types\n * **Transformer**: Implements specific optimizations for transformer-based architectures like BERT, GPT, and other attention-based models\n * **U-Net**: Applies specialized memory optimizations for U-Net architectures to prevent performance-impacting data transfers\n\n* **Distributed training support**: Enables efficient large language model (LLM) training through distribution strategies that shard parameters, gradients, and optimizer states across data-parallel workers.\n\n* **Advanced memory management**: Optimizes memory usage for large models through techniques like model sharding across multiple NeuronCores, with configurable logical NeuronCore settings to control sharding degree.\n\n* **Optimization levels**: Provides multiple optimization levels (1-3) to balance compilation time against runtime performance, allowing users to choose the appropriate tradeoff for their workflow.\n\n* **Mixed precision support**: Offers fine-grained control over precision and performance through auto-casting options, supporting multiple numeric formats (FP32, TF32, FP16, BF16, FP8) with different strengths in dynamic range and numeric precision.\n\nThe compilation process is typically transparent to users, as the compiler is invoked automatically within ML frameworks through Neuron Framework plugins. Models are analyzed, optimized, and compiled into a NEFF file (Neuron Executable File Format), which is then loaded by the :doc:`Neuron Runtime ` for execution on Neuron devices.\n\n.. grid:: 1 \n :gutter: 3\n\n .. grid-item-card:: Neuron Graph Compiler Component Release Notes\n :link: /release-notes/components/compiler\n :link-type: doc\n\n Review the Neuron Graph Compiler release notes for all versions of the Neuron SDK.\n\n.. tab-set::\n\n .. tab-item:: Neuron Graph Compiler (neuronx-cc) for Trn1 & Inf2\n\n .. grid:: 1 \n :gutter: 3\n\n .. grid-item-card:: CLI Reference Guide\n :link: neuron-compiler-cli-reference-guide\n :link-type: ref\n\n Neuron Compiler CLI Reference Guide\n\n .. grid-item-card:: Graph Compiler Developer Guide\n :link: neuronx-cc-training-mixed-precision\n :link-type: ref\n\n Mixed precision training guide\n\n .. grid-item-card:: Graph Compiler Error Code Reference\n :link: ncc-errors-home\n :link-type: ref\n\n Error code reference\n\n .. grid-item-card:: How To Convolute Kernels in UNet Training Models\n :link: implement-convolution-kernels-unet\n :link-type: ref\n\n Learn how to modify UNet training models to use convolution kernels with the AWS Neuron SDK. \n\n .. grid-item-card:: Graph Compiler FAQ\n :link: neuronx_compiler_faq\n :link-type: ref\n\n Frequently asked questions\n\n\n .. tab-item:: Neuron Graph Compiler (neuron-cc) for Inf1\n\n .. grid:: 1 \n :gutter: 3\n\n .. grid-item-card:: Graph Compiler API Reference Guide\n :link: neuron-compiler-cli-reference\n :link-type: ref\n\n Neuron Compiler CLI Reference\n\n .. grid-item-card:: Graph Compiler Developer Guide\n :link: neuron-cc-training-mixed-precision\n :link-type: ref\n\n Mixed precision training guide\n\n .. grid-item-card:: Graph Compiler FAQ\n :link: neuron_compiler_faq\n :link-type: ref\n\n Frequently asked questions\n\n\n.. toctree::\n :maxdepth: 2\n :hidden:\n\n /compiler/neuronx-cc\n /compiler/neuron-cc\n Error Codes \n Release Notes \n" }, { "path": "compiler/neuron-cc/api-reference-guide.rst", "content": "API Reference Guide\n===================\n\n.. toctree::\n :maxdepth: 1\n\n /compiler/neuron-cc/command-line-reference" }, { "path": "compiler/neuron-cc/command-line-reference.rst", "content": ".. _neuron-compiler-cli-reference:\n\nNeuron compiler CLI Reference Guide (``neuron-cc``)\n===================================================\n\nThis document describes the command line interface of the Neuron\ncompiler. This reference is not relevant for applications that run\nneuron-cc from within a machine learning framework (TensorFlow-Neuron\nfor example) since these options are passed from the framework directly\nto neuron-cc.\n\nUsing neuron-cc on the command line may be desirable for applications\nthat do not use a framework, or customize existing frameworks. It is\nalso possible to supply CLI commands to the framework as options to be\npassed through to the compiler.\n\nUsage\n--------\n\nOptional parameters are shown in square brackets. See the individual\nframework guides for the correct syntax.\n\n.. _neuron_cli:\n\n.. rubric:: Neuron Compiler CLI\n\n.. program:: neuron-cc\n\n.. option:: neuron-cc [options] [parameters]\n\nCommon options for the Neuron CLI:\n\n - :option:`--verbose` (string) default=“WARN”:\n\n Valid values:\n\n - :option:`DEBUG`\n - :option:`INFO`\n - :option:`WARN`\n - :option:`ERROR`\n\n\n\nUse :option:`neuron-cc --help` for information on a specific command.\n\nAvailable Commands:\n~~~~~~~~~~~~~~~~~~~\n\n- :option:`compile`\n- :option:`list-operators`\n\n\n.. option:: neuron-cc compile [parameters]\n\n Compile a model for use on the AWS Inferentia Machine Learning Accelerator.\n\n .. code-block::\n\n neuron-cc compile --framework --io-config [--neuroncore-pipeline-cores ] [--enable-saturate-infinity] [--enable-fast-loading-neuron-binaries] [--enable-fast-context-switch] [--fp32-cast cast-method] [--fast-math cast-method] [--output ]\n\n **Compile Parameters:**\n\n - :option:``: Input containing model specification. The number\n of arguments required varies between frameworks:\n\n - **TENSORFLOW**: A local filename or URI of a TensorFlow Frozen\n GraphDef (.pb); or the name of a local directory containing a\n TensorFlow SavedModel.\n\n See\n https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/graph.proto\n for the associated .proto schema for TensorFlow Frozen GraphDefs. See\n https://www.tensorflow.org/guide/saved_model for more information on\n the SavedModel format.\n\n - **MXNET**: List of local filenames or URIs where input architecture\n .json file and parameter .param file are stored. These contains\n information related to the architecture of your graph and associated\n parameters, respectively.\n\n\n - :option:`--framework` (string): Framework in which the model was trained.\n\n Valid values:\n\n - :option:`TENSORFLOW`\n - :option:`MXNET`\n - :option:`XLA`\n\n - :option:`--neuroncore-pipeline-cores` (int) (default=1): Number of neuron cores\n to be used in \"NeuronCore Pipeline\" mode. This is different from data\n parallel deployment (same model on multiple neuron cores). Refer to\n Runtime/Framework documentation for data parallel deployment options.\n\n Compile for the given number of\n neuron cores so as to leverage NeuronCore Pipeline mode.\n\n .. note::\n This is not used to define the number of Neuron Cores to be used in a data\n parallel deployment (ie the same model on multiple Neuron Cores). That\n is a runtime/framework configuration choice.\n\n - :option:`--output` (string) (default=“out.neff”): Filename where compilation\n output (NEFF archive) will be recorded.\n\n - :option:`--io-config` (string): Configuration containing the names and shapes\n of input and output tensors.\n\n The io-config can be specified as a local filename, a URI, or a string\n containing the io-config itself.\n\n The io-config must be formatted as a JSON object with two members\n “inputs” and “outputs”. “inputs” is an object mapping input tensor names\n to an array of shape and data type. “outputs” is an array of output\n tensor names. Consider the following example:\n\n .. code-block:: json\n\n {\n \"inputs\": {\n \"input0:0\": [[1,100,100,3], \"float16\"],\n \"input1:0\": [[1,100,100,3], \"float16\"]\n },\n \"outputs\": [\"output:0\"]\n }\n\n - :option:`--enable-saturate-infinity` : Convert +/- infinity values to MAX/MIN_FLOAT for certain computations that have a high risk of generating Not-a-Number (NaN) values. There is a potential performance impact during model execution when this conversion is enabled.\n\n\n - :option:`--enable-fast-loading-neuron-binaries` : Write the compilation\n output (NEFF archive) in uncompressed format which results\n in faster loading of the archive during inference.\n\n - :option:`--enable-fast-context-switch` : Optimize for faster model switching\n rather than inference latency. This results in overall faster system\n performance when your application switches between models frequently\n on the same neuron core (or set of cores). The optimization\n triggered by this option for example defers loading some weight\n constants until the start of inference.\n\n - :option:`--fast-math` : Controls tradeoff between performance and accuracy for fp32 operators. See more suggestions on how to use this option with the below arguments in :ref:`neuron-cc-training-mixed-precision`.\n\n\n - ``all`` (Default): enables all optimizations that improve performance. This option can potentially lower precision/accuracy.\n\n - ``none`` : Disables all optimizations that improve performance. This option will provide best precision/accuracy.\n\n - Tensor transpose options\n\n - ``fast-relayout``: Only enables fast relayout optimization to improve performance by using the matrix multiplier for tensor transpose. The data type used for the transpose is either FP16 or BF16, which is controlled by the ``fp32-cast-xxx`` keyword.\n\n - ``no-fast-relayout``: Disables fast relayout optimization which ensures that tensor transpose is bit-accurate (lossless) but slightly slower.\n\n\n - Casting options\n\n - ``fp32-cast-all`` (Default): Cast all FP32 operators to BF16 to achieve highest performance and preserve dynamic range. Same as setting ``--fp32-cast all``.\n\n - ``fp32-cast-all-fp16``: Cast all FP32 operators to FP16 to achieve speed up and increase precision versus BF16. Same setting as ``--fp32-cast all-fp16``.\n\n - ``fp32-cast-matmult``: Only cast FP32 operators that use Neuron Matmult engine to BF16 while using FP16 for matmult-based transpose to get better accuracy. Same as setting ``--fp32-cast matmult``.\n\n - ``fp32-cast-matmult-bf16``: Cast only FP32 operators that use Neuron Matmult engine (including matmult-based transpose) to BF16 to preserve dynamic range. Same as setting ``--fp32-cast matmult-bf16``.\n\n - ``fp32-cast-matmult-fp16``: Cast only FP32 operators that use Neuron Matmult engine (including matmult-based transpose) to fp16 to better preserve precision. Same as setting ``--fp32-cast matmult-fp16``.\n\n\n\n .. important ::\n\n * ``all`` and ``none`` are mutually exclusive\n\n * ``all`` is equivalent to using ``fp32-cast-all fast-relayout`` (best performance)\n\n * ``none`` is equivalent to using ``fp32-cast-matmult-bf16 no-fast-relayout`` (best accuracy)\n\n * ``fp32-cast-*`` options are mutually exclusive\n\n * ``fast-relayout`` and ``no-fast-relayout`` are mutually exclusive\n\n * The ``fp32-cast-*`` and ``*-fast-relayout`` options will overwrite the default behavior in ``all`` and ``none``.\n\n * For backward compatibility, the ``--fp32-cast`` option has higher priority over ``--fast-math``. It will overwrite the FP32 casting options in any of the ``--fast-math`` options if ``--fp32-cast`` option is present explicitly.\n\n\n - :option:`--fp32-cast` : Refine the automatic casting of fp32 tensors. This is being replaced by a newer --fast-math.\n\n .. important ::\n\n * ``--fp32-cast`` option is being deprecated and ``--fast-math`` will replace it in future releases.\n\n * ``--fast-math`` is introducing the ``no-fast-relayout`` option to enable lossless transpose operation.\n\n\n The ``--fp32-cast`` is an interface for controlling the performance and accuracy tradeoffs. Many of the ``--fast-math`` values invoke (override) it.\n\n - ``all`` (default): Cast all FP32 operators to BF16 to achieve speed up and preserve dynamic range.\n\n - ``matmult``: Cast only FP32 operators that use Neuron Matmult engine to BF16 while using fp16 for matmult-based transpose to get better accuracy.\n\n - ``matmult-fp16``: Cast only FP32 operators that use Neuron Matmult engine (including matmult-based transpose) to fp16 to better preserve precision.\n\n - ``matmult-bf16``: Cast only FP32 operators that use Neuron Matmult engine (including matmult-based transpose) to BF16 to preserve dynamic range.\n\n - ``all-fp16``: Cast all FP32 operators to FP16 to achieve speed up and better preserve precision.\n\n\n\n\n **Log Levels:**\n\n Logs at levels “trace”, “debug”, and “info” will be written to STDOUT.\n\n Logs at levels “warn”, “error”, and “fatal” will be written to STDERR.\n\n **Exit Status**\n\n **0** - Compilation succeeded\n\n **>0** - An error occurred during compilation.\n\n **Examples**\n\n\n Compiling a saved TensorFlow model:\n\n .. code-block:: shell\n\n neuron-cc compile test_graph_tfmatmul.pb --framework TENSORFLOW --io-config test_graph_tfmatmul.config\n\n Compiling a MXNet model:\n\n .. code-block:: shell\n\n neuron-cc compile lenet-symbol.json lenet-0001.params --framework MXNET --neuroncore-pipeline-cores 2 --output file.neff\n\n Compiling an XLA HLO:\n\n .. code-block:: shell\n\n neuron-cc compile bert-model.hlo --framework XLA --output file.neff\n\n.. _neuron-cc-list-operators:\n\n.. option:: neuron-cc list-operators [parameters]\n\n .. _description-1:\n\n Returns a newline ('n') separated list of operators supported by the NeuronCore.\n\n - **TENSORFLOW**: Operators will be formatted according to the value\n passed to the associated REGISTER_OP(“OperatorName”) macro.\n\n See https://www.tensorflow.org/guide/create_op#define_the_op_interface\n for more information regarding operator registration in TensorFlow.\n\n - **MXNET**: Operator names will be formatted according to the value\n passed to the associated NNVM_REGISTER_OP(operator_name) macro.\n\n - **XLA**: Operator names will be formatted according to the value used by XLA compiler in XlaBuilder.\n\n See https://www.tensorflow.org/xla/operation_semantics for more information regarding XLA operator semantics in XLA interface.\n\n .. code-block:: shell\n\n neuron-cc list-operators --framework \n\n .. _options-1:\n\n - :option:`--framework` (string): Framework in which the operators were\n registered.\n\n Valid values:\n\n - :option:`TENSORFLOW`\n - :option:`MXNET`\n - :option:`XLA`\n\n **Exit Status**\n\n **0** - Call succeeded\n\n **>0** - An error occurred\n\n **Example**\n\n .. code-block:: shell\n\n $ neuron-cc list-operators --framework TENSORFLOW\n AddN\n AdjustContrastv2\n CheckNumbers\n ...\n" }, { "path": "compiler/neuron-cc/developer-guide.rst", "content": "Developer Guide\n===================\n\n.. toctree::\n :maxdepth: 1\n \n /about-neuron/appnotes/neuron-cc/mixed-precision" }, { "path": "compiler/neuron-cc/faq.rst", "content": ".. _neuron_compiler_faq:\n\nNeuron Compiler FAQ (``neuron-cc``)\n===================================\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nWhere can I compile to Neuron?\n---------------------------------\n\nThe one-time compilation step from the standard framework-level model to\nNEFF binary may be performed on any EC2 instance or even\non-premises.\n\nWe recommend using a high-performance compute server of choice (C5 or\nz1d instance types), for the fastest compile times and ease of use with\na prebuilt `DLAMI `__.\nDevelopers can also install Neuron in their own environments; this\napproach may work well for example when building a large fleet for\ninference, allowing the model creation, training and compilation to be\ndone in the training fleet, with the NEFF files being distributed by a\nconfiguration management application to the inference fleet.\n\nMy current Neural Network is based on FP32, how can I use it with Neuron?\n-------------------------------------------------------------------------\n\nDevelopers who want to train their models in FP32 for best accuracy can\ncompile and deploy them with Neuron. The Neuron compiler automatically converts\nFP32 to internally supported datatypes, such as FP16 or BF16.\nYou can find more details about FP32 data type support\nand performance and accuracy tuning\nin :ref:`neuron-cc-training-mixed-precision`.\nThe Neuron compiler preserves the application interface - FP32 inputs and outputs.\nTransferring such large tensors may become a bottleneck for your application.\nTherefore, you can improve execution time by casting the inputs and outputs to\nFP16 or BF16 in the ML framework prior to compilation for Inferentia.\n\nWhat are some of the important compiler defaults I should be aware of?\n-----------------------------------------------------------------------\n\nThe compiler compiles the input graph for a single NeuronCore by default. Using the :option:`--neuroncore-pipeline-cores` option directs the compiler to\npartition so as to run on a specified number of NeuronCores. This number can\nbe less than the total available NeuronCores on an instance.\nSee :ref:`inferentia-arch` for more information on NeuronCores.\n\nWhich operators does Neuron support?\n---------------------------------------\n\nsee :ref:`neuron-supported-operators`.\n\nYou can also use the \"neuron-cc list-operators\" command on the cli to list the\noperators. See :ref:`neuron-cc-list-operators`\n\nIf your model contains operators missing from the above list, and you can't reach your performance goals, please\npost a message on the Neuron developer forum or open a github issue to let us know.\n\nAny operators that Neuron doesn't support?\n---------------------------------------------\n\nModels with control-flow and dynamic shapes are not supported. You will\nneed to partition the model using the framework prior to compilation.\nSee the :ref:`neuron-cc`.\n\nWill I need to recompile again if I updated runtime/driver version?\n----------------------------------------------------------------------\n\nThe compiler and runtime are committed to maintaining compatibility for\nmajor version releases with each other. The versioning is defined as\nmajor.minor, with compatibility for all versions with the same major\nnumber. If the versions mismatch, an error notification is logged and\nthe load will fail. This will then require the model to be recompiled.\n\nI have a NEFF binary, how can I tell which compiler version\n-----------------------------------------------------------\ngenerated it?** We will bring a utility out to help with this soon.\n\nHow long does it take to compile?\n------------------------------------\n\nIt depends on the model and its size and complexity, but this generally\ntakes a few minutes.\n" }, { "path": "compiler/neuron-cc.rst", "content": ".. _neuron-cc-index:\n\nNeuron Compiler for Inf1\n========================\n\n.. toctree::\n :maxdepth: 1\n\n API Reference Guide \n CLI Reference \n Developer Guide \n FAQ " }, { "path": "compiler/neuronx-cc/api-reference-guide/index.rst", "content": ".. _neuron-compiler-cli-reference-guide:\n\nNeuron Compiler CLI Reference Guide (``neuronx-cc``)\n====================================================\n\nThis document describes the command line interface of the Neuron Compiler.\n\nThis reference is not relevant for applications that run the Neuron Compiler from within a machine learning framework (:ref:`PyTorch-Neuron ` for example) since these options are passed from the framework directly to the compiler. Using the compiler command line may be desirable for applications that do not use a framework or customize existing frameworks. It is also possible to specify compiler options within the framework which will forward these options to the compiler using :ref:`NEURON_CC_FLAGS `.\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\nUsage\n-----\n\n*Optional parameters are shown in square brackets.*\n\n.. _neuron_cli:\n\n.. rubric:: Neuron Compiler Command-Line Interface\n\n.. program:: neuronx-cc\n\n.. option:: neuronx-cc [parameters]\n\nAvailable Commands\n------------------\n\n- ``compile``\n- ``list-operators``\n\nCommon parameters for the Neuron CLI:\n\n- ``--help``: Display a usage message of compiler options.\n Use ``neuronx-cc --help`` for information on a specific command.\n\n\n.. _neuronx-cc-compile:\n\n'compile' Command\n-----------------\n\n.. option:: neuronx-cc compile [parameters]\n\n .. _description-1:\n\n Compile a model for use on the AWS Machine Learning Accelerator.\n\n\n .. code-block:: shell\n\n neuronx-cc compile \n --framework \n --target \n [--model-type ]\n [--auto-cast ]\n [--auto-cast-type ]\n [--distribution-strategy ]\n [--logical-nc-config ], or [-lnc ]\n [--optlevel ], or [-O ]\n [--enable-mixed-precision-accumulation]\n [--enable-saturate-infinity]\n [--enable-fast-context-switch]\n [--enable-fast-loading-neuron-binaries]\n [--logfile ]\n [--output ]\n [--verbose ]\n\n\n Parameters\n ~~~~~~~~~~\n\n - ````: Input containing model specification.\n The number of arguments required varies between frameworks:\n\n - **XLA**: A local filename of a HLO file (hlo.pb) generated via XLA. See `hlo.proto `_ for the .proto description and `inspect-compiled-programs `_ for more information on how to generate such files.\n\n - ``--framework ``: Framework used to generate training model.\n\n Valid values:\n\n - ``XLA``\n\n - ``--target ``: Name of the Neuron instance family on which the compiled model will be run.\n\n Valid values:\n\n - ``inf2``\n - ``trn1``\n - ``trn1n``\n - ``trn2``\n\n - ``--model-type ``: Permit the compiler to attempt model-specific optimizations based upon type of model being compiled. (Default: ``generic``)\n\n Valid values:\n\n - ``generic``: Perform optimizations applicable to all types of inference and training models.\n - ``transformer``: Perform optimizations specific to `Transformer `_ models. \n - ``unet-inference``: Perform optimizations specific to certain `U-Net `_ model architectures when performing inference. U-Net models often have certain structures that result in excessive performance-impacting data transfers; this option allows the compiler to apply additional memory optimizations to prevent these data transfers and also allows the compiler to map larger normalization operators which would otherwise not successfully execute.\n\n - ``--auto-cast ``: Controls how the compiler makes tradeoffs between performance and accuracy for FP32 operations. (Default: ``none``)\n\n Valid values:\n\n - ``none``: (default) Leave all data types as defined in the model. Do not apply auto-casting data type optimizations.\n - ``matmult``: Only cast FP32 operations that use the Neuron matrix-multiplication engine.\n - ``all``: Cast all FP32 operations to achieve highest performance. This option can potentially lower precision/accuracy.\n\n A more complete discussion on how to use this option and its arguments is in :ref:`Mixed Precision and Performance-accuracy Tuning for Training `.\n\n .. note:: If the ``--auto-cast`` option is specified, the ``--auto-cast-type`` compiler flag can be optionally set to define which lower-precision data type the compiler should use.\n\n - ``--auto-cast-type ``: When auto-cast mode is enabled, cast the FP32 operators to the lower-precision data type specified by this option. (Default: ``bf16``)\n\n Valid values:\n\n - ``bf16``: Cast the FP32 operations selected via the ``--auto-cast`` option to BF16 to achieve highest performance and preserve dynamic range.\n - ``fp16``: Cast the FP32 operations selected via the ``--auto-cast`` option to FP16 to achieve improved performance relative to FP32 and increased precision relative to BF16.\n - ``tf32``: Cast the FP32 operations selected via the ``--auto-cast`` option to TensorFloat-32.\n - ``fp8_e4m3``: Cast the FP32 operations selected via the ``--auto-cast`` option to a signed 8-bit floating point represented as a 4-bit exponent and 3-bit mantissa. \n\n\n .. note:: If multiple competing options are specified then the option right-most on the command line will supercede previous options.\n\n - ``--distribution-strategy ``: Permit the compiler to attempt model-specific optimizations based upon type of model being compiled. (Default: ``generic``)\n\n Valid values:\n\n - ``llm-training``: Enable the compiler to perform optimizations applicable to large language model (LLMS) training runs that shard parameters, gradients, and optimizer states across data-parallel workers. This is equivalent to the previously documented option argument value of ``NEMO``, which will be deprecated in a future release.\n\n - ``--logical-nc-config ``: Instructs the compiler to shard the input graph across physical NeuronCore accelerators. Possible numeric values are {1, 2}. (Only available on trn2; Default: ``2``)\n\n Valid values:\n\n - ``1``: instructs the compiler to shard the input graph across 1 physical NeuronCore, i.e., do not perform any input graph sharding.\n - ``2``: [default on trn2] instructs the compiler to shard the input graph across 2 physical NeuronCores.\n\n - ``--optlevel ``: Specify the level of optimization the compiler should perform. Possible numeric values are {1, 2, 3}. (Default: ``2``)\n\n Valid values:\n\n - ``1``: enables the core performance optimizations in the compiler, while also minimizing compile time.\n - ``2``: [default] provides the best balance between model performance and compile time.\n - ``3``: may provide additional model execution performance but may incur longer compile times and higher host memory usage during model compilation.\n\n .. note:: This option supercedes, and deprecates, the ``—enable-experimental-O1`` option introduced in an earlier release.\n\n - ``--enable-mixed-precision-accumulation``: **Enabled by default**. Set to ``true`` by default. Perform intermediate calculations of accumulation operators (such as softmax and layernorm) in FP32 and cast the result to the model-designated datatype. This improves the operator's resulting accuracy.\n\n - ``--disable-mixed-precision-accumulation``: Disables mixed precision accumulation. Mixed precision accumulation is enabled by default; use this flag to disable it. Disabling mixed precision accumulation may improve performance at the cost of reduced accuracy for certain operators.\n\n - ``--enable-saturate-infinity``: Convert +/- infinity values to MAX/MIN_FLOAT for compiler-introduced matrix-multiply transpose computations that have a high risk of generating Not-a-Number (NaN) values. There is a potential performance impact during model execution when this conversion is enabled. (Only needed on trn1; while the trn2 compiler will accept this flag for compatibility reasons, it has no effect on the compilation.)\n\n - ``--enable-fast-context-switch``: Optimize for faster model switching rather than execution latency.\n This option will defer loading some weight constants until the start of model execution. This results in overall faster system performance when your application switches between models frequently on the same Neuron Core (or set of cores).\n\n - ``--enable-fast-loading-neuron-binaries``: Save the compilation output file in an uncompressed format.\n This creates executable files which are larger in size but faster for the Neuron Runtime to load into memory during model execution.\n\n - ``--logfile ``: Filename where compiler writes log messages. (Default: “log-neuron-cc.txt”).\n\n - ``--output ``: Filename where compilation output (NEFF archive) will be recorded. (Default: \"file.neff”)\n\n - ``--verbose ``: Specify the level of output produced by the compiler. (Default: ``warning``)\n\n Valid values:\n\n - ``info``: Informational messages regarding the progress of model compilation (written to stdout).\n - ``warning``: Diagnostic messages that report model code that is not inherently erroneous but may be risky or suggest there may have been an error (written to stderr).\n - ``error``: The compiler detected a condition causing it not complete the compilation successfully (written to stderr).\n - ``critical``: The compiler encountered an unrecoverable error terminates immediately (written to stderr).\n - ``debug``: Extensive information regarding the compiler's internal execution phases (written to stdout).\n\n *Example*:\n Compiling an XLA HLO:\n\n .. code-block:: shell\n\n neuronx-cc compile bert-model.hlo —-framework XLA -—target trn1 —-model-type transformer —-output bert.neff\n\n\n.. _neuronx-cc-list-operators:\n\n'list-operators' Command\n------------------------\n\n.. option:: neuronx-cc list-operators [parameters]\n\n .. _description-1:\n\n Returns a newline (‘\\\\n’) separated list of operators supported by the Neuron Compiler.\n\n .. code-block:: shell\n\n neuronx-cc list-operators\n --framework \n\n Parameters\n ~~~~~~~~~~\n\n - ``--framework ``: Framework in which the operators were registered.\n\n Valid values:\n\n - ``XLA``: Operator names will be formatted according to the value used by XLA compiler in XlaBuilder.\n\n\n *Example*:\n\n .. code-block:: shell\n\n neuronx-cc list-operators —framework XLA\n ...\n\n\nCompiler Exit Statuses\n----------------------\n\n- **0**: Compilation succeeded\n- **<>0**: An error occurred during compilation.\n" }, { "path": "compiler/neuronx-cc/developer-guide.rst", "content": ".. meta::\n :description: Developer guides for the Neuron Compiler (neuronx-cc), including mixed precision training, performance tuning, and custom kernel implementation for AWS Trainium and Inferentia.\n :keywords: neuronx-cc, Neuron Compiler, mixed precision, BF16, FP16, TF32, auto-cast, convolution kernels, UNet, performance optimization, Trainium, Inferentia\n\nDeveloper Guide\n===================\n\nLearn how to optimize your models with the Neuron Compiler (neuronx-cc). These guides cover mixed precision training, performance-accuracy tuning, and custom kernel implementations for AWS Trainium and Inferentia instances.\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Mixed Precision and Performance-Accuracy Tuning\n :link: /about-neuron/appnotes/neuronx-cc/neuronx-cc-training-mixed-precision\n :link-type: doc\n\n Learn how to use FP32, TF32, FP16, and BF16 data types with the Neuron Compiler's auto-cast options to balance performance and accuracy. Understand the tradeoffs between different data types and how to configure compiler settings for optimal model execution.\n\n .. grid-item-card:: How to Use Convolution Kernels in UNet Training Models\n :link: /compiler/neuronx-cc/how-to-convolution-in-unet\n :link-type: doc\n\n Modify UNet training models to use custom convolution kernels with NKI (Neuron Kernel Interface). This implementation helps avoid out-of-memory errors when training convolution-heavy models on Trainium instances.\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n \n /about-neuron/appnotes/neuronx-cc/neuronx-cc-training-mixed-precision\n /compiler/neuronx-cc/how-to-convolution-in-unet" }, { "path": "compiler/neuronx-cc/faq.rst", "content": ".. _neuronx_compiler_faq:\n\nNeuron Compiler FAQ (``neuronx-cc``)\n====================================\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nWhere can I compile to Neuron?\n---------------------------------\n\nThe one-time compilation step from the standard framework-level model to\nNEFF binary may be performed on any EC2 instance or even\non-premises.\n\nWe recommend using a high-performance compute server of choice (C5 or\nz1d instance types), for the fastest compile times and ease of use with\na prebuilt `DLAMI `__.\nDevelopers can also install Neuron in their own environments; this\napproach may work well for example when building a large fleet for\ninference, allowing the model creation, training and compilation to be\ndone in the training fleet, with the NEFF files being distributed by a\nconfiguration management application to the inference fleet.\n\n.. _neuron-vs-neuronx:\n\nWhat is the difference between ``neuron-cc`` and ``neuronx-cc``?\n----------------------------------------------------------------\n\n* ``neuron-cc`` is the Neuron Compiler with TVM front-end, ``neuron-cc`` supports only :ref:`neuroncores-v1-arch`.\n* ``neuronx-cc`` is the Neuron Compiler with XLA front-end, ``neuronx-cc`` currently supports \n :ref:`neuroncores-v2-arch`, ``neuronx-cc`` support of :ref:`neuroncores-v1-arch` is currently a \n :ref:`Roadmap Item `.\n\nShould I use ``neuron-cc`` or ``neuronx-cc``?\n---------------------------------------------\n\nSee :ref:`neuron-vs-neuronx`\n\nMy current neural network is based on FP32, how can I use it with Neuron?\n-------------------------------------------------------------------------\n\nDevelopers who want to train their models in FP32 for best accuracy can\ncompile and deploy them with Neuron. The Neuron compiler automatically converts\nFP32 to internally supported datatypes, such as FP16 or BF16.\nYou can find more details about FP32 data type support\nand performance and accuracy tuning\nin :ref:`neuronx-cc-training-mixed-precision` or :ref:`neuron-cc-training-mixed-precision`.\nThe Neuron compiler preserves the application interface - FP32 inputs and outputs.\nTransferring such large tensors may become a bottleneck for your application.\nTherefore, you can improve execution time by casting the inputs and outputs to\nFP16 or BF16 in the ML framework prior to compilation.\n\nWhich operators does Neuron support?\n---------------------------------------\n\nYou can use the ``neuronx-cc list-operators`` command on the cli to list the operators. See :ref:`neuron-compiler-cli-reference-guide`.\n\nTo request support for new operators, open an issue on our `GitHub forum `_.\n\nAny operators that Neuron Compiler doesn't support?\n---------------------------------------------------\n\nModels with control-flow and dynamic shapes are not supported now. You will\nneed to partition the model using the framework prior to compilation.\n\n.. note::\n\n Starting with :ref:`neuroncores-v2-arch` Neuron supports control-flow and dynamic shapes.\n\n Stay tuned and follow the :ref:`Neuron Roadmap `.\n\nWill I need to recompile again if I updated runtime/driver version?\n----------------------------------------------------------------------\n\nThe compiler and runtime are committed to maintaining compatibility for\nmajor version releases with each other. The versioning is defined as\nmajor.minor, with compatibility for all versions with the same major\nnumber. If the versions mismatch, an error notification is logged and\nthe load will fail. This will then require the model to be recompiled.\n\nI have a NEFF binary, how can I tell which compiler version generated it?\n-------------------------------------------------------------------------\n ** We will bring a utility out to help with this soon.\n\nHow long does it take to compile?\n------------------------------------\n\nIt depends on the model and its size and complexity, but this generally\ntakes a few minutes.\n\nWhy is my model producing different results compared to CPU/GPU?\n----------------------------------------------------------------\n\n:ref:`neuroncores-v2-arch` supports multiple casting modes for floating point numbers, each with\nassociated implications for performance and accuracy. The default casting mode\nis a pragmatic balance between performance and accuracy, however on some models\nit may result in loss of precision.\n\nSee the :option:`--auto-cast` and :option:`--auto-cast-type` options in :ref:`neuron-compiler-cli-reference-guide` for details on how to adjust the casting mode.\n\nDo you support model **?\n-------------------------------------------\n\n``neuronx-cc`` has explicit support for select model families using the :option:`--model-type` option, though many other model types are supported. You can also inspect supported operators using the :option:`list-operators` sub-command. See th :ref:`neuron-compiler-cli-reference-guide` for details.\nMore generally, support for new operators and models is continually being added. See our :ref:`neuron_roadmap` for details.\n" }, { "path": "compiler/neuronx-cc/how-to-convolution-in-unet.rst", "content": ".. meta::\n :description: Learn how to modify UNet training models to use convolution kernels with AWS Neuron SDK\n :date_updated: 2025-09-09\n\n.. _implement-convolution-kernels-unet:\n\n=======================================================\nHow to Use Convolution Kernels in UNet Training Models\n=======================================================\n\nTask overview\n-------------\nThis topic discusses how to modify UNet training models to use convolution kernels with the AWS Neuron SDK. This implementation helps avoid out-of-memory errors seen when performing training on the convolution-heavy UNet model.\n\nPrerequisites\n-------------\n- AWS Neuron SDK 2.26 or later: Required for kernel implementation support\n- trn1.32xlarge instance: Needed for model training \n- Existing UNet implementation: Base model to be modified\n- PyTorch-Neuron environment: Required for neural network operations\n\nInstructions\n------------\n\n**1: Import required dependencies**\n\n.. code-block:: python\n\n import torch\n import torch.nn as nn\n import torch.nn.functional as F\n from torch.autograd import Function\n import neuronxcc.nki as nki\n import neuronxcc.nki.language as nl\n from neuronxcc.nki._private_kernels.conv import conv2d_dw_fb01_io01_01bf_rep_nhwc_Pcinh\n\n**2: Create the convolution wrapper function**\n\n.. code-block:: python\n\n @nki.jit\n def conv_wrap(img_ref, filter_ref, out_shape):\n out_arr = nl.ndarray(shape=out_shape, dtype=img_ref.dtype, buffer=nl.hbm)\n conv2d_dw_fb01_io01_01bf_rep_nhwc_Pcinh(img_ref, filter_ref, out_arr, **{\n 'input': img_ref.shape,\n 'filter': filter_ref.shape, \n 'output': out_shape,\n 'in_perm': [0, 1, 2, 3],\n 'kern_perm': [0, 1, 2, 3],\n 'out_perm': [0, 1, 2, 3],\n 'stride': (1, 1),\n 'padding': ((1, 1), (1, 1))})\n return out_arr\n\n**3: Implement the custom Conv2d module**\n\n.. code-block:: python\n\n class BwdConv2dWithKernel(nn.Module):\n def __init__(self, in_channels, out_channels, kernel_size, padding, bias):\n super().__init__()\n assert padding == 1\n assert bias == False\n self.in_channels = in_channels\n self.out_channels = out_channels\n self.kernel_size = kernel_size\n self.weight = nn.Parameter(torch.randn(out_channels, in_channels, kernel_size, kernel_size))\n nn.init.kaiming_uniform_(self.weight, a=0.0, mode='fan_in', nonlinearity='leaky_relu')\n\n**4: Replace standard convolutions in the UNet model**\n\n.. code-block:: python\n\n class DoubleConvWithKernel(nn.Module):\n def __init__(self, in_channels, out_channels, mid_channels=None):\n super().__init__()\n if not mid_channels:\n mid_channels = out_channels\n self.double_conv = nn.Sequential(\n BwdConv2dWithKernel(in_channels, mid_channels, kernel_size=3, padding=1, bias=False),\n nn.BatchNorm2d(mid_channels),\n nn.ReLU(inplace=True),\n BwdConv2dWithKernel(mid_channels, out_channels, kernel_size=3, padding=1, bias=False),\n nn.BatchNorm2d(out_channels),\n nn.ReLU(inplace=True)\n )\n\n**5: Update the UNet model initialization**\n\n.. code-block:: python\n\n def __init__(self, n_channels, n_classes, bilinear=False):\n super().__init__()\n self.n_channels = n_channels\n self.n_classes = n_classes\n self.bilinear = bilinear\n self.inc = (DoubleConvWithKernel(n_channels, 64))\n # ... rest of initialization\n\nConfirm your work\n-----------------\n\nTo confirm successful implementation, verify the following:\n\n.. code-block:: bash\n\n Expected training output\n Training Device=xla:0 Epoch=1 Step=20 Loss=0.30803\n Training Device=xla:0 Epoch=2 Step=560 Loss=0.01826\n\nCheck for:\n\n- No out-of-memory errors during execution\n- Decreasing loss values across epochs\n\nCommon issues\n-------------\n\n.. rubric:: Memory Errors\n\n- Solution: Verify all standard convolutions are replaced with BwdConv2dWithKernel implementations\n\n.. rubric:: Compilation Errors\n\n- Solution: Confirm Neuron SDK version is 2.26 or later\n\n.. rubric:: Kernel Errors\n\n- Solution: Use the kernel for supported configurations. The kernel will error out in unsupported scenarios.\n\nRelated information\n-------------------\n\n.. toctree::\n :maxdepth: 1\n\n * `UNet training sample `_ - Sample UNet training implementation\n" }, { "path": "compiler/neuronx-cc.rst", "content": ".. _neuronx-cc-index:\n\nNeuronX Compiler for Trn1 & Inf2\n=================================\n\n.. toctree::\n :maxdepth: 1\n\n API Reference Guide \n How-to: Convolution \n Developer Guide \n FAQ " }, { "path": "conf.py", "content": "# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common options. For a full\n# list see the documentation:\n# https://www.sphinx-doc.org/en/master/usage/configuration.html\n\n# -- Path setup --------------------------------------------------------------\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n\nimport datetime\nimport os\nimport sys\n\nsys.path.append(os.path.abspath(\"./_ext\"))\nsys.path.append(os.path.abspath(\"./nki/api\"))\nsys.path.append(os.path.abspath(\"./nki/_ext\"))\nsys.path.append(os.path.abspath(\"./frameworks/torch/torch-neuron/\"))\nsys.path.append(os.path.abspath(\"./_static\"))\n\n\n# get environment variables\ndef get_env_vars_from_gh():\n project_name = os.environ.get(\"GIT_PROJECT_NAME\", \"aws-neuron-sdk\")\n branch_name = os.environ.get(\"GIT_BRANCH_NAME\", \"master\")\n branch_name = \"master\" if branch_name == \"latest\" else branch_name\n\n return project_name, branch_name\n\n\ndef get_env_vars_from_rtd():\n branch_name = os.environ.get(\"READTHEDOCS_VERSION_NAME\", \"master\")\n branch_name = \"master\" if branch_name == \"latest\" else branch_name\n\n project_name = \"aws-neuron-sdk\"\n if os.environ.get(\"READTHEDOCS_PROJECT\") == \"awsdocs-neuron-staging\":\n project_name = \"private-aws-neuron-sdk-staging\"\n\n return project_name, branch_name\n\n\ndef get_env_vars():\n \"\"\"Configure project and branch names based on environment\"\"\"\n if os.environ.get(\"READTHEDOCS\") == \"True\":\n return get_env_vars_from_rtd()\n return get_env_vars_from_gh()\n\n\nproject_name, branch_name = get_env_vars()\n# -- Project information -----------------------------------------------------\n\nproject = \"AWS Neuron\"\ncopyright = \"{}, Amazon.com\".format(datetime.datetime.now().year)\nauthor = \"AWS\"\nmaster_doc = \"index\"\nhtml_title = \"AWS Neuron Documentation\"\n\n# -- General configuration ---------------------------------------------------\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\n \"sphinxcontrib.contentui\",\n \"nbsphinx\",\n \"sphinx.ext.extlinks\",\n \"sphinx.ext.intersphinx\",\n \"sphinx_plotly_directive\",\n \"df_tables\",\n \"sphinxcontrib.programoutput\",\n \"neuron_tag\",\n \"sphinx_design\",\n \"ablog\",\n \"sphinx.ext.viewcode\",\n \"sphinx.ext.napoleon\",\n \"sphinx.ext.autodoc\",\n \"sphinx.ext.autosummary\",\n \"local_documenter\",\n \"archive\",\n \"sphinx_copybutton\",\n \"nki_directives\",\n \"sphinxcontrib.googleanalytics\",\n \"sphinxcontrib.datatemplates\",\n \"sphinxcontrib.spelling\",\n \"sphinx_tabs.tabs\",\n]\n\n\nhtml_sidebars = {\n \"**\": [\n \"navbar-logo.html\",\n \"search-field.html\",\n \"sbt-sidebar-nav.html\",\n ],\n \"about-neuron/announcements/*\": [\n \"navbar-logo.html\",\n \"search-field.html\",\n \"ablog/postcard.html\",\n \"ablog/recentposts.html\",\n \"ablog/tagcloud.html\",\n \"ablog/categories.html\",\n \"ablog/archives.html\",\n \"sbt-sidebar-nav.html\",\n ],\n}\n\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = [\n \"_templates\",\n \"nki/_templates/\",\n \"_content-types/\",\n \"libraries/nxd-inference/_templates\",\n]\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path.\nexclude_patterns = ['_build', '_backup-rn', '_backup-setup', '_content-types','**.ipynb_checkpoints','.venv','_utilities', 'nki/_templates']\nhtml_extra_path = ['static']\n\n# remove bash/python/ipython/jupyter prompts and continuations\ncopybutton_prompt_text = r\">>> |\\.\\.\\. |\\$ |In \\[\\d*\\]: | {2,5}\\.\\.\\.: | {5,8}: \"\ncopybutton_prompt_is_regexp = True\n\n# nbsphinx_allow_errors = True\nnbsphinx_execute = \"never\"\n\nhtml_logo = \"images/Site-Merch_Neuron-ML-SDK_Editorial.png\"\n\nnapoleon_google_docstring = True\n\n# Turn on figure/table numbering\nnumfig = True\n\n# -- autodoc/autosummary options -------------------------------------------------\n\nautosummary_generate = True # Turn on sphinx.ext.autosummary\n\n\n# -- more options -------------------------------------------------\n\n\nprojectblob = project_name + \"/blob/\" + branch_name\nprojecttree = project_name + \"/tree/\" + branch_name\n\nextlinks = {\n \"mxnet-neuron\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/neuron-guide/neuron-frameworks/mxnet-neuron/%s\",\n \"\",\n ),\n \"pytorch-neuron\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/neuron-guide/neuron-frameworks/pytorch-neuron/%s\",\n \"\",\n ),\n \"tensorflow-neuron\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/neuron-guide/neuron-frameworks/tensorflow-neuron/%s\",\n \"\",\n ),\n \"neuron-deploy\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/neuron-deploy/%s\",\n \"\",\n ),\n \"neuron-tools-tree\": (\n \"https://github.com/aws-neuron/\" + projecttree + \"/neuron-guide/neuron-tools/%s\",\n \"\",\n ),\n \"mxnet-neuron-src\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/src/examples/mxnet/%s\",\n \"\",\n ),\n \"pytorch-neuron-src\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/src/examples/pytorch/%s\",\n \"\",\n ),\n \"tensorflow-neuron-src\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/src/examples/tensorflow/%s\",\n \"\",\n ),\n \"neuron-gatherinfor-src\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/src/examples/neuron-gatherinfo/%s\",\n \"\",\n ),\n \"neuron-monitor-src\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/src/examples/neuron-monitor/%s\",\n \"\",\n ),\n \"compile-pt\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/archive/src/benchmark/pytorch/%s_compile.py\",\n \"\",\n ),\n \"benchmark-pt\": (\n \"https://github.com/aws-neuron/\" + projectblob + \"/archive/src/benchmark/pytorch/%s_benchmark.py\",\n \"\",\n ),\n \"llama-sample\": (\n \"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/transformers-neuronx/inference/%s.ipynb\",\n \"\",\n ),\n 'github':(f'https://github.com/aws-neuron/{project_name}/blob/{branch_name}/%s', '')\n}\n\n\nintersphinx_mapping = {\n \"python\": (\"https://docs.python.org/3\", None),\n \"numpy\": (\"https://numpy.org/doc/stable/\", None),\n \"torch\": (\"https://pytorch.org/docs/master/\", None),\n \"transformers\": (\"https://huggingface.co/docs/transformers/master/en/\", None),\n}\n\n# -- Options for Theme -------------------------------------------------\n\ntop_banner_message = \"Neuron 2.29.0 is released! Check the What's New and Release Notes for more details.\"\n\nhtml_theme = \"sphinx_book_theme\"\nhtml_theme_options = {\n \"repository_url\": \"https://github.com/aws-neuron/\" + project_name,\n \"use_issues_button\": True,\n \"use_repository_button\": True,\n \"use_download_button\": True,\n \"use_fullscreen_button\": True,\n \"use_edit_page_button\": True,\n \"home_page_in_toc\": False,\n \"repository_branch\": branch_name,\n \"announcement\": top_banner_message,\n # \"navbar_persistent\": [],\n}\n\nhtml_additional_pages = {\n \"search-google\": \"search-google.html\",\n}\n\nhtml_context = {\n # ...\n \"default_mode\": \"light\"\n}\n\n# The theme to use for HTML and HTML Help pages. See the documentation for\n# a list of builtin themes.\n#\n# html_theme = 'sphinx_rtd_theme'\n\n# html_theme_options = {\n#\n# 'navigation_depth': 3\n# }\n\n\n# html_theme = \"pydata_sphinx_theme\"\n# html_theme_options = {\n# \"use_edit_page_button\": True,\n# }\n\n# html_context = {\n# \"github_url\": \"https://github.com\",\n# \"github_user\": \"aws-neuron\",\n# \"github_repo\": \"private-aws-neuron-sdk-staging\",\n# \"github_version\": \"master\",\n# \"doc_path\": \"/\",\n# }\n\n# -- Options for HTML output -------------------------------------------------\n\nhtml_css_files = [\"css/custom.css\", \"styles/sphinx-book-theme.css\"]\n\n# def setup(app):\n# app.add_css_file('css/custom.css')\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = [\"_static\"]\n\nplotly_include_source = False\nplotly_html_show_source_link = False\nplotly_html_show_formats = False\nplotly_include_directive_source = False\n\n\n# -- ABlog config -------------------------------------------------\nblog_path = \"about-neuron/announcements/index\"\nblog_post_pattern = \"about-neuron/appnotes/*.rst\"\nblog_feed_length = 5\nfontawesome_included = True\npost_show_prev_next = False\npost_auto_image = 1\npost_auto_excerpt = 2\nexecution_show_tb = \"READTHEDOCS\" in os.environ\n\n# --- Google Analytics Sphinx extension ---\n\ngoogleanalytics_id = \"G-2Q13EGB80H\"\n\n# --- for neuron-tag directive ---\n\nrst_prolog = \"\"\"\n\n.. neuron-tag::\n\n\n\"\"\"\n\nrst_epilog = \"\"\"\n\n.. neuron-tag::\n\n\"\"\"\n\n# Exclude private github from linkcheck. Readthedocs only exposes the ssh-agent to the 'checkout' build step, which is too early for the linkchecker to run.\nlinkcheck_ignore = [\n r\"http://localhost:\\d+/\",\n r\"https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/dlami-pytorch-introduce.html\",\n r\"https://github\\.com/aws-neuron/private-aws-neuron-sdk-staging/\",\n r\"https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/dlami-pytorch-introduce.html\",\n r\"https://awsdocs-neuron-staging.readthedocs-hosted.com/en/latest/frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-neuronx-install.html#install-tensorflow-neuronx\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/tree/master/torch-neuronx#inference\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/tree/master/torch-neuronx#training\",\n r\"https://github.com/aws/deep-learning-containers/blob/master/available_images.md#neuron-containers\",\n r\"https://github.com/aws-neuron/aws-neuron-sagemaker-samples/tree/master/inference/inf2-bert-on-sagemaker\",\n r\"https://github.com/awslabs/multi-model-server/blob/master/docs/management_api.md\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/training/dp_bert_hf_pretrain/run_dp_bert_large_hf_pretrain_bf16_s128.sh\",\n r\" https://github.com/pytorch/xla/blob/master/test/test_train_mp_mnist.py\",\n r\"https://github.com/pytorch/xla/blob/v1.10.0/TROUBLESHOOTING.md\",\n r\"https://github.com/tensorflow/docs/blob/master/site/en/r1/guide/saved_model.md\",\n r\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/xla/g3doc/index.md\",\n r\"https://github.com/pytorch/xla/blob/master/test/test_train_mp_mnist.py\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/transformers-neuronx/inference/meta-llama-2-13b-sampling.ipynb\",\n r\"https://github.com/aws-neuron/aws-neuron-sdk/blob/master/src/examples/pytorch/torch-neuronx/t5-inference-tutorial.ipynb\",\n r\"https://github.com/aws-neuron/aws-neuron-parallelcluster-samples/blob/master/examples/jobs/neuronx-nemo-megatron-llamav2-job.md\",\n r\"https://github.com/pytorch/PiPPy/blob/main/pippy/IR.py#L697\",\n r\"https://github.com/pytorch/pytorch/blob/main/torch/fx/_symbolic_trace.py#L241\",\n r\"https://github.com/pytorch/xla/blob/master/torch_xla/utils/checkpoint.py#L129\",\n r\"https://github.com/aws-neuron/neuronx-distributed/blob/main/src/neuronx_distributed/parallel_layers/layer_norm.py#L32\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain.py#L273C1-L289C55\",\n r\"https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install.html#pytorch-neuronx-install\",\n r\"https://github.com/google-research/bert#user-content-pre-trained-models\",\n r\"https://github.com/google-research/bert#user-content-sentence-and-sentence-pair-classification-tasks\",\n r\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html\",\n r\"https://repost.aws/knowledge-center/eventbridge-notification-scheduled-events\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain/modeling_gpt_neox_nxd.py\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/training/tp_dp_gpt_neox_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain/tp_dp_gpt_neox_20b_hf_pretrain.py\",\n r\"https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/transformers-neuronx/inference/llama-3-8b-32k-sampling.ipynb\",\n]\nlinkcheck_exclude_documents = [\n r\"src/examples/.*\",\n \"about-neuron/announcements/neuron1.x/announcements\",\n r\"release-notes/.*\",\n r\"containers/.*\",\n]\nnitpicky = False\n\n" }, { "path": "containers/container-deployment-flows.rst", "content": ".. _container-deployment-flows:\n\nContainer Deployment Flows\n==========================\n\nYou can also choose one of the following combinations for running the neuron container:\n\n.. toctree::\n :maxdepth: 1\n \n dlc-then-ec2-devflow\n dlc-then-ecs-devflow\n dlc-then-eks-devflow\n container-sm-hosting-devflow\n" }, { "path": "containers/container-sm-hosting-devflow.rst", "content": ".. _containers-byoc-hosting-devflow:\n\n.. include:: /devflows/inference/byoc-hosting-devflow.rst\n" }, { "path": "containers/developerflows.rst", "content": "Containers - Developer Flows\n============================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /containers/dlc-then-ec2-devflow\n /containers/dlc-then-ecs-devflow\n /containers/dlc-then-eks-devflow\n /containers/container-sm-hosting-devflow\n /containers/dlc-then-customize-devflow\n\n\n\n.. include:: /containers/developerflows.txt\n" }, { "path": "containers/developerflows.txt", "content": ".. tab-set:: \n\n .. tab-item:: Inference\n \n * :ref:`containers-dlc-then-ec2-devflow`\n * :ref:`containers-dlc-then-ecs-devflow`\n * :ref:`containers-dlc-then-eks-devflow`\n * :ref:`containers-byoc-hosting-devflow`\n * :ref:`containers-dlc-then-customize-devflow`\n" }, { "path": "containers/dlc-then-customize-devflow.rst", "content": ".. _containers-dlc-then-customize-devflow:\n\n.. include:: /devflows/dlc-then-customize-devflow.rst\n" }, { "path": "containers/dlc-then-ec2-devflow.rst", "content": ".. _containers-dlc-then-ec2-devflow:\n\n.. include:: /devflows/inference/dlc-then-ec2-devflow.rst" }, { "path": "containers/dlc-then-ecs-devflow.rst", "content": ".. _containers-dlc-then-ecs-devflow:\n\n.. include:: /devflows/inference/dlc-then-ecs-devflow.rst" }, { "path": "containers/dlc-then-eks-devflow.rst", "content": ".. _containers-dlc-then-eks-devflow:\n\n.. include:: /devflows/inference/dlc-then-eks-devflow.rst" }, { "path": "containers/dlc-then-k8s-devflow.rst", "content": ".. _containers-dlc-then-k8s-devflow:\n\n\n.. include:: /devflows/inference/dlc-then-k8s-devflow.rst" }, { "path": "containers/docker-example/Dockerfile.device-plugin", "content": "FROM amazonlinux:2 \n\nRUN echo $'[neuron] \\n\\\nname=Neuron YUM Repository \\n\\\nbaseurl=https://yum.repos.neuron.amazonaws.com \\n\\\nenabled=1' > /etc/yum.repos.d/neuron.repo\n\nRUN rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB\n\nRUN dnf install -y aws-neuron-k8-plugin\nRUN dnf install -y tar gzip\n\nENV PATH=\"/opt/aws/neuron/bin/k8s-neuron-device-plugin:${PATH}\"\n\nCMD k8s-neuron-device-plugin\n" }, { "path": "containers/docker-example/index.rst", "content": "Example: Run containerized neuron application\n=============================================\n\nIntroduction:\n-------------\n\nWith this example you will learn how to run a Neuron application using\ndocker containers.\n\nPrerequisites:\n--------------\n\n- Please ensure the steps from the guide on :ref:`tensorflow-serving`\n were completed successfully before continuing.\n\nSteps:\n------\n\nStep 1: Start neuron-rtd container:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nYou may choose to use the following neuron-rtd image:\n[790709498068.dkr.ecr.us-east-1.amazonaws.com/neuron-rtd:latest], or\nbuild your own image as shown in :ref:`neuron-runtime-dockerfile`.\n\nRun neuron-rtd container as shown below. A volume must be mounted to\n:/sock where neuron-rtd will open a UDS socket. The application can\ninteract with runtime using this socket.\n\n.. code:: bash\n\n aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 790709498068.dkr.ecr.us-east-1.amazonaws.com\n \n docker pull 790709498068.dkr.ecr.us-east-1.amazonaws.com/neuron-rtd:1.1.1402.0\n docker tag 790709498068.dkr.ecr.us-east-1.amazonaws.com/neuron-rtd:1.1.1402.0 neuron-rtd\n mkdir /tmp/neuron_rtd_sock\n chmod o+rwx /tmp/neuron_rtd_sock\n docker run --device=/dev/neuron0 --cap-add IPC_LOCK -v /tmp/neuron_rtd_sock/:/sock -it neuron-rtd\n \n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n If using older version of neuorn(below 1.1):\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n docker pull 790709498068.dkr.ecr.us-east-1.amazonaws.com/neuron-rtd:1.0.9592.0\n docker tag 790709498068.dkr.ecr.us-east-1.amazonaws.com/neuron-rtd:1.0.9592.0 neuron-rtd\n mkdir /tmp/neuron_rtd_sock\n chmod o+rwx /tmp/neuron_rtd_sock\n docker run --env AWS_NEURON_VISIBLE_DEVICES=\"0\" --cap-add SYS_ADMIN --cap-add IPC_LOCK -v /tmp/neuron_rtd_sock/:/sock -it neuron-rtd\n\nStep 2: Start application (tensorflow serving) container:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nBuild tensorflow-model-server-neuron image using provided example\ndockerfile :ref:`tensorflow-model-server-neuron-dockerfile`.\n\nRun assuming a compiled saved model was stored in s3:///my_model/\n\n.. code:: bash\n\n\n # Note: the neuron-rtd socket directory must be mounted and pointed at using environment variable.\n # TensorFlow serving will use that socket to talk to Neuron-rtd\n docker run --env NEURON_RTD_ADDRESS=unix:/sock/neuron.sock \\\n -v /tmp/neuron_rtd_sock/:/sock \\\n -p 8501:8501 \\\n -p 8500:8500 \\\n --env MODEL_BASE_PATH=s3:///my_model/ \\\n --env MODEL_NAME=my_model\n tensorflow-model-server-neuron\n\nStep 3: Verify by running an inference!\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAs shown in :ref:`tensorflow-serving`\n" }, { "path": "containers/docker-example/inference/Dockerfile-inference", "content": "# Example pytorch neuron container\n# To build:\n# docker build . -f Dockerfile.pt -t neuron-container:pytorch\n# To run on EC2 Inf1 instances with AWS DLAMI:\n# docker run -it --device=/dev/neuron0 neuron-container:pytorch\n\nFROM ubuntu:24.04\n\nLABEL maintainer=\" \"\n\nRUN apt-get update -y \\\n && apt-get install -y --no-install-recommends \\\n gnupg2 \\\n wget \\\n python3-pip \\\n python3-setuptools \\\n && cd /usr/local/bin \\\n && pip3 --no-cache-dir install --upgrade pip \\\n && rm -rf /var/lib/apt/lists/* \\\n && apt-get clean\n\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com bionic main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\n# Installing Neuron Tools\nRUN apt-get update -y && apt-get install -y \\\n aws-neuronx-tools\n\n# Sets up Path for Neuron tools\nENV PATH=\"/opt/bin/:/opt/aws/neuron/bin:${PATH}\"\n\n# Include framework tensorflow-neuron or torch-neuronx and compiler (compiler not needed for inference)\nRUN pip3 install \\\n torch-neuronx \\\n --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n# Include your APP dependencies here.\n# RUN ...\n\n# Define the entrypoint script that has some application code (if needed) and executes the docker run command\n# For example you can use something like below\n# COPY dockerd-libmode-entrypoint.sh /opt/bin/dockerd-entrypoint.sh\n# RUN chmod +x /opt/bin/dockerd-entrypoint.sh\n# ENTRYPOINT [\"/opt/bin/dockerd-entrypoint.sh\"]\n\nCMD [\"neuron-top\"]\n" }, { "path": "containers/docker-example/inference/Dockerfile-inference-dlc", "content": "FROM ubuntu:24.04\n\n#SDK 1.17.1 has version 1. We skipped 1.18.0.\nLABEL dlc_major_version=\"2\"\nLABEL maintainer=\"Amazon AI\"\nLABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true\n\nARG PYTHON=python3.7\nARG PYTHON_VERSION=3.7.10\nARG TS_VERSION=0.5.2\nARG MAMBA_VERSION=4.12.0-0\n\n# See http://bugs.python.org/issue19846\nENV LANG C.UTF-8\nENV LD_LIBRARY_PATH /lib/x86_64-linux-gnu:/opt/conda/lib/:$LD_LIBRARY_PATH\nENV PATH /opt/conda/bin:$PATH\nENV SAGEMAKER_SERVING_MODULE sagemaker_pytorch_serving_container.serving:main\nENV TEMP=/home/model-server/tmp\n\nRUN apt-get update \\\n && apt-get install -y --no-install-recommends software-properties-common \\\n && add-apt-repository ppa:openjdk-r/ppa \\\n && apt-get update \\\n && apt-get install -y --no-install-recommends \\\n build-essential \\\n apt-transport-https \\\n ca-certificates \\\n cmake \\\n curl \\\n emacs \\\n git \\\n jq \\\n libgl1-mesa-glx \\\n libglib2.0-0 \\\n libsm6 \\\n libxext6 \\\n libxrender-dev \\\n openjdk-11-jdk \\\n vim \\\n wget \\\n unzip \\\n zlib1g-dev \\\n libcap-dev \\\n gpg-agent \\\n && rm -rf /var/lib/apt/lists/* \\\n && rm -rf /tmp/tmp* \\\n && apt-get clean\n\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com bionic main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\nRUN apt-get update \\\n && apt-get install -y \\\n aws-neuron-tools \\\n && rm -rf /var/lib/apt/lists/* \\\n && rm -rf /tmp/tmp* \\\n && apt-get clean\n\n\n# https://github.com/docker-library/openjdk/issues/261 https://github.com/docker-library/openjdk/pull/263/files\nRUN keytool -importkeystore -srckeystore /etc/ssl/certs/java/cacerts -destkeystore /etc/ssl/certs/java/cacerts.jks -deststoretype JKS -srcstorepass changeit -deststorepass changeit -noprompt; \\\n mv /etc/ssl/certs/java/cacerts.jks /etc/ssl/certs/java/cacerts; \\\n /var/lib/dpkg/info/ca-certificates-java.postinst configure;\n\nRUN curl -L -o ~/mambaforge.sh https://github.com/conda-forge/miniforge/releases/download/${MAMBA_VERSION}/Mambaforge-${MAMBA_VERSION}-Linux-x86_64.sh \\\n && chmod +x ~/mambaforge.sh \\\n && ~/mambaforge.sh -b -p /opt/conda \\\n && rm ~/mambaforge.sh \\\n && /opt/conda/bin/conda update conda \\\n && /opt/conda/bin/conda install -c conda-forge -y \\\n python=$PYTHON_VERSION \\\n cython \\\n mkl-include \\\n mkl \\\n parso \\\n scipy \\\n typing \\\n # Below 2 are included in miniconda base, but not mamba so need to install\n conda-content-trust \\\n charset-normalizer \\\n && /opt/conda/bin/conda clean -ya\n\nRUN conda install -c conda-forge \\\n opencv \\\n scikit-learn \\\n pandas \\\n h5py \\\n requests \\\n && conda clean -ya \\\n && pip install --upgrade pip --trusted-host pypi.org --trusted-host files.pythonhosted.org \\\n && ln -s /opt/conda/bin/pip /usr/local/bin/pip3 \\\n && pip install packaging==20.4 \\\n enum-compat==0.0.3 \\\n numpy==1.20.3 \\\n ipython \\\n # pyOpenSSL requires cryptography>=2.3, but all versions <3.3 have vulnerabilities\n \"cryptography>=3.3.2\"\n\nRUN pip install --no-cache-dir -U \\\n scipy \\\n six \\\n # install PyYAML>=5.4 to avoid conflict with latest awscli\n \"pyYAML>=5.4,<5.5\" \\\n \"pillow>=8.3\" \\\n \"awscli<2\" \\\n boto3\n\nRUN pip install neuron-cc[tensorflow] --extra-index-url https://pip.repos.neuron.amazonaws.com \\\n && pip install \"torch-neuron>=1.10.2,<1.10.3\" --extra-index-url https://pip.repos.neuron.amazonaws.com \\\n && pip install torchserve==$TS_VERSION \\\n && pip install --no-deps --no-cache-dir -U torchvision==0.11.3 \\\n # Install TF 1.15.5 to override neuron-cc[tensorflow]'s installation of tensorflow==1.15.0\n && pip install -U tensorflow==1.15.5 \\\n && pip install torch-model-archiver==$TS_VERSION\n\nRUN useradd -m model-server \\\n && mkdir -p /home/model-server/tmp /opt/ml/model \\\n && chown -R model-server /home/model-server /opt/ml/model\n\nCOPY torchserve-neuron.sh /usr/local/bin/entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.py \\\n && chmod +x /usr/local/bin/neuron-monitor.sh \\\n && chmod +x /usr/local/bin/entrypoint.sh\n\nADD https://raw.githubusercontent.com/aws/deep-learning-containers/master/src/deep_learning_container.py /usr/local/bin/deep_learning_container.py\n\nRUN chmod +x /usr/local/bin/deep_learning_container.py\n\nRUN pip install --no-cache-dir \"sagemaker-pytorch-inference==2.0.8\"\n\nRUN HOME_DIR=/root \\\n && curl -o ${HOME_DIR}/oss_compliance.zip https://aws-dlinfra-utilities.s3.amazonaws.com/oss_compliance.zip \\\n && unzip ${HOME_DIR}/oss_compliance.zip -d ${HOME_DIR}/ \\\n && cp ${HOME_DIR}/oss_compliance/test/testOSSCompliance /usr/local/bin/testOSSCompliance \\\n && chmod +x /usr/local/bin/testOSSCompliance \\\n && chmod +x ${HOME_DIR}/oss_compliance/generate_oss_compliance.sh \\\n && ${HOME_DIR}/oss_compliance/generate_oss_compliance.sh ${HOME_DIR} ${PYTHON} \\\n && rm -rf ${HOME_DIR}/oss_compliance*\n\nRUN curl https://aws-dlc-licenses.s3.amazonaws.com/pytorch-1.10/license.txt -o /license.txt\n\nEXPOSE 8080 8081\n\nCMD [\"/usr/local/bin/entrypoint.sh\"]\n" }, { "path": "containers/docker-example/inference/Dockerfile-inference-dlc.rst", "content": ".. _inference-dlc-dockerfile:\n\nDLC sample Dockerfile for Application Container\n==============================================\n\n.. literalinclude:: Dockerfile-inference-dlc\n :linenos:\n" }, { "path": "containers/docker-example/inference/Dockerfile-libmode", "content": "# Example pytorch neuron container\n# To build:\n# docker build . -f Dockerfile.pt -t neuron-container:pytorch\n# To run on EC2 Inf1 instances with AWS DLAMI:\n# docker run -it --device=/dev/neuron0 neuron-container:pytorch\n\nFROM ubuntu:24.04\n\nLABEL maintainer=\" \"\n\nRUN apt-get update -y \\\n && apt-get install -y --no-install-recommends \\\n gnupg2 \\\n wget \\\n python3-pip \\\n python3-setuptools \\\n && cd /usr/local/bin \\\n && pip3 --no-cache-dir install --upgrade pip \\\n && rm -rf /var/lib/apt/lists/* \\\n && apt-get clean\n\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com bionic main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\n# Installing Neuron Tools\nRUN apt-get update -y && apt-get install -y \\\n aws-neuron-tools\n\n# Sets up Path for Neuron tools\nENV PATH=\"/opt/bin/:/opt/aws/neuron/bin:${PATH}\"\n\n# Include framework tensorflow-neuron or torch-neuron and compiler (compiler not needed for inference)\nRUN pip3 install \\\n torch-neuron \\\n --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n# Include your APP dependencies here.\n# RUN ...\n\n# Define the entrypoint script that has some application code (if needed) and executes the docker run command\n# For example you can use something like below\n# COPY dockerd-libmode-entrypoint.sh /opt/bin/dockerd-entrypoint.sh\n# RUN chmod +x /opt/bin/dockerd-entrypoint.sh\n# ENTRYPOINT [\"/opt/bin/dockerd-entrypoint.sh\"]\n\nCMD [\"neuron-top\"]\n" }, { "path": "containers/docker-example/inference/Dockerfile-libmode.rst", "content": ".. _libmode-dockerfile:\n\nDockerfile for Application Container\n====================================\n\n.. literalinclude:: Dockerfile-inference\n :linenos:\n" }, { "path": "containers/docker-example/inference/Dockerfile-tf-serving.rst", "content": ".. _tensorflow-model-server-neuron-dockerfile:\n\ntensorflow-model-server-neuron Dockerfile\n=========================================\n\n.. literalinclude:: Dockerfile.tf-serving\n :linenos:\n" }, { "path": "containers/docker-example/inference/Dockerfile.mxnet-serving", "content": "# To build:\n# docker build . -f Dockerfile.mxnet-serving -t mxnet-model-server-neuron\n\nFROM amazonlinux:2\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN dnf install -y gcc-c++\nRUN dnf install -y python3-devel\nRUN dnf install -y java-1.8.0-openjdk\nRUN dnf install -y curl\nRUN cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python3 get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1\nRUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3 1\nRUN pip install mxnet-neuron --index-url=https://pip.repos.neuron.amazonaws.com\nRUN pip install multi-model-server\n\n\nRUN useradd -m model-server \\\n && mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\n\nRUN mkdir -p /home/model-server/tmp/models/\n#copy your model\nCOPY mxnet_model/resnet-50_compiled.mar /home/model-server/tmp/models/\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh \\\n && chown -R model-server /home/model-server\n\nEXPOSE 8080 8081\n\nUSER model-server\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n" }, { "path": "containers/docker-example/inference/Dockerfile.tf-serving", "content": "# Example tensorflow-model-server-neuron dockerfile.\n\n# Note: tensorflow_model_server_neuron must be pointed at the model location and name using MODEL_BASE_PATH and\n# MODEL_NAME env variables. MODEL_BASE_PATH may be an s3 location.\n\n# To build:\n# docker build . -f Dockerfile.tf-serving -t tensorflow-model-server-neuron\n\n\nFROM amazonlinux:2\n\n\n# Expose ports for gRPC and REST\nEXPOSE 8500 8501\n\nENV MODEL_BASE_PATH=/models \\\n MODEL_NAME=model\n\nRUN echo $'[neuron] \\n\\\nname=Neuron YUM Repository \\n\\\nbaseurl=https://yum.repos.neuron.amazonaws.com \\n\\\nenabled=1' > /etc/yum.repos.d/neuron.repo\n\nRUN rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB\n\nRUN dnf install -y tensorflow-model-server-neuron\nRUN mkdir -p /root/models/\n#copy your model\nCOPY tf_model/ /root/models/\nRUN ls -la /root/models/*\n\nCMD [\"/bin/sh\", \"-c\", \"/usr/local/bin/tensorflow_model_server_neuron --port=8500 --rest_api_port=8501 --model_name=${MODEL_NAME} --model_base_path=/root/models/${MODEL_NAME}\"]\n" }, { "path": "containers/docker-example/inference/config-properties.rst", "content": ".. _torchserve-config-properties:\n\nTorchserve config.properties example\n====================================\n\n.. literalinclude:: config.properties\n :linenos:\n" }, { "path": "containers/docker-example/inference/config.properties", "content": "vmargs=-XX:+UseContainerSupport -XX:InitialRAMPercentage=8.0 -XX:MaxRAMPercentage=10.0 -XX:-UseLargePages -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError\nmodel_store=/opt/ml/model\nload_models=ALL\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\n# management_address=unix:/tmp/management.sock\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_response_timeout=120\n# default_workers_per_model=0\n# job_queue_size=100\n# async_logging=false\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\n# private_key_file=src/test/resources/key.pem\n# certificate_file=src/test/resources/certs.pem\n# max_response_size=6553500\n# max_request_size=6553500\n# blacklist_env_vars=\n# decode_input_request=false\n# enable_envvars_config=false\n" }, { "path": "containers/docker-example/inference/dockerd-libmode-entrypoint.rst", "content": ".. _dockerd-libmode-entrypoint:\n\nDocker Entrypoint Example - Application container\n=================================================\n\n.. literalinclude:: dockerd-libmode-entrypoint.sh\n :linenos:\n" }, { "path": "containers/docker-example/inference/dockerd-libmode-entrypoint.sh", "content": "#!/bin/bash\nif [[ \"$1\" = \"serve\" ]]; then\n # Start your application here!\n # e.g: 'python my_server_app.py'\nelse\n eval \"$@\"\nfi\n\n# prevent docker exit\ntail -f /dev/null\n" }, { "path": "containers/docker-example/inference/torchserve-neuron.rst", "content": ".. _torchserve-neuron:\n\nTorchserve Example\n==================\n\n.. literalinclude:: torchserve-neuron.sh\n :linenos:\n" }, { "path": "containers/docker-example/inference/torchserve-neuron.sh", "content": "#!/bin/bash\n\nMODEL_STORE=/opt/ml/model\nTS_CONFIG=/home/model-server/config.properties\nMODEL_PATH=\"\"\n\nwhile getopts \":m:t:\" opt; do\n case $opt in\n m) MODEL_PATH=\"$OPTARG\"\n ;;\n t) TS_CONFIG=\"$OPTARG\"\n ;;\n \\?) echo \"Invalid option -$OPTARG\" >&2\n ;;\n esac\ndone\n\nprintf \"Model path: %s\\n\" \"$MODEL_PATH\"\nprintf \"TS_CONFIG: %s\\n\" \"$TS_CONFIG\"\n# Start the Model Server\nif [[ -z \"$MODEL_PATH\" ]]; then\n torchserve --start --ts-config /home/model-server/config.properties --model-store /opt/ml/model &\nelse\n torchserve --start --ts-config $TS_CONFIG --models $MODEL_PATH &\nfi\nstatus=$?\nif [ $status -ne 0 ]; then\n echo \"Failed to start TF Model Server: $status\"\n exit $status\nfi" }, { "path": "containers/docker-example/training/Dockerfile-training-dlc", "content": "# Example pytorch neuron container\n# To build:\n# docker build . -f Dockerfile.pt -t neuron-container:pytorch\n# To run on EC2 Inf1 instances with AWS DLAMI:\n# docker run -it --net=host --device=/dev/neuron0 neuron-container:pytorch\n\n# You can find the latest Pytorch Training Image here - https://gallery.ecr.aws/neuron/pytorch-training-neuronx\nFROM public.ecr.aws/neuron/pytorch-training-neuronx:2.9.0-neuronx-py310-sdk2.27.0-ubuntu24.04\nRUN mkdir -p /opt/ml\nCOPY model.py /opt/ml/model.py\nCOPY mlp_train.py /opt/ml/mlp_train.py " }, { "path": "containers/docker-example/training/Dockerfile-trainium-dlc.rst", "content": ".. _trainium-dlc-dockerfile:\n\nDockerfile for Application Container\n====================================\n\n.. literalinclude:: Dockerfile-training-dlc\n :linenos:\n" }, { "path": "containers/docker-example/training/mlp.rst", "content": ".. _mlp-train:\n\nSimple MLP train script\n========================\n\nSave the following contents as mlp_train.py\n\n.. literalinclude:: mlp_train.py\n :linenos:\n\n\nSave the following contents as model.py\n\n.. literalinclude:: model.py\n :linenos:" }, { "path": "containers/docker-example/training/mlp_train.py", "content": "import os\nimport time\nimport torch\nfrom model import MLP\n\nfrom torchvision.datasets import mnist\nfrom torch.utils.data import DataLoader\nfrom torchvision.transforms import ToTensor\n\n# XLA imports\nimport torch_xla.core.xla_model as xm\n\n# Global constants\nEPOCHS = 4\nWARMUP_STEPS = 2\nBATCH_SIZE = 32\n\n# Load MNIST train dataset\ntrain_dataset = mnist.MNIST(root='./MNIST_DATA_train',\n train=True, download=True, transform=ToTensor())\n\ndef main():\n # Prepare data loader\n train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE)\n\n # Fix the random number generator seeds for reproducibility\n torch.manual_seed(0)\n\n # XLA: Specify XLA device (defaults to a NeuronCore on Trn1 instance)\n device = 'xla'\n\n # Move model to device and declare optimizer and loss function\n model = MLP().to(device)\n optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n loss_fn = torch.nn.NLLLoss()\n\n # Run the training loop\n print('----------Training ---------------')\n model.train()\n for epoch in range(EPOCHS):\n start = time.time()\n for idx, (train_x, train_label) in enumerate(train_loader):\n optimizer.zero_grad()\n train_x = train_x.view(train_x.size(0), -1)\n train_x = train_x.to(device)\n train_label = train_label.to(device)\n output = model(train_x)\n loss = loss_fn(output, train_label)\n loss.backward()\n optimizer.step()\n xm.mark_step() # XLA: collect ops and run them in XLA runtime\n if idx < WARMUP_STEPS: # skip warmup iterations\n start = time.time()\n\n # Compute statistics for the last epoch\n interval = idx - WARMUP_STEPS # skip warmup iterations\n throughput = interval / (time.time() - start)\n print(\"Train throughput (iter/sec): {}\".format(throughput))\n print(\"Final loss is {:0.4f}\".format(loss.detach().to('cpu')))\n\n # Save checkpoint for evaluation\n os.makedirs(\"checkpoints\", exist_ok=True)\n checkpoint = {'state_dict': model.state_dict()}\n # XLA: use xm.save instead of torch.save to ensure states are moved back to cpu\n # This can prevent \"XRT memory handle not found\" at end of test.py execution\n xm.save(checkpoint,'checkpoints/checkpoint.pt')\n\n print('----------End Training ---------------')\n\nif __name__ == '__main__':\n main()" }, { "path": "containers/docker-example/training/model.py", "content": "import torch.nn as nn\nimport torch.nn.functional as F\n\n# Declare 3-layer MLP for MNIST dataset\nclass MLP(nn.Module):\n def __init__(self, input_size = 28 * 28, output_size = 10, layers = [120, 84]):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return F.log_softmax(x, dim=1)" }, { "path": "containers/docker-example/v1/inference/Dockerfile-app-rt-diff.rst", "content": ".. _app-rt-diff-dockerfile:\n\nDockerfile with Application and Runtime in different Container\n==============================================================\n\n.. literalinclude:: Dockerfile.app-rt-diff\n :linenos:\n" }, { "path": "containers/docker-example/v1/inference/Dockerfile-app-rt-same.rst", "content": ".. _app-rt-same-dockerfile:\n\nDockerfile with Application and Runtime in same Container\n=========================================================\n\n.. literalinclude:: Dockerfile.torch-neuron\n :linenos:\n" }, { "path": "containers/docker-example/v1/inference/Dockerfile-neuron-rtd.rst", "content": ".. _neuron-runtime-dockerfile:\n\nNeuron Runtime Dockerfile\n=========================\n\n.. literalinclude:: Dockerfile.neuron-rtd\n :linenos:\n" }, { "path": "containers/docker-example/v1/inference/Dockerfile-torch-neuron.rst", "content": ".. _torch-neuron-dockerfile:\n\ntorch-neuron Dockerfile\n=======================\n\n.. literalinclude:: Dockerfile.torch-neuron\n :linenos:\n" }, { "path": "containers/docker-example/v1/inference/Dockerfile.app-rt-diff", "content": "# Example pytorch neuron container\n# To build:\n# docker build . -f Dockerfile.pt -t neuron-container:pytorch\n# To run on EC2 Inf1 instances with AWS DLAMI:\n# sudo service neuron-rtd stop\n# docker run -it --device=/dev/neuron0 -v /run/:/run --cap-add IPC_LOCK neuron-container:pytorch\n\nFROM ubuntu:18.04\n\nLABEL maintainer=\" \"\n\nRUN apt-get update -y \\\n && apt-get install -y --no-install-recommends \\\n wget \\\n gnupg2 \\\n python3-pip \\\n python3-setuptools \\\n && cd /usr/local/bin \\\n && pip3 --no-cache-dir install --upgrade pip \\\n && rm -rf /var/lib/apt/lists/* \\\n && apt-get clean\n\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com bionic main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\n# Include framework tensorflow-neuron or torch-neuron and compiler (compiler not needed for inference)\nRUN pip3 install \\\n torch-neuron \\\n --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n# Include your APP dependencies here.\n# RUN/ENTRYPOINT/CMD ...\n\n" }, { "path": "containers/docker-example/v1/inference/Dockerfile.neuron-rtd", "content": "# Example neuron-rtd dockerfile.\n\n# To build:\n# docker build . -f Dockerfile.neuron-rtd -t neuron-rtd\n\n# Note: the container must start with CAP_IPC_LOCK capability\n\n# To run on EC2 Inf1 instances with AWS DLAMI:\n# sudo service neuron-rtd stop\n# docker run --env AWS_NEURON_VISIBLE_DEVICES=\"0\" --cap-add IPC_LOCK -v /tmp/neuron_rtd_sock/:/sock neuron-rtd\n\n\nFROM amazonlinux:2\n\nRUN echo $'[neuron] \\n\\\nname=Neuron YUM Repository \\n\\\nbaseurl=https://yum.repos.neuron.amazonaws.com \\n\\\nenabled=1' > /etc/yum.repos.d/neuron.repo\n\nRUN rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB\n\nRUN dnf install -y aws-neuron-tools\nRUN dnf install -y aws-neuron-runtime\nRUN dnf install -y tar gzip\n\nENV PATH=\"/opt/aws/neuron/bin:${PATH}\"\n\nCMD neuron-rtd -g unix:/sock/neuron.sock --log-console\n" }, { "path": "containers/docker-example/v1/inference/Dockerfile.torch-neuron", "content": "# Example pytorch neuron container\n# Note: a dockerd_entrypoint.sh script is required to succesfully build this image. Place the script on the same folder as the Dockerfile\n# To build:\n# docker build . -f Dockerfile.pt -t neuron-container:pytorch\n# To run on EC2 Inf1 instances with AWS DLAMI:\n# sudo service neuron-rtd stop\n# docker run -it --device=/dev/neuron0 --cap-add IPC_LOCK neuron-container:pytorch\n\nFROM ubuntu:18.04\n\nLABEL maintainer=\" \"\n\nRUN apt-get update -y \\\n && apt-get install -y --no-install-recommends \\\n gnupg2 \\\n wget \\\n python3-pip \\\n python3-setuptools \\\n libcap-dev \\\n && cd /usr/local/bin \\\n && pip3 --no-cache-dir install --upgrade pip \\\n && rm -rf /var/lib/apt/lists/* \\\n && apt-get clean\n\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com bionic main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\n# Installing Neuron Runtime and Tools\nRUN apt-get update -y && apt-get install -y \\\n aws-neuron-runtime \\\n aws-neuron-tools\n\n# Sets up Path for Neuron tools\nENV PATH=\"/opt/bin/:/opt/aws/neuron/bin:${PATH}\"\n\n# Include framework tensorflow-neuron or torch-neuron and compiler (compiler not needed for inference)\nRUN pip3 install \\\n torch-neuron \\\n --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n# Include your APP dependencies here.\n# RUN ...\n\n# Define the entrypoint script that starts the runtime and executes the docker run command\nCOPY dockerd-entrypoint.sh /opt/bin/dockerd-entrypoint.sh\nRUN chmod +x /opt/bin/dockerd-entrypoint.sh\nENTRYPOINT [\"/opt/bin/dockerd-entrypoint.sh\"]\n\nCMD [\"neuron-top\"]\n" }, { "path": "containers/docker-example/v1/inference/dockerd-entrypoint-app-rt-same.rst", "content": ".. _dockerd-entrypoint-app-rt-same:\n\nDocker Entrypoint Example - Application and Runtime in same Container\n=====================================================================\n\n.. literalinclude:: dockerd-entrypoint.sh\n :linenos:\n" }, { "path": "containers/docker-example/v1/inference/dockerd-entrypoint.sh", "content": "#!/bin/bash\nset -e\n\nwait_for_nrtd() {\n nrtd_sock=\"/run/neuron.sock\"\n SOCKET_TIMEOUT=300\n is_wait=true\n wait_time=0\n i=1\n sp=\"/-\\|\"\n echo -n \"Waiting for neuron-rtd \"\n pid=$1\n while $is_wait; do\n if [ -S \"$nrtd_sock\" ]; then\n echo \"$nrtd_sock Exist...\"\n is_wait=false\n else\n sleep 1\n wait_time=$((wait_time + 1))\n if [ \"$wait_time\" -gt \"$SOCKET_TIMEOUT\" ]; then\n echo \"neuron-rtd failed to start, exiting\"\n\t cat /tmp/nrtd.log\n exit 1\n fi\n printf \"\\b${sp:i++%${#sp}:1}\"\n fi\n done\n cat /tmp/nrtd.log\n}\n\n# Start neuron-rtd\n/opt/aws/neuron/bin/neuron-rtd -g unix:/run/neuron.sock --log-console >> /tmp/nrtd.log 2>&1 &\nnrtd_pid=$!\necho \"NRTD PID: \"$nrtd_pid\"\"\n#wait for nrtd to be up (5 minutes timeout)\nwait_for_nrtd $nrtd_pid\nexport NEURON_RTD_ADDRESS=unix:/run/neuron.sock\nnrtd_present=1\n\nif [[ \"$1\" = \"serve\" ]]; then\n # Start your application here!\n # e.g: 'python my_server_app.py'\nelse\n eval \"$@\"\nfi\n\n# prevent docker exit\ntail -f /dev/null\n" }, { "path": "containers/ec2-then-ec2-devflow.rst", "content": ".. _containers-ec2-then-ec2-devflow:\n\n.. include:: /devflows/inference/ec2-then-ec2-devflow.rst" }, { "path": "containers/ec2.rst", "content": ".. _ec2-instance:\n\nEC2 Instance\n============\n\nIntroduction\n------------\n\nUse of Neuron in containers on EC2 can be simple to achieve by following these steps\n\n - :ref:`tutorial-docker-env-setup-for-neuron`\n - More details on EC2 setup `can be found at `_\n\nDLC Images\n----------\n\n - The location for DLC images for Neuron can be obtained from `here `_\n - To get the list of images for neuron, the following commands can be used.\n\n ``aws ecr list-images --registry-id 763104351884 --repository-name tensorflow-inference-neuron``\n\n ``aws ecr list-images --registry-id 763104351884 --repository-name pytorch-inference-neuron``\n\nSetup recommendations\n---------------------\n\n - The EC2 Inf1 instance needs to have the aws-neuron-runtime-base and aws-neruon-dkms package installed.\n - The DLC inference container runs the framework server (like tensorflow-model-server or TorchServe) and also the neuron runtime that interacts with the neuron driver running in the host.\n - For more details on setting up the container, check the `tensorflow `_ or `pytorch `_. Make sure the appropriate framework container image is used.\n\nDebug Hints\n-----------\n - Use the docker log command to get the neuron rtd logs in the container.\n\n ``docker logs ``\n\n - Look for errors like the following\n - If we see *nrtd[8]: [TDRV:tdrv_init_mla_phase1] Could not open the device index:0*, it either means that some other container is using that device or the host is running the neuron-rtd process.\n - Check to see that host is not running neuron-rtd\n\n ``sudo systemctl status neuron-rtd``\n" }, { "path": "containers/faq-troubleshooting-releasenote.rst", "content": "Containers - FAQ, Troubleshooting & ReleaseNotes\n================================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n FAQ \n troubleshooting\n /release-notes/components/containers\n\n* :ref:`container-faq`\n* :ref:`container-troubleshooting`\n* :ref:`containers_rn`\n" }, { "path": "containers/faq.rst", "content": ".. _container-faq:\n\nNeuron Containers FAQ\n=====================\n\n.. contents:: Table of Contents\n :local:\n :depth: 1\n\nWhere can I find DLC images\n---------------------------\n* The Inference/Training DLC images can be found `here `_.\n* In the `DLC release page `_ do a search for neuron to get the ECR repo location of specific neuron DLC release.\n\n\nWhat is OCI Neuron Hook and do we need that\n-------------------------------------------\nNeuron devices are exposed to the containers using the --device option in the docker run command.\nDocker runtime (runc) does not yet support the ALL option to expose all neuron\ndevices to the container. \n\nWith OCI neuron hook support is added to expose ALL devices to container using an environment variable,\n“AWS_NEURON_VISIBLE_DEVICES=ALL\". For more details please refer :ref:`oci neuron hook `\n\nIn Kubernetes, if we are using the device plugin version 1.7 & below, then the oci neuron hook is needed. If\nusing device plugin version >= 1.8 then oci neuron hook is not needed\n\nWhat container runtimes are supported\n-------------------------------------\nNeuron containers have been tested to work with docker, containerd, cri-o runtimes without any changes.\nIf the oci neuron hook is used then they need to be enabled in the runtime config. For more details please refer :ref:`oci neuron hook `\n\n\nHow to expose Neuron Devices to Container\n-----------------------------------------\nNeuron Device: Represents the number of Inferentia/Trainium chips in the instance. Refer :ref:`Container Devices ` for more details\n\n\nHow to expose Neuron Cores to Container\n---------------------------------------\nNeuron Core: Represents the number of Neuron Cores in the instance. Refer :ref:`Container Cores ` for more details. Each Inferentia1\ndevice has 4 Neuron Cores and each Inferentia2 and Trainium1 device has 2 Neuron Cores.\nWhen the devices are exposed to the containers all the cores in the device are available\nfor use in the container. Please refer :ref:`nrt-configuration` to see how the environment variables NEURON_RT_VISIBLE_CORES and NEURON_RT_NUM_CORES \ncan be used to assign core to containers\n\nCan Neuron Devices be shared by different Containers running in the same Host\n-----------------------------------------------------------------------------\nYes, except in Kubernetes environment where the devices cannot be shared\n\nCan Neuron Cores be shared by different Containers running in the same Host\n-----------------------------------------------------------------------------\nNo\n\nWhen would you use Neuron K8 Scheduler Extension\n-------------------------------------------------\nThe neuron cores/devices that are exposed to the container needs to be contiguous. The kubernetes device plugin\ndoes not guarantee the devices to be contiguous. The K8 Neuron Scheduler Extension takes care of \nassigning contiguous devices to the containers.\n\nHow to add EFA devices to the container\n---------------------------------------\nThe EFA devices are exposed to the container using the --device option\n\n::\n\n --device /dev/infiniband/uverbs0 \n\nIn a Kubernetes environment, the EFA device plugin is used to detect and advertise \nthe available EFA interfaces. The EFA device plugin can be installed using the `Helm chart provided by Amazon EKS `_\n\n::\n\n helm repo add eks https://aws.github.io/eks-charts\n helm install aws-efa-k8s-device-plugin --namespace kube-system eks/aws-efa-k8s-device-plugin\n\nOnce the plugin is deployed, applications can use the resource type vpc.amazonaws.com/efa in a pod request spec\n\n::\n\n resources:\n limits:\n vpc.amazonaws.com/efa: 4\n\n\nCan distributed training jobs be run without EFA devices in container\n---------------------------------------------------------------------\nNo. For distributed training jobs on Trainium, all EFA interfaces provided by trn1.32xlarge need to be\nattached to the container\n" }, { "path": "containers/files/index-dra.rst", "content": ".. meta::\n :description: Templates supporting AWS Neuron Dynamic Resource Allocation (DRA) on Kubernetes.\n :keywords: AWS Neuron, Neuron DRA, Dynamic Resource Allocation, Kubernetes, K8s, Device Plugin\n :date-modified: 02/05/2026\n\nAWS Neuron Dynamic Resource Allocation (DRA) on Kubernetes: Support files\n=========================================================================\n\nThis page provides templates supporting AWS Neuron Dynamic Resource Allocation (DRA) on Kubernetes. You can view and download these files from the links below.\n\nResource Claim Specifications\n-----------------------------\n\nExample resource claim templates and pod specifications demonstrating different Neuron device allocation patterns for various workload requirements.\n\n.. list-table::\n :header-rows: 1\n :widths: 30 55 15\n\n * - File Name\n - Description\n - Download\n * - 1x4-connected-devices.yaml\n - Resource claim template for allocating 4 connected Neuron devices with topology constraints for optimal performance.\n - :download:`Download `\n * - 2-node-inference-us.yaml\n - Multi-node inference configuration for distributed workloads across 2 Trainium nodes.\n - :download:`Download `\n * - 4-node-inference-us.yaml\n - Large-scale inference setup for distributed workloads spanning 4 Trainium nodes.\n - :download:`Download `\n * - all-devices.yaml\n - Resource claim template that allocates all available Neuron devices on a trn2.48xlarge instance.\n - :download:`Download `\n * - lnc-setting-trn2.yaml\n - Logical NeuronCore configuration template optimized for Trainium2 instances.\n - :download:`Download `\n * - specific-driver-version.yaml\n - Example configuration for requesting specific Neuron driver versions in resource claims.\n - :download:`Download `\n * - us-and-lnc-config.yaml\n - Example configuration for requesting UltraServer node with Logical NeuronCore configuration.\n - :download:`Download `\n\n" }, { "path": "containers/files/manifests/clusterrole.yaml", "content": "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n name: neuron-dra-driver-clusterrole\nrules:\n# Required for DRA device plugin to manage ResourceSlices\n- apiGroups: [\"resource.k8s.io\"]\n resources: [\"resourceslices\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n# Required for DRA device plugin to read ResourceClaims\n- apiGroups: [\"resource.k8s.io\"]\n resources: [\"resourceclaims\"]\n verbs: [\"get\", \"list\", \"watch\"]\n# Required for DRA device plugin to read DeviceClasses\n- apiGroups: [\"resource.k8s.io\"]\n resources: [\"deviceclasses\"]\n verbs: [\"get\", \"list\", \"watch\"]\n # Required to read and modify node information\n- apiGroups: [\"\"]\n resources: [\"nodes\"]\n verbs: [\"get\", \"list\", \"watch\", \"patch\", \"update\"]\n # Required to modify node status\n- apiGroups: [\"\"]\n resources: [\"nodes/status\"]\n verbs: [\"patch\"]\n" }, { "path": "containers/files/manifests/clusterrolebinding.yaml", "content": "\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: neuron-dra-driver-binding\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: neuron-dra-driver-clusterrole\nsubjects:\n- kind: ServiceAccount\n name: neuron-dra-driver-sa\n namespace: neuron-dra-driver" }, { "path": "containers/files/manifests/daemonset.yaml", "content": "apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: neuron-dra-driver-kubelet-plugin\n namespace: neuron-dra-driver\n labels:\n app: neuron-dra-driver-kubelet-plugin\nspec:\n updateStrategy:\n type: RollingUpdate\n rollingUpdate:\n maxUnavailable: 0\n maxSurge: 1\n selector:\n matchLabels:\n app: neuron-dra-driver-kubelet-plugin\n template:\n metadata:\n labels:\n app: neuron-dra-driver-kubelet-plugin\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n - key: node.kubernetes.io/instance-type\n operator: In\n values:\n - trn1.2xlarge\n - trn1.32xlarge\n - trn1n.32xlarge\n - trn2.3xlarge\n - trn2.48xlarge\n - trn2n.48xlarge\n - key: eks.amazonaws.com/compute-type\n operator: NotIn\n values:\n - fargate\n - hybrid\n - auto\n serviceAccountName: neuron-dra-driver-sa\n hostNetwork: true\n containers:\n - name: neuron-dra-driver\n image: NEURON_DRA_IMAGE\n imagePullPolicy: Always\n command: [\"k8s-neuron-dra-driver\"]\n # args:\n # - --v=6\n env:\n - name: NODE_NAME\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n - name: POD_UID\n valueFrom:\n fieldRef:\n fieldPath: metadata.uid\n - name: CDI_ROOT\n value: \"/var/run/cdi\"\n - name: KUBELET_REGISTRAR_DIRECTORY_PATH\n value: \"/var/lib/kubelet/plugins_registry\"\n - name: KUBELET_PLUGINS_DIRECTORY_PATH\n value: \"/var/lib/kubelet/plugins\"\n - name: HEALTHCHECK_PORT\n value: \"51515\"\n - name: NEURON_DRA_DRIVER_EMULATION_MODE\n value: \"trn2u\"\n resources:\n limits:\n cpu: 20m\n memory: 256Mi\n requests:\n cpu: 10m\n memory: 128Mi\n securityContext:\n privileged: true\n volumeMounts:\n - name: kubelet-plugins-dir\n mountPath: /var/lib/kubelet/plugins\n - name: kubelet-registry-dir\n mountPath: /var/lib/kubelet/plugins_registry\n - name: cdi-dir\n mountPath: /var/run/cdi\n livenessProbe:\n grpc:\n port: 51515\n service: liveness\n failureThreshold: 3\n periodSeconds: 10\n initialDelaySeconds: 30\n timeoutSeconds: 5\n volumes:\n - name: kubelet-plugins-dir\n hostPath:\n path: /var/lib/kubelet/plugins\n - name: kubelet-registry-dir\n hostPath:\n path: /var/lib/kubelet/plugins_registry\n - name: cdi-dir\n hostPath:\n path: /var/run/cdi\n tolerations:\n - key: CriticalAddonsOnly\n operator: Exists\n - key: aws.amazon.com/neuron\n operator: Exists\n effect: NoSchedule\n - key: sagemaker.amazonaws.com/node-health-status\n operator: Equal\n value: Unschedulable\n effect: NoSchedule\n # - key: \"kwok.x-k8s.io/node\"\n # operator: \"Exists\"\n # effect: \"NoSchedule\"" }, { "path": "containers/files/manifests/deviceclass.yaml", "content": "apiVersion: resource.k8s.io/v1beta1\nkind: DeviceClass\nmetadata:\n name: neuron.aws.com\nspec:\n selectors:\n - cel:\n expression: device.driver == \"neuron.aws.com\"" }, { "path": "containers/files/manifests/namespace.yaml", "content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: neuron-dra-driver\n labels:\n name: neuron-dra-driver" }, { "path": "containers/files/manifests/serviceaccount.yaml", "content": "apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: neuron-dra-driver-sa\n namespace: neuron-dra-driver" }, { "path": "containers/files/scripts/install-dra-driver.sh", "content": "#!/bin/bash\n\n# Deploy Neuron DRA Driver\nset -e\n\necho \"🚀 Deploying Neuron DRA Driver...\"\n\n# Check argument\nif [ $# -ne 1 ]; then\n echo \"Usage: $0 \"\n echo \"Example: $0 123456789.dkr.ecr.us-west-2.amazonaws.com/neuron-dra-driver:v1.0\"\n exit 1\nfi\n\n# Get the script directory and set the manifests path\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nMANIFESTS_DIR=\"$SCRIPT_DIR/../../manifests\"\nDRA_IMAGE=\"$1\"\n\n# Apply all manifests in order\necho \"📝 Creating namespace...\"\nkubectl apply -f \"$MANIFESTS_DIR/namespace.yaml\"\n\necho \"🔐 Creating ServiceAccount and RBAC...\"\nkubectl apply -f \"$MANIFESTS_DIR/serviceaccount.yaml\"\nkubectl apply -f \"$MANIFESTS_DIR/clusterrole.yaml\"\nkubectl apply -f \"$MANIFESTS_DIR/clusterrolebinding.yaml\"\n\necho \"📱 Creating DeviceClass...\"\nkubectl apply -f \"$MANIFESTS_DIR/deviceclass.yaml\"\n\necho \"🔧 Deploying DRA DaemonSet...\"\n# Check if DaemonSet already exists before applying\nDAEMONSET_EXISTS=false\nif kubectl get daemonset neuron-dra-driver-kubelet-plugin -n neuron-dra-driver >/dev/null 2>&1; then\n DAEMONSET_EXISTS=true\n echo \"📋 DaemonSet already exists, will restart after applying...\"\nfi\n\necho \"🏷️ Using custom image: $DRA_IMAGE\"\nsed \"s|NEURON_DRA_IMAGE|$DRA_IMAGE|g\" \"$MANIFESTS_DIR/daemonset.yaml\" | kubectl apply -f -\n\n# If DaemonSet was already running, restart it to pull latest image\nif [ \"$DAEMONSET_EXISTS\" = true ]; then\n echo \"🔄 Restarting DaemonSet to pull latest image...\"\n kubectl rollout restart daemonset/neuron-dra-driver-kubelet-plugin -n neuron-dra-driver\n echo \"⏳ Waiting for rollout to complete...\"\n kubectl rollout status daemonset/neuron-dra-driver-kubelet-plugin -n neuron-dra-driver --timeout=300s\nelse\n echo \"⏳ Waiting until pods are in a running state...\"\n kubectl wait --for=condition=ready pod -l app=neuron-dra-driver-kubelet-plugin -n neuron-dra-driver --timeout=300s\nfi\n\necho \"✅ Deployment complete!\"\n\necho \"\"\necho \"📊 Recent logs from dra driver:\"\nkubectl logs -n neuron-dra-driver -l app=neuron-dra-driver-kubelet-plugin --tail=10\necho \"\"" }, { "path": "containers/files/specs/1x4-connected-devices.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: 1x4-connected-neurons\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n allocationMode: ExactCount\n count: 4\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\"\n constraints:\n - requests: [\"neurons\"]\n matchAttribute: \"resource.aws.com/devicegroup4_id\"\n\n---\napiVersion: v1\nkind: Pod\nmetadata:\n name: pod0\n labels:\n app: pod\nspec:\n containers:\n - name: ctr0\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command: [\"bash\", \"-c\"]\n args: [\"export; trap 'exit 0' TERM; sleep 9999 & wait\"]\n resources:\n claims:\n - name: neurons\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: 1x4-connected-neurons\n" }, { "path": "containers/files/specs/2-node-inference-us.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: us-2-node-config\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].resourceType == 'neuron_node'\"\n allocationMode: ExactCount\n count: 1\n config:\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: UltraServerConfig\n ultraserverMode: 2\n---\napiVersion: leaderworkerset.x-k8s.io/v1\nkind: LeaderWorkerSet\nmetadata:\n name: vllm\n annotations:\n leaderworkerset.sigs.k8s.io/exclusive-topology: neuron.amazonaws.com/ultraserver-server-id-2\nspec:\n rolloutStrategy:\n type: RollingUpdate\n rollingUpdateConfiguration:\n maxUnavailable: 1\n maxSurge: 1\n # Two replica groups of 2 nodes each\n replicas: 2\n leaderWorkerTemplate:\n size: 2\n restartPolicy: RecreateGroupOnPodRestart\n leaderTemplate:\n metadata:\n labels:\n role: leader\n spec:\n containers:\n - name: vllm-leader\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-2-node-config\n workerTemplate:\n metadata:\n labels:\n role: worker\n spec:\n containers:\n - name: vllm-worker\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-2-node-config\n" }, { "path": "containers/files/specs/4-node-inference-us.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: us-4-node-config\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].resourceType == 'neuron_node'\"\n allocationMode: ExactCount\n count: 1\n config:\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: UltraServerConfig\n ultraserverMode: 4\n---\napiVersion: leaderworkerset.x-k8s.io/v1\nkind: LeaderWorkerSet\nmetadata:\n name: vllm\n annotations:\n leaderworkerset.sigs.k8s.io/exclusive-topology: neuron.amazonaws.com/ultraserver-server-id-4\nspec:\n rolloutStrategy:\n type: RollingUpdate\n rollingUpdateConfiguration:\n maxUnavailable: 1\n maxSurge: 1\n # Two replica groups of 4 nodes each, i.e. two ultraservers\n replicas: 2\n leaderWorkerTemplate:\n size: 4\n restartPolicy: RecreateGroupOnPodRestart\n leaderTemplate:\n metadata:\n labels:\n role: leader\n spec:\n containers:\n - name: vllm-leader\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-4-node-config\n workerTemplate:\n metadata:\n labels:\n role: worker\n spec:\n containers:\n - name: vllm-worker\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-4-node-config\n" }, { "path": "containers/files/specs/all-devices.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: all-neurons\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\"\n allocationMode: All\n---\napiVersion: v1\nkind: Pod\nmetadata:\n name: pod0\n labels:\n app: pod\nspec:\n containers:\n - name: ctr0\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command: [\"bash\", \"-c\"]\n args: [\"export; trap 'exit 0' TERM; sleep 9999 & wait\"]\n resources:\n claims:\n - name: neurons\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: all-neurons" }, { "path": "containers/files/specs/lnc-setting-trn2.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: all-neurons-lnc-1\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\"\n allocationMode: All\n config:\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: NeuronConfig\n logicalNeuronCore: 1\n---\napiVersion: v1\nkind: Pod\nmetadata:\n name: pod0\n labels:\n app: pod\nspec:\n containers:\n - name: ctr0\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command: [\"bash\", \"-c\"]\n args: [\"export; trap 'exit 0' TERM; sleep 9999 & wait\"]\n resources:\n claims:\n - name: neurons\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: all-neurons-lnc-1\n" }, { "path": "containers/files/specs/specific-driver-version.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: driver-version-neuron\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].neuronDriverVersion == '2.25.4.0'\"\n allocationMode: All\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: my-app\nspec:\n replicas: 2\n selector:\n matchLabels:\n app: my-app\n template:\n metadata:\n labels:\n app: my-app\n spec:\n containers:\n - name: nginx\n image: public.ecr.aws/docker/library/nginx:alpine\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: driver-version-neuron" }, { "path": "containers/files/specs/us-and-lnc-config.yaml", "content": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n name: us-and-lnc-config\nspec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].resourceType == 'neuron_node'\"\n allocationMode: ExactCount\n count: 1\n config:\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: UltraServerConfig\n ultraserverMode: 2\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: NeuronConfig\n logicalNeuronCore: 1\n---\napiVersion: leaderworkerset.x-k8s.io/v1\nkind: LeaderWorkerSet\nmetadata:\n name: vllm\n annotations:\n leaderworkerset.sigs.k8s.io/exclusive-topology: neuron.amazonaws.com/ultraserver-server-id-2\nspec:\n rolloutStrategy:\n type: RollingUpdate\n rollingUpdateConfiguration:\n maxUnavailable: 1\n maxSurge: 1\n # Two replica groups of 2 nodes each\n replicas: 2\n leaderWorkerTemplate:\n size: 2\n restartPolicy: RecreateGroupOnPodRestart\n leaderTemplate:\n metadata:\n labels:\n role: leader\n spec:\n containers:\n - name: vllm-leader\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-and-lnc-config\n workerTemplate:\n metadata:\n labels:\n role: worker\n spec:\n containers:\n - name: vllm-worker\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-and-lnc-config\n" }, { "path": "containers/get-started/quickstart-configure-deploy-dlc.rst", "content": ".. meta::\n :description: Learn how to deploy a vLLM server using preconfigured Neuron Deep Learning Container with on Trainium and Inferentia instances.\n :date_updated: 01/26/2026\n\n.. _quickstart_vllm_dlc_deploy:\n\nQuickstart: Configure and deploy a vLLM server using Neuron Deep Learning Container (DLC)\n==========================================================================================\n\nThis topic guides you through deploying a vLLM server on Trainium and Inferentia instances using a Deep Learning Container preconfigured with AWS Neuron SDK artifacts. When you complete this tutorial, you will be able run a vLLM inference server on AWS Trainium and Inferentia instances.\n\nOverview\n--------\nIn this quickstart, you will pull a vLLM Docker image, configure it for Neuron devices, and start an inference server running vLLM. This process lets you deploy large language models on AWS ML accelerators for high-performance inference workloads.\n\nBefore you start\n----------------\n\nThis tutorial assumes that you have experience in the following areas:\n\n* Docker container management\n* AWS EC2 instance administration\n* Command-line interface operations\n\nPrerequisites\n-------------\n\nBefore you begin, ensure you have:\n\n* AWS Trainium or Inferentia instance access\n* Docker installed on your instance. You can set up docker environment according to :ref:`tutorial-docker-env-setup`\n* SSH access to your instance\n\nPrepare your environment\n------------------------\n\nLaunch an AWS Trainium or Inferentia instance with sufficient resources for your model requirements. We recommend using one of the base DLAMIs to launch your instance - `Neuron Base DLAMI <#>`.\n\nStep 1: Pull the vLLM Docker image\n-----------------------------------\n\nIn this step, you will download the vLLM Docker image from AWS ECR.\n\nGet the latest vLLM Docker image from Neuron's ECR public gallery `pytorch-inference-vllm-neuronx `_ repository, and then get the latest published image tag and use it in the command below:\n\n.. code-block:: bash\n\n docker pull public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:\n\nFor example, replace ```` with an SDK 2.28.0 released DLC image tag such as ``0.13.0-neuronx-py312-sdk2.28.0-ubuntu24.04``\n\nStep 2: Start the Docker container\n-----------------------------------\n\nIn this step, you will run the container with access to Neuron devices. For this tutorial, we are using an trn1.32xlarge instance.\n\nRun the container interactively with access to Neuron devices:\n\n.. code-block:: bash\n\n docker run -it \\\n --device=/dev/neuron0 \\\n --device=/dev/neuron1 \\\n --device=/dev/neuron2 \\\n --device=/dev/neuron3 \\\n --device=/dev/neuron4 \\\n --device=/dev/neuron5 \\\n --device=/dev/neuron6 \\\n --device=/dev/neuron7 \\\n --device=/dev/neuron8 \\\n --device=/dev/neuron9 \\\n --device=/dev/neuron10 \\\n --device=/dev/neuron11 \\\n --device=/dev/neuron12 \\\n --device=/dev/neuron13 \\\n --device=/dev/neuron14 \\\n --device=/dev/neuron15 \\\n --cap-add SYS_ADMIN \\\n --cap-add IPC_LOCK \\\n -p 8080:8080 \\\n --name \\\n \\\n bash\n\n.. note::\n The trn1.32xlarge instance provides 16 Neuron devices. Adjust the number of Neuron devices (``--device=/dev/neuronX``) based on your instance type and requirements.\n\nStep 3: Start the vLLM server\n------------------------------\n\nIn this step, you will launch the vLLM inference server inside the container.\n\nInside the container, start the vLLM inference server:\n\n.. code-block:: bash\n\n vllm serve \\\n --model='TinyLlama/TinyLlama-1.1B-Chat-v1.0' \\\n --max-num-seqs=4 \\\n --max-model-len=128 \\\n --tensor-parallel-size=2 \\\n --block-size=32 \\\n --num-gpu-blocks-override=16 \\\n --port=8080 \\\n --additional-config='{\"override_neuron_config\":{\"enable_bucketing\":false}}'\n\n.. note::\n **Version compatibility**: The command above is compatible with vLLM version 0.11.0 and later. If you are using an older version (such as 0.9.1), you must:\n \n * Replace ``--additional-config='{\"override_neuron_config\":{\"enable_bucketing\":false}}'`` with ``--override-neuron-config '{\"enable_bucketing\":false}'``\n \n.. important::\n * Choose the appropriate model for your use case\n * Set ``--tensor-parallel-size`` to be less than or equal to total number of NeuronCores (or TP ranks) available from your devices, accounting for cores per device and logical core configuration\n * Server startup typically takes 5-10 minutes\n\nStep 4: Verify server status\n-----------------------------\n\nIn this step, you will confirm the server starts successfully.\n\nWait for the server to fully initialize. You will see output showing available API routes:\n\n.. code-block:: text\n\n INFO 08-12 00:04:47 [launcher.py:28] Available routes are:\n INFO 08-12 00:04:47 [launcher.py:36] Route: /health, Methods: GET\n INFO 08-12 00:04:47 [launcher.py:36] Route: /v1/chat/completions, Methods: POST\n INFO 08-12 00:04:47 [launcher.py:36] Route: /v1/completions, Methods: POST\n\n.. note::\n During startup, you may see warning logs similar to the following, which can be safely ignored:\n\n .. code-block:: text\n\n No module named 'vllm._version'\n from .version import __version__, __version_tuple__ # isort:skip\n WARNING [__init__.py:25] The vLLM package was not found, so its version could not be inspected. This may cause platform detection to fail.\n INFO [__init__.py:243] Automatically detected platform neuron.\n WARNING [_custom_ops.py:21] Failed to import from vllm._C with ModuleNotFoundError(\"No module named 'vllm._C'\")\n\nAll complete! Now, let's confirm everything works.\n\nStep 5: Inference service confirmation\n---------------------------------------\n\nTest the API to confirm your setup works correctly.\n\nOpen a separate terminal and make an API call:\n\n.. code-block:: bash\n\n curl http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"What is the capital of Italy?\"\n }\n ]\n }'\n\nYou should receive a response similar to:\n\n.. code-block:: json\n\n {\n \"id\": \"chatcmpl-ac7551dd2f2a4be3bd2c1aabffa79b4c\",\n \"object\": \"chat.completion\",\n \"created\": 1754958455,\n \"model\": \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": \"assistant\",\n \"content\": \"The capital of Italy is Rome...\",\n \"tool_calls\": []\n },\n \"finish_reason\": \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 23,\n \"total_tokens\": 106,\n \"completion_tokens\": 83\n }\n }\n\nCongratulations! You have successfully deployed a vLLM inference server using a preconfigured Neuron DLC. If you encountered any issues, see the **Common issues** section below.\n\nAvailable API endpoints\n-----------------------\n\nThe server provides various endpoints for different use cases:\n\n* **Health Check**: ``GET /health``\n* **Chat Completions**: ``POST /v1/chat/completions``\n* **Text Completions**: ``POST /v1/completions``\n* **Models Info**: ``GET /v1/models``\n* **API Documentation**: ``GET /docs``\n\nCommon issues\n-------------\n\nDid you encounter an error while working through this tutorial? Here are common issues and solutions:\n\n- **Server won't start**: Check that you have sufficient Neuron devices allocated\n- **Connection refused**: Verify the container is running and port 8080 is properly mapped\n- **Slow performance**: Ensure your ``tensor-parallel-size`` matches your available Neuron devices\n- **Memory issues**: Consider using a larger instance type or reducing model size\n\nFor additional help, refer to the complete vLLM User Guide for NxD Inference documentation.\n\nClean up\n--------\n\nTo clean up resources after completing this tutorial:\n\n1. Stop the Docker container:\n\n .. code-block:: bash\n\n docker stop \n\n2. Remove the container:\n\n .. code-block:: bash\n\n docker rm \n\n3. Terminate your EC2 instance if no longer needed.\n\nNext steps\n----------\n\nNow that you've completed this tutorial, explore these related topics:\n\n* Learn more about vLLM configuration options in the vLLM User Guide for NxD Inference\n* Explore model optimization techniques for better performance\n* Set up production deployment with load balancing and monitoring\n\nFurther reading\n---------------\n\n- `vLLM User Guide for NxD Inference <#>`_ - Complete documentation for vLLM on Neuron\n- `AWS Neuron SDK Documentation `_ - Full Neuron SDK reference\n" }, { "path": "containers/get-started/quickstart-pytorch-inference-dlc.rst", "content": ".. meta::\n :description: Learn how to run PyTorch inference using preconfigured Neuron Deep Learning Container with Llama-2-7b on Trainium instances.\n :date_updated: 02/17/2026\n\n.. _quickstart_pytorch_inference_dlc:\n\nQuickstart: Run PyTorch inference using Neuron Deep Learning Container (DLC)\n=============================================================================\n\nThis topic guides you through running PyTorch inference on Trainium instances using a Deep Learning Container preconfigured with AWS Neuron SDK artifacts. When you complete this tutorial, you will be able to run inference with the Llama-2-7b model on AWS Trainium instances.\n\nOverview\n--------\nIn this quickstart, you will pull a PyTorch inference Docker image, download the Llama-2-7b model from S3, and run an inference demo that compiles, validates, and benchmarks the model. This process lets you deploy large language models on AWS ML accelerators for high-performance inference workloads.\n\nBefore you start\n----------------\n\nThis tutorial assumes that you have experience in the following areas:\n\n* Docker container management\n* AWS EC2 instance administration\n* Command-line interface operations\n* AWS S3 operations\n\nPrerequisites\n-------------\n\nBefore you begin, ensure you have:\n\n* AWS Trainium instance access (trn2.48xlarge recommended)\n* Docker installed on your instance. You can set up docker environment according to :ref:`tutorial-docker-env-setup`\n* SSH access to your instance\n* AWS credentials configured with access to the model S3 bucket\n\nPrepare your environment\n------------------------\n\nLaunch an AWS Trainium instance with sufficient resources for your model requirements. We recommend using one of the base DLAMIs to launch your instance - `Neuron Base DLAMI <#>`.\n\nStep 1: Pull the PyTorch inference Docker image\n------------------------------------------------\n\nIn this step, you will download the PyTorch inference Docker image from AWS ECR.\n\nGet the latest PyTorch inference Docker image from Neuron's ECR public gallery `pytorch-inference-neuronx `_ repository, and then get the latest published image tag and use it in the command below:\n\n.. code-block:: bash\n\n docker pull public.ecr.aws/neuron/pytorch-inference-neuronx:\n\nFor example, replace ```` with an SDK 2.28.0 released DLC image tag such as ``2.9.0-neuronx-py312-sdk2.28.0-ubuntu24.04``\n\nStep 2: Download the Llama-2-7b model\n--------------------------------------\n\nIn this step, you will download the Llama-2-7b model from HuggingFace to an S3 bucket, then copy it to your instance.\n\nFirst, download the model from HuggingFace and upload to your S3 bucket:\n\n.. code-block:: bash\n\n # Install HuggingFace CLI if not already installed\n pip install huggingface-hub\n\n # Login to HuggingFace (you'll need to accept the Llama-2 license first)\n hf auth login\n\n # Download the model\n hf download meta-llama/Llama-2-7b --local-dir ./Llama-2-7b\n\n # Upload to your S3 bucket\n aws s3 cp --recursive ./Llama-2-7b s3://your-bucket-name/models/Llama-2-7b/\n\nThen, on your Trainium instance, download the model from S3:\n\n.. note::\n Change ``/home/ec2-user`` to ``/home/ubuntu`` if you're using an Ubuntu AMI.\n\n.. code-block:: bash\n\n # Create directory for the model\n mkdir -p /home/ec2-user/model_hf/Llama-2-7b\n\n # Download from S3\n aws s3 cp --recursive s3://your-bucket-name/models/Llama-2-7b/ /home/ec2-user/model_hf/Llama-2-7b/\n\n # Verify the model downloaded successfully\n ls /home/ec2-user/model_hf/Llama-2-7b/config.json\n\n.. note::\n You must accept the Llama-2 license on HuggingFace before you can download the model. Visit https://huggingface.co/meta-llama/Llama-2-7b to request access.\n\nStep 3: Start the Docker container\n-----------------------------------\n\nIn this step, you will run the container with access to Neuron devices and mount the model directory. For this tutorial, we are using a trn2.48xlarge instance.\n\nRun the container interactively with access to all Neuron devices:\n\n.. code-block:: bash\n\n docker run -it \\\n --device=/dev/neuron0 \\\n --device=/dev/neuron1 \\\n --device=/dev/neuron2 \\\n --device=/dev/neuron3 \\\n --device=/dev/neuron4 \\\n --device=/dev/neuron5 \\\n --device=/dev/neuron6 \\\n --device=/dev/neuron7 \\\n --device=/dev/neuron8 \\\n --device=/dev/neuron9 \\\n --device=/dev/neuron10 \\\n --device=/dev/neuron11 \\\n -v /home/ec2-user/model_hf/Llama-2-7b:/root/model_hf/Llama-2-7b \\\n --cap-add SYS_ADMIN \\\n --cap-add IPC_LOCK \\\n --name pytorch-inference-demo \\\n public.ecr.aws/neuron/pytorch-inference-neuronx: \\\n bash\n\n.. note::\n The trn2.48xlarge instance provides 12 Neuron devices. Adjust the number of Neuron devices (``--device=/dev/neuronX``) based on your instance type and requirements.\n\nStep 4: Run the inference demo\n-------------------------------\n\nIn this step, you will run the inference demo script that compiles the model, checks accuracy, and benchmarks performance.\n\nInside the container, run the inference demo:\n\n.. code-block:: bash\n\n inference_demo \\\n --model-type llama \\\n --task-type causal-lm \\\n run \\\n --model-path /root/model_hf/Llama-2-7b/ \\\n --compiled-model-path /root/traced_model/Llama-2-7b-demo/ \\\n --torch-dtype bfloat16 \\\n --tp-degree 96 \\\n --batch-size 2 \\\n --max-context-length 32 \\\n --seq-len 64 \\\n --on-device-sampling \\\n --enable-bucketing \\\n --top-k 1 \\\n --do-sample \\\n --pad-token-id 2 \\\n --prompt 'I believe the meaning of life is' \\\n --prompt 'The color of the sky is' \\\n --check-accuracy-mode token-matching \\\n --benchmark\n\n.. important::\n * The inference demo takes approximately 20 minutes to complete on a trn2.48xlarge instance\n * The script will compile the model, validate accuracy, and run benchmarks\n * Set ``--tp-degree`` to match the number of NeuronCores you want to use (96 for trn2.48xlarge)\n\nStep 5: Verify the results\n---------------------------\n\nIn this step, you will confirm the inference demo completed successfully and review the benchmark results.\n\nWait for the demo to complete. You will see output showing benchmark results:\n\n.. code-block:: text\n\n Benchmark completed and its result is as following\n {\n \"e2e_model\": {\n \"latency_ms_p50\": 8539.34,\n \"latency_ms_p90\": 8627.43,\n \"latency_ms_p95\": 8646.97,\n \"latency_ms_p99\": 8652.62,\n \"latency_ms_p100\": 8654.03,\n \"latency_ms_avg\": 8533.13,\n \"throughput\": 480.01\n },\n \"context_encoding_model\": {\n \"latency_ms_p50\": 132.42,\n \"latency_ms_p90\": 133.47,\n \"latency_ms_p95\": 133.59,\n \"latency_ms_p99\": 133.81,\n \"latency_ms_p100\": 133.86,\n \"latency_ms_avg\": 132.52,\n \"throughput\": 30908.75\n },\n \"token_generation_model\": {\n \"latency_ms_p50\": 7.84,\n \"latency_ms_p90\": 8.39,\n \"latency_ms_p95\": 8.47,\n \"latency_ms_p99\": 8.63,\n \"latency_ms_p100\": 28.96,\n \"latency_ms_avg\": 7.87,\n \"throughput\": 520434.73\n }\n }\n Completed saving result to benchmark_report.json\n\n.. note::\n You may see several red ``ERROR NRT:nrt_tensor_free`` errors at the end of the script output. These can be safely ignored - the actual benchmark results appear above these error messages.\n\nAll complete! The benchmark results are saved to ``benchmark_report.json`` in the container.\n\nUnderstanding the results\n-------------------------\n\nThe benchmark output provides three key metrics:\n\n* **e2e_model**: End-to-end model performance including context encoding and token generation\n* **context_encoding_model**: Performance of processing the input prompt\n* **token_generation_model**: Performance of generating output tokens\n\nEach metric includes:\n\n* Latency percentiles (p50, p90, p95, p99, p100) in milliseconds\n* Average latency in milliseconds\n* Throughput in tokens per second\n\nCommon issues\n-------------\n\nDid you encounter an error while working through this tutorial? Here are common issues and solutions:\n\n- **Model download fails**: Verify you have accepted the Llama-2 license on HuggingFace and have valid AWS credentials\n- **Container won't start**: Check that you have sufficient Neuron devices allocated\n- **Compilation fails**: Ensure you have enough memory and the correct PyTorch version\n- **Slow performance**: Verify your ``tp-degree`` matches your available Neuron devices\n- **Memory issues**: Consider using a larger instance type or reducing batch size\n\nFor additional help, refer to the complete NeuronX Distributed Inference documentation.\n\nClean up\n--------\n\nTo clean up resources after completing this tutorial:\n\n1. Exit the container:\n\n .. code-block:: bash\n\n exit\n\n2. Stop and remove the container:\n\n .. code-block:: bash\n\n docker stop pytorch-inference-demo\n docker rm pytorch-inference-demo\n\n3. Remove the model files if no longer needed:\n\n .. code-block:: bash\n\n rm -rf /home/ec2-user/model_hf/Llama-2-7b\n\n4. Terminate your EC2 instance if no longer needed.\n\nNext steps\n----------\n\nNow that you've completed this tutorial, explore these related topics:\n\n* Learn more about NeuronX Distributed Inference configuration options\n* Explore different model architectures and optimization techniques\n* Set up production deployment with monitoring and logging\n\nFurther reading\n---------------\n\n- `NeuronX Distributed Inference Documentation <#>`_ - Complete documentation for inference on Neuron\n- `AWS Neuron SDK Documentation `_ - Full Neuron SDK reference\n- `Llama-2 Model Card `_ - Model details and license information\n" }, { "path": "containers/getting-started.rst", "content": ".. _containers-getting-started:\n\nGetting started with Neuron DLC using Docker\n============================================\n\n.. tab-set::\n\n .. tab-item:: Training\n\n\n .. dropdown:: Launch Trn1 Instance\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /setup/install-templates/launch-instance.txt\n\n .. dropdown:: Install Drivers\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. code:: bash\n\n\n # Configure Linux for Neuron repository updates\n\n sudo tee /etc/yum.repos.d/neuron.repo > /dev/null < /dev/null <= 2.26.26.0 (:ref:`tutorials/k8s-neuron-device-plugin`)\n* MPI operator installed on the cluster\n* An MPI job spec\n\nInstructions\n------------\n\nUltraServer Init Script\n~~~~~~~~~~~~~~~~~~~~~~~\n\nDownload the UltraServer init script :download:`k8s-ultraserver-init-script.sh `\n\nTo use the script, either:\n- add it to your MPI job Dockerfile and build the image OR\n- create a new Dockerfile and build a new image from your MPI job image\n\nExample:\n\n.. code-block:: dockerfile\n\n FROM 123456789012.dkr.ecr.us-west-2.amazonaws.com/ultraserver:mpijob\n COPY ultraserver-init-script.sh /tmp/\n RUN chmod +x /tmp/ultraserver-init-script.sh\n ENTRYPOINT [\"/tmp/ultraserver-init-script.sh\"]\n\nThen add the 2 required init containers to the launcher pod.\n\nThe first init container should utilize the /etc/mpi/discover_hosts.sh script to ensure that all worker pods are ready\nbefore continuing on to the UltraServer init script.\n\nThe second init container should use the image containing ultraserver-init-script.sh. You can specify a value for\nNEURON_ULTRASERVER_NODE_CONFIG, which determines what UltraServer node config your MPI job will use, i.e. how many\nUltraServer nodes to use. Possible values are 4, 2, and 1, and the default value is 4.\n\nExample:\n\n.. code-block:: yaml\n\n apiVersion: kubeflow.org/v2beta1\n kind: MPIJob\n metadata:\n name: &job_name \n namespace: default\n spec:\n mpiReplicaSpecs:\n Launcher:\n replicas: 1\n template:\n spec:\n containers:\n - name: mpitest\n image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/ultraserver:mpijob\n ...\n initContainers:\n - name: wait-hostfilename\n image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/ultraserver:mpijob\n command:\n - bash\n - -cx\n - |\n if [[ $(cat /etc/mpi/discover_hosts.sh | wc -l) != 1 ]]; then\n date\n echo \"Ready\"\n cat /etc/mpi/discover_hosts.sh\n else\n date\n echo \"not ready ...\"\n sleep 10\n exit 1\n fi\n while read host; do\n while ! ssh $host echo $host; do\n date\n echo \"Pod $host is not up ...\"\n sleep 10\n done\n date\n echo \"Pod $host is ready\"\n done <<< \"$(/etc/mpi/discover_hosts.sh)\"\n resources: {}\n volumeMounts:\n - mountPath: /etc/mpi\n name: mpi-job-config\n - mountPath: /root/.ssh\n name: ssh-auth\n - name: ultraserver-init-container\n image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/ultraserver:init-container\n env:\n - name: NEURON_ULTRASERVER_NODE_CONFIG\n value: <\"4\", \"2\", OR \"1\">\n volumeMounts:\n - mountPath: /etc/mpi\n name: mpi-job-config\n - mountPath: /root/.ssh\n name: ssh-auth\n - mountPath: /root/ultraserver_init\n name: ultraserver-init\n ...\n volumes:\n - name: ultraserver-init\n emptyDir: {}\n\nMPI Worker Pod Affinity\n~~~~~~~~~~~~~~~~~~~~~~~\n\nSingle-node Job\n^^^^^^^^^^^^^^^\n\n2-node job\n\n.. code-block:: yaml\n\n apiVersion: kubeflow.org/v2beta1\n kind: MPIJob\n metadata:\n name: &job_name \n namespace: default\n ...\n spec:\n mpiReplicaSpecs:\n Launcher:\n ...\n Worker:\n replicas: 2\n template:\n spec:\n nodeSelector:\n node.kubernetes.io/instance-type: trn2u.48xlarge\n affinity:\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchExpressions:\n - key: training.kubeflow.org/job-name\n operator: NotIn\n values:\n - *job_name\n matchLabels:\n training.kubeflow.org/job-role: worker\n topologyKey: neuron.amazonaws.com/ultraserver-server-id-2\n podAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n training.kubeflow.org/job-role: worker\n training.kubeflow.org/job-name: *job_name\n topologyKey: neuron.amazonaws.com/ultraserver-server-id-2\n ...\n\n4-node job\n\n.. code-block:: yaml\n\n apiVersion: kubeflow.org/v2beta1\n kind: MPIJob\n metadata:\n name: &job_name \n namespace: default\n ...\n spec:\n mpiReplicaSpecs:\n Launcher:\n ...\n Worker:\n replicas: 4\n template:\n spec:\n nodeSelector:\n node.kubernetes.io/instance-type: trn2u.48xlarge\n affinity:\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchExpressions:\n - key: training.kubeflow.org/job-name\n operator: NotIn\n values:\n - *job_name\n matchLabels:\n training.kubeflow.org/job-role: worker\n topologyKey: neuron.amazonaws.com/ultraserver-server-id-4\n podAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n training.kubeflow.org/job-role: worker\n training.kubeflow.org/job-name: *job_name\n topologyKey: neuron.amazonaws.com/ultraserver-server-id-4\n ...\n\nMulti-node job\n^^^^^^^^^^^^^^\n\n.. code-block:: yaml\n\n apiVersion: kubeflow.org/v2beta1\n kind: MPIJob\n metadata:\n name: &job_name \n namespace: default\n ...\n spec:\n mpiReplicaSpecs:\n Launcher:\n ...\n Worker:\n replicas: 16\n template:\n spec:\n nodeSelector:\n node.kubernetes.io/instance-type: trn2u.48xlarge\n affinity:\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchExpressions:\n - key: training.kubeflow.org/job-name\n operator: NotIn\n values:\n - *job_name\n matchLabels:\n training.kubeflow.org/job-role: worker\n topologyKey: neuron.amazonaws.com/ultraserver-server-id-4\n podAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 100\n podAffinityTerm:\n labelSelector:\n matchLabels:\n training.kubeflow.org/job-role: worker\n training.kubeflow.org/job-name: *job_name\n topologyKey: neuron.amazonaws.com/ultraserver-server-id-4\n ...\n\nTo use the affinity configuration, replace with your MPI job name and add it to your workload yaml spec.\n\nConfirm your work\n-----------------\n\nTo validate that the init container is working:\n\n.. code-block::\n\n # Find the worker pods associated with your MPI job\n kubectl get pods\n\n # Get the logs of the init container\n kubectl logs -c ultraserver-init-container\n\nYou should see logs under the init container.\n\nExample:\n\n.. code-block::\n\n $ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n demo-launcher-42lh9 0/1 Init:0/2 0 4s\n demo-worker-0 1/1 Running 0 4s\n demo-worker-1 1/1 Running 0 4s\n demo-worker-2 1/1 Running 0 4s\n demo-worker-3 1/1 Running 0 4s\n\n $ kubectl logs demo-launcher-42lh9 -c ultraserver-init-container\n Using 4-node config\n ...\n\nTo validate that the affinity configuration is working:\n\n.. code-block::\n\n # Find the worker pods and the nodes they are scheduled to\n kubectl get pods -o=custom-columns='POD_NAME:metadata.name,NODE_NAME:spec.nodeName'\n\n # Compare the labels of the nodes to the\n kubectl get nodes \\\n -l neuron.amazonaws.com/ultraserver-mode \\\n -o=custom-columns='NAME:metadata.name,MODE:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-mode,ULTRASERVER_SERVER_ID_2:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-server-id-2,ULTRASERVER_NODE_ID_2:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-node-id-2,ULTRASERVER_SERVER_ID_4:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-server-id-4,ULTRASERVER_NODE_ID_4:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-node-id-4' | awk 'NR==1{print;next}{print | \"sort -k3,3 -k4,4\"}'\n\nWhen looking at the nodes used by the worker pods, they should share the same ULTRASERVER_SERVER_ID_2 or\nULTRASERVER_SERVER_ID_4 label based on which config you chose.\n\nExample when choosing a 4-node config:\n\n.. code-block::\n\n $ kubectl get pods -o=custom-columns='POD_NAME:metadata.name,NODE_NAME:spec.nodeName'\n POD_NAME NODE_NAME\n demo-launcher-42lh9 ip-172-32-5-227.ap-southeast-4.compute.internal\n demo-worker-0 ip-172-32-5-227.ap-southeast-4.compute.internal\n demo-worker-1 ip-172-32-11-17.ap-southeast-4.compute.internal\n demo-worker-2 ip-172-32-13-57.ap-southeast-4.compute.internal\n demo-worker-3 ip-172-32-9-4.ap-southeast-4.compute.internal\n\n $ kubectl get nodes \\\n -l neuron.amazonaws.com/ultraserver-mode \\\n -o=custom-columns='NAME:metadata.name,MODE:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-mode,ULTRASERVER_SERVER_ID_2:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-server-id-2,ULTRASERVER_NODE_ID_2:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-node-id-2,ULTRASERVER_SERVER_ID_4:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-server-id-4,ULTRASERVER_NODE_ID_4:metadata.labels.neuron\\.amazonaws\\.com/ultraserver-node-id-4' | awk 'NR==1{print;next}{print | \"sort -k3,3 -k4,4\"}'\n\n NAME MODE ULTRASERVER_SERVER_ID_2 ULTRASERVER_NODE_ID_2 ULTRASERVER_SERVER_ID_4 ULTRASERVER_NODE_ID_4\n ip-172-32-11-17.ap-southeast-4.compute.internal 1_2_4 u5wy80u0o2saugxy 0 bog79p1y8tetj5uu 0\n ip-172-32-13-57.ap-southeast-4.compute.internal 1_2_4 u5wy80u0o2saugxy 1 bog79p1y8tetj5uu 1\n ip-172-32-5-227.ap-southeast-4.compute.internal 1_2_4 ygml2651y0lwdd46 0 bog79p1y8tetj5uu 2\n ip-172-32-9-4.ap-southeast-4.compute.internal 1_2_4 ygml2651y0lwdd46 1 bog79p1y8tetj5uu 3\n\nCommon issues\n-------------\n\nInit script fails to start\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf at least one of the worker pods isn't scheduled to a node, the init script will fail to start.\n\nExample:\n\n.. code-block::\n\n $ kubectl get pods -o=custom-columns='POD_NAME:metadata.name,NODE_NAME:spec.nodeName'\n POD_NAME NODE_NAME\n demo-launcher-96xsl ip-172-32-9-4.ap-southeast-4.compute.internal\n demo-worker-0 \n demo-worker-1 \n demo-worker-2 \n demo-worker-3 \n\n $ kubectl logs demo-launcher-96xsl -c ultraserver-init-container\n Error from server (BadRequest): container \"ultraserver-init-container\" in pod \"demo-launcher-96xsl\" is waiting to start: PodInitializing\n\nPossible solution: Check your pods for affinity/scheduling issues.\n\n.. code-block::\n\n $ kubectl describe pod demo-worker-0\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Warning FailedScheduling 3m13s default-scheduler 0/4 nodes are available: 4 node(s) didn't match pod affinity rules. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.\n\nRelated Information\n-------------------\n\n- :ref:`kubernetes-getting-started` - Information about how to use Neuron on EKS\n- :ref:`tutorials/k8s-neuron-device-plugin` - Information about Neuron Device Plugin\n- :ref:`aws-trn2-arch` - Information about trn2 UltraServer architecture\n- :ref:`general-troubleshooting` - Information about general troubleshooting for Neuron\n- `MPI Operator `_ - Information about MPI Operator\n- `MPI User Guide `_ - Information about MPI jobs\n- `Kubernetes Pod Affinity `_ - Information about pod affinity rules\n- `YAML anchors `_ - Information about YAML anchors\n" }, { "path": "containers/index.rst", "content": ".. meta::\n :description: AWS Neuron Deep Learning Containers (DLCs) are pre-configured Docker images for training and serving models on AWS Trainium and Inferentia instances with the Neuron SDK.\n :keywords: Neuron Containers, Deep Learning Containers, DLC, Docker, Kubernetes, EKS, ECS, AWS Neuron, Trainium, Inferentia, vLLM, Container Deployment\n :date-modified: 01/22/2026\n\n.. _neuron_containers:\n\nNeuron Containers\n=================\n\nThis section contains the technical documentation for using AWS Neuron Deep Learning Containers (DLCs) and containerized deployments on Inferentia and Trainium instances.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Getting Started \n Locate Neuron DLC Images \n Customize DLC \n Neuron Plugins \n Tutorials \n How-To Guides \n FAQ \n DRA \n Release Notes \n\nWhat are Neuron Deep Learning Containers?\n------------------------------------------\n\nAWS Neuron Deep Learning Containers (DLCs) are a set of pre-configured Docker images for training and serving models on AWS Trainium and Inferentia instances using the AWS Neuron SDK. Each DLC is optimized for specific ML frameworks and comes with all Neuron components pre-installed, enabling you to quickly deploy containerized workloads without manual setup.\n\nWith Neuron DLCs, developers can:\n\n* Deploy production-ready containers with pre-installed Neuron SDK and ML frameworks\n* Use containers across multiple deployment platforms including EC2, EKS, ECS, and SageMaker\n* Customize DLCs to fit specific project requirements\n* Leverage Neuron plugins for better observability and fault tolerance\n* Run distributed training and inference workloads with vLLM integration\n* Schedule MPI jobs on Trn2 UltraServers for improved performance\n\nNeuron DLCs support popular ML frameworks including PyTorch, TensorFlow, and JAX, and are available for both training and inference workloads on Inf1, Inf2, Trn1, Trn1n, and Trn2 instances.\n\n.. admonition:: Neuron DRA for Kubernetes\n\n Neuron has released support for Dynamic Resource Allocation (DRA) with Kubernetes. :doc:`Read more about it here `.\n\nQuickstarts\n-----------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n \n .. grid-item-card:: Quickstart: Deploy a DLC with vLLM\n :link: quickstart_vllm_dlc_deploy\n :link-type: ref\n :class-card: sd-rounded-3\n \n Get started by configuring and deploying a Deep Learning Container with vLLM for inference. Time to complete: ~30 minutes.\n\n .. grid-item-card:: Quickstart: Build a Custom Neuron Container\n :link: containers-getting-started\n :link-type: ref\n :class-card: sd-rounded-3\n \n Learn how to build a custom Neuron container using Docker for training or inference workloads.\n\nNeuron Containers Documentation\n--------------------------------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n \n .. grid-item-card:: Getting Started\n :link: containers-getting-started\n :link-type: ref\n :class-card: sd-rounded-3\n \n Step-by-step guide for building Neuron containers using Docker, including driver installation and container setup.\n\n .. grid-item-card:: Locate Neuron DLC Images\n :link: locate-neuron-dlc-image\n :link-type: ref\n :class-card: sd-rounded-3\n \n Find the right pre-configured Deep Learning Container image for your ML framework and instance type.\n\n .. grid-item-card:: Customize Neuron DLC\n :link: containers-dlc-then-customize-devflow\n :link-type: ref\n :class-card: sd-rounded-3\n \n Learn how to customize Neuron Deep Learning Containers to fit your specific project requirements.\n\n .. grid-item-card:: Neuron Plugins\n :link: neuron-container-plugins\n :link-type: ref\n :class-card: sd-rounded-3\n \n Explore Neuron plugins for containerized environments, providing better observability and fault tolerance.\n\n .. grid-item-card:: Tutorials\n :link: /containers/tutorials\n :link-type: doc\n :class-card: sd-rounded-3\n \n Hands-on tutorials for deploying containers on EC2, EKS, ECS, and other platforms with various configurations.\n\n .. grid-item-card:: How-To: Schedule MPI Jobs on UltraServers\n :link: containers-how-to-ultraserver\n :link-type: ref\n :class-card: sd-rounded-3\n \n Learn how to schedule MPI jobs to run on Neuron UltraServers in EKS for improved performance.\n\n .. grid-item-card:: FAQ & Troubleshooting\n :link: container-faq\n :link-type: ref\n :class-card: sd-rounded-3\n \n Frequently asked questions and solutions for common issues with Neuron containers.\n\n .. grid-item-card:: Neuron Containers Release Notes\n :link: /release-notes/components/containers\n :link-type: doc\n :class-card: sd-rounded-3\n \n Review the latest updates, new DLC images, and improvements in Neuron container releases.\n" }, { "path": "containers/k8.rst", "content": ".. _self-managed-kubernetes-service:\n\nSelf Managed Kubernetes Service\n===============================\nIntroduction\n------------\nUse of Neuron in containers on a Kubernetes cluster can be simple to achieve by following :ref:`tutorial-k8s-env-setup-for-neuron`\n\nKnown Limitations\n-----------------\nScheduling on k8s cluster requires contiguous neuron device-ids. Neuron provides a scheduler extension to solve this problem for self-managed k8 clusters. Read more about it here: :ref:`neuron-k8-scheduler-ext`.\n" }, { "path": "containers/kubernetes-getting-started.rst", "content": ".. _kubernetes-getting-started:\n\nUsing Neuron with Amazon EKS\n=============================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n.. _tutorial-k8s-env-setup-for-neuron:\n\nEKS Setup for Neuron\n--------------------\n\nCustomers that use Kubernetes can conveniently integrate Inf/Trn instances into their workflows. This section provides step-by-step instructions for setting up an EKS cluster with Neuron support.\n\nPrerequisites\n~~~~~~~~~~~~~\n\n.. include:: /containers/tutorials/k8s-prerequisite.rst\n\nNeuron Helm Chart\n~~~~~~~~~~~~~~~~~\n\n.. include:: /containers/tutorials/k8s-neuron-helm-chart.rst\n\n.. _k8s-neuron-device-plugin:\n\nNeuron Device Plugin\n~~~~~~~~~~~~~~~~~~~~\n\n.. include:: /containers/tutorials/k8s-neuron-device-plugin.rst\n\n.. _neuron_scheduler:\n\nNeuron Scheduler Extension\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. include:: /containers/tutorials/k8s-neuron-scheduler.rst\n\nNeuron Node Problem Detector and Recovery\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. include:: /containers/tutorials/k8s-neuron-problem-detector-and-recovery-irsa.rst\n\n.. include:: /containers/tutorials/k8s-neuron-problem-detector-and-recovery.rst\n\nNeuron Monitor Daemonset\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. include:: /containers/tutorials/k8s-neuron-monitor.rst\n" }, { "path": "containers/locate-neuron-dlc-image.rst", "content": ".. _locate-neuron-dlc-image:\n\nNeuron Deep Learning Containers\n===============================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nAWS Deep Learning Containers (DLCs) provide a set of Docker images that are pre-installed with deep learning frameworks.\nThe containers are optimized for performance and available in Amazon Elastic Container Registry (Amazon ECR).\nDLCs make it straightforward to deploy custom ML environments in a containerized manner,\nwhile taking advantage of the portability and reproducibility benefits of containers.\n\nAWS Neuron DLCs are a set of Docker images for training and serving models on AWS Trainium and Inferentia instances using AWS Neuron SDK.\nThe sections below list all of the AWS Neuron DLCs, as well as the AWS DLCs that come pre-installed with the Neuron SDK.\n\n\nInference Containers\n--------------------\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - DLC Name\n - DLC Link(s)\n - Tutorial(s)\n\n * - Neuron Inference Containers\n - | `Neuron PyTorch Inference Containers `_\n | `Neuronx PyTorch Inference Containers `_\n | `Neuronx PyTorch vLLM Inference Containers `_\n - | :ref:`tutorial-infer`\n | :ref:`torchserve-neuron`\n | :ref:`quickstart_vllm_dlc_deploy`\n\n * - Large Model Inference (LMI)/Deep Java Library (DJL) Containers\n - `LMI Containers `_\n -\n\n * - HuggingFace Inference Containers\n - | `HuggingFace Neuron Inference Containers `_\n | `HuggingFace Neuron vLLM Containers `_\n | `HuggingFace Text Generation Inference (TGI) Containers `_\n -\n\n * - Triton Inference Containers\n - `NVIDIA Triton Inference Containers `_\n -\n\n\nTraining Containers\n-------------------\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - DLC Name\n - DLC Link(s)\n - Tutorial(s)\n\n * - Neuron Training Containers\n - | `Neuronx PyTorch Training Containers `_\n | `Neuronx Jax Training Containers `_\n - :ref:`tutorial-training`\n\n * - HuggingFace Training Containers\n - `HuggingFace Neuron Training Containers `_\n -\n\n.. note::\n Latest HuggingFace Neuron containers are also available on the `HuggingFace Optimum website `_.\n\n\nGetting started with Neuron DLC using Docker\n----------------------------------------------\n\n:ref:`containers-getting-started`\n\n\nUsing containers on AWS services\n----------------------------------\n\n:ref:`Amazon EKS`\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n:ref:`Amazon ECS`\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n:ref:`Amazon SageMaker`\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n:ref:`AWS Batch`\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\nCustomizing Neuron Deep Learning Containers\n-------------------------------------------\nDeep Learning Containers can be customized to fit your specific project needs.\nTo read more, visit :ref:`containers-dlc-then-customize-devflow`.\n" }, { "path": "containers/neo-then-hosting-devflow.rst", "content": ".. include:: /devflows/inference/neo-then-hosting-devflow.rst" }, { "path": "containers/neuron-dra.rst", "content": ".. meta::\n :description: AWS Neuron Dynamic Resource Allocation (DRA) for Kubernetes\n :keywords: AWS, Neuron, DRA, Kubernetes, Dynamic Resource Allocation\n\n.. _neuron-dra:\n\n=================================================\nAWS Neuron Dynamic Resource Allocation (DRA)\n=================================================\n\nWhat is DRA?\n------------\n\nPrior to Kubernetes 1.33, Kubernetes used device plugins for resource management. The Neuron device plugin implements the\ndevice plugin interface to allow Kubernetes scheduler to manage Neuron resources. However, the device plugin framework\nonly tracks device count—the scheduler cannot see device attributes. Due to this limitation, the framework cannot natively\nfacilitate attribute-based filtering during device selection. For example, the default Kubernetes scheduler prior to DRA cannot\nsupport allocation of connected devices without additional mechanisms such as a scheduler extension.\n\nDynamic Resource Allocation (DRA) is a new framework for advanced resource management that addresses this limitation. DRA\nenables the scheduler to see the device attributes, allowing workloads to select devices based on specific attributes and\nachieve topology aware allocation. Hardware vendors determine which attributes are published for their hardware. The AWS\nNeuron DRA driver implements the kubelet plugin for DRA for AWS Trainium instances.\n\nFor more information on DRA, refer to `Kubernetes Dynamic Resource Allocation `_.\n\nWhere can I get the Neuron DRA driver and resource templates?\n-------------------------------------------------------------------\n\nTo review and download the individual resource claim templates, visit this page: \n\n* :doc:`/containers/files/index-dra`.\n\nWhat are the benefits of using DRA over device plugin?\n-------------------------------------------------------\n\n**Reduced developer complexity**\n\nDevice plugin-based workloads use node labels along with request and limits to allocate right resources. Example:\n\n.. code-block:: yaml\n\n Worker:\n replicas: 4\n template:\n spec:\n containers:\n - image: .dkr.ecr.us-west-2.amazonaws.com/neuronx_nemo:latest\n name: mpitest\n imagePullPolicy: Always\n resources:\n limits:\n aws.amazon.com/neuron: \"16\"\n vpc.amazonaws.com/efa: \"16\"\n requests:\n aws.amazon.com/neuron: \"16\"\n vpc.amazonaws.com/efa: \"16\"\n volumeMounts:\n - name: dshm\n mountPath: /dev/shm\n volumes:\n - name: dshm\n emptyDir:\n medium: Memory\n\nDRA introduces ``ResourceClaim`` and ``ResourceClaimTemplates`` which provide abstraction:\n\n.. code-block:: yaml\n\n Worker:\n replicas: 4\n template:\n spec:\n containers:\n - image: .dkr.ecr.us-west-2.amazonaws.com/neuronx_nemo:latest\n name: mpitest\n imagePullPolicy: Always\n resources:\n claims:\n - name: neurons\n volumeMounts:\n - name: dshm\n mountPath: /dev/shm\n volumes:\n - name: dshm\n emptyDir:\n medium: Memory\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: efa-neurons-4-devices\n\nThe ``ResourceClaimTemplate`` name is a given name and can be defined by the ML infra operators to be friendly to their developers. The RCT\ndefinition translates the name into the underlying allocation details - these are abstracted away from ML developers.\n\n**Rich interface for resource requests**\n\nWith DRA, resource requests can specify attribute-based selection. For example, RCT can follow requests, which was not possible to\ndo with device plugins without additional node labeling and extensions. This interface allows us to facilitate topology-aware scheduling.\n\n* Allocate connected neuron devices from trn2 instance type and the devices in the set need to be running specified Neuron driver version.\n* Allocate a specific set of neuron devices for my pod - I want the pod to use devices in row 1 of the topology.\n\n**Dynamic configuration**\n\nDRA allows end users to specify additional configuration for the device via RCT. The Neuron DRA driver leverages this capability to\nallow ResourceClaimTemplates to specify LNC size to be used for the allocation. An example is shown below. The end user need\nnot configure LNC via launch template while using Neuron devices with Neuron DRA driver.\n\n.. code-block:: yaml\n\n #Template will be vended by Neuron via documentation/code repo\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n namespace: neuron-test7\n name: lnc-neurons\n spec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: device.attributes['neuron.aws.com'].instanceType == \"trn2.48xlarge\"\n allocationMode: all\n config:\n - opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: NeuronConfig\n logicalNeuronCore: 1\n requests: [\"neurons\"]\n\nPrerequisites\n-----------------------------\n\n* **Kubernetes version** - Please use K8s control plane 1.34+\n* **Instance type** - Trn2.48xlarge launched with K8s version 1.34.2+\n\nFor instructions on how to setup an EKS cluster, please refer to :ref:`prerequisites`.\n\nInstallation via Helm\n---------------------\n\nConnect to your cluster from local box. The cluster should have at least one trn2.48xlarge node. \nDo not install the Neuron device plugin on the cluster! \n\nPlease confirm the cluster being used via:\n\n.. code-block:: bash\n\n kubectl config current-context\n\nThen install the DRA driver:\n\n.. code-block:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart \\\n --set \"devicePlugin.enabled=false\" --set \"npd.enabled=false\" --set \"draDriver.enabled=true\"\n\nExample 1 – Connected Neuron Devices\n--------------------------------------\n\nThis section will demonstrate how to run a workload that needs to request a subset of connected Neuron Devices from a trn2.48xlarge instance.\nBefore DRA, this use case required using Neuron Scheduler Extension. With DRA, this allocation is enabled natively.\n\n* [:download:`Download example YAML file `]\n\nThe supported subsets include set of 1, 4, 8 or 16. Specifically, these are ``resource.aws.com/devicegroup1_id``, ``resource.aws.com/devicegroup4_id``,\n``resource.aws.com/devicegroup8_id``, ``resource.aws.com/devicegroup16_id`` respectively.\n\nThe sets of 4 and 8 are selected as shown in diagram below:\n\n.. image:: /containers/images/neuron-dra-connected-devices.jpeg\n :alt: Connected Neuron Devices\n :width: 600px\n\nTo enable a workload to consume a connected subset of Neuron Devices, first create a ``ResourceClaimTemplate`` that requests a connected set of\nNeuron devices. From the package run:\n\n.. code-block:: bash\n\n kubectl apply -f specs/1x4-connected-devices.yaml\n\nThis workload definition (which includes the ``ResourceClaimTemplate``) is shown below for quick reference:\n\n.. code-block:: yaml\n\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n name: 1x4-connected-neurons\n spec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n allocationMode: ExactCount\n count: 4\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\"\n constraints:\n - requests: [\"neurons\"]\n matchAttribute: \"resource.aws.com/devicegroup4_id\"\n\nNext step is to reference the ``ResourceClaimTemplate`` in a pod definition as shown below:\n\n.. code-block:: yaml\n\n ---\n apiVersion: v1\n kind: Pod\n metadata:\n name: pod0\n labels:\n app: pod\n spec:\n containers:\n - name: ctr0\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command: [\"bash\", \"-c\"]\n args: [\"export; trap 'exit 0' TERM; sleep 9999 & wait\"]\n resources:\n claims:\n - name: neurons\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: 1x4-connected-neurons\n\n\nDeploy the above workload using ``kubectl apply``. When the pod is running, examine the related ``ResourceClaim`` using:\n\n.. code-block:: bash\n\n kubectl get resourceclaim -o yaml\n\nThe ``resourceclaim`` output will show the 4 Neuron Devices that were allocated to the pod. An example is shown below. These will be connected Neuron\nDevices.\n\n.. code-block:: bash\n\n [devbox]$ kubectl get pod\n \n NAME READY STATUS RESTARTS AGE\n ---------------------------------------\n pod0 1/1 Running 0 3s\n \n [devbox]$ kubectl get resourceclaim\n \n NAME STATE AGE\n ---------------------------------------------\n pod0-neurons-zdk76 allocated,reserved 9s\n \n [devbox]$ kubectl get resourceclaim pod0-neurons-zdk76 -o yaml\n\nStatus shown below:\n\n.. code-block:: yaml\n\n status:\n allocation:\n devices:\n results:\n - adminAccess: null\n device: neurondevice2\n driver: neuron.aws.com\n pool: ip-1-1-1-1.region.compute.internal\n request: neurons\n - adminAccess: null\n device: neurondevice3\n driver: neuron.aws.com\n pool: ip-1-1-1-1.region.compute.internal\n request: neurons\n - adminAccess: null\n device: neurondevice1\n driver: neuron.aws.com\n pool: ip-1-1-1-1.region.compute.internal\n request: neurons\n - adminAccess: null\n device: neurondevice0\n driver: neuron.aws.com\n pool: ip-1-1-1-1.region.compute.internal\n request: neurons\n\n.. note::\n The RCT name can be simplified to communicate the intent of the allocation and abstract the allocation details away from ML developers.\n\n**Example RCT1 - \"xl\" - Allocate All 16 devices**\n\n.. code-block:: yaml\n\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n name: xl-trn2\n spec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly: \n allocationMode: ExactCount\n count: 16\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\n\n**Example RCT2 - large - Allocate 8 devices**\n\n.. code-block:: yaml\n\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n name: l-trn2\n spec:\n spec:\n devices:\n constraints:\n - matchAttribute: resource.aws.com/devicegroup8_id\n requests:\n - neurons\n requests:\n - name: neurons\n exactly:\n allocationMode: ExactCount\n count: 8\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\n\n**Example RCT2 - 2.27-driver – Allocate 8 devices with driver version at the driver published by Neuron SDK 2.27**\n\n`Neuron 2.27.0 Runtime `_\n\n.. code-block:: yaml\n\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n name: 2.27-driver-trn2\n spec:\n spec:\n devices:\n constraints:\n - matchAttribute: resource.aws.com/devicegroup8_id\n requests:\n - neurons\n requests:\n - name: neurons\n exactly:\n allocationMode: ExactCount\n count: 8\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge' &&\n device.attributes['neuron.aws.com'].neuronDriverVersion == '2.25.4.0'\n\nExample 2 - Dynamic LNC config\n------------------------------\n\nThis example shows how to set LNC per workload. Earlier, overriding LNC on a Node required a node template. With DRA, workloads can\noverride default LNC via ``ResourceClaim.``\n\n* [:download:`Download example YAML file `]\n\n\nApply the following workload definition:\n\n.. code-block:: bash\n\n kubectl apply -f specs/lnc-setting-trn2.yaml\n\nThis workload definition (which includes the ``ResourceClaimTemplate``) is shown below for quick reference:\n\n.. code-block:: yaml\n\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n name: all-neurons-lnc-1\n spec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly:\n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].instanceType == 'trn2.48xlarge'\"\n allocationMode: All\n config:\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: NeuronConfig\n logicalNeuronCore: 1\n\nThen deploy a pod that references the above ``ResourceClaimTemplate`` as shown below:\n\n.. code-block:: yaml\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: pod0\n labels:\n app: pod\n spec:\n containers:\n - name: ctr0\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command: [\"bash\", \"-c\"]\n args: [\"export; trap 'exit 0' TERM; sleep 9999 & wait\"]\n resources:\n claims:\n - name: neurons\n resourceClaims:\n - name: neurons\n resourceClaimTemplateName: all-neurons-lnc-1\n\nExample 3 – Four Node Inference on trn2u.48xlarge\n--------------------------------------------------\n\nA trn2u.48xlarge Trn2 UltraServer has 4 Trn2 nodes interconnected by Neuron Links.\n\ntrn2u.48xlarge instances can be allocated in set of 1, 2, or 4. The Neuron DRA driver can utilize 1 or more ``ResourceClaimTemplate`` definitions to convey the\ndesired size of the set. The ``ResourceClaimTemplate`` allows end users to specify \"UltraServerConfig\" to declare their intent to use all 4 nodes of\nthe UltraServer. This configuration value is passed by the Neuron DRA driver to the Neuron runtime and collectives inside the container.\n\n* [:download:`Download example YAML file `]\n\nExample yaml for 4-node inference on trn2u.48xlarge:\n\n.. code-block:: yaml\n\n apiVersion: resource.k8s.io/v1\n kind: ResourceClaimTemplate\n metadata:\n name: us-4-node-config\n spec:\n spec:\n devices:\n requests:\n - name: neurons\n exactly: \n deviceClassName: neuron.aws.com\n selectors:\n - cel:\n expression: \"device.attributes['neuron.aws.com'].resourceType == 'neuron_node'\"\n allocationMode: ExactCount\n count: 1\n config:\n - requests: [\"neurons\"]\n opaque:\n driver: neuron.aws.com\n parameters:\n apiVersion: neuron.aws.com/v1\n kind: UltraServerConfig\n ultraserverMode: 4\n ---\n apiVersion: leaderworkerset.x-k8s.io/v1\n kind: LeaderWorkerSet\n metadata:\n name: vllm\n annotations:\n leaderworkerset.sigs.k8s.io/exclusive-topology: neuron.amazonaws.com/ultraserver-server-id-4\n spec:\n rolloutStrategy:\n type: RollingUpdate\n rollingUpdateConfiguration:\n maxUnavailable: 1\n maxSurge: 1\n # Two replica groups of 4 nodes each, i.e. two ultraservers\n replicas: 2\n leaderWorkerTemplate:\n size: 4\n restartPolicy: RecreateGroupOnPodRestart\n leaderTemplate:\n metadata:\n labels:\n role: leader\n spec:\n containers:\n - name: vllm-leader\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-4-node-config\n workerTemplate:\n metadata:\n labels:\n role: worker\n spec:\n containers:\n - name: vllm-worker\n image: public.ecr.aws/ubuntu/ubuntu:22.04\n command:\n - sh\n - -c\n - \"sleep infinity\"\n resources:\n claims:\n - name: one-node-from-ultraserver\n resourceClaims:\n - name: one-node-from-ultraserver\n resourceClaimTemplateName: us-4-node-config\n\n\nNeuron DRA Driver Attributes Reference\n---------------------------------------\n\nThe Neuron DRA driver publishes the following attributes in resource slices. These attributes can be used in ``ResourceClaimTemplate`` CEL expressions\nto filter and select specific devices for allocation.\n\nCommon Attributes\n^^^^^^^^^^^^^^^^^\n\nThese attributes are common to all Neuron instances and their devices:\n\n* ``deviceId`` - An integer value representing the ID of the Neuron device. Used to identify which device is chosen from allocation.\n* ``instanceType`` - A string value representing the EC2 instance type of the Neuron device. Used to specify devices of which instance(s) to choose for allocation.\n* ``neuronDriverVersion`` - A string value representing the Neuron driver version running on the instance. Used to claim instances with the same driver version for allocation.\n* ``draDriverVersion`` - A version value of the Neuron DRA driver version. Provides visibility on which Neuron DRA driver version published the resource slice.\n* ``resourceType`` - A string value to distinguish between devices and UltraServer nodes. For devices, this value is ``neuron_device``. For UltraServers, this value is ``neuron_node``.\n* ``networkNodeLayer1`` - A string value representing network node layer 1. Can be used during topology-aware scheduling to minimize network latency and optimize instance placement. See `EC2 Instance Topology `_.\n* ``networkNodeLayer2`` - A string value representing network node layer 2. Can be used to allocate workloads to nodes on the same spine. See `EC2 Instance Topology `_.\n* ``networkNodeLayer3`` - A string value representing network node layer 3. Can be used during topology-aware scheduling to minimize network latency and optimize instance placement. See `EC2 Instance Topology `_.\n\nTrn Non-UltraServer Attributes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThese attributes are only populated for Neuron instances that have grid topology (trn) and are not UltraServers:\n\n* ``topology_x`` - An integer value representing the row of the device in a grid topology. Only populated when the number of devices in the instance is greater than 1. Can be used to select a specific device or devices that belong to the same row.\n* ``topology_y`` - An integer value representing the column of the device in a grid topology. Only populated when the number of devices in the instance is greater than 1. Can be used to select a specific device or devices that belong to the same column.\n* ``topology4_id`` - An integer value representing the row of the device in a grid topology. Only populated when the number of devices in the instance is greater than 1. Can be used to select devices that belong to the same row.\n* ``topology8_id`` - An integer value representing the row of the device in a grid topology. Only populated when the number of devices in the instance is greater than or equal to 8. Can be used to select devices that belong to the same two rows.\n\nTrn UltraServer Attributes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThese attributes are only populated for Neuron instances that have grid topology (trn) and are UltraServers:\n\n* ``capacityBlockId`` - A string value representing the ID of the capacity block that the UltraServer instance is in. See `Instance Topology API `_.\n\nEFA-Enabled Instance Attributes\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThese attributes are only populated for Neuron instances that are EFA-enabled:\n\n* ``resource.aws.com/devicegroup1_id`` - A string value representing the EFA Bus:Device:Function (BDF) corresponding to that device.\n* ``resource.aws.com/devicegroup4_id`` - A string value representing a hash, ensuring Neuron devices in the same topology group of 4 get the same group ID.\n* ``resource.aws.com/devicegroup8_id`` - A string value representing a hash, ensuring Neuron devices in the same topology group of 8 get the same group ID.\n* ``resource.aws.com/devicegroup16_id`` - A string value representing a hash, ensuring Neuron devices in the same topology group of 16 get the same group ID.\n\nFAQs\n----\n\nCan DRA plugin co-exist with other device plugins?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDevice plugins and the DRA plugin can coexist in the same cluster, but **not** for the same node. As of now, the two mechanisms act independently. Neuron is preparing\nan upcoming feature that will allow device plugin based allocations to work with DRA, but the feature is still in alpha and not enabled on EKS.\nRef: `Extended Resource `_.\n\nIs DRA replacing Neuron Device Plugin and Scheduler Extension?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWe will continue to support the Neuron Device Plugin and Scheduler Extension as long as:\n\n1. Upstream Kubernetes continues to support device plugins.\n2. EKS continues to support Kubernetes versions below 1.34 (which do not support DRA).\n\nWhat Kubernetes versions are supported?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nKubernetes control plane must be on 1.34. For Node AMI, we support 1.34.2+. We do not support Node AMI for 1.34.0 or 1.34.1\nsince it had a regression in DRA. Upstream issue: `Kubernetes Issue #133920 `_\n\nWhere can I learn more about how to put together RCT using CEL expressions?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo learn more about RCTs, please visit `Kubernetes Dynamic Resource Allocation `_. To learn more\nabout CEL expressions, please visit `CEL Language `_. Send us feedback and let us know which additional RCT examples you would like\nus to provide in the source code.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Support Files \n" }, { "path": "containers/neuron-plugins.rst", "content": ".. _neuron-container-plugins:\n\nNeuron Plugins for Containerized Environments\n=============================================\n\nThis section provides an overview of the Neuron infrastructure components for containerized environments. For detailed setup instructions, see :ref:`tutorial-k8s-env-setup-for-neuron`.\n\nNeuron Device Plugin\n--------------------\n\nExposes Neuron hardware resources to Kubernetes as schedulable resources (``aws.amazon.com/neuron`` and ``aws.amazon.com/neuroncore``). The device plugin discovers Neuron devices on each node, advertises them to the scheduler, and manages allocation to Pods with exclusive access.\n\nNeuron Scheduler Extension\n---------------------------\n\nProvides topology-aware scheduling for optimal Neuron device allocation. It considers device connectivity and placement to ensure efficient utilization. This component is optional and most beneficial for workloads requesting specific subsets of Neuron devices or cores.\n\nNeuron Node Problem Detector and Recovery\n------------------------------------------\n\nMonitors Neuron device health and detects hardware and software errors. When unrecoverable issues occur, it can mark nodes as unhealthy and trigger node replacement. It also publishes CloudWatch metrics under the ``NeuronHealthCheck`` namespace for monitoring.\n\nFor ECS environments, see :ref:`ecs-neuron-problem-detector-and-recovery`.\n\nNeuron Monitor\n--------------\n\nCollects and exposes metrics from Neuron devices including hardware utilization, performance counters, memory usage, and device health. Supports integration with observability platforms like Prometheus for monitoring and alerting.\n\nNeuron Dynamic Resource Allocation (DRA) Driver\n-----------------------------------------------\n\nManages Neuron hardware resources in a Kubernetes environment. It integration with Kubernetes Dynamic Resource Allocation (DRA) framework to advertise Neuron devices and their attributes. This feature cannot be used alongside Neuron device plugin for nodes of the same cluster. For more information on Neuron DRA driver, please refer to :ref:`neuron-dra`\n\n" }, { "path": "containers/neuron_dlc_images.csv", "content": "Framework,Neuron Package,Job Type,Supported EC2 Instance Types,Python Version Options,ECR Public Repo URL,Image Details,Other Packages\nPyTorch 2.1.2,\"aws-neuronx-tools, neuronx_distributed, torch-neuronx, transformers-neuronx\",inference,trn1 and inf2,3.10 (py310),https://gallery.ecr.aws/neuron/pytorch-inference-neuronx,https://github.com/aws-neuron/deep-learning-containers#pytorch-inference-neuronx,torchserve\nPyTorch 2.1.2,\"aws-neuronx-tools, neuronx_distributed, torch-neuronx\",training,trn1 and inf2,3.10 (py310),https://gallery.ecr.aws/neuron/pytorch-training-neuronx,https://github.com/aws-neuron/deep-learning-containers#pytorch-training-neuronx,\nPyTorch 1.13.1,\"aws-neuronx-tools, torch-neuron\",inference,inf1,3.10 (py310),https://gallery.ecr.aws/neuron/pytorch-inference-neuron,https://github.com/aws-neuron/deep-learning-containers#pytorch-inference-neuron,torchserve\nPyTorch 1.13.1,\"aws-neuronx-tools, neuronx_distributed, torch-neuronx, transformers-neuronx\",inference,trn1 and inf2,3.10 (py310),https://gallery.ecr.aws/neuron/pytorch-inference-neuronx,https://github.com/aws-neuron/deep-learning-containers#pytorch-inference-neuronx,torchserve\nPyTorch 1.13.1,\"aws-neuronx-tools, neuronx_distributed, torch-neuronx\",training,trn1 and inf2,3.10 (py310),https://gallery.ecr.aws/neuron/pytorch-training-neuronx,https://github.com/aws-neuron/deep-learning-containers#pytorch-training-neuronx,\n" }, { "path": "containers/troubleshooting.rst", "content": ".. _container-troubleshooting:\n\nTroubleshooting Neuron Containers\n=================================\n\nThis document aims to provide more information on how to fix issues you\nmight encounter while using the Neuron Containers. For each\nissue we will provide an explanation of what happened and what can\npotentially correct the issue.\n\n\nIf your issue is not listed below or you have a more nuanced problem, contact\nus via `issues `__ posted\nto this repo, the `AWS Neuron developer\nforum `__, or\nthrough AWS support.\n\nNeuron Container includes the following Neuron Components. For issues relating to \nthese components inside the container refer the individual component troubleshooting\nguides :ref:`general-troubleshooting`\n\n* Neuron Runtime/Driver\n* Pytorch/Tenosrflow/MXNet frameworks\n* Libfabric/EFA \n\nThe following are container specific issues\n\nNeuron Device Not found\n-----------------------\n\nThe neuron container expects the neuron devices to be exposed to the container as\nreferenced in :ref:`container-devices`. \n\nPlease look at the container logs to see messages like below\n\n::\n\n 2022-Sep-08 17:55:23.0768 19:19 ERROR TDRV:tdrv_get_dev_info No neuron device available\n\n\nIf the above message is seen then devices are not exposed to container\n\nSolution\n''''''''\n\n* Refer :ref:`container-devices` and make sure the devices are exposed to container\n* If specific cores are being used refer :ref:`container-cores` and make sure the cores are exposed to container\n* In kubernetes environment refer :ref:`k8s-specify-devices` or :ref:`k8s-specify-cores` to make sure neuron devices/cores are there in pods container spec\n\n\nContiguous Device ID's\n-----------------------\n\nNeuron runtime expects the inferentia/trainium device id's to be contigious. If the device id's\nare not contiguous you might see error messages like below\n\n\n::\n\n 2022-Sep-08 21:52:11.0307 7:7 ERROR TDRV:tdrv_init_mla_phase1 Could not open the nd1\n\n::\n\n 2022-Sep-08 23:00:05.0667 8:8 ERROR NRT:nrt_allocate_neuron_cores Neuron cores are not contiguous\n\n\nSolution\n''''''''\n\n* In the docker run command make sure the devices specified using --device are all contiguous\n* If oci neuron hook is used and the env variable AWS_NEURON_VISIBLE_DEVICES is used then make sure the\ndevices specified are all contiguous\n* In kubernetes environment with just the neuron device plugin running there is no guarantee that\nthe devices allocated will be contiguous. Make sure to run the neuron scheduler extension as specified in :ref:`neuron-k8-scheduler-ext`" }, { "path": "containers/tutorial-docker-runtime1.0.rst", "content": ".. _tutorial-docker-environment-setup-for-neuron-runtime-10:\n\nTutorial: Docker environment setup for Neuron Runtime 1.x\n=========================================================\n\nIntroduction\n------------\n\nA Neuron application can be deployed using docker containers. This\ntutorial describes how to configure docker to expose Inferentia devices\nto containers.\n\nOnce the environment is setup, a container can be started with\n*AWS_NEURON_VISIBLE_DEVICES* environment variable to specify desired set\nof Inferentia devices to be exposed to the container.\nAWS_NEURON_VISIBLE_DEVICES is a set of contiguous comma-seperated\ninferentia logical ids. To find out the available logical ids on your\ninstance, run the neuron-ls tool. For example, on inf1.6xlarge instance\nwith 4 inferentia devices, you may set AWS_NEURON_VISIBLE_DEVICES=\"2,3\"\nto expose the last two devices to a container. When running neuron-ls\ninside a container, you will only see the set of exposed Inferentias.\nFor example:\n\n.. code:: bash\n\n docker run --env AWS_NEURON_VISIBLE_DEVICES=\"0\" neuron-test neuron-ls\n\nWould produce the following output:\n\n::\n\n +--------------+---------+--------+-----------+-----------+------+------+\n | PCI BDF | LOGICAL | NEURON | MEMORY | MEMORY | EAST | WEST |\n | | ID | CORES | CHANNEL 0 | CHANNEL 1 | | |\n +--------------+---------+--------+-----------+-----------+------+------+\n | 0000:00:1f.0 | 0 | 4 | 4096 MB | 4096 MB | 0 | 0 |\n +--------------+---------+--------+-----------+-----------+------+------+\n\nSteps:\n------\n\nThis tutorial starts from a fresh Ubuntu Server 16.04 LTS AMI\n\"ami-08bc77a2c7eb2b1da\".\n\nStep 1: install aws-neuron-runtime-base package\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFollow the :ref:`install-guide-index` to\nsetup access to Neuron repos. Then, install the aws-neuron-runtime-base\npackage.\n\n.. code:: bash\n\n sudo apt-get install aws-neuron-runtime-base\n\nStep 2: Make sure that the neuron-rtd service is not running\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIf neuron-rtd is running on the host, stop the neuron-rtd service before\nstarting the containerized neuron-rtd. This is needed to allow\nassignment of devices to containers:\n\n.. code:: bash\n\n sudo service neuron-rtd stop\n\nStep 3: install oci-add-hooks dependency\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n`oci-add-hooks `__ is an OCI\nruntime with the sole purpose of injecting OCI prestart, poststart, and\npoststop hooks into a container config.json before passing along to an\nOCI compatable runtime. oci-add-hooks is used to inject a hook that\nexposes Inferentia devices to the container.\n\n.. code:: bash\n\n sudo apt install -y golang && \\\n export GOPATH=$HOME/go && \\\n go get github.com/joeshaw/json-lossless && \\\n cd /tmp/ && \\\n git clone https://github.com/awslabs/oci-add-hooks && \\\n cd /tmp/oci-add-hooks && \\\n make build && \\\n sudo cp /tmp/oci-add-hooks/oci-add-hooks /usr/local/bin/\n\n.. _step-4-setup-docker-to-use-oci-neuron-oci-runtime:\n\nStep 4: setup Docker to use oci-neuron OCI runtime.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\noci-neuron is a script representing OCI compatible runtime. It wraps\noci-add-hooks, which wraps runc. In this step, we configure docker to\npoint at oci-neuron OCI runtime. Install dockerIO:\n\n.. code:: bash\n\n sudo apt install -y docker.io\n sudo usermod -aG docker $USER\n\nLogout and log back in to refresh membership. Place daemon.json Docker\nconfiguration file supplied by Neuron SDK in default location. This file\nspecifies oci-neuron as default docker runtime:\n\n.. code:: bash\n\n sudo cp /opt/aws/neuron/share/docker-daemon.json /etc/docker/daemon.json\n sudo service docker restart\n\nIf the docker restart command fails, make sure to check if the docker\nsystemd service is not masked. More information on this can be found\nhere: https://stackoverflow.com/a/37640824\n\nVerify docker:\n\n.. code:: bash\n\n docker run hello-world\n\nExpected result:\n\n::\n\n Hello from Docker!\n This message shows that your installation appears to be working correctly.\n\n To generate this message, Docker took the following steps:\n 1. The Docker client contacted the Docker daemon.\n 2. The Docker daemon pulled the \"hello-world\" image from the Docker Hub.\n (amd64)\n 3. The Docker daemon created a new container from that image which runs the\n executable that produces the output you are currently reading.\n 4. The Docker daemon streamed that output to the Docker client, which sent it\n to your terminal.\n\n To try something more ambitious, you can run an Ubuntu container with:\n $ docker run -it ubuntu bash\n\n Share images, automate workflows, and more with a free Docker ID:\n https://hub.docker.com/\n\n For more examples and ideas, visit:\n https://docs.docker.com/get-started/\n\nBuild a docker image using provided dockerfile :ref:`neuron-runtime-dockerfile`, and use to\nverify whitelisting:\n\n.. code:: bash\n\n docker build . -f Dockerfile.neuron-rtd -t neuron-test\n\nThen run:\n\n.. code:: bash\n\n docker run --env AWS_NEURON_VISIBLE_DEVICES=\"0\" neuron-test neuron-ls\n\nExpected result:\n\n::\n\n +--------------+---------+--------+-----------+-----------+------+------+\n | PCI BDF | LOGICAL | NEURON | MEMORY | MEMORY | EAST | WEST |\n | | ID | CORES | CHANNEL 0 | CHANNEL 1 | | |\n +--------------+---------+--------+-----------+-----------+------+------+\n | 0000:00:1f.0 | 0 | 4 | 4096 MB | 4096 MB | 0 | 0 |\n +--------------+---------+--------+-----------+-----------+------+------+\n" }, { "path": "containers/tutorials/build-run-neuron-container.rst", "content": ".. _how-to-build-neuron-container:\n\nTutorial How to Build and Run a Neuron Container\n================================================\n\nIntroduction\n------------\n\nThis document explains how to build a Neuron Container using an existing Dockerfile.\n\nPre-requisites\n--------------\n#. Docker version 18 or newer is configured according to :ref:`tutorial-docker-env-setup`\n#. Inf1/Trn1 instance with available :ref:`Neuron Devices`\n#. If running a serving application such as tensorflow-model-server, torchserve or multi-model-server, make sure the appropriate ports that the server listens to are exposed using EXPOSE in the Dockerfile or the arguments ``-p 80:8080`` on the ``docker run`` command.\n\n.. _running-application-container:\n\nBuild and Run the Application Container\n---------------------------------------\nFollow the steps below for creating neuron application containers.\n\n- Build a docker image using provided dockerfile :ref:`libmode-dockerfile` for Inf1 and :ref:`trainium-dlc-dockerfile` for Trn1 (also for Trn1 the dockerfile needs mlp train script found here at :ref:`mlp-train`\n\n.. code:: bash\n\n docker build . -f Dockerfile.pt -t neuron-container:pytorch\n\n- Run the container locally:\n\n.. code:: bash\n\n docker run -it --name pt17 --device=/dev/neuron0 neuron-container:pytorch neuron-ls\n\nExpected result for Inf1:\n\n::\n\n +--------------+---------+--------+-----------+-----------+------+------+\n | PCI BDF | LOGICAL | NEURON | MEMORY | MEMORY | EAST | WEST |\n | | ID | CORES | CHANNEL 0 | CHANNEL 1 | | |\n +--------------+---------+--------+-----------+-----------+------+------+\n | 0000:00:1f.0 | 0 | 4 | 4096 MB | 4096 MB | 0 | 0 |\n +--------------+---------+--------+-----------+-----------+------+------+\n\nExpected result for Trn1:\n\n::\n\n +--------+--------+--------+-----------+---------+\n | NEURON | NEURON | NEURON | CONNECTED | PCI |\n | DEVICE | CORES | MEMORY | DEVICES | BDF |\n +--------+--------+--------+-----------+---------+\n | 0 | 4 | 8 GB | 1 | 00:1f.0 |\n +--------+--------+--------+-----------+---------+\n\n\n.. note::\n\n If instead of the --device option above if the env variable AWS_NEURON_VISIBLE_DEVICES\n is to be used then the oci hook needs to installed by following instructions in :ref:`tutorial-oci-hook`\n\n\nImportant to know\n-----------------\n\n.. _container-devices:\n\nDevices\n^^^^^^^\n\n- The docker native way is to use --device /dev/neuron# for each of the Neuron Devices intended to be passed. When using --device option ALL/all is not supported.\n\n .. code:: bash\n\n docker run --device=/dev/neuron0 --device=/dev/neuron1\n\n- If you install the aws-neuronx-oci-hook package, you will have an OCI hook that also supports use of a container environment variable AWS_NEURON_VISIBLE_DEVICES=, which intends to make things easier for multi device scenarios. Following are some examples. For setting up oci hook please refer :ref:`oci neuron hook `\n\n .. code:: bash\n\n docker run -e “AWS_NEURON_VISIBLE_DEVICES=0,1”\n docker run -e “AWS_NEURON_VISIBLE_DEVICES=ALL”\n\n- In kubernetes environment, the neuron device plugin is used for exposing the neuron device to the containers in the pod. The number of devices can be adjusted using the *aws.amazon.com/neuron* resource in the pod specification. Refer :ref:`K8s setup ` for more details\n\n .. code:: bash\n\n resources:\n limits:\n aws.amazon.com/neuron: 1\n\n .. note::\n\n Only the number of devices can be specfied.\n When only the neuron device plugin is running that does not guaratee the devices to be\n contiguous. Make sure to run the neuron scheduler extension :ref:`neuron-k8-scheduler-ext`\n so that it makes sure that contigiuous devices are allocated to the containers\n\n\n- Multiple container applications running in the same host can share the devices but the cores cannot be shared. This is similar to running multiple applications in the host. \n- In the kubernetes environment the devices cannot be shared by multiple containers in the pod\n\n.. _container-cores:\n\nCores\n^^^^^\nEach neuron device has multiple cores. The cores allocated to process/container can be controlled by\nthe environment variable NEURON_RT_VISIBLE_CORES and NEURON_RT_NUM_CORES. Please refer :ref:`nrt-configuration` for more details.\n\n- The docker native way is to use --device /dev/neuron# for each of the Neuron Devices intended to be passed. Add --env NEURON_RT_VISIBLE_CORES-1,2 to use cores 1 and 2 to this container. For example in inf1.24xlarge with 64 cores, if we want to use cores 51 & 52, the appropriate device and NEURON_RT_VISIBLE_CORES needs to be used. With 4 cores in each device, core 51 is in device 12 and 52 is in device 13\n\n .. code:: bash\n\n docker run --device=/dev/neuron12 --device=/dev/neuron13 --env NEURON_RT_VISIBLE_CORES=51,52\n\n- In kubernetes environment, the neuron device plugin is used for exposing the neuron cores to the containers in the pod. The number of cores can be adjusted using the *aws.amazon.com/neuroncore* resource in the pod specification. Refer :ref:`K8s setup ` for more details.\n\n .. code:: bash\n\n resources:\n limits:\n aws.amazon.com/neuroncore: 1\n\n .. note::\n\n Only the number of cores can be specfied.\n When only the neuron device plugin is running that does not guaratee the cores to be\n contiguous. Make sure to run the neuron scheduler extension :ref:`neuron-k8-scheduler-ext`\n so that it makes sure that contigiuous cores are allocated to the containers\n\n- Multiple container applications running in the same host cannot share the cores. This is similar to running multiple applications in the host.\n- In the kubernetes environment the cores cannot be shared by multiple containers in the pod\n" }, { "path": "containers/tutorials/inference/index.rst", "content": "Containers -- Inference Tutorials\n=================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /containers/tutorials/inference/tutorial-infer\n /containers/tutorials/inference/k8s_rn50_demo\n\n\n.. include:: /containers/tutorials/inference/index.txt" }, { "path": "containers/tutorials/inference/index.txt", "content": "* :ref:`tutorial-infer`\n* :ref:`example-deploy-rn50-as-k8s-service`\n" }, { "path": "containers/tutorials/inference/k8s_rn50_demo.rst", "content": ".. _example-deploy-rn50-as-k8s-service:\n\nDeploy a TensorFlow Resnet50 model as a Kubernetes service\n----------------------------------------------------------\n\nThis tutorial uses Resnet50 model as a teaching example on how to deploy an\ninference application using Kubernetes on the Inf1 instances.\n\nPrerequisite:\n^^^^^^^^^^^^^\n\n- Please follow instructions at :ref:`tutorial-k8s-env-setup-for-neuron` to setup k8s support on your cluster.\n- Inf1 instances as worker nodes with attached roles allowing:\n\n - ECR read access policy to retrieve container images from ECR:\n **arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly**\n - S3 access to retrieve saved_model from within tensorflow serving\n container.\n\nDeploy a TensorFlow Serving application image\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nA trained model must be compiled to an Inferentia target before it can be deployed on Inferentia instances\\.\nTo continue, you will need a Neuron-optimized TensorFlow model saved in Amazon S3\\.\nIf you don’t already have a SavedModel, please follow the tutorial for `creating a Neuron compatible ResNet50 model `_\nand upload the resulting SavedModel to S3\\.\n\nResNet-50 is a popular machine learning model used for image\nclassification tasks\\. For more information about compiling Neuron models, see\n`The AWS Inferentia Chip With DLAMI `_\nin the AWS Deep Learning AMI Developer Guide\\.\n\nThe sample deployment manifest manages a pre-built inference serving container for TensorFlow provided by\nAWS Deep Learning Containers. Inside the container is the AWS Neuron Runtime and the TensorFlow Serving application.\nA complete list of pre-built Deep Learning Containers optimized for Neuron is maintained on GitHub under\n`Available Images `_.\nAt start\\-up, the DLC will fetch your model from Amazon S3, launch Neuron TensorFlow Serving with the saved model,\nand wait for prediction requests\\.\n\nThe number of Neuron devices allocated to your serving application can be adjusted by changing the\n`aws.amazon.com/neuron` resource in the deployment yaml\\. Please note that communication between TensorFlow Serving\nand the Neuron runtime happens over GRPC, which requires passing the `IPC_LOCK` capability to the container.\n\n1. Create a file named `rn50_deployment.yaml` with the contents below\\. Update the region\\-code and model path to match your desired settings. The model name is for identification purposes when a client makes a request to the TensorFlow server\\. This example uses a model name to match a sample ResNet50 client script that will be used in a later step for sending prediction requests\\.\n\n.. note::\n 1. Replace the s3 bucket name in model_base_path arg in the file with the location of the where the saved model was stored in s3.\n 2. In the image: add the appropriate location of the DLC tensorflow image\n\n\n::\n\n kind: Deployment\n apiVersion: apps/v1\n metadata:\n name: k8s-neuron-test\n labels:\n app: k8s-neuron-test\n role: master\n spec:\n replicas: 2\n selector:\n matchLabels:\n app: k8s-neuron-test\n role: master\n template:\n metadata:\n labels:\n app: k8s-neuron-test\n role: master\n spec:\n containers:\n - name: k8s-neuron-test\n image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference-neuron:1.15.4-neuron-py37-ubuntu18.04\n command:\n - /usr/local/bin/entrypoint.sh\n args:\n - --port=8500\n - --rest_api_port=9000\n - --model_name=resnet50_neuron\n - --model_base_path=s3://${your-bucket-of-models}/resnet50_neuron/\n ports:\n - containerPort: 8500\n - containerPort: 9000\n imagePullPolicy: IfNotPresent\n env:\n - name: AWS_REGION\n value: \"us-east-1\"\n - name: S3_USE_HTTPS\n value: \"1\"\n - name: S3_VERIFY_SSL\n value: \"0\"\n - name: S3_ENDPOINT\n value: s3.us-east-1.amazonaws.com\n - name: AWS_LOG_LEVEL\n value: \"3\"\n resources:\n limits:\n cpu: 4\n memory: 4Gi\n aws.amazon.com/neuron: 1\n requests:\n cpu: \"1\"\n memory: 1Gi\n securityContext:\n capabilities:\n add:\n - IPC_LOCK\n\n2. Deploy the model\\.\n\n::\n\n kubectl apply -f rn50_deployment.yaml\n\n3. Create a file named `rn50_service.yaml` with the following contents\\. The HTTP and gRPC ports are opened for accepting prediction requests\\.\n\n::\n\n kind: Service\n apiVersion: v1\n metadata:\n name: k8s-neuron-test\n labels:\n app: k8s-neuron-test\n spec:\n type: ClusterIP\n ports:\n - name: http-tf-serving\n port: 8500\n targetPort: 8500\n - name: grpc-tf-serving\n port: 9000\n targetPort: 9000\n selector:\n app: k8s-neuron-test\n role: master\n\n\n4. Create a Kubernetes service for your TensorFlow model Serving application\\.\n\n::\n\n kubectl apply -f rn50_service.yaml\n\nMake predictions against your TensorFlow Serving service\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n1. To test locally, forward the gRPC port to the `k8s-neuron-test` service\\.\n\n::\n\n kubectl port-forward service/k8s-neuron-test 8500:8500 &\n\n2. Create a Python script called `tensorflow-model-server-infer.py` with the following content. This script runs inference via gRPC, which is service framework.\n\n::\n\n import numpy as np\n import grpc\n import tensorflow as tf\n from tensorflow.keras.preprocessing import image\n from tensorflow.keras.applications.resnet50 import preprocess_input\n from tensorflow_serving.apis import predict_pb2\n from tensorflow_serving.apis import prediction_service_pb2_grpc\n from tensorflow.keras.applications.resnet50 import decode_predictions\n\n if __name__ == '__main__':\n channel = grpc.insecure_channel('localhost:8500')\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n img_file = tf.keras.utils.get_file(\n \"./kitten_small.jpg\",\n \"https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg\")\n img = image.load_img(img_file, target_size=(224, 224))\n img_array = preprocess_input(image.img_to_array(img)[None, ...])\n request = predict_pb2.PredictRequest()\n request.model_spec.name = 'resnet50_inf1'\n request.inputs['input'].CopyFrom(\n tf.make_tensor_proto(img_array, shape=img_array.shape))\n result = stub.Predict(request)\n prediction = tf.make_ndarray(result.outputs['output'])\n print(decode_predictions(prediction))\n\n3. Run the script to submit predictions to your service\\.\n::\n\n python3 tensorflow-model-server-infer.py\n\n Your output should look like the following:\n\n::\n\n [[(u'n02123045', u'tabby', 0.68817204), (u'n02127052', u'lynx', 0.12701613), (u'n02123159', u'tiger_cat', 0.08736559), (u'n02124075', u'Egyptian_cat', 0.063844085), (u'n02128757', u'snow_leopard', 0.009240591)]]\n" }, { "path": "containers/tutorials/inference/tutorial-infer.rst", "content": ".. _tutorial-infer:\n\nRun Inference in PyTorch Neuron Container\n==========================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nThis tutorial demonstrates how to run a pytorch DLC on an inferentia instance.\n\nBy the end of this tutorial you will be able to run the inference using the container\n\nYou will use an inf1.2xlarge to test your Docker configuration for Inferentia.\n\nTo find out the available neuron devices on your instance, use the command ``ls /dev/neuron*``.\n\nSetup Environment\n-----------------\n\n1. Launch an Inf1 Instance\n\n2. Set up docker environment according to :ref:`tutorial-docker-env-setup`\n\n3. Clone the `aws-neuron/deep-learning-containers `_ GitHub repository and use one of the PyTorch inference Dockerfiles found in the folders of the repo:\n\n.. code:: bash\n\n git clone https://github.com/aws-neuron/deep-learning-containers.git\n cd deep-learning-containers/docker/pytorch/inference/2.9.0\n\nFor additional prerequisites and setup requirements, see the `docker build prerequisites `_.\n\nThis tutorial requires the `torchserve entrypoint `_ and `torchserve config.properties `_ which are copied over to the same parent folder as part of prerequisites.\n\nWith the files in a local directory, build the image with the following command:\n\n.. code:: bash\n\n docker build . -f Dockerfile.neuronx -t neuron-container:pytorch\n\nRun the following command to start the container\n\n.. code:: bash\n\n docker run -itd --name pt-cont -p 80:8080 -p 8081:8081 --device=/dev/neuron0 neuron-container:pytorch /usr/local/bin/entrypoint.sh -m 'pytorch-resnet-neuron=https://aws-dlc-sample-models.s3.amazonaws.com/pytorch/Resnet50-neuron.mar' -t /home/model-server/config.properties" }, { "path": "containers/tutorials/k8s-default-scheduler.rst", "content": "This approach integrates the Neuron Scheduler Extension directly with the Kubernetes default scheduler. This method requires access to modify the default scheduler configuration.\n\n**Prerequisites**\n\nEnsure that the Neuron Device Plugin is running.\n\n**Step 1: Configure kube-scheduler**\n\nEnable the kube-scheduler to use a ConfigMap for scheduler policy. In your ``cluster.yml``, update the spec section with the following:\n\n.. code:: yaml\n\n spec:\n kubeScheduler:\n usePolicyConfigMap: true\n\n**Step 2: Launch the Cluster**\n\nCreate and launch the cluster:\n\n.. code:: bash\n\n kops create -f cluster.yml\n kops create secret --name neuron-test-1.k8s.local sshpublickey admin -i ~/.ssh/id_rsa.pub\n kops update cluster --name neuron-test-1.k8s.local --yes\n\n**Step 3: Install Neuron Scheduler Extension**\n\nInstall the Neuron Scheduler Extension and register it with kube-scheduler:\n\n.. code:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart \\\n --set \"scheduler.enabled=true\" \\\n --set \"scheduler.customScheduler.enabled=false\" \\\n --set \"scheduler.defaultScheduler.enabled=true\" \\\n --set \"npd.enabled=false\"\n" }, { "path": "containers/tutorials/k8s-multiple-scheduler.rst", "content": "This approach deploys a separate scheduler alongside the default Kubernetes scheduler. This is useful in environments where you don't have access to modify the default scheduler configuration, such as Amazon EKS.\n\nIn this setup, a new scheduler (``my-scheduler``) is deployed with the Neuron Scheduler Extension integrated. Pods that need to run Neuron workloads specify this custom scheduler in their configuration.\n\n.. note::\n\n Amazon EKS does not natively support modifying the default scheduler, so this multiple scheduler approach is required for EKS environments.\n\n**Prerequisites**\n\nEnsure that the Neuron Device Plugin is running.\n\n**Step 1: Install Neuron Scheduler Extension**\n\nInstall the Neuron Scheduler Extension as a custom scheduler:\n\n.. code:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart \\\n --set \"scheduler.enabled=true\" \\\n --set \"npd.enabled=false\"\n\n**Step 2: Verify Installation**\n\nCheck that there are no errors in the ``my-scheduler`` pod logs and that the ``k8s-neuron-scheduler`` pod is bound to a node:\n\n.. code:: bash\n\n kubectl logs -n kube-system my-scheduler-79bd4cb788-hq2sq\n\n**Expected output:**\n\n.. code:: bash\n\n I1012 15:30:21.629611 1 scheduler.go:604] \"Successfully bound pod to node\" pod=\"kube-system/k8s-neuron-scheduler-5d9d9d7988-xcpqm\" node=\"ip-192-168-2-25.ec2.internal\" evaluatedNodes=1 feasibleNodes=1\n\n**Step 3: Configure Pods to Use Custom Scheduler**\n\nWhen creating Pods that need to use the Neuron Scheduler Extension, specify ``my-scheduler`` as the scheduler name. Here's a sample Pod specification:\n\n.. code:: yaml\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: \n spec:\n restartPolicy: Never\n schedulerName: my-scheduler\n containers:\n - name: \n command: [\"\"]\n image: \n resources:\n limits:\n cpu: \"4\"\n memory: 4Gi\n aws.amazon.com/neuroncore: 9\n requests:\n cpu: \"1\"\n memory: 1Gi\n\n**Step 4: Verify Scheduling**\n\nAfter running a Neuron workload Pod, verify that the Neuron Scheduler successfully processed the filter and bind requests:\n\n.. code:: bash\n\n kubectl logs -n kube-system k8s-neuron-scheduler-5d9d9d7988-xcpqm\n\n**Expected output for filter request:**\n\n.. code:: bash\n\n 2022/10/12 15:41:16 POD nrt-test-5038 fits in Node:ip-192-168-2-25.ec2.internal\n 2022/10/12 15:41:16 Filtered nodes: [ip-192-168-2-25.ec2.internal]\n 2022/10/12 15:41:16 Failed nodes: map[]\n 2022/10/12 15:41:16 Finished Processing Filter Request...\n\n**Expected output for bind request:**\n\n.. code:: bash\n\n 2022/10/12 15:41:16 Executing Bind Request!\n 2022/10/12 15:41:16 Determine if the pod %v is NeuronDevice podnrt-test-5038\n 2022/10/12 15:41:16 Updating POD Annotation with alloc devices!\n 2022/10/12 15:41:16 Return aws.amazon.com/neuroncore\n 2022/10/12 15:41:16 neuronDevUsageMap for resource:aws.amazon.com/neuroncore in node: ip-192-168-2-25.ec2.internal is [false false false false false false false false false false false false false false false false]\n 2022/10/12 15:41:16 Allocated ids for POD nrt-test-5038 are: 0,1,2,3,4,5,6,7,8\n 2022/10/12 15:41:16 Try to bind pod nrt-test-5038 in default namespace to node ip-192-168-2-25.ec2.internal with &Binding{ObjectMeta:{nrt-test-5038 8da590b1-30bc-4335-b7e7-fe574f4f5538 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Target:ObjectReference{Kind:Node,Namespace:,Name:ip-192-168-2-25.ec2.internal,UID:,APIVersion:,ResourceVersion:,FieldPath:,},}\n 2022/10/12 15:41:16 Updating the DevUsageMap since the bind is successful!\n 2022/10/12 15:41:16 Return aws.amazon.com/neuroncore\n 2022/10/12 15:41:16 neuronDevUsageMap for resource:aws.amazon.com/neuroncore in node: ip-192-168-2-25.ec2.internal is [false false false false false false false false false false false false false false false false]\n 2022/10/12 15:41:16 neuronDevUsageMap for resource:aws.amazon.com/neurondevice in node: ip-192-168-2-25.ec2.internal is [false false false false]\n 2022/10/12 15:41:16 Allocated devices list 0,1,2,3,4,5,6,7,8 for resource aws.amazon.com/neuroncore\n 2022/10/12 15:41:16 Allocated devices list [0] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [0] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [0] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [0] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [1] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [1] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [1] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [1] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Allocated devices list [2] for other resource aws.amazon.com/neurondevice\n 2022/10/12 15:41:16 Return aws.amazon.com/neuroncore\n 2022/10/12 15:41:16 Succesfully updated the DevUsageMap [true true true true true true true true true false false false false false false false] and otherDevUsageMap [true true true false] after alloc for node ip-192-168-2-25.ec2.internal\n 2022/10/12 15:41:16 Finished executing Bind Request...\n" }, { "path": "containers/tutorials/k8s-neuron-device-plugin.rst", "content": "The Neuron Device Plugin is a Kubernetes device plugin that exposes Neuron hardware resources to the cluster's scheduler. It discovers available Neuron devices on each node, advertises them as allocatable resources, and manages their lifecycle. When Pods request Neuron resources, the device plugin handles the allocation and ensures exclusive access to the assigned devices. This integration enables Kubernetes to treat Neuron accelerators as first-class schedulable resources, similar to GPUs or other specialized hardware.\n\nThe device plugin registers two resource types with Kubernetes:\n\n* ``aws.amazon.com/neuroncore`` - Used for allocating individual Neuron cores to containers\n* ``aws.amazon.com/neuron`` - Used for allocating entire Neuron devices to containers (all cores belonging to the device)\n\n**Deploy Neuron Device Plugin**\n\n**Prerequisites**\n\nEnsure that all :ref:`prerequisites` are satisfied before proceeding.\n\n**Installation**\n\nApply the Neuron Device Plugin as a DaemonSet on the cluster:\n\n.. code:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart \\\n --set \"npd.enabled=false\"\n\n**Verify Installation**\n\nVerify that the Neuron Device Plugin is running:\n\n.. code:: bash\n\n kubectl get ds neuron-device-plugin -n kube-system\n\nExpected output (example with 2 nodes in cluster):\n\n.. code:: bash\n\n NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE\n neuron-device-plugin 2 2 2 2 2 18h\n\n**Verify Allocatable Resources**\n\nVerify that nodes have allocatable Neuron cores:\n\n.. code:: bash\n\n kubectl get nodes \"-o=custom-columns=NAME:.metadata.name,NeuronCore:.status.allocatable.aws\\.amazon\\.com/neuroncore\"\n\nExpected output:\n\n.. code:: bash\n\n NAME NeuronCore\n ip-192-168-65-41.us-west-2.compute.internal 32\n ip-192-168-87-81.us-west-2.compute.internal 32\n\nVerify that nodes have allocatable Neuron devices:\n\n.. code:: bash\n\n kubectl get nodes \"-o=custom-columns=NAME:.metadata.name,NeuronDevice:.status.allocatable.aws\\.amazon\\.com/neuron\"\n\nExpected output:\n\n.. code:: bash\n\n NAME NeuronDevice\n ip-192-168-65-41.us-west-2.compute.internal 16\n ip-192-168-87-81.us-west-2.compute.internal 16\n" }, { "path": "containers/tutorials/k8s-neuron-helm-chart.rst", "content": ".. _k8s-neuron-helm-chart:\n\nThe Neuron Helm Chart simplifies the deployment and management of Neuron infrastructure components on Kubernetes clusters. It provides a unified installation method for all essential Neuron components, streamlining the setup process and ensuring consistent configuration across your cluster.\n\nComponents Included\n^^^^^^^^^^^^^^^^^^^\n\nThe Neuron Helm Chart includes the following components:\n\n* Neuron Device Plugin\n* Neuron Scheduler Extension\n* :ref:`Neuron Node Problem Detector and Recovery `\n* Neuron DRA (Dynamic Resource Allocation) Driver. Refer to :ref:`neuron-dra`.\n\nInstallation\n^^^^^^^^^^^^\n\nTo install the Neuron Helm Chart:\n\n.. code:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart\n\nFor detailed information on configuration options, advanced deployment scenarios, and troubleshooting, please refer to the official Neuron Helm Charts repository: https://github.com/aws-neuron/neuron-helm-charts/\n" }, { "path": "containers/tutorials/k8s-neuron-monitor.rst", "content": ".. _k8s-neuron-monitor:\n\nNeuron Monitor is a monitoring solution that collects and exposes metrics from Neuron devices and the Neuron runtime. It provides visibility into hardware utilization, performance counters, memory usage, and device health status. The monitor can export metrics in formats compatible with popular observability platforms like Prometheus, enabling integration with existing monitoring and alerting infrastructure. This allows operators to track Neuron device performance, identify bottlenecks, and troubleshoot issues in production environments.\n\nFor detailed information about Neuron Monitor, see the `Neuron Monitor User Guide `_.\n\n.. note::\n\n Neuron Monitor does not currently support environments using the Neuron DRA (Dynamic Resource Allocation) Driver.\n\nDeploy Neuron Monitor DaemonSet\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n**Step 1: Download the Configuration**\n\nDownload the Neuron Monitor YAML file: :download:`k8s-neuron-monitor-daemonset.yml `\n\n**Step 2: Apply the Configuration**\n\nApply the Neuron Monitor YAML to create a DaemonSet on the cluster:\n\n.. code:: bash\n\n kubectl apply -f k8s-neuron-monitor-daemonset.yml\n\n**Step 3: Verify Installation**\n\nVerify that the Neuron Monitor DaemonSet is running:\n\n.. code:: bash\n\n kubectl get ds neuron-monitor --namespace neuron-monitor\n\nExpected output (example with 2 nodes in cluster):\n\n.. code:: bash\n\n NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE\n neuron-monitor 2 2 2 2 2 27h\n\n**Step 4: Get Pod Names**\n\nRetrieve the Neuron Monitor pod names:\n\n.. code:: bash\n\n kubectl get pods --namespace neuron-monitor\n\nExpected output:\n\n.. code:: bash\n\n NAME READY STATUS RESTARTS AGE\n neuron-monitor-slsxf 1/1 Running 0 17m\n neuron-monitor-wc4f5 1/1 Running 0 17m\n\n**Step 5: Verify Prometheus Endpoint**\n\nVerify that the Prometheus metrics endpoint is available:\n\n.. code:: bash\n\n kubectl exec neuron-monitor-wc4f5 --namespace neuron-monitor -- wget -q --output-document - http://127.0.0.1:8000\n\nExpected output (sample metrics):\n\n.. code:: bash\n\n # HELP python_gc_objects_collected_total Objects collected during gc\n # TYPE python_gc_objects_collected_total counter\n python_gc_objects_collected_total{generation=\"0\"} 362.0\n python_gc_objects_collected_total{generation=\"1\"} 0.0\n python_gc_objects_collected_total{generation=\"2\"} 0.0\n # HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC\n # TYPE python_gc_objects_uncollectable_total counter\n" }, { "path": "containers/tutorials/k8s-neuron-problem-detector-and-recovery-irsa.rst", "content": ".. _k8s-neuron-problem-detector-and-recovery-irsa:\n\nPermissions for Neuron Node Problem Detector and Recovery\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe Neuron Node Problem Detector and Recovery requires IAM roles for service accounts (IRSA) for authorization. For more information, see `IAM roles for service accounts `__ in the Amazon EKS User Guide.\n\nThis section shows how to configure an IAM role for service accounts using the ``eksctl`` command-line tool.\n\n**Step 1: Install eksctl**\n\nInstall the ``eksctl`` CLI using the instructions at https://eksctl.io/installation/.\n\n**Step 2: Create IAM Policy**\n\nCreate an IAM policy that grants the necessary permissions for the Neuron Node Problem Detector.\n\n.. code:: json\n\n {\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": [\n \"autoscaling:SetInstanceHealth\",\n \"autoscaling:DescribeAutoScalingInstances\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": \"\"\n },\n {\n \"Action\": [\n \"ec2:DescribeInstances\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": \"*\",\n \"Condition\": {\n \"ForAllValues:StringEquals\": {\n \"ec2:ResourceTag/aws:autoscaling:groupName\": \"\"\n }\n }\n },\n {\n \"Action\": [\n \"cloudwatch:PutMetricData\"\n ],\n \"Effect\": \"Allow\",\n \"Resource\": \"*\",\n \"Condition\": {\n \"StringEquals\": {\n \"cloudwatch:Namespace\": \"NeuronHealthCheck\"\n }\n }\n }\n ]\n }\n\nSave the policy template above to a file named ``npd-policy.json`` (replacing the placeholder values), then run:\n\n.. code:: bash\n\n aws iam create-policy \\\n --policy-name NeuronProblemDetectorPolicy \\\n --policy-document file://npd-policy.json\n\n**Step 3: Create Namespace and Service Account**\n\nCreate a dedicated namespace for the Neuron Node Problem Detector:\n\n.. code:: bash\n\n kubectl create ns neuron-healthcheck-system\n\n**Step 4: Associate IAM Role with Service Account**\n\nUse the following script to create the service account and associate it with the IAM role:\n\n.. code:: bash\n\n #!/bin/bash\n CLUSTER_NAME=\n REGION_CODE=$(aws configure get region)\n POLICY_ARN=\n\n eksctl create iamserviceaccount \\\n --name node-problem-detector \\\n --namespace neuron-healthcheck-system \\\n --cluster $CLUSTER_NAME \\\n --attach-policy-arn $POLICY_ARN \\\n --approve \\\n --role-name neuron-problem-detector-role-$CLUSTER_NAME \\\n --region $REGION_CODE \\\n --override-existing-serviceaccounts\n\n**Step 5: Verify Service Account Configuration**\n\nVerify that the service account is annotated correctly with the IAM role:\n\n.. code:: bash\n\n kubectl describe sa node-problem-detector -n neuron-healthcheck-system\n\nExpected output:\n\n.. code:: bash\n\n Name: node-problem-detector\n Namespace: neuron-healthcheck-system\n Labels: app.kubernetes.io/managed-by=eksctl\n Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/neuron-problem-detector-role-cluster1\n Image pull secrets: \n Mountable secrets: \n Tokens: \n Events: \n\n**Cleanup**\n\nTo remove the service account and associated IAM role, use the following command:\n\n.. code:: bash\n\n #!/bin/bash\n CLUSTER_NAME=\n REGION_CODE=$(aws configure get region)\n\n eksctl delete iamserviceaccount \\\n --name node-problem-detector \\\n --namespace neuron-healthcheck-system \\\n --cluster $CLUSTER_NAME \\\n --approve \\\n --region $REGION_CODE\n" }, { "path": "containers/tutorials/k8s-neuron-problem-detector-and-recovery.rst", "content": ".. _k8s-neuron-problem-detector-and-recovery:\n\nDeploy Neuron Node Problem Detector and Recovery\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe Neuron Node Problem Detector and Recovery is a critical resiliency component that continuously monitors the health of Neuron devices on each Kubernetes node by detecting hardware and software errors such as device failures, driver problems, and runtime errors. It integrates with the Kubernetes Node Problem Detector framework to report Neuron-specific conditions. When unrecoverable issues are detected, it can automatically remediate problems by marking nodes as unhealthy and triggering node replacement to prevent workload scheduling on faulty hardware. The component can also publish CloudWatch metrics under the ``NeuronHealthCheck`` namespace for monitoring and alerting purposes.\n\n**Requirements**\n\nBefore deploying the Neuron Node Problem Detector and Recovery, ensure the following requirements are met:\n\n* **Neuron Driver:** Version 2.15 or later\n* **Neuron Runtime:** SDK 2.18 or later\n* **Prerequisites:** All prerequisites for Kubernetes containers and the Neuron Node Problem Detector must be satisfied\n\n**Installation**\n\nInstall the Neuron Node Problem Detector and Recovery as a DaemonSet using Helm:\n\n.. note::\n\n The installation pulls the container image from the upstream Node Problem Detector repository at ``registry.k8s.io/node-problem-detector``.\n\n.. code:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart\n\n**Enable Node Recovery**\n\nBy default, the Neuron Node Problem Detector runs in **monitor-only mode**. To enable automatic node recovery functionality:\n\n.. code:: bash\n\n helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart \\\n --set \"npd.nodeRecovery.enabled=true\"\n\n**Verify Installation**\n\nVerify that the Node Problem Detector pods are running:\n\n.. code:: bash\n\n kubectl get pod -n neuron-healthcheck-system\n\nExpected output (example with 4 nodes in cluster):\n\n.. code:: bash\n\n NAME READY STATUS RESTARTS AGE\n node-problem-detector-7qcrj 1/1 Running 0 59s\n node-problem-detector-j45t5 1/1 Running 0 59s\n node-problem-detector-mr2cl 1/1 Running 0 59s\n node-problem-detector-vpjtk 1/1 Running 0 59s\n\n**Monitoring and Metrics**\n\nWhen an unrecoverable error occurs, the Neuron Node Problem Detector:\n\n* Publishes metrics to CloudWatch under the ``NeuronHealthCheck`` namespace\n* Updates the node's ``NodeCondition``, which can be viewed using:\n\n .. code:: bash\n\n kubectl describe node \n" }, { "path": "containers/tutorials/k8s-neuron-scheduler-flow.rst", "content": ".. _k8s-neuron-scheduler-flow:\n\nNeuron Scheduler Extension Flow Diagram\n---------------------------------------\n\n::\n\n\n\n\n +----------------------------+\n | POD Manifest |\n | with Request |\n | aws.amazon.com/neuroncore:2|\n | |\n | |\n 2 +-------------+--------------+\n +--------------------------------+ |\n | | |\n | | | 3\n +------------------------------+-----+ | |\n | Kubelet in INF1/TRN1 Node| | |\n | +<-----------+ | |\n +-----+---------------------+--------+ | +-----v-----------v--------------+\n | ^ | | Kube-Scheduler |\n | | | | |\n | | | +--^------+---------------+------+\n 9 | 1 | | | | |\n | | 8| 5| |4 |\n | | | | | |\n | | | | | |6\n v | | | | |\n +-----+---------------------+--------+ | +--+------v---------------v------+\n | neuron-device-plugin | +-------+ neuron|scheduler|ext |\n | in INF1/TRN1 node | +---------------------+----------+\n +----+----------------------+--------+ |\n | | |7\n | |10 |\n | | v\n 11| | +---------+-------+\n | | |POD Manifest: |\n | | |Annotation: |\n | | |NEURON_CORES:2,3 |\n v +---------------------------------------->+ |\n --device=/dev/neuron1 --env NEURON_RT_VISIBLE_CORES=2,3 | |\n | |\n +-----------------+\n\n 1. neuron-device-plugin returns the list of Neuron cores/devices to kublet\n 2. Kubelet advertises the Core/Device list to K8s API server (in turn to kube-scheduler)\n 3. POD Request for neuron cores/devices [Kube-Scheduler picks up the POD creation request]\n 4. kube-scheduler calls the neuron-scheduler-extn filter function with list of nodes and POD Specification\n 5. neuron-scheduler-extn scans through the nodes and filters out nodes with non\n contiguous cores/devices and returns the nodes that are capable of supporing the given POD specification\n 6. kube-scheduler calls the neuron-scheduler-extn bind function with pod and node\n 7. neuron-scheduler-extn updates the POD annotation with allocated neuron core/device Ids (contiguous)\n 8. neuron-scheduler-extn sends the bind request to kubelet of the selected node\n 9. Kubelet calls the Alloc function of the neuron-device-plugin\n 10. neuron-device-plugin queries the POD Annotation for allocated core/device Ids\n 11. neuron-device-plugin exports the devices & visisble cores to container runtime\n" }, { "path": "containers/tutorials/k8s-neuron-scheduler.rst", "content": "The Neuron Scheduler Extension is a Kubernetes scheduler plugin that provides intelligent, topology-aware scheduling for Neuron workloads. While the device plugin handles basic resource allocation, the scheduler extension optimizes Pod placement by considering Neuron core topology, NeuronCore-to-NeuronCore connectivity, and workload requirements. It ensures efficient utilization of Neuron devices by placing Pods on nodes where the requested Neuron cores are optimally configured. This component is optional and primarily beneficial for workloads that require specific subsets of Neuron devices or cores rather than consuming all available resources on a node.\n\nThe scheduler extension is required for scheduling Pods that request more than one Neuron core or device resource. It finds sets of directly connected devices with minimal communication latency when scheduling containers, ensuring optimal performance for multi-device workloads.\n\nFor a graphical depiction of how the Neuron Scheduler Extension works, see :ref:`k8s-neuron-scheduler-flow`.\n\n**Device Allocation by Instance Type**\n\nThe Neuron Scheduler Extension applies topology-aware scheduling rules based on instance type to ensure consistent and high performance regardless of which cores and devices are assigned to containers.\n\n**Inf1 and Inf2 Instances (Ring Topology)**\n\nDevices are connected through a ring topology with no restrictions on the number of devices requested (as long as it is fewer than the total devices on a node). When N devices are requested, the scheduler finds a node where N contiguous devices are available to minimize communication latency. It will never allocate non-contiguous devices to the same container.\n\nFor example, when a container requests 3 Neuron devices, the scheduler might assign devices 0, 1, 2 if available, but never devices 0, 2, 4 because those devices are not directly connected.\n\nThe figure below shows examples of device sets on an Inf2.48xlarge node that could be assigned to a container requesting 2 devices:\n\n|eks-inf2-device-set|\n\n**Trn1.32xlarge and Trn1n.32xlarge Instances (2D Torus Topology)**\n\nDevices are connected via a 2D torus topology. The scheduler enforces that containers request 1, 4, 8, or all 16 devices. If your container requires a different number of devices (such as 2 or 5), we recommend using an Inf2 instance instead to benefit from more flexible topology support.\n\nIf you request an invalid number of devices (such as 7), your Pod will not be scheduled and you will receive a warning:\n\n``Instance type trn1.32xlarge does not support requests for device: 7. Please request a different number of devices.``\n\nWhen requesting 4 devices, your container will be allocated one of the following device sets if available:\n\n|eks-trn1-device-set4|\n\nWhen requesting 8 devices, your container will be allocated one of the following device sets if available:\n\n|eks-trn1-device-set8|\n\n.. note::\n\n For all instance types, requesting one or all Neuron cores or devices is always valid.\n\n**Deploy Neuron Scheduler Extension**\n\n.. tab-set::\n\n .. tab-item:: Multiple Scheduler Approach\n\n .. include:: /containers/tutorials/k8s-multiple-scheduler.rst\n\n .. tab-item:: Default Scheduler Approach\n\n .. include:: /containers/tutorials/k8s-default-scheduler.rst\n\n\n.. |eks-inf2-device-set| image:: /images/eks-inf2-device-set.png\n.. |eks-trn1-device-set4| image:: /images/eks-trn1-device-set4.png\n.. |eks-trn1-device-set8| image:: /images/eks-trn1-device-set8.png\n" }, { "path": "containers/tutorials/k8s-prerequisite.rst", "content": ".. _k8s-prerequisite:\n\n.. meta::\n :description: Learn how to create an Amazon EKS cluster with AWS Trainium instances (Trn1, Trn2) for machine learning workloads using AWS Neuron SDK. Step-by-step guide with eksctl and CloudFormation templates.\n :keywords: EKS, Kubernetes, Trainium, Trn1, Trn2, Neuron, AWS, machine learning, distributed training, eksctl, CloudFormation, EFA, node group\n\nBefore setting up Neuron components on your EKS cluster, you must create an EKS cluster and add Neuron-enabled nodes. This section guides you through creating an Amazon Elastic Kubernetes Service (EKS) cluster with AWS Trainium-enabled nodes (Trn1 or Trn2 instances) using CloudFormation templates and the eksctl command-line tool. You'll configure optimized networking with Elastic Fabric Adapter (EFA) support and pre-configured Neuron components for distributed training and inference workloads.\n\nFor detailed information, refer to:\n\n* `EKS Cluster Creation Guide `_\n* `EKS Compute Resources Guide `_\n* `eksctl Getting Started `_\n\n**Step 1: Download Node Group Template**\n\nDownload the node group CloudFormation template for your instance type.\n\n.. tab-set::\n\n .. tab-item:: Trn1\n\n .. code-block:: bash\n\n wget https://raw.githubusercontent.com/aws-neuron/aws-neuron-eks-samples/master/dp_bert_hf_pretrain/cfn/eks_trn1_ng_stack.yaml\n\n .. tab-item:: Trn2\n\n .. code-block:: bash\n\n wget https://raw.githubusercontent.com/aws-neuron/aws-neuron-eks-samples/master/dp_bert_hf_pretrain/cfn/eks_trn2_ng_stack_al2023.yaml\n\n**Important template configuration information**\n\n* **Placement Group:** Optimizes network speed between nodes\n* **EFA Driver:** Installed automatically (ensure ``libfabric`` version matches between AMI and workload containers)\n* **AMI:** Uses `EKS optimized accelerated AMI `_ with Neuron components pre-installed\n* **Instance Type:** Configured for trn1.32xlarge or trn2.48xlarge (update to your desired instance type)\n* **Kubernetes Version:** Trn1 templates use Kubernetes 1.25+, Trn2 templates use Kubernetes 1.34+ (update as needed)\n\nTrn2 LNC configuration (Optional):\n\nTrn2 instances use a default Logical NeuronCore Configuration (LNC) of ``2``. To change it to ``1``, update the ``UserData`` section of the launch template:\n\n.. code-block:: bash\n\n --==BOUNDARY==\n Content-Type: text/x-shellscript; charset=\"us-ascii\"\n\n #!/bin/bash\n set -ex\n config_dir=/opt/aws/neuron\n config_file=${config_dir}/logical_nc_config\n [ -d \"$config_dir\" ] || mkdir -p \"$config_dir\"\n [ -f \"$config_file\" ] || touch \"$config_file\"\n if ! grep -q \"^NEURON_LOGICAL_NC_CONFIG=1$\" \"$config_file\" 2>/dev/null; then\n printf \"NEURON_LOGICAL_NC_CONFIG=1\" >> \"$config_file\"\n fi\n --==BOUNDARY==--\n\n**Step 2: Create Cluster Parameter Script**\n\nCreate a bash script to capture the parameters needed for the node template:\n\n.. tab-set::\n\n .. tab-item:: Trn1\n\n .. code-block:: bash\n\n #!/bin/bash\n\n CLUSTER_NAME=$1\n CLUSTER_SG=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].ResourcesVpcConfig.ClusterSecurityGroupId\")\n VPC_ID=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].ResourcesVpcConfig.VpcId\")\n\n cat < cfn_params.json\n [\n {\n \"ParameterKey\": \"ClusterName\",\n \"ParameterValue\": \"$CLUSTER_NAME\"\n },\n {\n \"ParameterKey\": \"ClusterControlPlaneSecurityGroup\",\n \"ParameterValue\": \"$CLUSTER_SG\"\n },\n {\n \"ParameterKey\": \"VpcId\",\n \"ParameterValue\": \"$VPC_ID\"\n }\n ]\n EOF\n\n .. tab-item:: Trn2\n\n .. code-block:: bash\n\n #!/bin/bash\n\n CLUSTER_NAME=$1\n CLUSTER_SG=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].ResourcesVpcConfig.ClusterSecurityGroupId\")\n VPC_ID=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].ResourcesVpcConfig.VpcId\")\n CLUSTER_ENDPOINT=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].Endpoint\")\n CLUSTER_SERVICE_CIDR=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].KubernetesNetworkConfig.ServiceIpv4Cidr\")\n CLUSTER_CA=$(eksctl get cluster $CLUSTER_NAME -o json | jq -r \".[0].CertificateAuthority.Data\")\n\n cat < cfn_params.json\n [\n {\n \"ParameterKey\": \"ClusterName\",\n \"ParameterValue\": \"$CLUSTER_NAME\"\n },\n {\n \"ParameterKey\": \"ClusterControlPlaneSecurityGroup\",\n \"ParameterValue\": \"$CLUSTER_SG\"\n },\n {\n \"ParameterKey\": \"VpcId\",\n \"ParameterValue\": \"$VPC_ID\"\n },\n {\n \"ParameterKey\": \"ClusterEndpoint\",\n \"ParameterValue\": \"$CLUSTER_ENDPOINT\"\n },\n {\n \"ParameterKey\": \"ClusterServiceCidr\",\n \"ParameterValue\": \"$CLUSTER_SERVICE_CIDR\"\n },\n {\n \"ParameterKey\": \"ClusterCertificateAuthority\",\n \"ParameterValue\": \"$CLUSTER_CA\"\n }\n ]\n EOF\n\n\nThis script captures the cluster name, security group for control plane connectivity, and VPC ID.\n\n**Step 3: Create CloudFormation Stack**\n\nCreate the CloudFormation stack for the node group.\n\n.. tab-set::\n\n .. tab-item:: Trn1\n\n .. code-block:: bash\n\n aws cloudformation create-stack \\\n --stack-name eks-trn1-ng-stack \\\n --template-body file://eks_trn1_ng_stack.yaml \\\n --parameters file://cfn_params.json \\\n --capabilities CAPABILITY_IAM\n\n .. tab-item:: Trn2\n\n .. code-block:: bash\n\n aws cloudformation create-stack \\\n --stack-name eks-trn2-ng-stack \\\n --template-body file://eks_trn2_ng_stack_al2023.yaml \\\n --parameters file://cfn_params.json \\\n --capabilities CAPABILITY_IAM\n\nWait for the stack creation to complete before proceeding. You can monitor the progress in the AWS CloudFormation console.\n\n**Step 4: Determine Availability Zones**\n\nIdentify the availability zones for your cluster:\n\n.. code-block:: bash\n\n aws ec2 describe-availability-zones \\\n --region $REGION_CODE \\\n --query \"AvailabilityZones[]\" \\\n --filters \"Name=zone-id,Values=$1\" \\\n --query \"AvailabilityZones[].ZoneName\" \\\n --output text\n\n**Step 5: Generate Node Group Configuration**\n\nCreate a script named ``create_ng_yaml.sh`` to generate the node group YAML configuration. The script requires: region, availability zones, cluster name, and CloudFormation stack name.\n\n.. tab-set::\n\n .. tab-item:: Trn1\n\n .. code-block:: bash\n\n #!/bin/bash\n\n REGION_CODE=$1\n EKSAZ1=$2\n EKSAZ2=$3\n CLUSTER_NAME=$4\n STACKNAME=$5\n\n LT_ID_TRN1=$(aws cloudformation describe-stacks --stack-name $STACKNAME \\\n --query \"Stacks[0].Outputs[?OutputKey=='LaunchTemplateIdTrn1'].OutputValue\" \\\n --output text)\n\n cat < trn1_nodegroup.yaml\n apiVersion: eksctl.io/v1alpha5\n kind: ClusterConfig\n\n metadata:\n name: $CLUSTER_NAME\n region: $REGION_CODE\n version: \"1.28\"\n\n iam:\n withOIDC: true\n\n availabilityZones: [\"$EKSAZ1\",\"$EKSAZ2\"]\n\n managedNodeGroups:\n - name: trn1-32xl-ng1\n launchTemplate:\n id: $LT_ID_TRN1\n minSize: 1\n desiredCapacity: 1\n maxSize: 1\n availabilityZones: [\"$EKSAZ1\"]\n privateNetworking: true\n efaEnabled: true\n EOF\n\n .. tab-item:: Trn2\n\n .. code-block:: bash\n\n #!/bin/bash\n\n REGION_CODE=$1\n EKSAZ1=$2\n EKSAZ2=$3\n CLUSTER_NAME=$4\n STACKNAME=$5\n\n LT_ID_TRN2=$(aws cloudformation describe-stacks --stack-name $STACKNAME \\\n --query \"Stacks[0].Outputs[?OutputKey=='LaunchTemplateIdTrn2'].OutputValue\" \\\n --output text)\n\n cat < trn2_nodegroup.yaml\n apiVersion: eksctl.io/v1alpha5\n kind: ClusterConfig\n\n metadata:\n name: $CLUSTER_NAME\n region: $REGION_CODE\n version: \"1.34\"\n\n iam:\n withOIDC: true\n\n availabilityZones: [\"$EKSAZ1\",\"$EKSAZ2\"]\n\n managedNodeGroups:\n - name: trn2-48xl-ng1\n launchTemplate:\n id: $LT_ID_TRN2\n minSize: 1\n desiredCapacity: 1\n maxSize: 1\n availabilityZones: [\"$EKSAZ1\"]\n privateNetworking: true\n efaEnabled: true\n EOF\n\nRun the script to generate the configuration file. Update the Kubernetes version as needed for your environment.\n\nExample output:\n\n.. tab-set::\n\n .. tab-item:: Trn1\n\n .. code-block:: yaml\n\n apiVersion: eksctl.io/v1alpha5\n kind: ClusterConfig\n\n metadata:\n name: nemo2\n region: us-west-2\n version: \"1.28\"\n\n iam:\n withOIDC: true\n\n availabilityZones: [\"us-west-2d\",\"us-west-2c\"]\n\n managedNodeGroups:\n - name: trn1-32xl-ng1\n launchTemplate:\n id: lt-093c222b35ea89009\n minSize: 1\n desiredCapacity: 1\n maxSize: 1\n availabilityZones: [\"us-west-2d\"]\n privateNetworking: true\n efaEnabled: true\n\n .. tab-item:: Trn2\n\n .. code-block:: yaml\n\n apiVersion: eksctl.io/v1alpha5\n kind: ClusterConfig\n\n metadata:\n name: nemo2\n region: us-west-2\n version: \"1.34\"\n\n iam:\n withOIDC: true\n\n availabilityZones: [\"us-west-2d\",\"us-west-2c\"]\n\n managedNodeGroups:\n - name: trn2-48xl-ng1\n launchTemplate:\n id: lt-093c222b35ea89010\n minSize: 1\n desiredCapacity: 1\n maxSize: 1\n availabilityZones: [\"us-west-2d\"]\n privateNetworking: true\n efaEnabled: true\n\n**Step 6: Create Node Group**\n\nCreate the node group using the generated configuration.\n\n.. tab-set::\n\n .. tab-item:: Trn1\n\n .. code-block:: bash\n\n eksctl create nodegroup -f trn1_nodegroup.yaml\n\n .. tab-item:: Trn2\n\n .. code-block:: bash\n\n eksctl create nodegroup -f trn2_nodegroup.yaml\n\nWait for the nodes to reach the ``Ready`` state. Verify using:\n\n.. code-block:: bash\n\n kubectl get nodes\n\n**Step 7: Install EFA Device Plugin (Optional)**\n\nIf you plan to run distributed training or inference jobs, install the EFA device plugin following the instructions at the `EFA device plugin repository `_.\n" }, { "path": "containers/tutorials/k8s-setup.rst", "content": ".. _tutorial-k8s-env-setup-for-neuron-to-remove:\n\nKubernetes environment setup for Neuron\n=======================================\n\nIntroduction\n------------\n\nCustomers that use Kubernetes can conveniently integrate Inf1/Trn1 instances into their workflows. This tutorial will go through deploying the neuron device plugin daemonset and also how to allocate neuron cores or devices to application pods.\n\n.. dropdown:: Prerequisite\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /containers/tutorials/k8s-prerequisite.rst\n\n.. dropdown:: Deploy Neuron Device Plugin\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /containers/tutorials/k8s-neuron-device-plugin.rst\n\n.. dropdown:: Deploy Neuron Scheduler Extension\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. include:: /containers/tutorials/k8s-neuron-scheduler.rst\n" }, { "path": "containers/tutorials/training/index.rst", "content": "Containers -- Training Tutorials\n=================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /containers/tutorials/training/tutorial-training\n /containers/tutorials/training/k8s_mlp_train_demo\n\n\n.. include:: /containers/tutorials/training/index.txt\n" }, { "path": "containers/tutorials/training/index.txt", "content": "* :ref:`tutorial-training`\n* :ref:`example-deploy-mlp-train-pod`\n" }, { "path": "containers/tutorials/training/k8s_mlp_train_demo.rst", "content": ".. _example-deploy-mlp-train-pod:\n\nDeploy a simple mlp training script as a Kubernetes job\n----------------------------------------------------------\n\nThis tutorial uses mlp train as a teaching example on how to deploy an\ntraining application using Kubernetes on the Trn1 instances. For more advanced example, please refer to `Tutorial: Launch a Multi-Node PyTorch Neuron Training Job on Trainium Using TorchX and EKS `__\n\nPrerequisite:\n^^^^^^^^^^^^^\n\n- :ref:`tutorial-k8s-env-setup-for-neuron`: to setup k8s support on your cluster.\n- Trn1 instances as worker nodes with attached roles allowing:\n\n - ECR read access policy to retrieve container images from ECR:\n **arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly**\n- Have a container image that is build using :ref:`tutorial-training`\n\nDeploy a mlp training image\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n1. Create a file named `mlp_train.yaml` with the contents below\\. \n\n.. note::\n In the image: add the appropriate location of the image\n\n\n::\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: trn1-mlp\n spec:\n restartPolicy: Never\n schedulerName: default-scheduler\n hostNetwork: true\n nodeSelector:\n beta.kubernetes.io/instance-type: trn1.32xlarge\n beta.kubernetes.io/instance-type: trn1.2xlarge\n containers:\n - name: trn1-mlp\n command: [\"/usr/local/bin/python3\"]\n args: [\"/opt/ml/mlp_train.py\"]\n image: 647554078242.dkr.ecr.us-east-1.amazonaws.com/sunda-pt:k8s_mlp_0907\n imagePullPolicy: IfNotPresent\n env:\n - name: NEURON_RT_LOG_LEVEL\n value: \"INFO\"\n resources:\n limits: \n aws.amazon.com/neuron: 2\n requests:\n aws.amazon.com/neuron: 2\n\n2. Deploy the pod.\n\n::\n\n kubectl apply -f mlp_train.yaml\n\n3. Check the logs to make sure training completed\n::\n\n kubectl logs \n\n Your log should have the following\n\n::\n\n Final loss is 0.1977\n ----------End Training ---------------\n" }, { "path": "containers/tutorials/training/tutorial-training.rst", "content": ".. _tutorial-training:\n\nRun Training in PyTorch Neuron Container\n========================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nThis tutorial demonstrates how to run a pytorch container on an trainium instance.\n\nBy the end of this tutorial you will be able to run simple mlp training using the container\n\nYou will use an trn1.2xlarge to test your Docker configuration for Trainium.\n\nTo find out the available neuron devices on your instance, use the command ``ls /dev/neuron*``.\n\nSetup Environment\n-----------------\n\n1. Launch an Trn1 Instance\n .. include:: /setup/install-templates/launch-instance.txt\n\n2. Set up docker environment according to :ref:`tutorial-docker-env-setup`\n\n3. A sample Dockerfile for for torch-neuron can be found here :ref:`trainium-dlc-dockerfile`.\nThis dockerfile needs the mlp train script found here :ref:`mlp-train`\n\nWith the files in a dir, build the image with the following command:\n\n.. code:: bash\n\n docker build . -f Dockerfile.pt -t neuron-container:pytorch\n\nRun the following command to start the container\n\n.. code:: bash\n\n docker run -it --name pt-cont --net=host --device=/dev/neuron0 neuron-container:pytorch python3 /opt/ml/mlp_train.py" }, { "path": "containers/tutorials/tutorial-docker-env-setup.rst", "content": ".. _tutorial-docker-env-setup:\n\nTutorial Docker environment setup\n=================================\n\nIntroduction\n------------\n\nA Neuron application can be deployed using docker containers. This\ntutorial describes how to configure docker on Amazon Linux 2023 to expose Inferentia/Trainium devices\nto containers.\n\n\n.. tab-set::\n\n .. tab-item:: Training\n\n .. dropdown:: Install Drivers\n :class-title: sphinx-design-class-title-small\n :class-body: sphinx-design-class-body-small\n :animate: fade-in\n\n .. code:: bash\n\n # Configure Linux for Neuron repository updates\n\n sudo tee /etc/yum.repos.d/neuron.repo > /dev/null < /dev/null <`__ is an OCI\nruntime with the sole purpose of injecting OCI prestart, poststart, and\npoststop hooks into a container config.json before passing along to an\nOCI compatable runtime. oci-add-hooks is used to inject a hook that\nexposes Inferentia devices to the container.\n\n.. code:: bash\n\n sudo apt install -y golang && \\\n export GOPATH=$HOME/go && \\\n go get github.com/joeshaw/json-lossless && \\\n cd /tmp/ && \\\n git clone https://github.com/awslabs/oci-add-hooks && \\\n cd /tmp/oci-add-hooks && \\\n make build && \\\n sudo cp /tmp/oci-add-hooks/oci-add-hooks /usr/local/bin/\n\nInstall the package that has oci hook software\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. important::\n\n This step should run on the Linux host and not inside the container.\n\nFor Inf1 install the following package\n\n.. code:: bash\n\n sudo apt-get install aws-neuron-runtime-base -y\n\nFor Trn1 install the following package\n\n.. code:: bash\n\n sudo apt-get install aws-neuronx-oci-hook -y\n\nFor docker runtime setup Docker to use oci-neuron OCI runtime.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\noci-neuron is a script representing OCI compatible runtime. It wraps\noci-add-hooks, which wraps runc. In this step, we configure docker to\npoint at oci-neuron OCI runtime. Install dockerIO:\n\n.. code:: bash\n\n sudo cp /opt/aws/neuron/share/docker-daemon.json /etc/docker/daemon.json\n sudo service docker restart\n\nIf the docker restart command fails, make sure to check if the docker\nsystemd service is not masked. More information on this can be found\nhere: https://stackoverflow.com/a/37640824\n\nFor containerd runtime, setup containerd to use oci-neuron OCI runtime.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUpdate the following fields in the /etc/containerd/config.toml to configure\ncontainerd to use the neuron oci hook\n\n.. code:: bash\n\n default_runtime_name = \"neuron\"\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.neuron]\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.neuron.options]\n BinaryName = \"/opt/aws/neuron/bin/oci_neuron_hook_wrapper.sh\"\n\n\nAfter that restart the containerd daemon\n\n.. code:: bash\n\n sudo systemctl restart containerd\n\nFor cri-o runtime, setup cri-o to use oci-neuron OCI runtime.\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUpdate the following fields in the /etc/crio/crio.conf to configure\ncri-o to use the neuron oci hook\n\n.. code:: bash\n\n default_runtime_name = \"neuron\"\n [crio.runtime.runtimes.neuron]\n runtime_path = \"/opt/aws/neuron/bin/oci_neuron_hook_wrapper.sh\"\n\nAfter that restart the containerd daemon\n\n.. code:: bash\n\n sudo systemctl restart cri-o\n\n.. _oci-hook-workarounds:\n\nOCI hook workarounds\n^^^^^^^^^^^^^^^^^^^^\n\n**ECS (EC2)**\n\nAdd the following to your ECS task definition:\n\n.. code:: json\n\n \"linuxParameters\": {\n \"devices\": [\n {\n \"containerPath\": \"/dev/neuron0\",\n \"hostPath\": \"/dev/neuron0\",\n \"permissions\": [\n \"read\",\n \"write\"\n ]\n },\n {\n \"containerPath\": \"/dev/neuron1\",\n \"hostPath\": \"/dev/neuron1\",\n \"permissions\": [\n \"read\",\n \"write\"\n ]\n },\n ...,\n ],\n },\n\nThe linuxParameters parameter can be found under containerDefinition. More information can be found here:\nhttps://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_linuxparameters.\nExpose as many Neuron devices as needed, up to the max number of devices for the specified instance.\nFor example, the trn1.32xlarge instance type contains 16 neuron devices, so the devices that can be exposed are\n/dev/neuron0, /dev/neuron1, up to /dev/neuron15.\nTo see an example of an ECS task definition exposing Neuron devices,\nsee https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-inference-task-def.html.\n" }, { "path": "containers/tutorials.rst", "content": ".. meta::\n :description: Comprehensive tutorials for deploying AWS Neuron SDK in containers with Docker and Kubernetes. Learn to build Neuron containers, configure EKS clusters, deploy device plugins, and set up monitoring for Trainium and Inferentia instances.\n :keywords: Neuron containers, Docker, Kubernetes, EKS, Trainium, Inferentia, device plugin, scheduler, monitoring, tutorials, AWS, machine learning\n\nContainers - Tutorials\n=======================\n\nLearn how to deploy and manage AWS Neuron workloads in containerized environments. These tutorials cover everything from building Docker containers with Neuron support to deploying production-ready Kubernetes clusters with device plugins, schedulers, and monitoring solutions. Whether you're running inference or training workloads on AWS Trainium or Inferentia instances, these step-by-step guides will help you configure your container infrastructure for optimal performance and reliability.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n Inference \n Training \n /containers/tutorials/tutorial-docker-env-setup\n /containers/tutorials/build-run-neuron-container\n /containers/tutorials/tutorial-oci-hook\n /containers/tutorials/k8s-setup\n /containers/tutorials/k8s-neuron-helm-chart\n /containers/tutorials/k8s-neuron-scheduler-flow\n /containers/tutorials/k8s-neuron-monitor\n /containers/tutorials/k8s-neuron-problem-detector-and-recovery\n /containers/tutorials/k8s-neuron-problem-detector-and-recovery-irsa\n\n\nGeneral Container Tutorials\n----------------------------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Docker Environment Setup\n :link: /containers/tutorials/tutorial-docker-env-setup\n :link-type: doc\n\n Configure Docker on Amazon Linux 2023 to expose Inferentia and Trainium devices to containers. Install Neuron drivers, runtime, and configure the Docker daemon for Neuron device access.\n\n .. grid-item-card:: Build and Run Neuron Containers\n :link: /containers/tutorials/build-run-neuron-container\n :link-type: doc\n\n Learn how to build Docker images with Neuron support using provided Dockerfiles and run containerized applications on Inf1 and Trn1 instances with proper device exposure.\n\n .. grid-item-card:: Docker Neuron OCI Hook Setup\n :link: /containers/tutorials/tutorial-oci-hook\n :link-type: doc\n\n Install and configure the Neuron OCI hook to enable the AWS_NEURON_VISIBLE_DEVICES environment variable for exposing all Neuron devices to containers without explicit device flags.\n\nKubernetes Setup and Configuration\n-----------------------------------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Kubernetes Environment Setup\n :link: /containers/tutorials/k8s-setup\n :link-type: doc\n\n Complete guide to setting up Kubernetes for Neuron, including EKS cluster creation with Trainium nodes, device plugin installation, scheduler extension setup, and resource allocation configuration.\n\n .. grid-item-card:: Neuron Helm Chart\n :link: /containers/tutorials/k8s-neuron-helm-chart\n :link-type: doc\n\n Simplify Neuron infrastructure deployment with the unified Helm chart that installs device plugins, scheduler extensions, node problem detector, and DRA driver in a single command.\n\nKubernetes Device Management\n-----------------------------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Scheduler Flow Diagram\n :link: /containers/tutorials/k8s-neuron-scheduler-flow\n :link-type: doc\n\n Visual diagram showing how the Neuron Scheduler Extension integrates with Kubernetes components to schedule Pods with Neuron resource requests.\n\nKubernetes Monitoring and Recovery\n-----------------------------------\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Neuron Monitor\n :link: /containers/tutorials/k8s-neuron-monitor\n :link-type: doc\n\n Deploy Neuron Monitor to collect and expose metrics from Neuron devices and runtime. Integrate with Prometheus for observability, performance tracking, and troubleshooting.\n\n .. grid-item-card:: Node Problem Detector and Recovery\n :link: /containers/tutorials/k8s-neuron-problem-detector-and-recovery\n :link-type: doc\n\n Monitor Neuron device health and automatically remediate issues by detecting hardware failures, driver problems, and runtime errors. Enable automatic node replacement for faulty hardware.\n\n .. grid-item-card:: NPD Permissions (IRSA)\n :link: /containers/tutorials/k8s-neuron-problem-detector-and-recovery-irsa\n :link-type: doc\n\n Configure IAM roles for service accounts (IRSA) to grant the Neuron Node Problem Detector necessary permissions for Auto Scaling group operations and CloudWatch metrics.\n\n\nTraining and Inference Container Tutorials\n------------------------------------------ \n.. tab-set:: \n\n .. tab-item:: Training\n\n .. include:: /containers/tutorials/training/index.txt\n\n.. tab-set:: \n\n .. tab-item:: Inference\n \n .. include:: /containers/tutorials/inference/index.txt" }, { "path": "devflows/aws-batch-flows.rst", "content": ".. _aws_batch_flow:\n\nAWS Batch\n=========\n\n.. toctree::\n :maxdepth: 1\n\n /devflows/training/batch/batch-training\n\n \n" }, { "path": "devflows/aws-batch-flows.txt", "content": ".. tab-set:: \n\n .. tab-item:: Inference\n\n .. include:: /devflows/inference/aws-batch-flows.txt\n\n\n.. tab-set:: \n\n .. tab-item:: Training\n\n .. include:: /devflows/training/aws-batch-flows.txt" }, { "path": "devflows/dlc-then-customize-devflow.rst", "content": ".. _dlc-then-customize-devflow:\n\nCustomize Neuron DLC\n==============================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nDescription\n-----------\n\nThis guide covers how to customize and extend the Neuron Deep Learning Container (DLC) to fit your specific project needs. You can customize the DLC either by using the DLC as a base image in your Dockerfile or by modifying published Dockerfiles on GitHub.\n\nMethod 1: Using DLC as a Base Image\n-----------------\n\n1. Create a New Dockerfile. In your Dockerfile, specify the Neuron DLC as your base image using the FROM directive.\n\n2. Complete the Dockerfile. You can add additional packages, change the base environment, or any other modifications that suit your project. `AWS Batch Training `_ is a good example which needs customize Neuron DLC by using it as the base image. From its `Dockerfile `_, we can find the customized container copies llama_batch_training.sh to the container and runs it.\n\n3. Navigate to the directory containing your Dockerfile and build your custom container.\n\nMethod 2: Modifying Published Dockerfiles\n-----------------\n\n1. Visit the `Neuron DLC Github repo `_ and locate the Dockerfile for the container you wish to customize.\n\n2. Modify the Dockerfile as needed. You can add additional packages, change the base environment, or any other modifications that suit your project. For example, if you do not need to use Neuron tools in your scenario and want to make the container smaller, you can remove aws-neuronx-tools at this `line `_.\n\n3. Navigate to the directory containing your Dockerfile and build your custom container.\n" }, { "path": "devflows/ec2-flows.rst", "content": ".. _amazon-ec2:\n\nAmazon EC2\n==========\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Inference \n Training \n\n\n.. include:: /devflows/ec2-flows.txt\n" }, { "path": "devflows/ec2-flows.txt", "content": ".. tab-set:: \n\n .. tab-item:: Inference\n\n .. include:: /devflows/inference/ec2-flows.txt\n\n.. tab-set:: \n\n .. tab-item:: Training\n \n .. include:: /devflows/training/ec2-flows.txt\n" }, { "path": "devflows/ecs-flows.rst", "content": ".. _ecs_flow:\n\nAmazon ECS\n==========\n\n.. toctree::\n :maxdepth: 1\n\n /devflows/plugins/npd-ecs-flows\n /devflows/inference/dlc-then-ecs-devflow\n /devflows/training/dlc-then-ecs-devflow\n\nIn this section, you'll find resources to help you use Neuron with ECS cluster, deploying inference and training workloads on Inferentia and Trainium ECS clusters.\n\n\nUsing Neuron Node Problem Detector Plugin with ECS\n--------------------------------------------------\n\nNeuron node problem detector and recovery plugin enhances resiliency by detecting and remediating errors.\nTo get started with using Neuron node problem detector plugin and recovery plugin on an ECS cluster, please refer to :ref:`ecs-neuron-problem-detector-and-recovery`.\n\n\nRunning Inference workload\n--------------------------\n\nThis guide walks you through the end-to-end process of building and running a Docker container with your model and deploying it on an ECS cluster with Inferentia instances.\nFor running machine learning inference workloads on Amazon ECS using AWS Deep Learning Containers, please refer to :ref:`inference-dlc-then-ecs-devflow`.\n\n\nRunning Training workload\n-------------------------\n\nThis guide walks you through the end-to-end process of building and running a Docker container with your model and deploying it on an ECS cluster with Trainium instances.\nFor running machine learning training workloads on Amazon ECS using AWS Deep Learning Containers, please refer to :ref:`training-dlc-then-ecs-devflow`.\n" }, { "path": "devflows/eks-flows.rst", "content": ".. _eks_flow:\n\nAmazon EKS\n==========\n\n.. toctree::\n :maxdepth: 1\n\n /containers/kubernetes-getting-started\n /devflows/inference/dlc-then-eks-devflow\n /containers/tutorials/training/k8s_mlp_train_demo\n\n\nIn this section, you'll find resources to help you use Neuron with EKS cluster, deploying inference and training workloads on Inferentia and Trainium EKS clusters.\n\n\nEKS Setup\n------------\n\nThis guide covers setting up the Neuron device plugin, scheduler extension, node problem detector, and monitoring plugins.\nThese components enable efficient resource utilization, monitoring, and resilience when using Inferentia and Trainium instances for inference and training workloads on Kubernetes clusters.\nTo get started with using AWS Neuron and setting up the required plugins on an EKS cluster, please refer to :ref:`tutorial-k8s-env-setup-for-neuron`.\n\n\nRunning Inference workload\n--------------------------\n\nThis guide walks you through the end-to-end process of building and running a Docker container with your model and deploying it on an EKS cluster with Inferentia instances.\nFor running machine learning inference workloads on Amazon EKS using AWS Deep Learning Containers, please refer to :ref:`dlc-then-eks-devflow`.\n\n\nRunning Training workload\n-------------------------\n\nThis guide walks you through the end-to-end process of building and running a Docker container with your model and deploying it on an EKS cluster with Trainium instances.\nFor running machine learning training workloads on Amazon EKS using AWS Deep Learning Containers, please refer to :ref:`example-deploy-mlp-train-pod`.\n" }, { "path": "devflows/index.rst", "content": ".. _neuron-devflows:\n\n.. meta::\n :description:\n :date-modified:\n\nAWS Workload Orchestration\n==========================\n\nAWS Neuron integrates seamlessly with various AWS compute and orchestration services to accelerate deep learning workloads. This section provides deployment patterns and best practices for running Neuron-powered applications across different AWS services, from container orchestration to high-performance computing clusters.\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: Amazon EKS\n :link: /devflows/eks-flows\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Deploy Neuron workloads on Kubernetes with Amazon Elastic Kubernetes Service\n\n .. grid-item-card:: Amazon ECS\n :link: /devflows/ecs-flows\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Run containerized Neuron applications using Amazon Elastic Container Service\n\n .. grid-item-card:: AWS ParallelCluster\n :link: /devflows/parallelcluster-flows\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Set up HPC clusters for distributed training and inference workloads\n\n .. grid-item-card:: AWS Batch\n :link: /devflows/aws-batch-flows\n :link-type: doc\n :class-body: sphinx-design-class-title-small\n\n Execute batch ML jobs with automatic scaling and resource management\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /devflows/eks-flows\n /devflows/ecs-flows\n /devflows/parallelcluster-flows\n /devflows/aws-batch-flows\n Amazon SageMaker \n Third-party Solutions \n\n" }, { "path": "devflows/inference/aws-batch-flows.rst", "content": "AWS Batch Flows - Inference\n===========================\n\n\n.. include:: /devflows/inference/aws-batch-flows.txt" }, { "path": "devflows/inference/aws-batch-flows.txt", "content": ".. note::\n\n AWS Batch supports Inf1.\n\n An example of how to deploy a model with Neuron using Batch is coming soon.\n\n" }, { "path": "devflows/inference/byoc-hosting-devflow-inf2.rst", "content": ".. _byoc-hosting-devflow-inf2:\n\nBring Your Own Neuron Container to Sagemaker Hosting (inf2 or trn1)\n====================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/byoc-then-hosting-dev-flow.png\n :width: 850\n :alt: Neuron developer flow on SageMaker Neo\n :align: middle\n\nYou can use a SageMaker Notebook or an EC2 instance to compile models and build your own containers for deployment on SageMaker Hosting using ml.inf2 instances. In this developer flow, you provision a Sagemaker Notebook or an EC2 instance to train and compile your model to Inferentia. Then you deploy your model to SageMaker Hosting using the `SageMaker Python SDK `_. \n\nYou may not need to create a container to bring your own **code** to Amazon SageMaker. When you are using a framework such as TensorFlow or PyTorch that has direct support in SageMaker, you can simply supply the Python code that implements your algorithm using the SDK entry points for that framework.\n\nFollow the steps bellow to setup your environment. Once your environment is set you'll be able to follow the `Compiling and Deploying HuggingFace Pretrained BERT on Inf2 on Amazon SageMaker Sample `_.\n\n\n.. _byoc-hosting-setenv:\n\nSetup Environment\n-----------------\n\n1. Create a Compilation Instance:\n\tIf using an **EC2 instance for compilation only** you can use any instances to compile a model. It is recommended that you start with an c5.4xlarge instance. If using an **EC2 instance for compilation and test a model** you can use an Inf2 instance. Follow these steps to launch an Inf2 instance:\n\t\t\n\t\t.. include:: /setup/install-templates/inf2/launch-inf2-dlami.rst\n\t\n\n\tIf using an **SageMaker Notebook for compilation**, follow the instructions in `Get Started with Notebook Instances `_ to provision the environment. \n\n\tIt is recommended that you start with an ml.c5.4xlarge instance for the compilation. Also, increase the volume size of you SageMaker notebook instance, to accomodate the models and containers built locally. A volume of 10GB is sufficient.\n\t\n\t\t.. note::\n\t\t\t\n\t\t\tTo compile the model in the SageMaker Notebook instance, you'll need to install the Neuron Compiler and Neuron Framework Extensions. Follow the `Compiling and Deploying HuggingFace Pretrained BERT on Inf2 on Amazon SageMaker Sample `_ to install the environments. \n\n\n2. Set up the environment to compile a model, build your own container and deploy:\n To compile your model on EC2 or SageMaker Notebook, follow the *Set up a development environment* section on the EC2 :ref:`ec2-then-ec2-setenv` documentation.\n\n Refer to `Adapting Your Own Inference Container `_ documentation for information on how to bring your own containers to SageMaker Hosting.\n\n Make sure to add the **AmazonEC2ContainerRegistryPowerUser** role to your IAM role ARN, so you're able to build and push containers from your SageMaker Notebook instance.\n\n .. note::\n The container image can be created using :ref:`how-to-build-neuron-container`.\n" }, { "path": "devflows/inference/byoc-hosting-devflow.rst", "content": ".. _byoc-hosting-devflow:\n\nBring Your Own Neuron Container to Sagemaker Hosting (inf1)\n====================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/byoc-then-hosting-dev-flow.png\n :width: 850\n :alt: Neuron developer flow on SageMaker Neo\n :align: middle\n\nYou can use a SageMaker Notebook or an EC2 instance to compile models and build your own containers for deployment on SageMaker Hosting using ml.inf1 instances. In this developer flow, you provision a Sagemaker Notebook or an EC2 instance to train and compile your model to Inferentia. Then you deploy your model to SageMaker Hosting using the SageMaker Python SDK. Follow the steps bellow to setup your environment. Once your environment is set you'll be able to follow the :ref:`BYOC HuggingFace pretrained BERT container to Sagemaker Tutorial ` .\n\n.. _byoc-hosting-setenv:\n\nSetup Environment\n-----------------\n\n1. Create a Compilation Instance:\n\tIf using an **EC2 instance for compilation** you can use an Inf1 instance to compile and test a model. Follow these steps to launch an Inf1 instance:\n\t\t\n\t\t.. include:: /setup/install-templates/inf1/launch-inf1-ami.rst\n\t\n\n\tIf using an **SageMaker Notebook for compilation**, follow the instructions in `Get Started with Notebook Instances `_ to provision the environment. \n\n\tIt is recommended that you start with an ml.c5.4xlarge instance for the compilation. Also, increase the volume size of you SageMaker notebook instance, to accomodate the models and containers built locally. A volume of 10GB is sufficient.\n\t\n\t\t.. note::\n\t\t\t\n\t\t\tTo compile the model in the SageMaker Notebook instance, you'll need to update the conda environments to include the Neuron Compiler and Neuron Framework Extensions. Follow the installation guide on the section :ref:`how-to-update-to-latest-Neuron-Conda-Env` to update the environments. \n\n\n2. Set up the environment to compile a model, build your own container and deploy:\n To compile your model on EC2 or SageMaker Notebook, follow the *Set up a development environment* section on the EC2 :ref:`ec2-then-ec2-setenv` documentation.\n\n Refer to `Adapting Your Own Inference Container `_ documentation for information on how to bring your own containers to SageMaker Hosting.\n\n Make sure to add the **AmazonEC2ContainerRegistryPowerUser** role to your IAM role ARN, so you're able to build and push containers from your SageMaker Notebook instance.\n\n .. note::\n The container image can be created using :ref:`how-to-build-neuron-container`.\n" }, { "path": "devflows/inference/container-sm-hosting-devflow.rst", "content": ".. _container-sm-hosting-devflow:\n\nDeploy on Sagemaker Hosting\n===========================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\nYou can use `Sagemaker Hosted Endpoint `_ to do inference on Inf1 instances." }, { "path": "devflows/inference/dev-flows.rst", "content": ".. _neuron1-devflows:\n.. _compilation-flow-target:\n.. _deploym-flow-target:\n\nDeveloper Flows Introduction\n============================\n\n|image|\n\n \n.. |image| image:: /images/neuron-devflow.jpg\n :width: 500\n :alt: Neuron developer flow\n \nA typical Neuron developer flow includes compilation phase and then deployment (inference) on inf1 instance/s. You can develop on Neuron using one of the following combinations of developer flows:\n\n\n\n.. toctree::\n :maxdepth: 1\n\n ec2-then-ec2-devflow\n ec2-then-ec2-devflow-inf2\n neo-then-hosting-devflow\n byoc-hosting-devflow\n dlc-then-ec2-devflow\n dlc-then-ecs-devflow\n dlc-then-eks-devflow\n\n\n\n" }, { "path": "devflows/inference/dlc-then-ec2-devflow.rst", "content": ".. _dlc-then-ec2-devflow:\n\nDeploy Neuron Container on EC2\n==============================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/dlc-on-ec2-dev-flow.png\n :width: 500\n :alt: Neuron developer flow for DLC on EC2\n :align: middle\n\nYou can use the Neuron version of the `AWS Deep Learning Containers `_ to run inference on inf1 instances. In this developer flow, you provision an EC2 inf1 instance using a Deep Learming AMI (DLAMI), pull the container image with the Neuron version of the desired framework, and run the container as a server for the already compiled model. This developer flow assumes the model has already has been compiled through a :ref:`compilation developer flow ` \n\n.. _dlc-then-ec2-setenv:\n\nSetup Environment\n-----------------\n\n1. Launch an Inf1 Instance\n\t.. include:: /setup/install-templates/inf1/launch-inf1-ami.rst\n\n2. Once you have your EC2 environment set according to :ref:`tutorial-docker-env-setup`, you can build and run a Neuron container using the :ref:`how-to-build-neuron-container` section above.\n\n.. [DLC specific flow, uncomment when DLC available] Follow the `Getting Started with Deep Learning Containers for Inference on EC2 `_ and use the appropriate DLC container.\n\n\n.. note:: \n\n\t**Prior to running the container**, make sure that the Neuron runtime on the instance is turned off, by running the command:\n\n\t.. code:: bash\n\n\t\tsudo service neuron-rtd stop\n\n\n\n" }, { "path": "devflows/inference/dlc-then-ecs-devflow.rst", "content": ".. _inference-dlc-then-ecs-devflow:\n\nDeploy Neuron Container on Elastic Container Service (ECS) for Inference\n========================================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/dlc-on-ecs-dev-flow.png\n :width: 750\n :alt: Neuron developer flow for DLC on ECS\n :align: middle\n\nYou can use the Neuron version of the `AWS Deep Learning Containers `_ to run inference on Amazon Elastic Container Service (ECS). In this developer flow, you set up an ECS cluster with inf1/inf2 instances, create a task description for your inference service and deploy it to your cluster. This developer flow assumes:\n\n1. The model has already been compiled through :ref:`Compilation with Framework API on EC2 instance ` or through :ref:`Compilation with Sagemaker Neo `. \n\n2. You already set up your container to retrieve it from storage.\n\n.. _inference-dlc-then-ecs-setenv:\n\nSetup Environment\n-----------------\n\n\n1. Set up an Amazon ECS cluster:\n\tFollow the instructions on `Setting up Amazon ECS for Deep Learning Containers `_\n\n2. Define an Inference Task:\n\tUse the instruction on the `DLC Inference on ECS Tutorial `_ to define a task and create a service for the appropriate framework.\n\n\tWhen creating tasks for inferentia instances on ECS, be aware of the considerations and requirements listed in `Working with inference workloads on Amazon ECS `_. \n\n\n3. Use the container image created using :ref:`how-to-build-neuron-container` as the ``image`` in your task definition.\n\n .. _inference-push_to_ecr_note:\n\n .. note::\n\n Before deploying your task definition to your ECS cluster, make sure to push the image to ECR. Refer to `Pushing a Docker image `_ for more information.\n" }, { "path": "devflows/inference/dlc-then-eks-devflow.rst", "content": ".. _dlc-then-eks-devflow:\n\nDeploy Neuron Container on Elastic Kubernetes Service (EKS) for Inference\n=========================================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/dlc-on-eks-dev-flow.png\n :width: 750\n :alt: Neuron developer flow for DLC on ECS\n :align: middle\n\nYou can use the Neuron version of the `AWS Deep Learning Containers `_ to run inference on Amazon Elastic Kubernetes Service (EKS). In this developer flow, you set up an EKS cluster with Inf1 instances, create a Kubernetes manifest for your inference service and deploy it to your cluster. This developer flow assumes:\n\n1. The model has already been compiled through :ref:`Compilation with Framework API on EC2 instance ` or through :ref:`Compilation with Sagemaker Neo `. \n\n2. You already set up your container to retrieve it from storage.\n\n.. _dlc-then-eks-setenv:\n\nSetup Environment\n-----------------\n\nPlease add inferentia nodes using instructions at :ref:`tutorial-k8s-env-setup-for-neuron` . \n\nUsing the YML deployment manifest shown `in the EKS documentation for inferentia `_, replace the `image` in the `containers` specification with the one you built using :ref:`how-to-build-neuron-container`.\n\n .. note::\n\n Before deploying the yaml to your EKS cluster, make sure to push the image to ECR. Refer to `Pushing a Docker image `_ for more information.\n\n\nInference Example\n-----------------\nPlease refer to :ref:`example-deploy-rn50-as-k8s-service` run a simple inference example. Note that the container image referenced in the YML manifest is created using :ref:`how-to-build-neuron-container`.\n" }, { "path": "devflows/inference/dlc-then-k8s-devflow.rst", "content": ".. _dlc-then-k8s-devflow:\n\nDeploy Neuron Container on Kubernetes\n======================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\nUse of Neuron in containers on Kubernetes cluster can be simple to achieve by following :ref:`tutorial-k8s-env-setup-for-neuron`\n\nKnown Limitations\n-----------------\nScheduling on k8s cluster requires contiguous neuron device-ids\n" }, { "path": "devflows/inference/ec2-flows.rst", "content": "EC2 Flows - Inference\n=====================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /devflows/inference/ec2-then-ec2-devflow\n /devflows/inference/ec2-then-ec2-devflow-inf2\n\n \n\n.. include:: /devflows/inference/ec2-flows.txt\n\n" }, { "path": "devflows/inference/ec2-flows.txt", "content": "* :ref:`ec2-then-ec2-devflow`\n* :ref:`ec2-then-ec2-devflow-inf2`\n \n" }, { "path": "devflows/inference/ec2-then-ec2-devflow-inf2.rst", "content": ".. _ec2-then-ec2-devflow-inf2:\n\nCompile with Framework API and Deploy on EC2 Inf2\n=================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/ec2-then-ec2-dev-flow-inf2.png\n :width: 500\n :alt: Neuron developer flow on EC2\n :align: middle\n\nYou can use a single inf2 instance as a development environment to compile and deploy Neuron models. In this developer flow, you provision an EC2 inf2 instance using a Deep Learning AMI (DLAMI) and execute the two steps of the development flow in the same instance. The DLAMI comes pre-packaged with the Neuron frameworks, compiler, and required runtimes to complete the flow. Development happens through Jupyter Notebooks or using a secure shell (ssh) connection in terminal. Follow the steps below to setup your environment. \n\n.. note::\n\t**Model compilation can be executed on a non-inf2 instance** for later deployment. \n\tFollow the same EC2 Developer Flow Setup using other instance families and leverage `Amazon Simple Storage Service `_ (S3) to share the compiled models between different instances. \n\n.. _ec2-then-ec2-setenv:\n\nSetup Environment\n-----------------\n\n1. Launch an Inf2 Instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n .. include:: /setup/install-templates/inf2/launch-inf2-dlami.rst\n \n\n2. Set up a development environment\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \nEnable PyTorch-Neuron\n~~~~~~~~~~~~~~~~~~~~~\n\n.. include :: /setup/install-templates/inf2/note-setup-libnrt-warning.rst\n\n.. include:: /setup/install-templates/inf2/dlami-enable-neuron-pytorch.rst\n\n3. Set up Jupyter notebook\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo develop from a Jupyter notebook see :ref:`setup-jupyter-notebook-steps-troubleshooting` \n\nYou can also run a Jupyter notebook as a script, first enable the ML framework Conda or Python environment of your choice and see :ref:`running-jupyter-notebook-as-script` for instructions. \n" }, { "path": "devflows/inference/ec2-then-ec2-devflow.rst", "content": ".. _ec2-then-ec2-devflow:\n\nCompile with Framework API and Deploy on EC2 Inf1\n=================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/ec2-then-ec2-dev-flow.png\n :width: 500\n :alt: Neuron developer flow on EC2\n :align: middle\n\nYou can use a single inf1 instance as a development environment to compile and deploy Neuron models. In this developer flow, you provision an EC2 inf1 instance using a Deep Learming AMI (DLAMI) and execute the two steps of the development flow in the same instance. The DLAMI comes pre-packaged with the Neuron frameworks, compiler, and required runtimes to complete the flow. Development happens through Jupyter Notebooks or using a secure shell (ssh) connection in terminal. Follow the steps bellow to setup your environment. \n\n.. note::\n\t**Model compilation can be executed on a non-inf1 instance** for later deployment. \n\tFollow the same EC2 Developer Flow Setup using other instance families and leverage `Amazon Simple Storage Service `_ (S3) to share the compiled models between different instances. \n\n.. _ec2-then-ec2-setenv:\n\nSetup Environment\n-----------------\n\n1. Launch an Inf1 Instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n .. include:: /setup/install-templates/inf1/launch-inf1-dlami.rst\n \n\n2. Set up a development environment\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \nEnable PyTorch-Neuron\n~~~~~~~~~~~~~~~~~~~~~\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. include:: /setup/install-templates/inf1/dlami-enable-neuron-pytorch.rst\n\nEnable TensorFlow-Neuron\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. include:: /setup/install-templates/inf1/dlami-enable-neuron-tensorflow.rst\n\nEnable Apache MXNet\n~~~~~~~~~~~~~~~~~~~~\n\n.. include :: /setup/install-templates/inf1/note-setup-libnrt-warning.rst\n\n.. include:: /setup/install-templates/inf1/dlami-enable-neuron-mxnet.rst\n\n3. Set up Jupyter notebook\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo develop from a Jupyter notebook see :ref:`setup-jupyter-notebook-steps-troubleshooting` \n\nYou can also run a Jupyter notebook as a script, first enable the ML framework Conda or Python environment of your choice and see :ref:`running-jupyter-notebook-as-script` for instructions. \n" }, { "path": "devflows/inference/env-setup-text.rst", "content": "A typical Neuron developer flow includes compilation phase and then deployment (inference) on inf1 instance/s.\n\n\nYou can also choose one of the following combinations for compilation and deployment:\n\n\n \n\n" }, { "path": "devflows/inference/neo-then-hosting-devflow.rst", "content": ".. _neo-then-hosting-devflow:\n\nCompile with Sagemaker Neo and Deploy on Sagemaker Hosting (inf1)\n==========================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/neo-then-hosting-dev-flow.png\n :width: 700\n :alt: Neuron developer flow on SageMaker Neo\n :align: middle\n\nYou can use SageMaker Neo to compile models for deployment on SageMaker Hosting using ml.inf1 instances. In this developer flow, you provision a Sagemaker Notebook instance to train, compile and deploy your model using the SageMaker Python SDK. Follow the steps bellow to setup your environment. \n\n.. _neo-then-hosting-setenv:\n\nSetup Environment\n-----------------\n\n1. Create an Amazon SageMaker Notebook Instance:\n\n\tFollow the instructions in `Get Started with Notebook Instances `_\n\n\tThe Notebook instance created provides the required Python SDK for training, compiling and deploying models with Amazon SageMaker.\n\n2. Compile a model using the Amazon SageMaker SDK:\n\n\tRefer to `Supported Instances Types and Frameworks `_ for information on the framework versions currently supported by Amazon SageMaker Neo on AWS Inferentia. \n\n\tMore information about compiling and deploying models with Amazon SageMaker Neo can be found on `Use Neo to Compile a Model `_\n\n\n\n\n\n\n" }, { "path": "devflows/inference/parallelcluster-flows.rst", "content": "Parallel Cluster Flows - Inference\n===================================\n\n\n.. include:: /devflows/inference/parallelcluster-flows.txt" }, { "path": "devflows/inference/parallelcluster-flows.txt", "content": ".. note::\n\n AWS ParallelCluster support is coming soon." }, { "path": "devflows/inference/sagemaker-flows.rst", "content": "Sagemaker Flows - Inference\n===========================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n \n /devflows/inference/byoc-hosting-devflow-inf2\n /devflows/inference/byoc-hosting-devflow \n /devflows/inference/neo-then-hosting-devflow\n\n \n\n.. include:: /devflows/inference/sagemaker-flows.txt" }, { "path": "devflows/inference/sagemaker-flows.txt", "content": "* :ref:`byoc-hosting-devflow-inf2`\n* :ref:`byoc-hosting-devflow`\n* :ref:`neo-then-hosting-devflow`\n* `AWS Neuron Sagemaker Samples GitHub Repository `_\n" }, { "path": "devflows/parallelcluster-flows.rst", "content": "AWS ParallelCluster\n===================\n\n.. toctree::\n :maxdepth: 1\n\n /devflows/training/parallelcluster-flows\n\n\n.. .. include:: /devflows/parallelcluster-flows.txt\n" }, { "path": "devflows/parallelcluster-flows.txt", "content": ".. tab-set:: \n\n .. tab-item:: Training\n \n .. include:: /devflows/training/parallelcluster-flows.txt\n \n\n.. tab-set:: \n\n .. tab-item:: Inference\n\n .. note::\n\n AWS ParallelCluster support is coming soon.\n\n\n" }, { "path": "devflows/plugins/npd-ecs-flows.rst", "content": ".. _ecs-neuron-problem-detector-and-recovery:\n\nNeuron Problem Detector And Recovery\n====================================\n\n.. include:: /devflows/plugins/npd-ecs-flows.txt\n" }, { "path": "devflows/plugins/npd-ecs-flows.txt", "content": "Neuron node problem detector and recovery artifact checks the health of Neuron devices on each ECS instance. After detecting an unrecoverable Neuron error, it triggers an instance replacement. In order to get started with Neuron node problem detector and recovery, make sure that the following requirements are satisfied:\n\n* The Neuron node problem detector and recovery requires Neuron driver 2.15+, and it requires the runtime to be at SDK 2.18 or later.\n\nCreating a Task Definition\n--------------------------\n\nConfiguration\n~~~~~~~~~~~~~\n\nThe task definition includes two containers:\n\n- **npd-container**: This container is responsible for enabling Problem detection functionality in the ECS cluster.\n- **recovery-container**: This container handles recovery operations in case of failures detected by Neuron Problem Detector.\n\nThe **recovery-container** has an environment variable called ``ENABLE_RECOVERY`` that controls whether recovery is enabled or disabled. Set the value to ``true`` to enable recovery, or ``false`` to disable it.\n\nFollow these steps to create a task definition for NPD and recovery:\n\n1. Go to the `ECS console `_ and select **Task Definitions** in the navigation pane.\n2. Click **Create new Task Definition** and choose **Create new Task Definition with JSON**.\n3. Paste the task definition JSON provided, replacing the placeholders with your account-specific values.\n\n .. code-block:: json\n\n {\n \"family\": \"neuron-npd-and-recovery\",\n \"containerDefinitions\": [\n {\n \"name\": \"npd\",\n \"image\": \"registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.19\",\n \"cpu\": 0,\n \"portMappings\": [\n {\n \"name\": \"npd-80-tcp\",\n \"containerPort\": 80,\n \"hostPort\": 80,\n \"protocol\": \"tcp\",\n \"appProtocol\": \"http\"\n }\n ],\n \"essential\": true,\n \"entryPoint\": [\n \"/bin/sh\",\n \"-c\"\n ],\n \"command\": [\n \"echo '{\\\"plugin\\\":\\\"kmsg\\\",\\\"logPath\\\":\\\"/dev/kmsg\\\",\\\"lookback\\\":\\\"5m\\\",\\\"bufferSize\\\":10,\\\"source\\\":\\\"kernel-monitor\\\",\\\"conditions\\\":[{\\\"type\\\":\\\"NeuronHealth\\\",\\\"reason\\\":\\\"NeuronHasNoError\\\",\\\"message\\\":\\\"Neuronhasnoerror\\\"}],\\\"rules\\\":[{\\\"type\\\":\\\"permanent\\\",\\\"condition\\\":\\\"NeuronHealth\\\",\\\"reason\\\":\\\"NeuronHasError_SRAM_UNCORRECTABLE_ERROR\\\",\\\"pattern\\\":\\\".*NEURON_HW_ERR=SRAM_UNCORRECTABLE_ERROR.*\\\"},{\\\"type\\\":\\\"permanent\\\",\\\"condition\\\":\\\"NeuronHealth\\\",\\\"reason\\\":\\\"NeuronHasError_NC_UNCORRECTABLE_ERROR\\\",\\\"pattern\\\":\\\".*NEURON_HW_ERR=NC_UNCORRECTABLE_ERROR.*\\\"},{\\\"type\\\":\\\"permanent\\\",\\\"condition\\\":\\\"NeuronHealth\\\",\\\"reason\\\":\\\"NeuronHasError_HBM_UNCORRECTABLE_ERROR\\\",\\\"pattern\\\":\\\".*NEURON_HW_ERR=HBM_UNCORRECTABLE_ERROR.*\\\"},{\\\"type\\\":\\\"permanent\\\",\\\"condition\\\":\\\"NeuronHealth\\\",\\\"reason\\\":\\\"NeuronHasError_DMA_ERROR\\\",\\\"pattern\\\":\\\".*NEURON_HW_ERR=DMA_ERROR.*\\\"}]}' > /config/kernel-monitor.json && /node-problem-detector --v=2 --logtostderr --enable-k8s-exporter=false --config.system-log-monitor=/config/kernel-monitor.json\"\n ],\n \"environment\": [],\n \"mountPoints\": [],\n \"volumesFrom\": [],\n \"linuxParameters\": {\n \"devices\": [\n {\n \"hostPath\": \"/dev/kmsg\",\n \"containerPath\": \"/dev/kmsg\",\n \"permissions\": [\n \"read\",\n \"write\"\n ]\n }\n ]\n },\n \"privileged\": true,\n \"logConfiguration\": {\n \"logDriver\": \"awslogs\",\n \"options\": {\n \"awslogs-group\": \"/ecs/npd\",\n \"awslogs-create-group\": \"true\",\n \"awslogs-region\": \"us-west-2\",\n \"awslogs-stream-prefix\": \"ecs\"\n },\n \"secretOptions\": []\n },\n \"systemControls\": []\n },\n {\n \"name\": \"recovery\",\n \"image\": \"public.ecr.aws/neuron/neuron-node-recovery:1.3.0\",\n \"cpu\": 0,\n \"portMappings\": [],\n \"essential\": true,\n \"entryPoint\": [\n \"/bin/sh\",\n \"-c\"\n ],\n \"command\": [\n \"python scripts/check-health.py\"\n ],\n \"environment\": [\n {\n \"name\": \"ENABLE_RECOVERY\",\n \"value\": \"false\"\n }\n ],\n \"mountPoints\": [],\n \"volumesFrom\": [],\n \"readonlyRootFilesystem\": true,\n \"logConfiguration\": {\n \"logDriver\": \"awslogs\",\n \"options\": {\n \"awslogs-create-group\": \"true\",\n \"awslogs-group\": \"/ecs/recovery\",\n \"awslogs-region\": \"us-west-2\",\n \"awslogs-stream-prefix\": \"ecs\"\n }\n },\n \"systemControls\": []\n }\n ],\n \"executionRoleArn\": \"arn:aws:iam::012345678910:role/ecsTaskExecutionRole\",\n \"taskRoleArn\": \"arn:aws:iam::012345678910:role/ecsTaskExecutionRole\",\n \"networkMode\": \"awsvpc\",\n \"requiresCompatibilities\": [\n \"EC2\"\n ],\n \"cpu\": \"1024\",\n \"memory\": \"3072\",\n \"runtimePlatform\": {\n \"cpuArchitecture\": \"X86_64\",\n \"operatingSystemFamily\": \"LINUX\"\n }\n }\n\n4. Review the task definition and click **Create**.\n\nFor more details on task definitions, refer to the `AWS documentation `_.\n\n.. _deploy-service:\n\nDeploying the Service\n---------------------\n\nAfter creating the task definition, follow these steps to deploy the service:\n\n1. In the ECS console, select the task definition and click **Deploy** → **Create Service**.\n2. Select your ECS cluster, set the launch type to **EC2**, and the service type to **Daemon**.\n3. Click **Create** to deploy the service.\n\nFor more details on deploying services, refer to the `AWS documentation `_.\n\nPermissions\n~~~~~~~~~~~\n\nEnsure the ECS task execution role and task role have permissions to:\n\n- Publish metrics to CloudWatch\n- Read and set health status of EC2 instances in the Auto Scaling group\n\nRefer to the `AWS documentation on IAM roles for ECS tasks `_ for more information.\n\nWhen any unrecoverable error occurs, Neuron node problem detector and recovery publishes a metric under the CloudWatch namespace NeuronHealthCheck. It also reflects in NodeCondition and can be seen with kubectl describe node." }, { "path": "devflows/sagemaker-flows.rst", "content": ".. _sagemaker_flow:\n\nAmazon SageMaker\n================\n\nAmazon SageMaker is a fully managed machine learning (ML) platform that streamlines the end-to-end ML workflow at scale. AWS Neuron integrates \nwith Amazon SageMaker to provide optimized performance for ML workloads on AWS Inferentia and AWS Trainium chips.\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nSageMaker JumpStart\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\nUse `Amazon SageMaker JumpStart `_ to train and deploy models using Neuron. SageMaker JumpStart is an ML hub that accelerates model \nselection and deployment. It provides support for fine-tuning and deploying popular models such as Meta’s Llama family of models. \nUsers can customize pre-trained models with their data and easily deploy them.\n\nSageMaker HyperPod\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\nUse `Amazon SageMaker HyperPod `_ to streamline ML infrastructure setup and optimization with AWS Neuron. SageMaker HyperPod leverages \npre-configured distributed training libraries to split workloads across numerous AI accelerators, enhancing model performance. \nHyperPod ensures uninterrupted training through automatic checkpointing, fault detection, and recovery.\n\nSageMaker Training\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n`Amazon SageMaker Model Training `_ reduces the time and cost to train and tune ML models at scale without the need to manage infrastructure.\n\nSageMaker Inference\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\nWith `Amazon SageMaker `_ , you can start getting predictions, or inferences, from your trained ML models. SageMaker \nprovides a broad selection of ML infrastructure and model deployment options to help meet all your ML inference needs." }, { "path": "devflows/setup/ecs-flows.rst", "content": "ECS Flows - Setup\n=================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /devflows/plugins/npd-ecs-flows\n\n.. include:: /devflows/setup/ecs-flows.txt" }, { "path": "devflows/setup/ecs-flows.txt", "content": "* :ref:`ecs-neuron-problem-detector-and-recovery`" }, { "path": "devflows/setup/eks-flows.rst", "content": "EKS - Setup\n=====================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /containers/kubernetes-getting-started\n\n\n\n.. include:: /devflows/setup/eks-flows.txt\n" }, { "path": "devflows/setup/eks-flows.txt", "content": "* :ref:`kubernetes-getting-started`\n" }, { "path": "devflows/third-party-solutions.rst", "content": ".. _third-party-devflow-solutions:\n\nThird-party solutions\n====================\n\nAWS Neuron integrates with multiple third-party partner solutions that alow you to run deep learning workloads on Amazon EC2 \ninstances powered by AWS Trainium and AWS Inferentia chips. The following list gives an overview of third-party solutions \nthat work with AWS Neuron.\n\n.. contents:: Table of contents\n :local:\n :depth: 1\n\nRay \n\"\"\"\nRay, by Anyscale, is the open source AI Compute Engine at the center of the world's most powerful AI Platforms. It precisely \norchestrates infrastructure for any distributed AI workload like data processing, model training, and serving on any accelerator at \nany scale. Ray simplifies the complexity of distributed computing, improves efficiency, lower costs, and accelerates developer \nproductivity.\n\n`Ray Train documentation `_\n\nDomino\n\"\"\"\"\"\"\nDomino is an open enterprise platform for data science, machine learning, and AI research. It works with an expansive list of \nindustry leading tools and technologies to enrich data science research, development, and deployment processes. Domino works with a \nwide range of data sources, languages, IDEs, tools, libraries, and publication targets.\n\n`Domino documentation `_\n" }, { "path": "devflows/training/aws-batch-flows.rst", "content": "AWS Batch Flows- Training\n=========================\n\n\n.. include:: /devflows/training/aws-batch-flows.txt" }, { "path": "devflows/training/aws-batch-flows.txt", "content": "* :ref:`batch-training`" }, { "path": "devflows/training/batch/batch-training.rst", "content": ".. _batch-training:\n\nTrain your model on AWS Batch\n=============================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\nDescription\n------------\n\nAWS Batch provides a scalable and cost-effective solution for running batch computing workloads in the AWS Cloud. Integrating Trainium with AWS Batch provides an efficient and cost-effective way of training deep learning models at scale.\nOnce you configure your training job, AWS Batch effectively manages the orchestration, execution, and dynamic scaling of compute resources for your extensive machine learning workloads. To learn more about AWS Batch, see `the AWS Batch documentation `_.\n\n\nHow does AWS Batch work with Trainium\n-------------------------------------\n\n.. image:: /images/batch-setup.png\n\n\nAs depicted in the illustration above, our workflow begins by building a ``Docker container image for Trainium`` and pushing it to Amazon Elastic Container Registry (ECR). Following this, we configure our AWS Batch environment with the required capabilities, and subsequently submit the training job.\n\nPlease follow the below mentioned steps to run your training jobs on ``AWS Batch`` with ``Trainium``.\n\n#. **Before you begin, please ensure that you have the following prerequisites completed:**\n\n * ``AWS VPC`` with at least one ``Subnet`` and ``EFA Enabled Security Group`` (learn more about EFA-enabled security group `the AWS EFA User Guide `_). Please make sure subnet needs to be private, and the VPC needs to have a NAT gateway to allow internet connectivity for the private subnet.\n * ``AWS ECR`` repository\n * ``AWS CLI`` installed and configured with permissions for the above mentioned AWS resources\n * ``Docker``\n * ``jq``\n\n#. **Setup to start working with AWS Batch**\n\n Connect to your EC2 instance(``x86_64-based Linux instance``) and clone the ``aws-neuron-samples`` repo. Once done, navigate to aws batch scripts directory.\n\n .. code:: shell\n\n cd ~/\n git clone https://github.com/aws-neuron/aws-neuron-samples.git\n cd ~/aws-neuron-samples/torch-neuronx/training/aws-batch/all-reduce\n\n#. **Configure resource requirements**\n\n Update the ``build_configs_and_setup.sh`` with your environment variables. Once done, execute the bash script using the command ``./build_configs_and_setup.sh``.\n\n#. **Build the required docker image and publish it to ECR**\n\n Run ``./build_docker_image.sh`` to build a Neuron Deep-Learning Container image using the latest Neuron packages and push this image to ECR.\n\n#. **Prepare the AWS infrastructure required to submit the batch job**\n\n Run ``./create_resources.sh`` to create all AWS Batch resources needed for your training workload. Below is the brief description of various AWS Batch components this script will create for you -\n\n * ``Placement Group`` enables you to influence the placement of your EC2 (Elastic Compute Cloud) instances within the AWS infrastructure.\n * ``Launch Template`` allows you to define a set of instance configuration parameters, including the Amazon Machine Image (AMI), instance type, key pair, security groups, and other settings, in a template format.\n * ``Compute Environment`` helps you to specify configuration that specifies the type of compute resources you want to use for your batch jobs. It includes details such as the EC2 instance types, the minimum and maximum number of instances, the VPC configuration, and other settings related to the compute environment.\n * ``Job Definition`` is a blueprint that specifies how a batch job should be run. It encapsulates information about the job, such as the Docker image to be used, the command to execute within the container, the CPU and memory requirements, job dependencies, and other settings.\n * ``Job Queue`` acts as a queueing mechanism for managing and scheduling the execution of batch computing workloads. By using job queues, AWS Batch provides a scalable and efficient way to process batch workloads, managing the allocation of resources and ensuring optimal use of compute capacity.\n\n#. **Submit the job to AWS-Batch**\n\n Run ``./submit_job.sh`` to submit a basic all-reduce job in the provisioned AWS Batch environment\n\n#. **Monitor the AWS-Batch job**\n\n You can use Amazon CloudWatch Logs to monitor, store, and view all your logs from AWS Batch job. To learn more about it, please see `the AWS docs on using Batch and EKS with CloudWatch `_.\n\n.. note::\n * You could run a full model training job using this setup. For example, `this sample `_ runs the Llama2-7B tutorial on AWS Batch using the same setup.\n * You can further tailor your ``Dockerfile`` to include any additional dependencies specific to your needs.\n * You have the option to leverage ``trn1n.32xlarge`` instances as an alternative to ``trn1.32xlarge``. To make this transition, you only need to make adjustments to the ``launch template`` and ``job definition`` in order to accommodate the use of 16 EFA (Elastic Fabric Adapter) devices, whereas the current setup for ``trn1`` employs 8 EFA devices. Please check out `this document `_ to start with ``trn1n.32xlarge`` for multi-node execution." }, { "path": "devflows/training/dlc-then-ecs-devflow.rst", "content": ".. _training-dlc-then-ecs-devflow:\n\nDeploy Neuron Container on Elastic Container Service (ECS) for Training\n=======================================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/dlc-on-ecs-dev-flow.png\n :width: 750\n :alt: Neuron developer flow for DLC on ECS\n :align: middle\n\nYou can use the Neuron version of the `AWS Deep Learning Containers `_ to run training on Amazon Elastic Container Service (ECS). In this developer flow, you set up an ECS cluster with trn1 instances, create a task description for your training container and deploy it to your cluster. This developer flow assumes:\n\n1. The model has already been compiled through :ref:`Compilation with Framework API on EC2 instance ` or through :ref:`Compilation with Sagemaker Neo `.\n\n2. You already set up your container to retrieve it from storage.\n\n.. _training-dlc-then-ecs-setenv:\n\nSetup Environment\n-----------------\n\n\n1. Set up an Amazon ECS cluster:\n\tFollow the instructions on `Setting up Amazon ECS for Deep Learning Containers `_\n\n2. Define a Training Task:\n\tUse the instruction on the `DLC Training on ECS Tutorial `_ to define a task and create a service for the appropriate framework.\n\n\tWhen creating tasks for trn1 instances on ECS, be aware of the considerations and requirements listed in `Working with training workloads on Amazon ECS `_.\n\n\n3. Use the container image created using :ref:`how-to-build-neuron-container` as the ``image`` in your task definition.\n\n .. _training_push_to_ecr_note:\n\n .. note::\n\n Before deploying your task definition to your ECS cluster, make sure to push the image to ECR. Refer to `Pushing a Docker image `_ for more information.\n" }, { "path": "devflows/training/ec2/ec2-training.rst", "content": ".. _ec2-training:\n\nTrain your model on EC2\n=======================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n \nDescription\n-----------\n\n|image|\n \n.. |image| image:: /images/trn1-on-ec2-dev-flow.png\n :width: 500\n :alt: Neuron developer flow on EC2\n :align: middle\n \nYou can use a single Trn1 instance as a development environment to compile and train Neuron models. In this developer flow, you provision an EC2 Trn1 instance using a Deep Learming AMI (DLAMI) and execute the two steps of the development flow in the same instance. The DLAMI comes pre-packaged with the Neuron frameworks, compiler, and required runtimes to complete the flow. Development happens through Jupyter Notebooks or using a secure shell (ssh) connection in terminal. Follow the steps bellow to setup your environment.\n\nSetup Environment\n-----------------\n\n1. Launch an Trn1 Instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n .. include:: /setup/install-templates/launch-trn1-dlami.rst\n\n2. Set up a development environment\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \nEnable PyTorch-Neuron\n~~~~~~~~~~~~~~~~~~~~~\n\n .. include:: /frameworks/torch/torch-neuronx/setup/install-templates/pytorch-dev-install.txt\n\n3. Set up Jupyter notebook\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo develop from a Jupyter notebook see :ref:`setup-jupyter-notebook-steps-troubleshooting` \n\nYou can also run a Jupyter notebook as a script, first enable the ML framework Conda or Python environment of your choice and see :ref:`running-jupyter-notebook-as-script` for instructions. \n" }, { "path": "devflows/training/ec2-flows.rst", "content": "EC2 Flows- Training\n====================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /devflows/training/ec2/ec2-training\n\n \n\n.. include:: /devflows/training/ec2-flows.txt" }, { "path": "devflows/training/ec2-flows.txt", "content": "* :ref:`ec2-training`\n" }, { "path": "devflows/training/parallelcluster/parallelcluster-training.rst", "content": ".. _parallelcluster-training:\n\nTrain your model on ParallelCluster\n===================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\nDescription\n------------\n\nThis document explains how to use AWS ParallelCluster to build HPC compute environment \nthat uses Trn1 compute nodes to run your distributed ML training job. Once the nodes are \nlaunched, we will run a training task to confirm that the nodes are working, and use \nslurm commands to check the job status. In this tutorial, we will use AWS `pcluster` command\nto run a yaml file in order to generate the cluster. As an example, we are going to launch\nmultiple Trn1.32xl nodes in our cluster.\n\nWe are going to set up our ParallelCluster infrastructure as below:\n\n.. image:: /images/vpc-setup.png\n\nAs shown in the figure above, inside a VPC, there are two subnets, a public and a private\nones. Head Node resides in the public subnet, while the compute fleet (in this case, trn1\ninstances) are in the private subnet. A Network Address Translation (NAT) gateway is also \nneeded in order for nodes in the private subnet to connect to clients outside the VPC. In \nthe next section, we are going to describe how to set up all the necessary infrastructure \nfor trn1 ParallelCluster.\n\n\nSetup environment\n-----------------\n\n1. Install prerequisite infrastructure:\n\nFollow `these setup `_ instructions to install VPC and all the necessary components for ParallelCluster. \n\n2. Install AWS ParallelCluster in a virtual environment (recommended)\n\nFollow `https://docs.aws.amazon.com/parallelcluster/latest/ug/install-v3-virtual-environment.html`\n\n3. Create and launch ParallelCluster \n\nFollow `these creating cluster `_ instructions to launch ParallelCluster in the VPC.\n\n1. Launch training job\n\nFollow `these running training `_ instructions to submit a model training script as a slurm job.\n" }, { "path": "devflows/training/parallelcluster-flows.rst", "content": "Parallel Cluster Flows- Training\n================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /devflows/training/parallelcluster/parallelcluster-training\n\n\n \n\n.. include:: /devflows/training/parallelcluster-flows.txt" }, { "path": "devflows/training/parallelcluster-flows.txt", "content": "* :ref:`parallelcluster-training`\n \n\n" }, { "path": "devflows/training/sagemaker-flows.rst", "content": "Sagemaker Flows- Training\n=========================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /devflows/training/sm-devflow/sm-training-devflow\n\n \n\n.. include:: /devflows/training/sagemaker-flows.txt" }, { "path": "devflows/training/sagemaker-flows.txt", "content": "* :ref:`sm-training-devflow`\n* `AWS Neuron Sagemaker Samples GitHub Repository `_\n\n" }, { "path": "devflows/training/sm-devflow/sm-training-devflow.rst", "content": ".. _sm-training-devflow:\n\nTrain your model on SageMaker\n===================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 3\n\nDescription\n------------\n\nSageMaker Training helps you manage cloud computing resources in Amazon EC2, data storage services\nsuch as S3, EFS, and FSx, and security management services such as IAM and VPC. SageMaker Training \nprovides you a complete end-to-end experience of training classical ML and state-of-the-art DL models. \n\nYou can use SageMaker to train models using Trn1 instances (ml.trn1 instance types). \nIn this developer flow, you provision a SageMaker Notebook instance or SageMaker Studio to train \nyour model using the `SageMaker Python SDK `_.\n\nThe Amazon SageMaker Python SDK lets you launch training jobs in just a few lines of code with ease. \nAs shown in the below diagram Amazon SageMaker launches Trn1 instances, copies both data and code \nonto the instance. It then runs the training script to generate model artifacts. The trained model \nartifacts are then uploaded to S3 and finally SageMaker will terminate the provisioned instances. \nIn order to speed up the training process for successive runs you can copy the `Neuron Persistent Cache\n`_\nto S3 and then copied by future training jobs as they will leverage the cached artifacts. \n(See `Hugging Face fine tuning BERT base model on Amazon SageMaker Tutorial \n`_\nfor an example on how to reuse the compiled cache.)\n\n.. image:: /images/trn1-on-sm-dev-flow.png\n\n\nSetup environment\n-----------------\n\n1. Create an Amazon SageMaker Notebook Instance\n\n Follow the instructions in `Get Started with Notebook Instances \n `_ or \n `Use Amazon SageMaker Studio Notebooks `_.\n The Notebook instance provides the required Python SDK for training models with Amazon SageMaker.\n Please make sure SageMaker Python SDK version is 2.116.0 or later.\n\n2. Train a model using the Amazon SageMaker SDK\n\n Follow the instructions in `Distributed Training with PyTorch Neuron on Trn1 instances\n `_.\n You’ll be able to follow the `Hugging Face fine tuning BERT base model on Amazon SageMaker Tutorial\n `_.\n\n .. note::\n SageMaker support for EC2 Trn1 instance is currently available only for PyTorch Estimator. \n HuggingFace Estimator will be available in future release.\n" }, { "path": "dlami/index.rst", "content": ".. meta::\n :description: Neuron Deep Learning AMIs (DLAMIs) are pre-configured Amazon Machine Images with the Neuron SDK for easy deployment on AWS Inferentia and Trainium instances.\n :keywords: Neuron DLAMI, Deep Learning AMI, AWS Neuron SDK, Inferentia, Trainium, PyTorch, JAX, TensorFlow, vLLM, SSM Parameters\n :date-modified: 01/22/2026\n\n.. _neuron-dlami-overview:\n.. _setup-ubuntu22-multi-framework-dlami:\n.. _setup-ubuntu24-multi-framework-dlami:\n\nNeuron DLAMI User Guide\n=======================\n\nThis guide helps you select, configure, and deploy AWS Neuron Deep Learning AMIs (DLAMIs) for running machine learning workloads on AWS Inferentia and Trainium instances. Learn about the different DLAMI types available, pre-installed virtual environments for popular ML frameworks like PyTorch and JAX, and how to automate DLAMI deployment.\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nWhat are Neuron DLAMIs?\n------------------------\n\nNeuron Deep Learning AMIs (DLAMIs) are pre-configured Amazon Machine Images that provide the easiest way to get started with the AWS Neuron SDK. Each DLAMI comes with Neuron drivers, frameworks, and libraries pre-installed, enabling you to quickly launch and run deep learning workloads on AWS Inferentia and Trainium instances without manual setup.\n\nNeuron currently supports three types of DLAMIs to meet different deployment needs:\n\n* **Multi-Framework DLAMIs**: Support multiple ML frameworks (PyTorch, JAX, vLLM) with separate virtual environments for each\n* **Single Framework DLAMIs**: Optimized for a specific framework version with focused virtual environments\n* **Base DLAMIs**: Include only Neuron drivers, EFA, and tools - ideal for containerized applications and custom builds\n\nAll Neuron DLAMIs support automated discovery through AWS Systems Manager (SSM) parameters, making them easy to integrate into cloud automation workflows and infrastructure-as-code deployments.\n\n.. note::\n Starting with version 2.26.1, Neuron DLAMIs no longer support ``Inf1`` instance types due to an incompatibility with the Neuron driver. \n If you'd like to run ``Inf1`` workloads, use previous DLAMIs released up to SDK version 2.26.\n\n----\n\nNeuron Multi Framework DLAMI\n----------------------------\n\nNeuron Multi-Framework DLAMIs provide the most comprehensive environment, supporting multiple ML frameworks and libraries in isolated virtual environments. Each DLAMI is pre-installed with Neuron drivers and supports all current Neuron instance types (Inf2, Trn1, Trn1n, Trn2, Trn3). This is the recommended option for teams working with multiple frameworks or exploring different ML libraries.\n\n.. note::\n Starting with version 2.27.1, AL2023 DLAMIs no longer support ``PyTorch 2.9+`` due to an incompatibility issue with the default GLIB.c installed on AL2023.\n PyTorch requires GLIB.c 2.35+ and upgrading the version within AL2023 can break other system dependencies. This is the error message:\n \n ``ImportError: /lib64/libm.so.6: version `GLIBC_2.35' not found``\n\n Since the latest vLLM version depends on PyTorch 2.9, we have also removed that environment from the DLAMI.\n \n For a workaround, use the latest Ubuntu-based AMIs instead.\n\n\nMulti Framework DLAMIs supported\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Operating System\n - Neuron Instances Supported\n - DLAMI Name\n\n * - Ubuntu 24.04\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning AMI Neuron (Ubuntu 24.04)\n\n.. _neuron-dlami-multifw-venvs:\n\n\nVirtual Environments pre-installed\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Neuron Framework/Libraries supported\n - Virtual Environment\n\n * - PyTorch 2.9 Torch NeuronX, NxD Core (Ubuntu 24.04)\n - /opt/aws_neuronx_venv_pytorch_2_9\n\n * - PyTorch 2.9 NxD Training, Torch NeuronX (Ubuntu 24.04)\n - /opt/aws_neuronx_venv_pytorch_2_9_nxd_training\n\n * - PyTorch 2.9 NxD Inference, Torch NeuronX (Ubuntu 24.04)\n - /opt/aws_neuronx_venv_pytorch_2_9_nxd_inference\n\n * - JAX 0.7 NeuronX (Ubuntu 24.04)\n - /opt/aws_neuronx_venv_jax_0_7\n\n * - vLLM 0.16.0 NxD Inference, Torch NeuronX (Ubuntu 24.04)\n - /opt/aws_neuronx_venv_pytorch_inference_vllm_0_16\n\n\nWe have included a setup script that installs required dependencies for the package within the PyTorch 2.9 NxD Training virtual environment. To run this script,\nactivate the virtual environment and run ``setup_nxdt.sh`` and this will run :ref:`the setup steps here `.\n\nYou can easily get started with the multi-framework DLAMI through AWS console by following this :doc:`setup guide `. If you are looking to \nuse the Neuron DLAMI in your cloud automation flows, Neuron also supports :ref:`SSM parameters ` to easily retrieve the latest DLAMI id.\n\n----\n\nNeuron Single Framework DLAMI\n-----------------------------\n\nNeuron Single Framework DLAMIs are optimized for specific framework versions, providing a streamlined environment when you know exactly which framework you'll be using. Each DLAMI is pre-installed with Neuron drivers and supports all Neuron instance types. These DLAMIs are ideal for production deployments where you want a focused, framework-specific environment. \n\n\nSingle Framework DLAMIs supported\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Framework\n - Operating System\n - Neuron Instances Supported\n - DLAMI Name\n\n * - PyTorch 2.9\n - Ubuntu 24.04\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning AMI Neuron PyTorch 2.9 (Ubuntu 24.04)\n\n * - JAX 0.7\n - Amazon Linux 2023\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning AMI Neuron JAX 0.7 (Amazon Linux 2023)\n\n * - JAX 0.7\n - Ubuntu 24.04\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning AMI Neuron JAX 0.7 (Ubuntu 24.04)\n\n * - vLLM 0.16.0\n - Ubuntu 24.04\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning AMI Neuron PyTorch Inference vLLM 0.16 (Ubuntu 24.04)\n\n\nVirtual Environments pre-installed\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - DLAMI Name\n - Neuron Libraries supported\n - Virtual Environment\n \n * - Deep Learning AMI Neuron PyTorch 2.9 (Ubuntu 24.04) \n - PyTorch 2.9 Torch NeuronX, NxD Core\n - /opt/aws_neuronx_venv_pytorch_2_9\n\n * - Deep Learning AMI Neuron PyTorch 2.9 (Ubuntu 24.04) \n - PyTorch 2.9 NxD Training, Torch NeuronX\n - /opt/aws_neuronx_venv_pytorch_2_9_nxd_training\n\n * - Deep Learning AMI Neuron PyTorch 2.9 (Ubuntu 24.04) \n - PyTorch 2.9 NxD Inference, Torch NeuronX\n - /opt/aws_neuronx_venv_pytorch_2_9_nxd_inference\n\n * - Deep Learning AMI Neuron JAX 0.7 (Ubuntu 24.04, Amazon Linux 2023) \n - JAX NeuronX 0.7\n - /opt/aws_neuronx_venv_jax_0_7\n\n * - Deep Learning AMI Neuron PyTorch Inference vLLM 0.16 (Ubuntu 24.04) \n - vLLM NeuronX 0.16.0\n - /opt/aws_neuronx_venv_pytorch_inference_vllm_0_16\n\n\nGet started with the single framework DLAMI through AWS console by following one of the corresponding setup guides. If you want to\nuse the Neuron DLAMI in your cloud automation flows, Neuron also supports :ref:`SSM parameters ` to retrieve the latest DLAMI id.\n\n----\n\nNeuron Base DLAMI\n-----------------\n\nNeuron Base DLAMIs provide a minimal foundation with only the essential components: Neuron driver, EFA (Elastic Fabric Adapter), and Neuron tools. These DLAMIs are designed for advanced users who want to build custom environments, create containerized applications, or have specific framework version requirements not covered by the pre-configured DLAMIs.\n\n\nBase DLAMIs supported\n^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - Operating System\n - Neuron Instances Supported\n - DLAMI Name\n\n * - Amazon Linux 2023\n - Inf2, Trn1, Trn1n, Trn2, Trn3 \n - Deep Learning Base Neuron AMI (Amazon Linux 2023)\n\n * - Ubuntu 24.04\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning Base Neuron AMI (Ubuntu 24.04)\n\n * - Ubuntu 22.04\n - Inf2, Trn1, Trn1n, Trn2, Trn3\n - Deep Learning Base Neuron AMI (Ubuntu 22.04)\n\n\n.. _ssm-parameter-neuron-dlami:\n\n----\n\nUsing SSM Parameters for Cloud Automation\n------------------------------------------\n\nNeuron DLAMIs support AWS Systems Manager (SSM) parameters for automated DLAMI discovery and deployment. This enables you to always use the latest Neuron SDK release in your infrastructure-as-code templates, CI/CD pipelines, and auto-scaling configurations without hardcoding AMI IDs.\n\nSSM parameters provide several key benefits:\n\n* **Always up-to-date**: Automatically reference the latest DLAMI with the newest Neuron SDK release\n* **Infrastructure-as-code friendly**: Use in CloudFormation, Terraform, and other IaC tools\n* **Auto Scaling integration**: Update Auto Scaling groups without modifying launch templates\n* **Multi-region support**: Available across all AWS regions where Neuron instances are supported\n\nCurrently, SSM parameters support finding the latest DLAMI ID for each DLAMI type. Support for finding specific Neuron SDK version DLAMIs will be added in future releases.\n\n\nFinding specific DLAMI image id with the latest neuron release\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nYou can find the DLAMI that supports latest Neuron SDK by using the SSM get-parameter.\n\n\n.. code-block::\n\n aws ssm get-parameter \\\n --region us-east-1 \\\n --name /latest/image_id \\\n --query \"Parameter.Value\" \\\n --output text\n\n\nThe SSM parameter prefix for each currently supported DLAMI can be seen below. To discover SSM parameters for older or end-of-life DLAMIs, you can filter by framework, version, or operating system using the path structure ``/aws/service/neuron/dlami/-/``:\n\n.. code-block::\n\n # List all Neuron DLAMI SSM parameters\n aws ssm get-parameters-by-path --region us-east-1 --path /aws/service/neuron --recursive\n\n # Filter by framework (e.g., all PyTorch 2.8 DLAMIs)\n aws ssm get-parameters-by-path --region us-east-1 --path /aws/service/neuron/dlami/pytorch-2.8 --recursive\n\n # Filter by framework and OS\n aws ssm get-parameters-by-path --region us-east-1 --path /aws/service/neuron/dlami/pytorch-2.8/ubuntu-22.04 --recursive\n\n\nSSM Parameter Prefix\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n.. list-table::\n :widths: 20 39\n :header-rows: 1\n :align: left\n :class: table-smaller-font-size\n\n * - AMI Name\n - SSM parameter Prefix\n\n * - Deep Learning AMI Neuron (Ubuntu 24.04)\n - /aws/service/neuron/dlami/multi-framework/ubuntu-24.04\n\n * - Deep Learning AMI Neuron PyTorch 2.9 (Ubuntu 24.04)\n - /aws/service/neuron/dlami/pytorch-2.9/ubuntu-24.04\n\n * - Deep Learning AMI Neuron JAX 0.7 (Amazon Linux 2023)\n - /aws/service/neuron/dlami/jax-0.7/amazon-linux-2023\n\n * - Deep Learning AMI Neuron JAX 0.7 (Ubuntu 24.04)\n - /aws/service/neuron/dlami/jax-0.7/ubuntu-24.04\n\n * - Deep Learning AMI Neuron PyTorch Inference vLLM 0.16 (Ubuntu 24.04)\n - /aws/service/neuron/dlami/pytorch-inference-vllm-0.16/ubuntu-24.04\n\n * - Deep Learning Base Neuron AMI (Amazon Linux 2023)\n - /aws/service/neuron/dlami/base/amazon-linux-2023\n\n * - Deep Learning Base Neuron AMI (Ubuntu 24.04)\n - /aws/service/neuron/dlami/base/ubuntu-24.04\n\n * - Deep Learning Base Neuron AMI (Ubuntu 22.04)\n - /aws/service/neuron/dlami/base/ubuntu-22.04\n\n\nFor example to find the latest DLAMI id for Multi-Framework DLAMI (Ubuntu 24.04) you can use the following:\n\n.. code-block::\n\n aws ssm get-parameter \\\n --region us-east-1 \\\n --name /aws/service/neuron/dlami/multi-framework/ubuntu-24.04/latest/image_id \\\n --query \"Parameter.Value\" \\\n --output text\n\n\nYou can find all available parameters supported in Neuron DLAMis via CLI\n\n.. code-block::\n\n aws ssm get-parameters-by-path \\\n --region us-east-1 \\\n --path /aws/service/neuron \\\n --recursive\n\n\nYou can also view the SSM parameters supported in Neuron through AWS parameter store by selecting the \"Neuron\" service.\n\n\n\nUse SSM Parameter to launch instance directly via CLI\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nYou can use the AWS CLI to resolve the latest DLAMI ID and launch an instance in a single command. This is particularly useful for scripting and automation workflows.\n\nBelow is an example of launching an Inf2 instance using the TensorFlow 2.10 single-framework DLAMI: \n\n\n.. code-block::\n\n aws ec2 run-instances \\\n --region us-east-1 \\\n --image-id resolve:ssm:/aws/service/neuron/dlami/tensorflow-2.10/ubuntu-22.04/latest/image_id \\\n --count 1 \\\n --instance-type inf2.48xlarge \\\n --key-name \\\n --security-groups \n\n\n\nUse SSM alias in EC2 launch templates\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nSSM Parameters can be used directly in EC2 launch templates, enabling your Auto Scaling groups to automatically use the latest AMI IDs without requiring updates to launch templates or creating new versions each time an AMI ID changes. This significantly simplifies AMI lifecycle management in production environments.\n\nFor more information, see: https://docs.aws.amazon.com/autoscaling/ec2/userguide/using-systems-manager-parameters.html\n\n----\n\nOther Resources\n---------------\n\nLearn more about AWS Deep Learning AMIs and Systems Manager:\n\n* `AWS Deep Learning AMI Developer Guide `_\n* `AWS DLAMI Release Notes `_\n* `AWS Systems Manager Parameter Store `_\n* :doc:`Neuron DLAMI Release Notes `\n" }, { "path": "frameworks/index.rst", "content": ".. meta::\n :description: ML Framework support on AWS Neuron SDK - PyTorch and JAX integration for high-performance machine learning on AWS Inferentia and Trainium.\n :date-modified: 2026-03-12\n :keywords: AWS Neuron, machine learning\n\n.. _frameworks-neuron-sdk:\n\nML framework support on AWS Neuron SDK\n=======================================\n\nAWS Neuron provides integration with popular machine learning frameworks, enabling you to accelerate your existing models on AWS Inferentia and Trainium with minimal code changes. Choose from our comprehensive framework support to optimize your inference and training workloads.\n\nFrameworks\n-----------\n\n.. grid:: 2\n :gutter: 2\n\n .. grid-item-card:: PyTorch on AWS Neuron\n :link: torch/index\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Complete PyTorch integration for both inference and training on all Neuron hardware.\n\n * **TorchNeuron Native** - Native PyTorch backend with eager execution and ``torch.compile``\n * **PyTorch NeuronX (torch-neuronx)** - ``Inf2``, ``Trn1``, ``Trn2`` (inference & training)\n * See: :doc:`/frameworks/torch/pytorch-native-overview`\n\n .. grid-item-card:: JAX on AWS Neuron\n :link: jax/index\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n **Beta release**\n\n Experimental JAX support with Neuron Kernel Interface (NKI) integration.\n\n * **JAX NeuronX** - Neuron hardware support\n * Research and development focus\n * **Status**: Beta - active\n\n.. note::\n\n Looking for TensorFlow, MXNet, or torch-neuron (Inf1) documentation? These frameworks\n have been archived. See :doc:`/archive/index` for legacy framework documentation.\n\nHardware compatibility matrix\n-----------------------------\n\n.. list-table::\n :header-rows: 1\n :class: compatibility-matrix\n\n * - Framework\n - Inf2\n - Trn1/Trn1n\n - Trn2\n - Inference\n - Training\n * - **torch-neuronx**\n - ✅\n - ✅\n - ✅\n - ✅\n - ✅\n * - **JAX NeuronX**\n - ✅\n - ✅\n - N/A\n - ✅\n - N/A\n" }, { "path": "frameworks/jax/api-reference-guide/index.rst", "content": ".. _jax-neuronx-api-reference-guide:\n\n\n.. meta::\n :description: API Reference Guide for JAX Neuronx - AWS Neuron SDK documentation\n :keywords: API reference, AWS Neuron, JAX, JAX NeuronX\n :date-modified: 2026-03-13\n\n\nAPI Reference Guide for JAX Neuronx\n====================================================\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n /frameworks/jax/api-reference-guide/neuron-envvars\n\n* :ref:`jax-neuronx-envvars`\n" }, { "path": "frameworks/jax/api-reference-guide/neuron-envvars.rst", "content": ".. _jax-neuronx-envvars:\n\n\n.. meta::\n :description: JAX NeuronX Environment Variables - AWS Neuron SDK documentation\n :keywords: API reference, AWS Neuron, JAX, JAX NeuronX\n :date-modified: 2026-03-13\n\n\nJAX NeuronX Environment Variables\n======================================\n\nEnvironment variables allow modifications to JAX NeuronX behavior\nwithout requiring code change to user script. It is recommended to set\nthem in code or just before invoking the python process, such as\n``NEURON_RT_VISIBLE_CORES=8 python3 \n\n\nGet Started with PyTorch Neuron (\"torch-neuron\") on Ubuntu 20\n==============================================================" }, { "path": "setup/torch-neuron.rst", "content": ".. _setup-torch-neuron:\n\nPyTorch Neuron (``torch-neuron``) Setup\n=======================================\n\n.. warning::\n\n ``torch-neuron`` is for Inf1 instances only (legacy NeuronCore v1).\n For new projects, use Inf2, Trn1, Trn2, or Trn3 with ``torch-neuronx``.\n See :doc:`/setup/pytorch/index` for current setup.\n\nFor Inf1 setup instructions, see :doc:`/setup/legacy-inf1/pytorch`.\n" }, { "path": "setup/torch-neuronx.rst", "content": ".. _setup-torch-neuronx:\n\n.. meta::\n :description: Install PyTorch NeuronX (torch-neuronx) on AWS Trainium and Inferentia instances using DLAMI, DLC, or manual pip installation\n :keywords: pytorch, neuron, torch-neuronx, installation, setup, trainium, inferentia, trn1, trn2, trn3, inf2, DLAMI, pip\n :date-modified: 2026-03-30\n\nPyTorch Neuron (``torch-neuronx``) Setup \n========================================\n\nInstall PyTorch with Neuron support for training and inference on Inf2, Trn1, Trn2, and Trn3 instances. Choose from a pre-configured DLAMI, a Docker container, or a manual pip installation.\n\nFor the full setup guide with all options, see :doc:`Install PyTorch for Neuron `.\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: 🚀 DLAMI Installation\n :link: /setup/pytorch/dlami\n :link-type: doc\n :class-card: sd-border-2\n\n Pre-configured environment with all dependencies. Recommended for most users.\n\n .. grid-item-card:: 🚀 Multi-Framework DLAMI\n :link: /setup/multiframework-dlami\n :link-type: doc\n :class-card: sd-border-2\n\n Pre-configured AMI with PyTorch, JAX, and vLLM virtual environments ready to use.\n\n .. grid-item-card:: � Deep Learning Container\n :link: /setup/pytorch/dlc\n :link-type: doc\n :class-card: sd-border-2\n\n Pre-configured Docker images from AWS ECR for containerized deployments.\n\n .. grid-item-card:: 🔧 Manual Installation\n :link: /setup/pytorch/manual\n :link-type: doc\n :class-card: sd-border-2\n\n Install on bare OS AMIs or existing systems with full control over dependencies.\n\n .. grid-item-card:: Rocky Linux 9\n :link: setup-rocky-linux-9\n :link-type: ref\n :class-card: sd-border-2\n\n Install on Rocky Linux 9 using the Rocky-9-EC2-Base AMI.\n" }, { "path": "setup/troubleshooting.rst", "content": ".. meta::\n :description: Troubleshooting guide for AWS Neuron SDK installation issues\n :keywords: neuron, troubleshooting, installation, errors, debugging\n :content-type: troubleshooting\n :date-modified: 2026-03-03\n\nInstallation Troubleshooting\n=============================\n\nCommon issues and solutions for Neuron SDK installation.\n\nModule Import Errors\n--------------------\n\nModuleNotFoundError: No module named 'torch_neuronx'\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: Python cannot find torch_neuronx module after installation.\n\n**Causes**:\n\n- Virtual environment not activated\n- Wrong Python version\n- Installation failed silently\n- Multiple Python installations\n\n**Solutions**:\n\n1. **Verify virtual environment**:\n \n .. code-block:: bash\n \n which python\n # Should show virtual environment path, not system Python\n\n2. **Check Python version**:\n \n .. code-block:: bash\n \n python --version\n # Should be 3.10, 3.11, or 3.12\n\n3. **Reinstall torch-neuronx**:\n \n .. code-block:: bash\n \n pip install --force-reinstall torch-neuronx --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\n4. **Verify installation**:\n \n .. code-block:: bash\n \n pip list | grep neuron\n\nImportError: cannot import name 'neuron' from 'torch'\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: Import error when trying to use Neuron features.\n\n**Cause**: Using PyTorch/XLA syntax with Native PyTorch backend.\n\n**Solution**: Update code to use Native PyTorch syntax:\n\n.. code-block:: python\n \n # Old (PyTorch/XLA)\n import torch_xla.core.xla_model as xm\n device = xm.xla_device()\n \n # New (Native PyTorch)\n import torch\n device = torch.device('neuron')\n\nSee :doc:`/frameworks/torch/index` for complete migration guide.\n\nDevice and Runtime Errors\n--------------------------\n\nNo Neuron devices found\n~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: ``neuron-ls`` shows no devices or returns error.\n\n**Causes**:\n\n- Wrong instance type\n- Neuron driver not loaded\n- Runtime not started\n\n**Solutions**:\n\n1. **Verify instance type**:\n \n .. code-block:: bash\n \n curl http://169.254.169.254/latest/meta-data/instance-type\n # Should show inf2.*, trn1.*, trn2.*, trn3.*, or inf1.*\n\n2. **Check Neuron driver**:\n \n .. code-block:: bash\n \n lsmod | grep neuron\n # Should show neuron driver loaded\n\n3. **Install/reload driver**:\n \n .. code-block:: bash\n \n # Ubuntu/Debian\n sudo apt-get install -y aws-neuronx-dkms\n \n # Amazon Linux\n sudo yum install -y aws-neuronx-dkms\n\n4. **Restart runtime**:\n \n .. code-block:: bash\n \n sudo systemctl restart neuron-monitor\n neuron-ls\n\nRuntimeError: Neuron runtime initialization failed\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: Runtime fails to initialize when running models.\n\n**Causes**:\n\n- Insufficient permissions\n- Runtime version mismatch\n- Corrupted runtime state\n\n**Solutions**:\n\n1. **Check runtime status**:\n \n .. code-block:: bash\n \n sudo systemctl status neuron-monitor\n\n2. **Verify permissions**:\n \n .. code-block:: bash\n \n ls -l /dev/neuron*\n # Should be accessible by current user\n\n3. **Reinstall runtime**:\n \n .. code-block:: bash\n \n sudo apt-get install --reinstall aws-neuronx-runtime-lib\n\nVersion Compatibility Issues\n-----------------------------\n\nCompiler version mismatch\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: Error about incompatible compiler version.\n\n**Cause**: neuronx-cc version incompatible with framework version.\n\n**Solution**: Install compatible versions:\n\n.. code-block:: bash\n \n # For PyTorch 2.9\n pip install neuronx-cc==2.15.* --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\nSee :doc:`/release-notes/index` for version compatibility matrix.\n\nPackage dependency conflicts\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: pip reports conflicting dependencies.\n\n**Solution**: Use fresh virtual environment:\n\n.. code-block:: bash\n \n python3 -m venv ~/fresh_neuron_venv\n source ~/fresh_neuron_venv/bin/activate\n pip install -U pip\n # Install packages in correct order\n pip install torch==2.9.0\n pip install torch-neuronx neuronx-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com\n\nNetwork and Repository Issues\n------------------------------\n\nCannot connect to Neuron repository\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Symptoms**: apt-get or pip cannot reach Neuron repositories.\n\n**Solutions**:\n\n1. **Verify network connectivity**:\n \n .. code-block:: bash\n \n curl -I https://apt.repos.neuron.amazonaws.com\n curl -I https://pip.repos.neuron.amazonaws.com\n\n2. **Check proxy settings** (if behind corporate proxy):\n \n .. code-block:: bash\n \n export https_proxy=http://proxy.example.com:8080\n export http_proxy=http://proxy.example.com:8080\n\n3. **Use alternative index URL**:\n \n .. code-block:: bash\n \n pip install torch-neuronx --index-url=https://pip.repos.neuron.amazonaws.com\n\nGPG key expired\n~~~~~~~~~~~~~~~\n\n**Symptoms**: \"EXPKEYSIG\" error during apt-get update.\n\n**Solution**:\n\n.. code-block:: bash\n \n wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -\n sudo apt-get update -y\n\nGetting Help\n------------\n\nIf issues persist:\n\n1. **Check release notes**: :doc:`/release-notes/index`\n2. **Review documentation**: :doc:`/frameworks/torch/index`\n3. **GitHub Issues**: `aws-neuron-sdk/aws-neuron-sdk `_\n4. **AWS Support**: Open support case if you have AWS Support plan\n\nDiagnostic Information\n----------------------\n\nWhen reporting issues, include:\n\n.. code-block:: bash\n \n # System information\n uname -a\n cat /etc/os-release\n \n # Instance type\n curl http://169.254.169.254/latest/meta-data/instance-type\n \n # Neuron devices\n neuron-ls\n \n # Package versions\n pip list | grep -E \"(torch|neuron)\"\n \n # Driver status\n lsmod | grep neuron\n sudo systemctl status neuron-monitor\n" }, { "path": "src/benchmark/helper_scripts/llmperf_dp.patch", "content": "diff --git a/src/llmperf/ray_clients/openai_chat_completions_client.py b/src/llmperf/ray_clients/openai_chat_completions_client.py\nindex f2e0a91..74c4027 100644\n--- a/src/llmperf/ray_clients/openai_chat_completions_client.py\n+++ b/src/llmperf/ray_clients/openai_chat_completions_client.py\n@@ -1,5 +1,6 @@\n import json\n import os\n+import random\n import time\n from typing import Any, Dict\n \n@@ -14,6 +15,9 @@ from llmperf import common_metrics\n @ray.remote\n class OpenAIChatCompletionsClient(LLMClient):\n \"\"\"Client for OpenAI Chat Completions API.\"\"\"\n+ def __init__(self):\n+ self.addr_id = 0\n+ self.addr_select_strategy = 'round-robin'\n \n def llm_request(self, request_config: RequestConfig) -> Dict[str, Any]:\n prompt = request_config.prompt\n@@ -50,6 +54,13 @@ class OpenAIChatCompletionsClient(LLMClient):\n address = os.environ.get(\"OPENAI_API_BASE\")\n if not address:\n raise ValueError(\"the environment variable OPENAI_API_BASE must be set.\")\n+ # if several addresses of model server exist, select one for each request (1) randomly or (2) round-robin\n+ address_list = address.split(\";\")\n+ if self.addr_select_strategy == 'round-robin':\n+ address = address_list[self.addr_id]\n+ self.addr_id = (self.addr_id + 1) % len(address_list)\n+ else:\n+ address = random.choice(address_list)\n key = os.environ.get(\"OPENAI_API_KEY\")\n if not key:\n raise ValueError(\"the environment variable OPENAI_API_KEY must be set.\")\n" }, { "path": "src/benchmark/helper_scripts/llmperf_reasoning.patch", "content": "diff --git a/src/llmperf/ray_clients/openai_chat_completions_client.py b/src/llmperf/ray_clients/openai_chat_completions_client.py\nindex aeb5fbf..f1b4473 100644\n--- a/src/llmperf/ray_clients/openai_chat_completions_client.py\n+++ b/src/llmperf/ray_clients/openai_chat_completions_client.py\n@@ -100,7 +100,7 @@ class OpenAIChatCompletionsClient(LLMClient):\n raise RuntimeError(data[\"error\"][\"message\"])\n\n delta = data[\"choices\"][0][\"delta\"]\n- if delta.get(\"content\", None):\n+ if delta.get(\"content\", None) or delta.get(\"reasoning_content\", None):\n if not ttft:\n ttft = time.monotonic() - start_time\n # time_to_next_token.append(ttft)\n@@ -109,7 +109,11 @@ class OpenAIChatCompletionsClient(LLMClient):\n time.monotonic() - most_recent_received_token_time\n )\n most_recent_received_token_time = time.monotonic()\n- generated_text += delta[\"content\"]\n+ if \"reasoning_content\" in delta and delta[\"reasoning_content\"]:\n+ chunk_content = delta[\"reasoning_content\"]\n+ else:\n+ chunk_content = delta[\"content\"]\n+ generated_text += chunk_content\n\n total_request_time = time.monotonic() - start_time\n output_throughput = tokens_received / total_request_time" }, { "path": "src/benchmark/helper_scripts/neuron_perf.patch", "content": "diff --git a/src/llmperf/ray_clients/openai_chat_completions_client.py b/src/llmperf/ray_clients/openai_chat_completions_client.py\nindex f2e0a91..644d5a6 100644\n--- a/src/llmperf/ray_clients/openai_chat_completions_client.py\n+++ b/src/llmperf/ray_clients/openai_chat_completions_client.py\n@@ -92,7 +92,7 @@ class OpenAIChatCompletionsClient(LLMClient):\n if delta.get(\"content\", None):\n if not ttft:\n ttft = time.monotonic() - start_time\n- time_to_next_token.append(ttft)\n+ # time_to_next_token.append(ttft)\n else:\n time_to_next_token.append(\n time.monotonic() - most_recent_received_token_time\ndiff --git a/token_benchmark_ray.py b/token_benchmark_ray.py\nindex 63216b1..11e0116 100644\n--- a/token_benchmark_ray.py\n+++ b/token_benchmark_ray.py\n@@ -32,6 +32,7 @@ def get_token_throughput_latencies(\n stddev_input_tokens: int,\n mean_output_tokens: int,\n stddev_output_tokens: int,\n+ tokenizer: str,\n additional_sampling_params: Optional[Dict[str, Any]] = None,\n num_concurrent_requests: int = 1,\n max_num_completed_requests: int = 500,\n@@ -60,10 +61,8 @@ def get_token_throughput_latencies(\n \"\"\"\n random.seed(11111)\n \n- tokenizer = LlamaTokenizerFast.from_pretrained(\n- \"hf-internal-testing/llama-tokenizer\"\n- )\n- get_token_length = lambda text: len(tokenizer.encode(text))\n+ hf_tokenizer = LlamaTokenizerFast.from_pretrained(tokenizer)\n+ get_token_length = lambda text: len(hf_tokenizer.encode(text))\n \n if not additional_sampling_params:\n additional_sampling_params = {}\n@@ -84,7 +83,7 @@ def get_token_throughput_latencies(\n prompt_tokens_mean=mean_input_tokens,\n prompt_tokens_stddev=stddev_input_tokens,\n expect_output_tokens=num_output_tokens,\n- tokenizer=tokenizer\n+ tokenizer=hf_tokenizer\n ))\n start_time = time.monotonic()\n pbar = tqdm(total=max_num_completed_requests)\n@@ -118,7 +117,7 @@ def get_token_throughput_latencies(\n with completed_requests_lock:\n if num_completed_requests < max_num_completed_requests:\n if num_output_tokens:\n- request_metrics[common_metrics.INTER_TOKEN_LAT] /= request_metrics[common_metrics.NUM_OUTPUT_TOKENS]\n+ request_metrics[common_metrics.INTER_TOKEN_LAT] /= num_output_tokens - 1\n else:\n request_metrics[common_metrics.INTER_TOKEN_LAT] = 0\n request_metrics[common_metrics.NUM_OUTPUT_TOKENS] = num_output_tokens\n@@ -155,7 +154,7 @@ def get_token_throughput_latencies(\n with completed_requests_lock:\n if num_completed_requests < max_num_completed_requests:\n if num_output_tokens:\n- request_metrics[common_metrics.INTER_TOKEN_LAT] /= num_output_tokens\n+ request_metrics[common_metrics.INTER_TOKEN_LAT] /= num_output_tokens - 1\n else:\n request_metrics[common_metrics.INTER_TOKEN_LAT] = 0\n request_metrics[common_metrics.NUM_OUTPUT_TOKENS] = num_output_tokens\n@@ -292,6 +291,7 @@ def run_token_benchmark(\n additional_sampling_params: str,\n results_dir: str,\n user_metadata: Dict[str, Any],\n+ tokenizer: str,\n ):\n \"\"\"\n Args:\n@@ -327,6 +327,7 @@ def run_token_benchmark(\n stddev_output_tokens=stddev_output_tokens,\n num_concurrent_requests=num_concurrent_requests,\n additional_sampling_params=json.loads(additional_sampling_params),\n+ tokenizer=tokenizer,\n )\n \n if results_dir:\n@@ -462,6 +463,11 @@ args.add_argument(\n \"name=foo,bar=1. These will be added to the metadata field of the results. \"\n ),\n )\n+args.add_argument(\n+ \"--tokenizer\",\n+ type=str,\n+ default=\"hf-internal-testing/llama-tokenizer\",\n+)\n \n if __name__ == \"__main__\":\n env_vars = dict(os.environ)\n@@ -488,4 +494,5 @@ if __name__ == \"__main__\":\n additional_sampling_params=args.additional_sampling_params,\n results_dir=args.results_dir,\n user_metadata=user_metadata,\n+ tokenizer=args.tokenizer,\n )\n" }, { "path": "src/benchmark/tensorflow/distilbert-base-uncased-finetuned-sst-2-english_benchmark.py", "content": "# Add to these lists or change as needed\nmodel_names = [\"distilbert-base-uncased-finetuned-sst-2-english\"]\nsequence_lengths = [128]\nbatch_sizes = [128]\npipeline_sizes = [1]\n\n# Silence an irrelevant warning from transformers library\nimport os\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n\nimport numpy as np\nimport neuronperf as npf\nimport neuronperf.tensorflow\nfrom transformers import AutoTokenizer, TFAutoModelForSequenceClassification\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence = \"I am sorry. I really want to like it, but I just can not stand sushi.\"\n paraphrase = tokenizer.encode_plus(\n sequence,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"np\",\n )\n inputs = {\n \"input_ids\": np.concatenate([paraphrase[\"input_ids\"]] * batch_size, axis=0),\n \"attention_mask\": np.concatenate([paraphrase[\"attention_mask\"]] * batch_size, axis=0),\n }\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Benchmark\n print(\"Benchmarking {}\".format(filename))\n reports = npf.tensorflow.benchmark(filename, inputs)\n\n # View and save results\n print(\"======== {} ========\".format(filename))\n npf.print_reports(reports)\n npf.write_csv(reports)\n npf.write_json(reports)\n" }, { "path": "src/benchmark/tensorflow/distilbert-base-uncased-finetuned-sst-2-english_compile.py", "content": "# Add to these lists or change as needed\nmodel_names = [\"distilbert-base-uncased-finetuned-sst-2-english\"]\nsequence_lengths = [128]\nbatch_sizes = [128]\npipeline_sizes = [1]\n\n# Silence an irrelevant warning from transformers library\nimport os\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n\nimport numpy as np\nimport neuronperf as npf\nimport neuronperf.tensorflow\nfrom transformers import AutoTokenizer, TFAutoModelForSequenceClassification\n\n\ndef get_batch(tokenizer, sequence_length, batch_size):\n sequence = \"I am sorry. I really want to like it, but I just can not stand sushi.\"\n paraphrase = tokenizer.encode_plus(\n sequence,\n max_length=sequence_length,\n padding=\"max_length\",\n truncation=True,\n return_tensors=\"np\",\n )\n inputs = {\n \"input_ids\": np.concatenate([paraphrase[\"input_ids\"]] * batch_size, axis=0),\n \"attention_mask\": np.concatenate([paraphrase[\"attention_mask\"]] * batch_size, axis=0),\n }\n return inputs\n\n\nif __name__ == \"__main__\":\n for model_name in model_names:\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n model = TFAutoModelForSequenceClassification.from_pretrained(model_name, return_dict=False)\n for sequence_length in sequence_lengths:\n inputs = [\n get_batch(tokenizer, sequence_length, batch_size) for batch_size in batch_sizes\n ]\n filename = f\"{model_name}_sl{sequence_length}.json\"\n\n # Compile\n print(\"Compiling {}\".format(filename))\n npf.tensorflow.compile(\n model,\n inputs,\n batch_sizes=batch_sizes,\n pipeline_sizes=pipeline_sizes,\n filename=filename,\n model_name=model_name,\n )\n" }, { "path": "src/examples/mxnet/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/mxnet/data_parallel/benchmark_utils.py", "content": "import math\nfrom collections import Counter\n\nimport numpy as np\n\nclass Results():\n\n def __init__(self, batch_size, num_cores=1):\n self.latency_array = []\n self.end_times = []\n self.start_times = []\n self.batch_size = batch_size\n self.num_cores = num_cores\n\n def add_result(self, latency_array, end_times, start_times):\n self.latency_array.extend(latency_array)\n self.end_times.extend(end_times)\n self.start_times.extend(start_times)\n\n def report(self, f, window_size=1):\n assert(len(self.latency_array) != 0)\n p50_latency = np.percentile(self.latency_array, 50)\n p90_latency = np.percentile(self.latency_array, 90)\n p95_latency = np.percentile(self.latency_array, 95)\n p99_latency = np.percentile(self.latency_array, 99)\n p100_latency = np.percentile(self.latency_array, 100)\n\n\n def get_bucket(start, end):\n bucketed_start = math.floor(start / window_size) * window_size\n bucketed_end = math.ceil(end / window_size) * window_size\n # The check is to make sure that we ignore timestamps that are larger than the window size\n if bucketed_end - bucketed_start == window_size:\n return bucketed_start\n else:\n return None\n \n # Divide the timestamps into different buckets\n bucketed_timestamps = [get_bucket(start, end)\n for start, end in zip(self.start_times, self.end_times)]\n # Count the values in each bucket\n counted_buckets = Counter(\n item for item in bucketed_timestamps if item is not None)\n # Normalize each bucket\n bucket_throughputs = [(key, value / window_size)\n for key, value in sorted(counted_buckets.items())]\n \n busy_throughputs = [value for _, value in bucket_throughputs]\n max_throughput = max(busy_throughputs) * self.batch_size\n avg_throughput = sum(busy_throughputs) * self.batch_size / len(busy_throughputs)\n \n f.write(\"\\n\")\n f.write(\n \"Maximum throughput = {} sentences/sec\\n\".format(int(max_throughput)))\n f.write(\"Average throughput = {} sentences/sec\\n\".format(int(avg_throughput)))\n\n f.write(\"\\n\")\n f.write(\"Latency Percentiles:\\n\")\n f.write(\"===\\n\")\n f.write(\"P50 = {} milliseconds\\n\".format(int(1000*p50_latency)))\n f.write(\"P90 = {} milliseconds\\n\".format(int(1000*p90_latency)))\n f.write(\"P95 = {} milliseconds\\n\".format(int(1000*p95_latency)))\n f.write(\"P99 = {} milliseconds\\n\".format(int(1000*p99_latency)))\n f.write(\"P100 = {} milliseconds\\n\".format(int(1000*p100_latency)))\n f.write(\"\\n\")\n f.write(\"Sanity test:\\n\")\n f.write(\"===\\n\")\n f.write(\"Processed - num batches {}\\n\".format(len(self.latency_array)))\n f.write(\" - batch size {}\\n\".format(self.batch_size))\n f.write(\" - num cores {}\\n\".format(self.num_cores))" }, { "path": "src/examples/mxnet/data_parallel/data_parallel_tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Using Data Parallel Mode with Gluon MXNet\\n\",\n \"\\n\",\n \"In this tutorial, you will compile a Gluon BERT model and run in data-parallel mode to completely utilize the NeuronCores. Here you will benchmark a multi-worker setup and compare it with a single worker.\\n\",\n \"\\n\",\n \"This tutorial is intended only for MXNet-1.8.\\n\",\n \"\\n\",\n \"In this tutorial, we will be using an inf1.2xlarge with the latest AWS Deep Learning AMI (DLAMI). The inf1.2xlarge instance has 1 AWS Inferentia Chip with 4 NeuronCores.\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Setting up your environment\\n\",\n \"\\n\",\n \"To run this tutorial, please make sure you deactivate any existing MXNet conda environments you already using. Install MXNet 1.8 by following the instructions at [MXNet Setup Guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-intro/mxnet-setup/mxnet-install.html#develop-on-aws-ml-accelerator-instance). You would also need to change your kernel to use the correct Python environment setup earlier by clicking Kerenel->Change Kernel->Python (Neuron MXNet)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Install dependencies\\n\",\n \"\\n\",\n \"We have to install gluon-nlp to get the BERT model. Run the following command to install:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!python -m pip install gluonnlp\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Compiling BERT Model\\n\",\n \"\\n\",\n \"Next, we compile the Gluon BERT model and save it. Once the model is compiled, we use the same model across the entire tutorial.\\n\",\n \"In this tutorial, we will be using a BERT model with sequence length 32\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import mxnet as mx\\n\",\n \"import mx_neuron\\n\",\n \"import gluonnlp as nlp\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"BERT_MODEL = 'bert_12_768_12'\\n\",\n \"BERT_DATA = 'book_corpus_wiki_en_uncased'\\n\",\n \"batch_size = 1\\n\",\n \"seq_len = 32\\n\",\n \"num_cores = 1\\n\",\n \"dtype = 'float32'\\n\",\n \"\\n\",\n \"compiled_model_path = '{}.compiled.{}.{}'.format(BERT_MODEL, batch_size, seq_len)\\n\",\n \"\\n\",\n \"model, vocab = nlp.model.get_model(BERT_MODEL,\\n\",\n \" dataset_name=BERT_DATA,\\n\",\n \" use_classifier=False,\\n\",\n \" use_decoder=False, ctx=mx.cpu())\\n\",\n \" \\n\",\n \"# Create sample inputs for compilation\\n\",\n \"words = mx.nd.ones([batch_size, seq_len], name='words', dtype=dtype)\\n\",\n \"valid_len = mx.nd.ones([batch_size,], name='valid_len', dtype=dtype)\\n\",\n \"segments = mx.nd.ones([batch_size, seq_len], name='segments', dtype=dtype)\\n\",\n \"inputs = {'data0': words, 'data1': segments, 'data2': valid_len}\\n\",\n \"\\n\",\n \"# Compiler Args ~~ \\n\",\n \"options = {}\\n\",\n \"embeddingNames = ['bertmodel0_word_embed_embedding0_fwd', 'bertmodel0_token_type_embed_embedding0_fwd', 'bertencoder0_embedding0']\\n\",\n \"options.update({'force_incl_node_names': embeddingNames})\\n\",\n \"options.update({'flags': ['--fp32-cast matmult']}) \\n\",\n \"\\n\",\n \"# Compile and save ~~ \\n\",\n \"model = mx_neuron.compile(model, inputs=inputs, **options)\\n\",\n \"model.export(compiled_model_path)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Data Parallel Mode\\n\",\n \"\\n\",\n \"Data Parallel Mode is a setup in which you launch multiple copies of the same model, such that each model is running independently of the other. In other words, each model has its own resources to run inference. \\n\",\n \"\\n\",\n \"On an inf1.2xlarge instance, we have 4 NeuronCores. Hence, we can launch 4 models such that each model is loaded on a single NeuronCore. This unables us to process 4 request concurrently without linear increase in latency. As a result, the throughput of the system increases when compared to a single model inference. This would also allow us to utilize all the 4 NeuronCores on the instance.\\n\",\n \"\\n\",\n \"Run through the next set of cells to see the difference in throughput as we scale from one model to 4 models running in parallel.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import numpy as np\\n\",\n \"\\n\",\n \"def get_sample_inputs(batch_size, seq_len):\\n\",\n \" words = np.ones([batch_size, seq_len], dtype=np.float32)\\n\",\n \" valid_len = np.ones([batch_size,], dtype=np.float32)\\n\",\n \" segments = np.ones([batch_size, seq_len], dtype=np.float32)\\n\",\n \" inputs = {'data0': words, 'data1': segments, 'data2': valid_len}\\n\",\n \" return inputs\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Next for comparison purposes, we run the setup with 1 worker. To do this, we set the num_cores=1. This would launch only 1 model running on a single NeuronCore. After running the below cell, note down the latency and throughput for the system\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from parallel import NeuronSimpleDataParallel\\n\",\n \"from benchmark_utils import Results\\n\",\n \"import time\\n\",\n \"import functools\\n\",\n \"import os\\n\",\n \"import numpy as np\\n\",\n \"import warnings\\n\",\n \"\\n\",\n \"num_cores = 1\\n\",\n \"batch_size=1\\n\",\n \"\\n\",\n \"# Each worker process should use one core, hence we set\\n\",\n \"# os.environ['NEURON_RT_NUM_CORES'] = \\\"1\\\"\\n\",\n \"os.environ[\\\"NEURON_RT_NUM_CORES\\\"] = \\\"1\\\"\\n\",\n \"\\n\",\n \"#Result aggregation class (code in bert_benchmark_utils.py)\\n\",\n \"results = Results(batch_size, num_cores)\\n\",\n \"def result_handler(output, start, end):\\n\",\n \" elapsed = end - start\\n\",\n \" results.add_result([elapsed], [end], [start])\\n\",\n \"\\n\",\n \"inputs = get_sample_inputs(batch_size, seq_len)\\n\",\n \"parallel_neuron_model = NeuronSimpleDataParallel(compiled_model_path, num_cores, inputs)\\n\",\n \"\\n\",\n \"#Starting the inference threads\\n\",\n \"parallel_neuron_model.start_continuous_inference()\\n\",\n \"\\n\",\n \"# Warm up the cores\\n\",\n \"for _ in range(num_cores*4):\\n\",\n \" parallel_neuron_model.warmup(inputs)\\n\",\n \" \\n\",\n \"# Need to run for high number of iterations to benchmark the models\\n\",\n \"for _ in range(1000):\\n\",\n \" parallel_neuron_model.infer(inputs)\\n\",\n \" # Passing the result_handler as a callback function\\n\",\n \" parallel_neuron_model.add_result(result_handler)\\n\",\n \"\\n\",\n \"# Stop inference \\n\",\n \"parallel_neuron_model.stop()\\n\",\n \"# Since we are using a multi-process execution with a shared queue, some inferences\\n\",\n \"# may still be in execution phase. Hence we need to wait till all the inputs are processed\\n\",\n \"# add_all_results() will collect all the results of requests which are in this state\\n\",\n \"parallel_neuron_model.add_all_results(result_handler)\\n\",\n \"\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"w\\\") as f:\\n\",\n \" results.report(f, window_size=1)\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"r\\\") as f:\\n\",\n \" for line in f:\\n\",\n \" print(line)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now we run the setup with 4 workers. To do this, we set the num_cores=4. This would launch 4 model running each running on individual NeuronCore. All the 4 models are running in individual processes, in other words the models are running in parallel. \\n\",\n \"\\n\",\n \"To feed the models efficiently, we use the producer-consumer setup, in which all processes running a model act as consumers. All consumers are fed using a sharing input queue.\\n\",\n \"\\n\",\n \"Now we run the below setup. You may notice, that the throughput increase by >2x when compared to a single worker setup.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from parallel import NeuronSimpleDataParallel\\n\",\n \"from benchmark_utils import Results\\n\",\n \"import time\\n\",\n \"import functools\\n\",\n \"import os\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"num_cores = 4\\n\",\n \"batch_size=1\\n\",\n \"\\n\",\n \"os.environ[\\\"NEURON_RT_NUM_CORES\\\"] = \\\"1\\\"\\n\",\n \"\\n\",\n \"#Result aggregation class (code in bert_benchmark_utils.py)\\n\",\n \"results = Results(batch_size, num_cores)\\n\",\n \"def result_handler(output, start, end):\\n\",\n \" elapsed = end - start\\n\",\n \" results.add_result([elapsed], [end], [start])\\n\",\n \"\\n\",\n \"inputs = get_sample_inputs(batch_size, seq_len)\\n\",\n \"parallel_neuron_model = NeuronSimpleDataParallel(compiled_model_path, num_cores, inputs)\\n\",\n \"\\n\",\n \"#Starting the inference threads\\n\",\n \"parallel_neuron_model.start_continuous_inference()\\n\",\n \"\\n\",\n \"# Warm up the cores\\n\",\n \"for _ in range(num_cores*4):\\n\",\n \" parallel_neuron_model.warmup(inputs)\\n\",\n \" \\n\",\n \"# Need to run for high number of iterations to benchmark the models\\n\",\n \"for _ in range(5000):\\n\",\n \" parallel_neuron_model.infer(inputs)\\n\",\n \" # Passing the result_handler as a callback function\\n\",\n \" parallel_neuron_model.add_result(result_handler)\\n\",\n \"\\n\",\n \"# Stop inference \\n\",\n \"parallel_neuron_model.stop()\\n\",\n \"# Since we are using a multi-process execution with a shared queue, some inferences\\n\",\n \"# may still be in execution phase. Hence we need to wait till all the inputs are processed\\n\",\n \"# add_all_results() will collect all the results of requests which are in this state\\n\",\n \"parallel_neuron_model.add_all_results(result_handler)\\n\",\n \"\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"w\\\") as f:\\n\",\n \" results.report(f, window_size=1)\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"r\\\") as f:\\n\",\n \" for line in f:\\n\",\n \" print(line)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.9\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/mxnet/data_parallel/parallel.py", "content": "import mxnet as mx\nimport mx_neuron\nimport os\nfrom time import time\nfrom queue import Queue\nfrom multiprocessing import Process, Manager\n\n\ndef consumer(model_file, sample_input, input_queue, result_queue):\n sym, args, aux = mx.model.load_checkpoint(model_file, 0)\n sample_input = {key: mx.nd.array(v) for key, v in sample_input.items()}\n args.update(sample_input)\n model = sym.bind(mx.cpu(), args=args, aux_states=aux, grad_req=\"null\")\n\n while True:\n inputs, input_id = input_queue.get()\n input_queue.task_done()\n # Stop execution if stopping condition is recieved\n if inputs == \"stop\":\n break\n inputs = {key: mx.nd.array(v) for key, v in inputs.items()}\n start = time()\n results = model.forward(**inputs)\n results[0].wait_to_read()\n\n # Make the output iterable - if it is not already a tuple or list\n if not isinstance(results, tuple) or isinstance(results, list):\n results = [results]\n end = time()\n\n if input_id != -1:\n result_queue.put((results, start, end, input_id))\n\n\nclass NeuronSimpleDataParallel:\n def __init__(self, model_file, num_neuron_cores, sample_input):\n self.num_neuron_cores = num_neuron_cores\n self.sample_input = sample_input\n self.model_path = model_file\n # Create shared input queue and output queue\n manager = Manager()\n self.input_queue = manager.Queue(maxsize=num_neuron_cores * 16)\n self.result_queue = manager.Queue(maxsize=num_neuron_cores * 16)\n\n self.processes = [\n Process(\n target=consumer,\n args=(\n self.model_path,\n self.sample_input,\n self.input_queue,\n self.result_queue,\n ),\n )\n for _ in range(num_neuron_cores)\n ]\n self.input_id = 0\n self.input_dict = set()\n\n def start_continuous_inference(self):\n for p in self.processes:\n p.start()\n\n def warmup(self, batch):\n self.input_queue.put((batch, -1))\n\n def infer(self, batch):\n self.input_id += 1\n self.input_dict.add(self.input_id)\n self.input_queue.put((batch, self.input_id))\n\n def stop(self):\n for _ in range(self.num_neuron_cores):\n self.input_queue.put((\"stop\", -1))\n\n def add_result(self, callback_fn):\n if not self.result_queue.empty():\n result, start, end, input_id = self.result_queue.get()\n self.input_dict.remove(input_id)\n self.result_queue.task_done()\n callback_fn(result, start, end)\n\n def add_all_results(self, callback_fn):\n results = []\n while len(self.input_dict):\n self.add_result(callback_fn)\n for p in self.processes:\n p.join()\n" }, { "path": "src/examples/mxnet/mxnet-gluon-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"4dcf9bb1\",\n \"metadata\": {},\n \"source\": [\n \"## MXNet 1.8: Getting Started with Gluon Tutorial\\n\",\n \"\\n\",\n \"In this tutorial you will compile and deploy resnet-50 using the newly supported MXNet 1.8 and Gluon API on an Inf1 instance. This tutorial is only supported with MXNet 1.8.\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on an inf1.6xlarge instance since you will be loading and compiling several large models.\\n\",\n \"\\n\",\n \"To run this tutorial, please make sure you deactivate any existing MXNet conda environments you already using. Install MXNet 1.8 by following the instructions at [MXNet Setup Guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-intro/mxnet-setup/mxnet-install.html#install-neuron-mxnet). You would also need to change your kernel to use the correct Python environment setup earlier by clicking Kerenel->Change Kernel->Python (Neuron MXNet)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"83eb578b\",\n \"metadata\": {},\n \"source\": [\n \"## Compile\\n\",\n \"\\n\",\n \"A trained model must be compiled to Inferentia target before it can run on Inferentia. In this step we compile a pre-trained ResNet50 and export it as a compiled MXNet checkpoint.\\n\",\n \"\\n\",\n \"Compilation will take a few minutes. At the end of compilation, the files resnet-50_compiled-0000.params and resnet-50_compiled-symbol.json will be created in local directory.\\n\",\n \"\\n\",\n \"To check the supported operations for the uncompiled model or information on Neuron subgraphs for the compiled model, please see [Neuron Check Model](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-tools/tutorial-neuron-check-model.html#neuron-check-model).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"88c41e01\",\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import mxnet as mx\\n\",\n \"import mx_neuron as neuron\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"path='http://data.mxnet.io/models/imagenet/'\\n\",\n \"mx.test_utils.download(path+'resnet/50-layers/resnet-50-0000.params')\\n\",\n \"mx.test_utils.download(path+'resnet/50-layers/resnet-50-symbol.json')\\n\",\n \"block = mx.gluon.nn.SymbolBlock.imports('resnet-50-symbol.json',\\\\\\n\",\n \" ['data', 'softmax_label'], 'resnet-50-0000.params', ctx=mx.cpu())\\n\",\n \"\\n\",\n \"block.hybridize()\\n\",\n \"\\n\",\n \"# Compile for Inferentia using Neuron\\n\",\n \"inputs = { \\\"data\\\" : mx.nd.ones([1,3,224,224], name='data', dtype='float32'), 'softmax_label' : mx.nd.ones([1], name='data', dtype='float32') }\\n\",\n \"block = neuron.compile(block, inputs=inputs)\\n\",\n \"\\n\",\n \"#save compiled model\\n\",\n \"block.export(\\\"resnet-50_compiled\\\", 0, block)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"6337e0ec\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!ls\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5a9af0c7\",\n \"metadata\": {},\n \"source\": [\n \"## Deploy\\n\",\n \"\\n\",\n \"Deply on Infenrentia to see the inference results as below:\\n\",\n \"```\\n\",\n \"probability=0.643591, class=n02123045 tabby, tabby cat\\n\",\n \"probability=0.184392, class=n02123159 tiger cat\\n\",\n \"probability=0.105063, class=n02124075 Egyptian cat\\n\",\n \"probability=0.030101, class=n02127052 lynx, catamount\\n\",\n \"probability=0.016112, class=n02129604 tiger, Panthera tigris\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"960c6aa9\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import numpy as np\\n\",\n \"import mxnet as mx\\n\",\n \"import mx_neuron as neuron\\n\",\n \"\\n\",\n \"path='http://data.mxnet.io/models/imagenet/'\\n\",\n \"mx.test_utils.download(path+'synset.txt')\\n\",\n \"\\n\",\n \"fname = mx.test_utils.download('https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg?raw=true')\\n\",\n \"img = mx.image.imread(fname)# convert into format (batch, RGB, width, height)\\n\",\n \"img = mx.image.imresize(img, 224, 224) # resize\\n\",\n \"img = img.transpose((2, 0, 1)) # Channel first\\n\",\n \"img = img.expand_dims(axis=0) # batchify\\n\",\n \"img = img.astype(dtype='float32')\\n\",\n \"\\n\",\n \"block = mx.gluon.nn.SymbolBlock.imports('resnet-50_compiled-symbol.json',\\\\\\n\",\n \" ['data', 'softmax_label'], 'resnet-50_compiled-0000.params', ctx=mx.cpu())\\n\",\n \"softmax = mx.nd.random_normal(shape=(1,))\\n\",\n \"\\n\",\n \"out = block(img, softmax).asnumpy()\\n\",\n \"\\n\",\n \"with open('synset.txt', 'r') as f:\\n\",\n \" labels = [l.rstrip() for l in f]\\n\",\n \"\\n\",\n \"out = block(img, softmax).asnumpy()\\n\",\n \"\\n\",\n \"prob = np.squeeze(out)\\n\",\n \"a = np.argsort(prob)[::-1]\\n\",\n \"for i in a[0:5]:\\n\",\n \" print('probability=%f, class=%s' %(prob[i], labels[i]))\"\n ]\n },\n {\n \"cell_type\": \"raw\",\n \"id\": \"4f15e776\",\n \"metadata\": {},\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 2\",\n \"language\": \"python\",\n \"name\": \"python2\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.9\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/mxnet/resnet50/resnet50.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"wrapped-soccer\",\n \"metadata\": {},\n \"source\": [\n \"# Running Neuron Apache MXNet ResNet50 on Inferentia \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"appreciated-daily\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction:\\n\",\n \"In this tutorial we will compile and deploy ResNet50 model for Inferentia.\\n\",\n \"In this tutorial we provide two main sections:\\n\",\n \"\\n\",\n \"1.Compile the ResNet50 model.\\n\",\n \"\\n\",\n \"2.Infer the compiled model.\\n\",\n \"\\n\",\n \"Before running the following verify this Jupyter notebook is running “conda_aws_neuron_mxnet_p36” kernel. You can select the Kernel from the “Kernel -> Change Kernel” option on the top of this Jupyter notebook page.\\n\",\n \"Neuron supports Python module, Symbol APIs and the C predict API. The following quick start example uses the Symbol API.\\n\",\n \"\\n\",\n \"### Warning\\n\",\n \"This tutorial was tested on MXNet-1.5\\n\",\n \"\\n\",\n \"MXNet-1.5 entered maintenance mode and require Neuron runtime 1.0, please see : [MXNet-1.5 enters maintainence mode](../../../../release-notes/maintenance.html)\\n\",\n \"\\n\",\n \"To setup development environment for MXNet-1.5 see installation instructions for Neuron 1.15.1 : [Neuron-1.15.1 MXNet install](../../../../archive/mxnet-neuron/setup/mxnet-install.html)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"advance-rebound\",\n \"metadata\": {},\n \"source\": [\n \"## Compile model on Neuron\\n\",\n \"The following step will compile the resnet50 model. Compilation will take a few minutes on inf1.6xlarge. At the end of compilation, the files resnet-50_compiled-0000.params and resnet-50_compiled-symbol.json will be created in local directory.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"alpha-publication\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import mxnet as mx\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"path='http://data.mxnet.io/models/imagenet/'\\n\",\n \"mx.test_utils.download(path+'resnet/50-layers/resnet-50-0000.params')\\n\",\n \"mx.test_utils.download(path+'resnet/50-layers/resnet-50-symbol.json')\\n\",\n \"sym, args, aux = mx.model.load_checkpoint('resnet-50', 0)\\n\",\n \"\\n\",\n \"# Compile for Inferentia using Neuron\\n\",\n \"inputs = { \\\"data\\\" : mx.nd.ones([1,3,224,224], name='data', dtype='float32') }\\n\",\n \"sym, args, aux = mx.contrib.neuron.compile(sym, args, aux, inputs)\\n\",\n \"\\n\",\n \"#save compiled model\\n\",\n \"mx.model.save_checkpoint(\\\"resnet-50_compiled\\\", 0, sym, args, aux)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"technical-reason\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!ls\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"meaningful-substance\",\n \"metadata\": {},\n \"source\": [\n \"## Deploy on Inferentia\\n\",\n \"Using same instance to deploy the model. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"cooked-jonathan\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import mxnet as mx\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"path='http://data.mxnet.io/models/imagenet/'\\n\",\n \"mx.test_utils.download(path+'synset.txt')\\n\",\n \"\\n\",\n \"fname = mx.test_utils.download('https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg?raw=true')\\n\",\n \"img = mx.image.imread(fname)# convert into format (batch, RGB, width, height)\\n\",\n \"img = mx.image.imresize(img, 224, 224) # resize\\n\",\n \"img = img.transpose((2, 0, 1)) # Channel first\\n\",\n \"img = img.expand_dims(axis=0) # batchify\\n\",\n \"img = img.astype(dtype='float32')\\n\",\n \"\\n\",\n \"sym, args, aux = mx.model.load_checkpoint('resnet-50_compiled', 0)\\n\",\n \"softmax = mx.nd.random_normal(shape=(1,))\\n\",\n \"args['softmax_label'] = softmax\\n\",\n \"args['data'] = img\\n\",\n \"\\n\",\n \"# Inferentia context\\n\",\n \"ctx = mx.neuron()\\n\",\n \"\\n\",\n \"exe = sym.bind(ctx=ctx, args=args, aux_states=aux, grad_req='null')\\n\",\n \"\\n\",\n \"with open('synset.txt', 'r') as f:\\n\",\n \" labels = [l.rstrip() for l in f]\\n\",\n \"\\n\",\n \"exe.forward(data=img)\\n\",\n \"prob = exe.outputs[0].asnumpy()# print the top-5\\n\",\n \"prob = np.squeeze(prob)\\n\",\n \"a = np.argsort(prob)[::-1]\\n\",\n \"for i in a[0:5]:\\n\",\n \" print('probability=%f, class=%s' %(prob[i], labels[i]))\\n\",\n \" \\n\",\n \"# Sample output will look like below:\\n\",\n \"#probability=0.634792, class=n02123045 tabby, tabby cat\\n\",\n \"#probability=0.193601, class=n02123159 tiger cat\\n\",\n \"#probability=0.103627, class=n02124075 Egyptian cat\\n\",\n \"#probability=0.031604, class=n02127052 lynx, catamount\\n\",\n \"#probability=0.015892, class=n02129604 tiger, Panthera tigris\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Environment (conda_aws_neuron_mxnet_p36)\",\n \"language\": \"python\",\n \"name\": \"conda_aws_neuron_mxnet_p36\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/mxnet/resnet50_neuroncore_groups.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Neuron Apache MXNet - Configurations for NeuronCore Groups Using Resnet50\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"## Introduction:\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy Resnet-50 model in parallel using the concept of NeuronCore Groups on an Inf1 instance. This Jupyter notebook should be run on an instance which is inf1.6xlarge or larger. For simplicity we will run this tutorial on inf1.6xlarge but in real life scenario the compilation should be done on a compute instance and the deployment on inf1 instance to save costs. \\n\",\n \"\\n\",\n \"Set environment variable NEURON_RT_NUM_CORES to the total number of Neuron cores that will be utilized. The consecutive NeuronCore groups will be created by Neuron Runtime and place the models to the cores according to the compiled size.\\n\",\n \"\\n\",\n \"Note that in order to map a model to a group, the model must be compiled to fit within the group size. To limit the number of NeuronCores during compilation, use compiler_args dictionary with field “–neuroncore-pipeline-cores“ set to the group size. For exmaple, if NEURON_RT_NUM_CORES=4 and two models compiled with “–neuroncore-pipeline-cores=3“ and “–neuroncore-pipeline-cores=1“ were loaded, the first model would occupy NC0-2 and the second model would occupy NC3. \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"```\\n\",\n \"compile_args = {'--neuroncore-pipeline-cores' : 2}\\n\",\n \"sym, args, auxs = neuron.compile(sym, args, auxs, inputs, **compile_args)\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"In this tutorial we provide two main sections:\\n\",\n \"\\n\",\n \"1. Compile the Resnet50 model for Neuron\\n\",\n \"\\n\",\n \"2. Run inference using NeuronCore Groups\\n\",\n \"\\n\",\n \"Please use environment `conda_aws_neuron_mxnet_p36`.\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Compile model for Neuron\\n\",\n \"\\n\",\n \"Model must be compiled to Inferentia target before it can be used on Inferentia. In the following we will compile the the flag, --neuroncore-pipeline-cores set to 2 and run it. The files resnet-50_compiled-0000.params and resnet-50_compiled-symbol.json will be created in local directory\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from packaging import version\\n\",\n \"import mxnet as mx\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"import mx_neuron as neuron\\n\",\n \"\\n\",\n \"path='http://data.mxnet.io/models/imagenet/'\\n\",\n \"mx.test_utils.download(path+'resnet/50-layers/resnet-50-0000.params')\\n\",\n \"mx.test_utils.download(path+'resnet/50-layers/resnet-50-symbol.json')\\n\",\n \"sym, args, aux = mx.model.load_checkpoint('resnet-50', 0)\\n\",\n \"\\n\",\n \"# Compile for Inferentia using Neuron, fit to NeuronCore group size of 2\\n\",\n \"inputs = { \\\"data\\\" : mx.nd.ones([1,3,224,224], name='data', dtype='float32') }\\n\",\n \"compile_args = {'--neuroncore-pipeline-cores' : 2}\\n\",\n \"sym, args, aux = neuron.compile(sym, args, aux, inputs, **compile_args)\\n\",\n \"\\n\",\n \"#save compiled model\\n\",\n \"mx.model.save_checkpoint(\\\"resnet-50_compiled\\\", 0, sym, args, aux)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference using NeuronCore Groups\\n\",\n \"\\n\",\n \"Within the framework, the model can be mapped to specific cores using ```ctx=mx.neuron(N)``` context where N specifies the index of the Neuron core to deploy. For more information, see https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/appnotes/perf/flex-eg.html .\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import warnings\\n\",\n \"\\n\",\n \"mx.test_utils.download(path+'synset.txt')\\n\",\n \"\\n\",\n \"fname = mx.test_utils.download('https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg?raw=true')\\n\",\n \"img = mx.image.imread(fname) # convert into format (batch, RGB, width, height)\\n\",\n \"img = mx.image.imresize(img, 224, 224) # resize\\n\",\n \"img = img.transpose((2, 0, 1)) # Channel first\\n\",\n \"img = img.expand_dims(axis=0) # batchify\\n\",\n \"img = img.astype(dtype='float32')\\n\",\n \"\\n\",\n \"sym, args, aux = mx.model.load_checkpoint('resnet-50_compiled', 0)\\n\",\n \"softmax = mx.nd.random_normal(shape=(1,))\\n\",\n \"args['softmax_label'] = softmax\\n\",\n \"args['data'] = img\\n\",\n \"\\n\",\n \"os.environ[\\\"NEURON_RT_NUM_CORES\\\"] = '4'\\n\",\n \"\\n\",\n \"\\n\",\n \"# Inferentia context - group index 1 (size 2) would skip NC0 and place the \\n\",\n \"# compiled model onto NC1,2\\n\",\n \"ctx = mx.neuron(1)\\n\",\n \"\\n\",\n \"exe = sym.bind(ctx=ctx, args=args, aux_states=aux, grad_req='null')\\n\",\n \"\\n\",\n \"with open('synset.txt', 'r') as f:\\n\",\n \" labels = [l.rstrip() for l in f]\\n\",\n \"\\n\",\n \"exe.forward(data=img)\\n\",\n \"prob = exe.outputs[0].asnumpy()# print the top-5\\n\",\n \"prob = np.squeeze(prob)\\n\",\n \"a = np.argsort(prob)[::-1]\\n\",\n \"for i in a[0:5]:\\n\",\n \" print('probability=%f, class=%s' %(prob[i], labels[i]))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can experiment with different Neuron core group combinations and different models.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Troubleshooting\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"If not enough NeuronCores are provided, an error message will be displayed:\\n\",\n \"\\n\",\n \"```\\n\",\n \"mxnet.base.MXNetError: [04:01:39] src/operator/subgraph/neuron/./neuron_util.h:541: Check failed: rsp.status().code() == 0: Failed load model with Neuron-RTD Error. Neuron-RTD Status Code: 9, details: \\\"\\\"\\n\",\n \"\\n\",\n \"```\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Environment (conda_aws_neuron_mxnet_p36)\",\n \"language\": \"python\",\n \"name\": \"conda_aws_neuron_mxnet_p36\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/neuron-monitor/neuron-monitor-grafana.json", "content": "{\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": 2,\n \"iteration\": 1605138719380,\n \"links\": [],\n \"panels\": [\n {\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {\n \"align\": null,\n \"filterable\": false\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"Value\"\n },\n \"properties\": [\n {\n \"id\": \"custom.width\",\n \"value\": 163\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"Field\"\n },\n \"properties\": [\n {\n \"id\": \"custom.width\",\n \"value\": 450\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"ami_id\"\n },\n \"properties\": [\n {\n \"id\": \"custom.width\",\n \"value\": 217\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"instance_type\"\n },\n \"properties\": [\n {\n \"id\": \"custom.width\",\n \"value\": 391\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"Prometheus instance\"\n },\n \"properties\": [\n {\n \"id\": \"custom.width\",\n \"value\": 641\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 8,\n \"options\": {\n \"showHeader\": true,\n \"sortBy\": []\n },\n \"pluginVersion\": \"7.2.1\",\n \"repeat\": null,\n \"targets\": [\n {\n \"expr\": \"instance_info\",\n \"format\": \"table\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Instance Info\",\n \"transformations\": [\n {\n \"id\": \"organize\",\n \"options\": {\n \"excludeByName\": {\n \"Time\": true,\n \"Value\": true,\n \"__name__\": true,\n \"ami_id\": false,\n \"instance\": true,\n \"job\": true\n },\n \"indexByName\": {\n \"Time\": 0,\n \"Value\": 7,\n \"__name__\": 1,\n \"availability_zone\": 8,\n \"instance\": 5,\n \"instance_id\": 2,\n \"instance_name\": 3,\n \"instance_type\": 4,\n \"job\": 6,\n \"region\": 9,\n \"subnet_id\": 10\n },\n \"renameByName\": {\n \"Value\": \"\",\n \"availability_zone\": \"Availability Zone\",\n \"instance\": \"\",\n \"instance_id\": \"Instance ID\",\n \"instance_name\": \"Instance Name\",\n \"instance_type\": \"Instance Type\",\n \"region\": \"Region\",\n \"subnet_id\": \"Subnet\"\n }\n }\n }\n ],\n \"type\": \"table\"\n },\n {\n \"datasource\": null,\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"super-light-yellow\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 0,\n \"y\": 8\n },\n \"id\": 36,\n \"options\": {\n \"colorMode\": \"value\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"last\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"count(instance_info)\\n\",\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Instance Count\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n }\n ]\n },\n \"unit\": \"none\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 3,\n \"y\": 8\n },\n \"id\": 10,\n \"options\": {\n \"colorMode\": \"value\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"center\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"sum (system_vcpu_count)\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"vCPU Count\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": null,\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"percentage\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"#EAB839\",\n \"value\": 70\n },\n {\n \"color\": \"orange\",\n \"value\": 80\n },\n {\n \"color\": \"semi-dark-red\",\n \"value\": 90\n }\n ]\n },\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 6,\n \"y\": 8\n },\n \"id\": 20,\n \"options\": {\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showThresholdLabels\": true,\n \"showThresholdMarkers\": true\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"avg(sum by (instance_id) (system_vcpu_usage_ratio))\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"vCPU Utilization\",\n \"type\": \"gauge\"\n },\n {\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"percentage\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"yellow\",\n \"value\": 70\n },\n {\n \"color\": \"orange\",\n \"value\": 80\n },\n {\n \"color\": \"red\",\n \"value\": 90\n }\n ]\n },\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 9,\n \"y\": 8\n },\n \"id\": 16,\n \"options\": {\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showThresholdLabels\": true,\n \"showThresholdMarkers\": true\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"avg(system_memory_used_bytes / system_memory_total_bytes)\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Host Memory Usage\",\n \"type\": \"gauge\"\n },\n {\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"rgb(191, 151, 105)\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 12,\n \"y\": 8\n },\n \"id\": 12,\n \"options\": {\n \"colorMode\": \"value\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"center\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"count(neuroncore_utilization_ratio > 0)\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"NeuronCores in Use\",\n \"transformations\": [],\n \"type\": \"stat\"\n },\n {\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {\n \"align\": null,\n \"filterable\": false\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"percentage\",\n \"steps\": [\n {\n \"color\": \"red\",\n \"value\": null\n },\n {\n \"color\": \"orange\",\n \"value\": 5\n },\n {\n \"color\": \"yellow\",\n \"value\": 20\n },\n {\n \"color\": \"green\",\n \"value\": 35\n }\n ]\n },\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 15,\n \"y\": 8\n },\n \"id\": 4,\n \"interval\": \"\",\n \"options\": {\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showThresholdLabels\": true,\n \"showThresholdMarkers\": true\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"avg(neuroncore_utilization_ratio)\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"NeuronCore Utilization\",\n \"type\": \"gauge\"\n },\n {\n \"datasource\": \"Prometheus\",\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"percentage\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n },\n \"unit\": \"cps\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 18,\n \"y\": 8\n },\n \"id\": 6,\n \"options\": {\n \"colorMode\": \"value\",\n \"graphMode\": \"area\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"sum(rate(execution_status_total{status_type=\\\"completed\\\"}[1m]))\",\n \"hide\": false,\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Execution Success Rate\",\n \"transformations\": [],\n \"type\": \"stat\"\n },\n {\n \"datasource\": \"Prometheus\",\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 1\n }\n ]\n },\n \"unit\": \"cps\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 5,\n \"w\": 3,\n \"x\": 21,\n \"y\": 8\n },\n \"id\": 18,\n \"options\": {\n \"colorMode\": \"value\",\n \"graphMode\": \"area\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"7.2.1\",\n \"targets\": [\n {\n \"expr\": \"sum(rate(execution_status_total{status_type!=\\\"completed\\\"}[1m]))\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Execution Error Rate\",\n \"type\": \"stat\"\n },\n {\n \"aliasColors\": {\n \"Inf Error Rate\": \"semi-dark-red\",\n \"Inf Success Rate\": \"light-green\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": null,\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {}\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 12,\n \"x\": 0,\n \"y\": 13\n },\n \"hiddenSeries\": false,\n \"id\": 32,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"sum(rate(execution_status_total{status_type=\\\"completed\\\"}[1m]))\",\n \"interval\": \"\",\n \"legendFormat\": \"Execution Success Rate\",\n \"refId\": \"A\"\n },\n {\n \"expr\": \"sum(rate(execution_status_total{status_type!=\\\"completed\\\"}[1m]))\",\n \"interval\": \"\",\n \"legendFormat\": \"Execution Error Rate\",\n \"refId\": \"B\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Execution Status Rates\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:547\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:548\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {\n \"p0\": \"dark-green\",\n \"p1\": \"semi-dark-green\",\n \"p100\": \"semi-dark-red\",\n \"p25\": \"light-green\",\n \"p50\": \"super-light-green\",\n \"p75\": \"super-light-red\",\n \"p99\": \"light-red\",\n \"{percentile=\\\"p0\\\"}\": \"dark-green\",\n \"{percentile=\\\"p1\\\"}\": \"semi-dark-green\",\n \"{percentile=\\\"p100\\\"}\": \"dark-red\",\n \"{percentile=\\\"p25\\\"}\": \"light-green\",\n \"{percentile=\\\"p50\\\"}\": \"super-light-green\",\n \"{percentile=\\\"p75\\\"}\": \"light-red\",\n \"{percentile=\\\"p99\\\"}\": \"semi-dark-red\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": null,\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"s\"\n },\n \"overrides\": []\n },\n \"fill\": 0,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 12,\n \"x\": 12,\n \"y\": 13\n },\n \"hiddenSeries\": false,\n \"id\": 34,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 1,\n \"points\": true,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg by (percentile) (execution_latency_seconds)\",\n \"interval\": \"\",\n \"legendFormat\": \"{{percentile}}\",\n \"refId\": \"A\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Execution Latency\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:61\",\n \"format\": \"s\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:62\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {},\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": null,\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 8,\n \"x\": 0,\n \"y\": 25\n },\n \"hiddenSeries\": false,\n \"id\": 30,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg by (neuroncore) (neuroncore_utilization_ratio)\",\n \"interval\": \"\",\n \"legendFormat\": \"nc{{neuroncore}}\",\n \"refId\": \"A\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"NeuronCore Utilization\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:493\",\n \"format\": \"percentunit\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": \"1\",\n \"min\": \"0\",\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:494\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": \"100\",\n \"min\": \"0\",\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {\n \"Runtime system CPU usage \": \"light-red\",\n \"Runtime user CPU usage \": \"light-green\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 8,\n \"x\": 8,\n \"y\": 25\n },\n \"hiddenSeries\": false,\n \"id\": 2,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": true,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg by (usage_type) (neuron_runtime_vcpu_usage_ratio)\",\n \"format\": \"time_series\",\n \"instant\": false,\n \"interval\": \"\",\n \"legendFormat\": \"Neuron Runtime {{usage_type}} CPU usage \",\n \"refId\": \"A\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Neuron Runtime vCPU Usage\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:385\",\n \"format\": \"percentunit\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": \"1\",\n \"min\": \"0\",\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:386\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {\n \"host\": \"rgb(0, 217, 255)\",\n \"neuron_device\": \"super-light-orange\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": null,\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"unit\": \"bytes\"\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 8,\n \"x\": 16,\n \"y\": 25\n },\n \"hiddenSeries\": false,\n \"id\": 28,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg by (memory_location) (sum by (instance_id, memory_location) (neuron_runtime_memory_used_bytes))\",\n \"interval\": \"\",\n \"legendFormat\": \"{{memory_location}}\",\n \"refId\": \"A\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Neuron Runtime Used Memory\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:439\",\n \"format\": \"bytes\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:440\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {\n \"Memory Usage\": \"rgb(0, 217, 255)\",\n \"NeuronCore Usage\": \"light-orange\",\n \"vCPU Usage\": \"light-blue\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": null,\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 8,\n \"x\": 0,\n \"y\": 37\n },\n \"hiddenSeries\": false,\n \"id\": 22,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg(system_memory_used_bytes / system_memory_total_bytes)\",\n \"instant\": false,\n \"interval\": \"\",\n \"legendFormat\": \"Memory Usage\",\n \"refId\": \"A\"\n },\n {\n \"expr\": \"avg(sum by (instance_id) (system_vcpu_usage_ratio))\",\n \"instant\": false,\n \"interval\": \"\",\n \"legendFormat\": \"vCPU Usage\",\n \"refId\": \"B\"\n },\n {\n \"expr\": \"avg(neuroncore_utilization_ratio)\",\n \"instant\": false,\n \"interval\": \"\",\n \"legendFormat\": \"NeuronCore Usage\",\n \"refId\": \"C\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Host System Utilization\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:664\",\n \"format\": \"percentunit\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": \"1\",\n \"min\": \"0\",\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:665\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {\n \"system\": \"light-red\",\n \"user\": \"light-green\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"unit\": \"percentunit\"\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 8,\n \"x\": 8,\n \"y\": 37\n },\n \"hiddenSeries\": false,\n \"id\": 24,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [],\n \"spaceLength\": 10,\n \"stack\": true,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg by (usage_type) (system_vcpu_usage_ratio)\",\n \"interval\": \"\",\n \"legendFormat\": \"{{usage_type}}\",\n \"refId\": \"A\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Host vCPU Usage\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:876\",\n \"format\": \"percentunit\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": \"1\",\n \"min\": \"0\",\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:877\",\n \"format\": \"short\",\n \"label\": null,\n \"logBase\": 1,\n \"max\": null,\n \"min\": null,\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n },\n {\n \"aliasColors\": {\n \"Memory Usage Bytes\": \"rgb(223, 180, 0)\",\n \"Memory Usage Percent\": \"rgb(0, 217, 255)\"\n },\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": \"Prometheus\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {},\n \"unit\": \"short\"\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"Memory Usage Percent\"\n },\n \"properties\": [\n {\n \"id\": \"unit\",\n \"value\": \"percentunit\"\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"Memory Usage Bytes\"\n },\n \"properties\": [\n {\n \"id\": \"unit\",\n \"value\": \"bytes\"\n }\n ]\n }\n ]\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 12,\n \"w\": 8,\n \"x\": 16,\n \"y\": 37\n },\n \"hiddenSeries\": false,\n \"id\": 26,\n \"legend\": {\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"show\": true,\n \"total\": false,\n \"values\": false\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"7.2.1\",\n \"pointradius\": 2,\n \"points\": false,\n \"renderer\": \"flot\",\n \"seriesOverrides\": [\n {\n \"$$hashKey\": \"object:711\"\n },\n {\n \"$$hashKey\": \"object:931\",\n \"alias\": \"Memory Usage Bytes\",\n \"yaxis\": 2\n }\n ],\n \"spaceLength\": 10,\n \"stack\": false,\n \"steppedLine\": false,\n \"targets\": [\n {\n \"expr\": \"avg(system_memory_used_bytes / system_memory_total_bytes)\",\n \"instant\": false,\n \"interval\": \"\",\n \"legendFormat\": \"Memory Usage Percent\",\n \"refId\": \"A\"\n },\n {\n \"expr\": \"avg(system_memory_used_bytes)\",\n \"instant\": false,\n \"interval\": \"\",\n \"legendFormat\": \"Memory Usage Bytes\",\n \"refId\": \"B\"\n }\n ],\n \"thresholds\": [],\n \"timeFrom\": null,\n \"timeRegions\": [],\n \"timeShift\": null,\n \"title\": \"Host Memory Usage\",\n \"tooltip\": {\n \"shared\": true,\n \"sort\": 0,\n \"value_type\": \"individual\"\n },\n \"type\": \"graph\",\n \"xaxis\": {\n \"buckets\": null,\n \"mode\": \"time\",\n \"name\": null,\n \"show\": true,\n \"values\": []\n },\n \"yaxes\": [\n {\n \"$$hashKey\": \"object:689\",\n \"format\": \"percentunit\",\n \"label\": \"\",\n \"logBase\": 1,\n \"max\": \"1\",\n \"min\": \"0\",\n \"show\": true\n },\n {\n \"$$hashKey\": \"object:690\",\n \"decimals\": null,\n \"format\": \"bytes\",\n \"label\": \"\",\n \"logBase\": 1,\n \"max\": null,\n \"min\": \"0\",\n \"show\": true\n }\n ],\n \"yaxis\": {\n \"align\": false,\n \"alignLevel\": null\n }\n }\n ],\n \"refresh\": \"5s\",\n \"schemaVersion\": 26,\n \"style\": \"dark\",\n \"tags\": [],\n \"templating\": {\n \"list\": [\n {\n \"datasource\": \"Prometheus\",\n \"filters\": [],\n \"hide\": 0,\n \"label\": \"\",\n \"name\": \"Filters\",\n \"skipUrlSync\": false,\n \"type\": \"adhoc\"\n }\n ]\n },\n \"time\": {\n \"from\": \"now-6h\",\n \"to\": \"now\"\n },\n \"timepicker\": {},\n \"timezone\": \"\",\n \"title\": \"neuron-monitor\",\n \"uid\": \"EqWNYf5Mz\",\n \"version\": 68\n}" }, { "path": "src/examples/pytorch/bert_tutorial/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/pytorch/bert_tutorial/THIRD", "content": "" }, { "path": "src/examples/pytorch/bert_tutorial/THIRD PARTY LICENSE.txt", "content": "** transformers; version 2.8.0 -- https://github.com/huggingface/transformers\r\nCopyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.\r\nCopyright (c) 2018, NVIDIA CORPORATION. All rights reserved.\r\n\r\n Apache License\r\n Version 2.0, January 2004\r\n http://www.apache.org/licenses/\r\n\r\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\r\n\r\n 1. Definitions.\r\n\r\n \"License\" shall mean the terms and conditions for use, reproduction,\r\n and distribution as defined by Sections 1 through 9 of this document.\r\n\r\n \"Licensor\" shall mean the copyright owner or entity authorized by\r\n the copyright owner that is granting the License.\r\n\r\n \"Legal Entity\" shall mean the union of the acting entity and all\r\n other entities that control, are controlled by, or are under common\r\n control with that entity. For the purposes of this definition,\r\n \"control\" means (i) the power, direct or indirect, to cause the\r\n direction or management of such entity, whether by contract or\r\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\r\n outstanding shares, or (iii) beneficial ownership of such entity.\r\n\r\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\r\n exercising permissions granted by this License.\r\n\r\n \"Source\" form shall mean the preferred form for making modifications,\r\n including but not limited to software source code, documentation\r\n source, and configuration files.\r\n\r\n \"Object\" form shall mean any form resulting from mechanical\r\n transformation or translation of a Source form, including but\r\n not limited to compiled object code, generated documentation,\r\n and conversions to other media types.\r\n\r\n \"Work\" shall mean the work of authorship, whether in Source or\r\n Object form, made available under the License, as indicated by a\r\n copyright notice that is included in or attached to the work\r\n (an example is provided in the Appendix below).\r\n\r\n \"Derivative Works\" shall mean any work, whether in Source or Object\r\n form, that is based on (or derived from) the Work and for which the\r\n editorial revisions, annotations, elaborations, or other modifications\r\n represent, as a whole, an original work of authorship. For the purposes\r\n of this License, Derivative Works shall not include works that remain\r\n separable from, or merely link (or bind by name) to the interfaces of,\r\n the Work and Derivative Works thereof.\r\n\r\n \"Contribution\" shall mean any work of authorship, including\r\n the original version of the Work and any modifications or additions\r\n to that Work or Derivative Works thereof, that is intentionally\r\n submitted to Licensor for inclusion in the Work by the copyright owner\r\n or by an individual or Legal Entity authorized to submit on behalf of\r\n the copyright owner. For the purposes of this definition, \"submitted\"\r\n means any form of electronic, verbal, or written communication sent\r\n to the Licensor or its representatives, including but not limited to\r\n communication on electronic mailing lists, source code control systems,\r\n and issue tracking systems that are managed by, or on behalf of, the\r\n Licensor for the purpose of discussing and improving the Work, but\r\n excluding communication that is conspicuously marked or otherwise\r\n designated in writing by the copyright owner as \"Not a Contribution.\"\r\n\r\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\r\n on behalf of whom a Contribution has been received by Licensor and\r\n subsequently incorporated within the Work.\r\n\r\n 2. Grant of Copyright License. Subject to the terms and conditions of\r\n this License, each Contributor hereby grants to You a perpetual,\r\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\r\n copyright license to reproduce, prepare Derivative Works of,\r\n publicly display, publicly perform, sublicense, and distribute the\r\n Work and such Derivative Works in Source or Object form.\r\n\r\n 3. Grant of Patent License. Subject to the terms and conditions of\r\n this License, each Contributor hereby grants to You a perpetual,\r\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\r\n (except as stated in this section) patent license to make, have made,\r\n use, offer to sell, sell, import, and otherwise transfer the Work,\r\n where such license applies only to those patent claims licensable\r\n by such Contributor that are necessarily infringed by their\r\n Contribution(s) alone or by combination of their Contribution(s)\r\n with the Work to which such Contribution(s) was submitted. If You\r\n institute patent litigation against any entity (including a\r\n cross-claim or counterclaim in a lawsuit) alleging that the Work\r\n or a Contribution incorporated within the Work constitutes direct\r\n or contributory patent infringement, then any patent licenses\r\n granted to You under this License for that Work shall terminate\r\n as of the date such litigation is filed.\r\n\r\n 4. Redistribution. You may reproduce and distribute copies of the\r\n Work or Derivative Works thereof in any medium, with or without\r\n modifications, and in Source or Object form, provided that You\r\n meet the following conditions:\r\n\r\n (a) You must give any other recipients of the Work or\r\n Derivative Works a copy of this License; and\r\n\r\n (b) You must cause any modified files to carry prominent notices\r\n stating that You changed the files; and\r\n\r\n (c) You must retain, in the Source form of any Derivative Works\r\n that You distribute, all copyright, patent, trademark, and\r\n attribution notices from the Source form of the Work,\r\n excluding those notices that do not pertain to any part of\r\n the Derivative Works; and\r\n\r\n (d) If the Work includes a \"NOTICE\" text file as part of its\r\n distribution, then any Derivative Works that You distribute must\r\n include a readable copy of the attribution notices contained\r\n within such NOTICE file, excluding those notices that do not\r\n pertain to any part of the Derivative Works, in at least one\r\n of the following places: within a NOTICE text file distributed\r\n as part of the Derivative Works; within the Source form or\r\n documentation, if provided along with the Derivative Works; or,\r\n within a display generated by the Derivative Works, if and\r\n wherever such third-party notices normally appear. The contents\r\n of the NOTICE file are for informational purposes only and\r\n do not modify the License. You may add Your own attribution\r\n notices within Derivative Works that You distribute, alongside\r\n or as an addendum to the NOTICE text from the Work, provided\r\n that such additional attribution notices cannot be construed\r\n as modifying the License.\r\n\r\n You may add Your own copyright statement to Your modifications and\r\n may provide additional or different license terms and conditions\r\n for use, reproduction, or distribution of Your modifications, or\r\n for any such Derivative Works as a whole, provided Your use,\r\n reproduction, and distribution of the Work otherwise complies with\r\n the conditions stated in this License.\r\n\r\n 5. Submission of Contributions. Unless You explicitly state otherwise,\r\n any Contribution intentionally submitted for inclusion in the Work\r\n by You to the Licensor shall be under the terms and conditions of\r\n this License, without any additional terms or conditions.\r\n Notwithstanding the above, nothing herein shall supersede or modify\r\n the terms of any separate license agreement you may have executed\r\n with Licensor regarding such Contributions.\r\n\r\n 6. Trademarks. This License does not grant permission to use the trade\r\n names, trademarks, service marks, or product names of the Licensor,\r\n except as required for reasonable and customary use in describing the\r\n origin of the Work and reproducing the content of the NOTICE file.\r\n\r\n 7. Disclaimer of Warranty. Unless required by applicable law or\r\n agreed to in writing, Licensor provides the Work (and each\r\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\r\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\r\n implied, including, without limitation, any warranties or conditions\r\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\r\n PARTICULAR PURPOSE. You are solely responsible for determining the\r\n appropriateness of using or redistributing the Work and assume any\r\n risks associated with Your exercise of permissions under this License.\r\n\r\n 8. Limitation of Liability. In no event and under no legal theory,\r\n whether in tort (including negligence), contract, or otherwise,\r\n unless required by applicable law (such as deliberate and grossly\r\n negligent acts) or agreed to in writing, shall any Contributor be\r\n liable to You for damages, including any direct, indirect, special,\r\n incidental, or consequential damages of any character arising as a\r\n result of this License or out of the use or inability to use the\r\n Work (including but not limited to damages for loss of goodwill,\r\n work stoppage, computer failure or malfunction, or any and all\r\n other commercial damages or losses), even if such Contributor\r\n has been advised of the possibility of such damages.\r\n\r\n 9. Accepting Warranty or Additional Liability. While redistributing\r\n the Work or Derivative Works thereof, You may choose to offer,\r\n and charge a fee for, acceptance of support, warranty, indemnity,\r\n or other liability obligations and/or rights consistent with this\r\n License. However, in accepting such obligations, You may act only\r\n on Your own behalf and on Your sole responsibility, not on behalf\r\n of any other Contributor, and only if You agree to indemnify,\r\n defend, and hold each Contributor harmless for any liability\r\n incurred by, or claims asserted against, such Contributor by reason\r\n of your accepting any such warranty or additional liability.\r\n\r\n END OF TERMS AND CONDITIONS\r\n\r\n APPENDIX: How to apply the Apache License to your work.\r\n\r\n To apply the Apache License to your work, attach the following\r\n boilerplate notice, with the fields enclosed by brackets \"[]\"\r\n replaced with your own identifying information. (Don't include\r\n the brackets!) The text should be enclosed in the appropriate\r\n comment syntax for the file format. We also recommend that a\r\n file or class name and description of purpose be included on the\r\n same \"printed page\" as the copyright notice for easier\r\n identification within third-party archives.\r\n\r\n Copyright [yyyy] [name of copyright owner]\r\n\r\n Licensed under the Apache License, Version 2.0 (the \"License\");\r\n you may not use this file except in compliance with the License.\r\n You may obtain a copy of the License at\r\n\r\n http://www.apache.org/licenses/LICENSE-2.0\r\n\r\n Unless required by applicable law or agreed to in writing, software\r\n distributed under the License is distributed on an \"AS IS\" BASIS,\r\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r\n See the License for the specific language governing permissions and\r\n limitations under the License." }, { "path": "src/examples/pytorch/bert_tutorial/bert_benchmark_utils.py", "content": "import torch\nimport torch.neuron\nimport os\nimport sys\nimport csv\nimport math\nfrom collections import Counter\n\nimport numpy as np\n\nclass BertTestDataset(torch.utils.data.Dataset):\n \"\"\"Bert test dataset.\"\"\"\n\n def __init__(self, tsv_file, tokenizer, max_length=128, transform=None):\n \"\"\"\n Args:\n csv_file (string): Path to the csv file with annotations.\n tokenizer (callable = hugging face tokenizer): Takes a string and encodes to standard input tensor set\n max_length (int): Maximum length that all input tensors will be padded to\n transform (callable, optional): Optional transform to be applied\n on a sample.\n \"\"\"\n with open(tsv_file, \"r\") as f:\n reader = csv.reader(f, delimiter=\"\\t\", quotechar=None)\n self.lines = list(reader)\n\n self.lines.pop(0)\n\n self.tokenizer = tokenizer\n self.max_length = max_length\n self.transform = transform\n\n def __len__(self):\n return len(self.lines)\n\n def __getitem__(self, idx):\n if torch.is_tensor(idx):\n idx = idx.tolist()\n\n s1_raw = self.lines[idx][3]\n if isinstance(s1_raw, bytes):\n s1_raw = s1_raw.decode(\"utf-8\", \"ignore\")\n s2_raw = self.lines[idx][4]\n if isinstance(s2_raw, bytes):\n s2_raw = s2_raw.decode(\"utf-8\", \"ignore\")\n\n quality = self.lines[idx][0]\n\n encoded = self.tokenizer.encode_plus(s1_raw, s2_raw, add_special_tokens=True,\n return_tensors='pt', max_length=self.max_length, \n padding='max_length', truncation=True)\n\n sample = {'encoded': encoded, 'quality': quality}\n\n if self.transform:\n sample = self.transform(sample)\n\n return sample\n\n\nclass BertResults():\n\n def __init__(self, batch_size, num_cores=1):\n self.correct_count = 0\n self.inference_count = 0\n self.latency_array = []\n self.end_times = []\n self.start_times = []\n self.batch_size = batch_size\n self.num_cores = num_cores\n\n def add_result(self, correct_count, inference_count, latency_array, end_times, start_times):\n self.correct_count += correct_count\n\n self.inference_count += inference_count\n self.latency_array.extend(latency_array)\n self.end_times.extend(end_times)\n self.start_times.extend(start_times)\n\n def report(self, f, window_size=1):\n assert(len(self.latency_array) != 0)\n p50_latency = np.percentile(self.latency_array, 50)\n p90_latency = np.percentile(self.latency_array, 90)\n p95_latency = np.percentile(self.latency_array, 95)\n p99_latency = np.percentile(self.latency_array, 99)\n p100_latency = np.percentile(self.latency_array, 100)\n\n\n def get_bucket(start, end):\n bucketed_start = math.floor(start / window_size) * window_size\n bucketed_end = math.ceil(end / window_size) * window_size\n # The check is to make sure that we ignore timestamps that are larger than the window size\n if bucketed_end - bucketed_start == window_size:\n return bucketed_start\n else:\n return None\n \n # Divide the timestamps into different buckets\n bucketed_timestamps = [get_bucket(start, end)\n for start, end in zip(self.start_times, self.end_times)]\n # Count the values in each bucket\n counted_buckets = Counter(\n item for item in bucketed_timestamps if item is not None)\n # Normalize each bucket\n bucket_throughputs = [(key, value / window_size)\n for key, value in sorted(counted_buckets.items())]\n \n busy_throughputs = [value for _, value in bucket_throughputs]\n max_throughput = max(busy_throughputs) * self.batch_size\n avg_throughput = sum(busy_throughputs) * self.batch_size / len(busy_throughputs)\n \n f.write(\"\\n\")\n f.write(\n \"Maximum throughput = {} sentences/sec\\n\".format(int(max_throughput)))\n f.write(\"Average throughput = {} sentences/sec\\n\".format(int(avg_throughput)))\n\n f.write(\"\\n\")\n f.write(\"Latency Percentiles:\\n\")\n f.write(\"===\\n\")\n f.write(\"P50 = {} milliseconds\\n\".format(int(1000*p50_latency)))\n f.write(\"P90 = {} milliseconds\\n\".format(int(1000*p90_latency)))\n f.write(\"P95 = {} milliseconds\\n\".format(int(1000*p95_latency)))\n f.write(\"P99 = {} milliseconds\\n\".format(int(1000*p99_latency)))\n f.write(\"P100 = {} milliseconds\\n\".format(int(1000*p100_latency)))\n f.write(\"\\n\")\n f.write(\"Accuracy:\\n\")\n f.write(\"===\\n\")\n if self.inference_count == 0:\n self.inference_count = 1\n accuracy = float(self.correct_count) / float(self.inference_count)\n f.write(\"Accuracy = {}% \\n\".format(round(100*accuracy, 2)))\n f.write(\"\\n\")\n f.write(\"Sanity test:\\n\")\n f.write(\"===\\n\")\n f.write(\"Processed - num batches {}\\n\".format(len(self.latency_array)))\n f.write(\" - batch size {}\\n\".format(self.batch_size))\n f.write(\" - num cores {}\\n\".format(self.num_cores))\n" }, { "path": "src/examples/pytorch/bert_tutorial/glue_mrpc_dev.tsv", "content": "Quality\t#1 ID\t#2 ID\t#1 String\t#2 String\n1\t1355540\t1355592\tHe said the foodservice pie business doesn 't fit the company 's long-term growth strategy .\t\" The foodservice pie business does not fit our long-term growth strategy .\n0\t2029631\t2029565\tMagnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war .\tHis wife said he was \" 100 percent behind George Bush \" and looked forward to using his years of training in the war .\n0\t487993\t487952\tThe dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat .\tThe dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .\n1\t1989515\t1989458\tThe AFL-CIO is waiting until October to decide if it will endorse a candidate .\tThe AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .\n0\t1783137\t1782659\tNo dates have been set for the civil or the criminal trial .\tNo dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .\n1\t3039165\t3039036\tWal-Mart said it would check all of its million-plus domestic workers to ensure they were legally employed .\tIt has also said it would review all of its domestic employees more than 1 million to ensure they have legal status .\n0\t1490811\t1490840\tWhile dioxin levels in the environment were up last year , they have dropped by 75 percent since the 1970s , said Caswell .\tThe Institute said dioxin levels in the environment have fallen by as much as 76 percent since the 1970s .\n1\t426112\t426210\tThis integrates with Rational PurifyPlus and allows developers to work in supported versions of Java , Visual C # and Visual Basic .NET.\tIBM said the Rational products were also integrated with Rational PurifyPlus , which allows developers to work in Java , Visual C # and VisualBasic .Net.\n1\t1439663\t1439808\tThe top rate will go to 4.45 percent for all residents with taxable incomes above $ 500,000 .\tFor residents with incomes above $ 500,000 , the income-tax rate will increase to 4.45 percent .\n1\t3147370\t3147525\tThe results appear in the January issue of Cancer , an American Cancer Society journal , being published online today .\tThe results appear in the January issue of Cancer , an American Cancer Society ( news - web sites ) journal , being published online Monday .\n1\t3300040\t3299992\tThe delegates said raising and distributing funds has been complicated by the U.S. crackdown on jihadi charitable foundations , bank accounts of terror-related organizations and money transfers .\tBin Laden ’ s men pointed out that raising and distributing funds has been complicated by the U.S. crackdown on jihadi charitable foundations , bank accounts of terror-related organizations and money transfers .\n0\t524136\t524119\t\" Sanitation is poor ... there could be typhoid and cholera , \" he said .\t\" Sanitation is poor , drinking water is generally left behind . . . there could be typhoid and cholera . \"\n0\t969512\t969295\tThe broader Standard & Poor 's 500 Index .SPX gave up 11.91 points , or 1.19 percent , at 986.60 .\tThe technology-laced Nasdaq Composite Index was down 25.36 points , or 1.53 percent , at 1,628.26 .\n1\t1685339\t1685429\tThe only announced Republican to replace Davis is Rep. Darrell Issa of Vista , who has spent $ 1.71 million of his own money to force a recall .\tSo far the only declared major party candidate is Rep. Darrell Issa , a Republican who has spent $ 1.5 million of his own money to fund the recall .\n1\t1967578\t1967664\tThe decision to issue new guidance has been prompted by intelligence passed to Britain by the FBI in a secret briefing in late July .\tScotland Yard 's decision to issue new guidance has been prompted by new intelligence passed to Britain by the FBI in late July .\n1\t2047034\t2046820\tUnable to find a home for him , a judge told mental health authorities they needed to find supervised housing and treatment for DeVries somewhere in California .\tThe judge had told the state Department of Mental Health to find supervised housing and treatment for DeVries somewhere in California .\n1\t2046630\t2046644\tThe decision came a year after Whipple ended federal oversight of the district 's racial balance , facilities , budget , and busing .\tThe decision came a year after Whipple ended federal oversight of school busing as well as the district 's racial balance , facilities and budget .\n0\t2221603\t2221633\tIn midafternoon trading , the Nasdaq composite index was up 8.34 , or 0.5 percent , to 1,790.47 .\tThe Nasdaq Composite Index .IXIC dipped 8.59 points , or 0.48 percent , to 1,773.54 .\n1\t129995\t129864\tMorgan Stanley raised its rating on the beverage maker to \" overweight \" from \" equal-weight \" saying in part that pricing power with its bottlers should improve in 2004 .\tMorgan Stanley raised its rating on the company to \" overweight \" from \" equal-weight , \" saying the beverage maker 's pricing power with bottlers should improve in 2004 .\n0\t919683\t919782\tThe pound also made progress against the dollar , reached fresh three-year highs at $ 1.6789 .\tThe British pound flexed its muscle against the dollar , last up 1 percent at $ 1.6672 .\n0\t970740\t971209\tFriday , Stanford ( 47-15 ) blanked the Gamecocks 8-0 .\tStanford ( 46-15 ) has a team full of such players this season .\n1\t2745055\t2745022\tLast month Intel raised its revenue guidance for the quarter to between $ 7.6 billion and $ 7.8 billion .\tAt the end of the second quarter , Intel initially predicted sales of between $ 6.9 billion and $ 7.5 billion .\n0\t2199097\t2199072\tThe driver , Eugene Rogers , helped to remove children from the bus , Wood said .\tAt the accident scene , the driver was \" covered in blood \" but helped to remove children , Wood said .\n1\t1609290\t1609098\tONG KONG , July 9 Tens of thousands of demonstrators gathered tonight before the legislature building here to call for free elections and the resignation of Hong Kong 's leader .\tTens of thousands of demonstrators gathered yesterday evening to stand before this city 's legislature building and call for free elections and the resignation of Hong Kong 's leader .\n1\t1597193\t1597119\tSaddam loyalists have been blamed for sabotaging the nation 's infrastructure , as well as frequent attacks on U.S. soldiers .\tHussein loyalists have been blamed for sabotaging the nation 's infrastructure and attacking US soldiers .\n1\t2758944\t2758975\tIts closest living relatives are a family frogs called sooglossidae that are found only in the Seychelles in the Indian Ocean .\tIts closest relative is found in the Seychelles Archipelago , near Madagascar in the Indian Ocean .\n0\t2584416\t2584653\tCooley said he expects Muhammad will similarly be called as a witness at a pretrial hearing for Malvo .\tLee Boyd Malvo will be called as a witness Wednesday in a pretrial hearing for fellow sniper suspect John Allen Muhammad .\n1\t86007\t86373\t\" Instead of pursuing the most imminent and real threats - international terrorists , \" Graham said , \" this Bush administration chose to settle old scores . \"\t\" Instead of pursuing the most imminent and real threats - international terrorists - this Bush administration has chosen to settle old scores , \" Graham said .\n1\t1602860\t1602844\tHe said they lied on a sworn affidavit that requires them to list prior marriages .\tMorgenthau said the women , all U.S. citizens , lied on a sworn affidavit that requires them to list prior marriages .\n1\t1201306\t1201329\tThe association said 28.2 million DVDs were rented in the week that ended June 15 , compared with 27.3 million VHS cassettes .\tThe Video Software Dealers Association said 28.2 million DVDs were rented out last week , compared to 27.3 million VHS cassettes .\n0\t461779\t461815\tWith these assets , Funny Cide has a solid chance to become the first Triple Crown winner since Affirmed in 1978 .\tFunny Cide is looking to become horse racing 's first Triple Crown winner in a generation .\n1\t1438666\t1438643\tIntel was disappointed and assessing its \" options in the event Mr. Hamidi resumes his spamming activity against Intel , \" spokesman Chuck Mulloy said .\tIntel spokesman Chuck Mulloy said the company was disappointed and assessing its \" options in the event Mr. Hamidi resumes his spamming activity against Intel . \"\n1\t3261484\t3261306\tMr Annan also warned the US should not use the war on terror as an excuse to suppress \" long-cherished freedoms \" .\tAnnan warned that the dangers of extremism after September 11 should not be used as an excuse to suppress \" long-cherished \" freedoms .\n1\t1277539\t1277527\tAt community colleges , tuition will jump to $ 2,800 from $ 2,500 .\tCommunity college students will see their tuition rise by $ 300 to $ 2,800 or 12 percent .\n1\t3035788\t3035918\tHe made a point of saying during Tuesdays debate that the Confederate flag was a racist symbol .\tThough Dean made a point of saying during the debate that the Confederate flag is a racist symbol .\n0\t132553\t132725\tBush wanted \" to see an aircraft landing the same way that the pilots saw an aircraft landing , \" White House press secretary Ari Fleischer said yesterday .\tOn Tuesday , before Byrd 's speech , Fleischer said Bush wanted ' ' to see an aircraft landing the same way that the pilots saw an aircraft landing .\n0\t2259788\t2259747\tOn Monday the Palestinian Prime Minister , Mahmoud Abbas , will report to the Palestinian parliament on his Government 's achievements in its first 100 days in office .\tPalestinian Prime Minister Mahmoud Abbas must defend the record of his first 100 days in office before Parliament today as the death toll in the occupied territories continues to rise .\n0\t2307064\t2307235\tThe civilian unemployment rate improved marginally last month -- slipping to 6.1 percent -- even as companies slashed payrolls by 93,000 .\tThe civilian unemployment rate improved marginally last month _ sliding down to 6.1 percent _ as companies slashed payrolls by 93,000 amid continuing mixed signals about the nation 's economic health .\n1\t3046488\t3046824\tPer-user pricing is $ 29 for Workplace Messaging , $ 89 for Team Collaboration and $ 35 for Collaborative Learning .\tWorkplace Messaging is $ 29 , Workplace Team Collaboration is $ 89 , and Collaborative Learning is $ 35 .\n1\t86020\t86007\t\" Instead of pursuing the most imminent and real threats – international terrorism – this Bush administration chose to settle old scores , \" Mr. Graham said .\t\" Instead of pursuing the most imminent and real threats - international terrorists , \" Graham said , \" this Bush administration chose to settle old scores . \"\n0\t1100998\t1100441\tSARS has killed about 800 people and affected more than 8400 since being detected in China in November .\tSARS has killed about 800 people and sickened more than 8,400 worldwide , mostly in Asia .\n1\t2268396\t2268480\tAuthorities had no evidence to suggest the two incidents were connected .\tThere was no immediate evidence that the two incidents were connected , police said .\n0\t1984039\t1983986\t\" Jeremy 's a good guy , \" Barber said , adding : \" Jeremy is living the dream life of the New York athlete .\tHe also said Shockey is \" living the dream life of a New York athlete .\n0\t2697659\t2697747\tRatliff 's daughters , Margaret and Martha Ratliff , were adopted by Peterson after their mother 's death .\tPeterson helped raise Ratliff 's two daughters , Margaret and Martha Ratliff , who supported him throughout the trial .\n0\t2175939\t2176090\tAfter losing as much as 84.56 earlier , the Dow Jones industrial average closed up 22.81 , or 0.2 percent , at 9,340.45 .\tIn midday trading , the Dow Jones industrial average lost 68.84 , or 0.7 percent , to 9,248.80 .\n1\t886618\t886456\tRumsfeld , who has been feuding for two years with Army leadership , passed over nine active-duty four-star generals .\tRumsfeld has been feuding for a long time with Army leadership , and he passed over nine active-duty four-star generals .\n1\t588637\t588864\tConsumers who said jobs are difficult to find jumped from 29.4 to 32.6 , while those claiming work was plentiful slipped from 13 to 12.6 .\tConsumers who said jobs are difficult to find jumped to 32.6 from 29.4 , while those saying work was plentiful slipped to 12.6 from 13 in April .\n0\t2252795\t2252970\tHe has no immediate plans for television advertising , believing it is unnecessary this early .\tA Lieberman aide said there were no immediate plans for television advertising .\n1\t1756329\t1756394\t\" I think it happened very quickly , \" Houston Police Department homicide investigator Phil Yochum said of the crime .\t\" I think it happened very quickly , \" said Investigator Phil Yochum of the Houston Police Department 's homicide division .\n1\t1673112\t1673068\tUnited issued a statement saying it will \" work professionally and cooperatively with all its unions . \"\tSenior vice president Sara Fields said the airline \" will work professionally and cooperatively with all our unions . \"\n1\t2357324\t2357271\t\" But they never climb out of the pot of beer again . \"\tIt 's just that they never climb out of the beer again . \"\n1\t780408\t780363\tChief financial officer Andy Bryant has said that hike had a greater affect volume than officials expected .\tBryant has said that hike had a greater effect on demand than officials expected .\n1\t821523\t821385\tRobert Liscouski , the Assistant Secretary of Homeland Security for Infrastructure Protection , will oversee NCSD .\tNCSD 's chief will be Robert Liscouski , the assistant secretary of Homeland Security for Infrastructure Protection .\n1\t2304696\t2304863\tHP 's shipments increased 48 percent year-over-year , compared to an increase of 31 percent for Dell .\tHPs shipments increased 48 per cent year-on-year , compared to an increase of 31 per cent for Dell .\n1\t2531749\t2531607\tChirac , who can pardon a law-breaker , refused Humbert 's request last year but kept in close touch with the family .\tChirac , who has the authority to pardon law-breakers , refused Humbert 's request to be allowed to die last year but kept in close touch with the family .\n1\t3180014\t3179967\tThe charges allege that he was part of the conspiracy to kill and kidnap persons in a foreign country .\tThe government now charges that Sattar conspired with Rahman to kill and kidnap individuals in foreign countries .\n1\t726966\t726945\tIn the 2002 study , the margin of error ranged from 1.8 to 4.4 percentage points .\tIt has a margin of error of plus or minus three to four percentage points .\n1\t2638861\t2638982\tMr. Clinton 's national security adviser , Sandy Berger , said that the White House wasn 't informed of the FBI activities .\tClinton ’ s national security adviser , Sandy Berger , said in an interview that the White House was not informed of the FBI activities .\n1\t2495223\t2495307\t\" This decision is clearly incorrect , \" FTC Chairman Timothy Muris said in a written statement .\tThe decision is \" clearly incorrect , \" FTC Chairman Tim Muris said .\n1\t55187\t54831\tProsecutors allege that Nichols and co-conspirator Timothy McVeigh worked together to prepare a bomb that destroyed the Alfred P. Murrah Federal Building .\tProsecutors allege that Nichols and coconspirator Timothy McVeigh worked together to prepare a 4,000-pound fuel-and-fertilizer bomb that destroyed the Murrah building .\n0\t2763381\t2763517\tTerri Schiavo , 39 , is expected to die sometime in the next two weeks in the Tampa-area hospice where she has spent the past several years .\tTerri Schiavo , 39 , underwent the procedure at the Tampa Bay area hospice where she has been living for several years , said her father , Bob Schindler .\n1\t1990975\t1991132\tSecretary of State Colin Powell designated the Chechen leader believed responsible for last year 's hostage standoff in a Moscow theater as a threat to U.S. security Friday .\tU.S. Secretary of State Colin Powell on Friday designated Chechen rebel leader Shamil Basayev a threat to the security of the United States and to U.S. citizens .\n1\t2204353\t2204418\t\" Today , we are trying to convey this problem to Russian President Vladimir Putin and US President George W Bush . \"\t\" Today , we are trying to convey this problem to Russian President Vladimir Putin ( news - web sites ) and President Bush ( news - web sites ) . \"\n1\t60122\t60445\tThat would be a potential setback to Chief Executive Phil Condit 's strategy of bolstering defense-related sales during a slump in jetliner deliveries .\tThe inquiry may hinder Chief Executive Phil Condit 's strategy of bolstering defense-related sales during a slump in jetliner deliveries .\n1\t961836\t962243\tPeopleSoft also said its board had officially rejected Oracle 's offer .\tThursday morning , PeopleSoft 's board rejected the Oracle takeover offer .\n0\t3140260\t3140288\tThe Dow Jones industrial average ended the day down 10.89 at 9,837.94 , after advancing 111.04 Wednesday .\tThe Dow Jones industrial average fell 10.89 points , or 0.11 percent , to 9,837.94 .\n1\t1720166\t1720115\tCortisol levels in the saliva of day care children were highest and rose most steeply in those judged by day care center personnel to be the shyest .\tCortisol levels in the saliva of day-care children were highest and rose most steeply in those whom day-care centre staffed judged to be the shyest .\n1\t2573262\t2573319\t\" The idea that Tony Abbott is in some way a one-dimensional political head-kicker couldn 't be more wrong , \" Mr Howard said .\t\" The idea that Tony Abbott is in some way a one-dimensional political head kicker couldn 't be more wrong . \"\n0\t1353356\t1353174\t\" Biotech products , if anything , may be safer than conventional products because of all the testing , \" Fraley said , adding that 18 countries have adopted biotechnology .\t\" Biotech products , if anything , may be safer than conventional products because of all the testing , \" said Robert Fraley , Monsanto 's executive vice president .\n1\t2738677\t2738741\tThe rate of skin cancer has tripled since the 1950s in Norway and Sweden , according to the study .\tThe study also found that skin cancer nearly tripled in Norway and Sweden since the 1950s .\n1\t1638813\t1639087\tWe acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , \" Rumsfeld said .\tRather , the US acted because the administration saw \" existing evidence in a new light , through the prism of our experience on September 11 \" .\n1\t1605350\t1605425\tTrans fat makes up only 1 percent to 3 percent of the total fat Americans consume , compared with 14 percent for saturated fat .\tTrans fat accounts for 2.5 percent of Americans ' daily calories , compared to 11 percent to 12 percent for saturated fat .\n1\t2494149\t2494073\tHowever , a recent slide in prices and OPEC 's expectations of a surge in oil inventories have compounded its fears about a further softening of the market .\tA 14 percent slide in crude prices this month and expectations of a build up in oil inventories compounded OPEC 's fears of a further softening of the market .\n1\t3023029\t3023229\tPeterson , 31 , is now charged with murder in the deaths of his 27-year-old wife and their unborn son .\tPeterson , 31 , is charged with two counts of first-degree murder in the slayings of his wife , Laci , and their unborn son , Conner .\n1\t1351550\t1351155\tCarlson on Tuesday said he would not recuse himself from the case .\tService officials said Carlson refused to recuse himself from the case .\n1\t981185\t981234\tThe program will grow to include ports in Dubai , Turkey and Malaysia , among others .\tThe program will be expanded to include areas of the Middle East such as Dubai , Turkey and Malaysia , Mr. Ridge said .\n0\t2111629\t2111786\tMcCabe said he was considered a witness , not a suspect .\t\" He is not considered a suspect , \" McCabe said .\n1\t655498\t655391\tThe woman was exposed to the SARS virus while in the hospital but was not a health care worker , said Dr. Colin D ’ Cunha , Ontario ’ s commissioner of public health .\tThe woman was exposed to the SARS virus while in the hospital but was not a health-care worker , said Dr Colin D 'Cunha , Ontario 's commissioner of public health .\n1\t533823\t533909\tHe added that those \" are not solely American principles , nor are they exclusively Western . \"\t\" These are not solely American principles nor are they exclusively Western , \" Rumsfeld said .\n1\t581592\t581570\t\" If we don 't march into Tehran , I think we will be in pretty good shape , \" he said .\t\" As long as we don 't march on Tehran , I think we are going to be in pretty good shape , \" he said .\n0\t1010655\t1010430\tOn Saturday , a 149mph serve against Agassi equalled Rusedski 's world record .\tOn Saturday , Roddick equalled the world record with a 149 m.p.h. serve in beating Andre Agassi .\n1\t2241925\t2242066\tChad Kolton , emergency management spokesman with the Department of Homeland Security , said the government is open to new technologies and methods to communicate more quickly and efficiently .\tChad Kolton , emergency management spokesman with the Department of Homeland Security , said the government is open to new ways to communicate .\n1\t2796978\t2797024\t\" APEC leaders are painfully aware that security and prosperity are inseparable , \" Thai Prime Minister Thaksin Shinawatra told business leaders .\t\" APEC leaders are painfully aware that security and prosperity are inseparable , \" Thaksin said .\n0\t101746\t101775\tDanbury prosecutor Warren Murray could not be reached for comment Monday .\tProsecutors could not be reached for comment after the legal papers were obtained late Monday afternoon .\n1\t327839\t327748\tWittig resigned last year after being indicted on federal bank fraud charges involving a real estate loan unrelated to Westar business .\tWittig resigned in late November about two weeks after being indicted on bank fraud charges in a real estate case unrelated to the company .\n0\t2988297\t2988555\tShattered Glass , \" starring Hayden Christensen as Stephen Glass , debuted well with $ 80,000 in eight theaters .\t\" Shattered Glass \" _ starring Hayden Christensen as Stephen Glass , The New Republic journalist fired for fabricating stories _ debuted well with $ 80,000 in eight theaters .\n1\t2217613\t2217659\tHe was arrested Friday night at an Alpharetta seafood restaurant while dining with his wife , singer Whitney Houston .\tHe was arrested again Friday night at an Alpharetta restaurant where he was having dinner with his wife .\n0\t2128530\t2128455\tHowever , EPA officials would not confirm the 20 percent figure .\tOnly in the past few weeks have officials settled on the 20 percent figure .\n1\t2208376\t2208198\tUniversity of Michigan President Mary Sue Coleman said in a statement on the university 's Web site , \" Our fundamental values haven 't changed .\t\" Our fundamental values haven 't changed , \" Mary Sue Coleman , president of the university , said in a statement in Ann Arbor .\n1\t1980654\t1980641\tThe first products are likely to be dongles costing between US $ 100 and US $ 150 that will establish connections between consumer electronics devices and PCs .\tThe first products will likely be dongles costing $ 100 to $ 150 that will establish connections between consumer electronics devices and PCs .\n0\t589579\t589557\tHowever , Lapidus expects foreign brands ' sales to be up 4 percent , driven by strong truck sales at Honda Motor Co .\tLapidus expects Ford to be down 5 percent , Chrysler down 10 percent and foreign brands up 4 percent driven by strong truck sales at Honda .\n1\t1636060\t1635946\tMichel , who remains in the government , denied that US pressure had provoked the government 's move .\tMichel , who has stayed in the new government , denied that it was U.S. pressure which had provoked the government 's move .\n1\t1630585\t1630657\tSome of the computers also are used to send spam e-mail messages to drum up traffic to the sites .\tSome are also used to send spam e-mail messages to boost traffic to the sites .\n0\t447728\t447699\tIndonesia 's army has often been accused of human rights abuses during GAM 's battle for independence , charges it has generally denied while accusing the separatists of committing rights violations .\tIndonesia 's army has been accused of human rights abuses during its earlier battles with GAM , charges it has generally denied .\n1\t1606495\t1606619\tBush also hoped to polish his anti-AIDS credentials in Uganda , which has been hailed as an African pioneer in fighting the killer disease .\tPresident Bush flies to Uganda Friday hoping to polish his anti- AIDS credentials in a country hailed as an African pioneer in fighting the epidemic .\n1\t1550897\t1550977\tLater this year , the command will send trainers with soldiers from four North African nations on patrolling and intelligence gathering missions .\tThis fall the command will send trainers to work with soldiers from four North African nations on patrolling and gathering intelligence .\n0\t490376\t490490\tThe reports helped overcome investor jitters after the euro briefly hit an all-time high against the dollar Tuesday .\tStocks slipped at the open after the euro hit record highs against the dollar .\n1\t3084554\t3084612\tSales for the quarter beat expectations , rising 37 percent year-on-year to 1.76 billion euros .\tSales rose 37 per cent year-on-year to 1.76bn , beating expectations .\n1\t315647\t315778\tIf the MTA 's appeal to a higher court is successful , the $ 2 bus and subway base fare won 't be rolled back .\tIf the MTA 's appeal is successful , the $ 2 bus and subway base fare won 't change .\n1\t3428298\t3428362\tRobert Walsh , 40 , remained in critical but stable condition Friday at Staten Island University Hospital 's north campus .\tWalsh , also 40 , was in critical but stable condition at Staten Island University Hospital last night .\n1\t2523564\t2523358\tThe Guru microcontroller serves four functions : hardware monitoring , overclocking management , BIOS ( Basic Input Output System ) update and a troubleshooting-assistance feature called Black Box .\tThe µGuru microcontroller serves four functions : hardware monitoring , overclocking management , BIOS update and a troubleshooting-assistance feature called Black Box .\n1\t2079200\t2079131\tU.S. corporate bond yield spreads tightened in spotty trading on Friday as Wall Street labored to get back on its feet after the largest power outage ever in North America .\tU.S. stocks rose slightly on feather-light volume on Friday , as Wall Street regrouped after the biggest-ever power outage in North America .\n1\t818091\t817811\tThe company said it would issue revised guidance for the full fiscal year next month when it releases its Q2 results .\tThe company said it would renew its guidance for 2003 when it announces its second quarter results in mid-July .\n1\t1580638\t1580663\t\" I stand 100 percent by it , and I think our intelligence services gave us the correct information at the time . \"\tI stand 100 percent by it , and I think that our intelligence services gave us the correct intelligence and information at the time , \" Blair said .\n0\t1919740\t1919926\t\" I don 't know if the person I 'm talking to now may end up being someone else at another time that may not follow the rules , \" Parrish said .\t\" I don 't know whether the person I 'm talking to now may end up being someone else , \" Parrish said .\n1\t2748287\t2748550\t\" I think it 's going to be a close vote , but I think the grant proposal is going to win , \" McConnell said .\t\" I think it 's going to be a close vote , but I think the grant proposal 's going to win , \" said Sen. Mitch McConnell , assistant majority leader .\n1\t3394891\t3394775\tTwenty-eight people were believed to have been spending Christmas Day with the caretaker of the St Sophia 's camp , when the mudslide smashed into two cabins .\tTwenty-seven people were believed to have been spending Christmas Day with the caretaker of Saint Sophia Camp , a Greek Orthodox facility , when the mudslide roared through .\n0\t2963943\t2963880\tOne , Capt. Doug McDonald , remained hospitalized in critical condition on Thursday .\tHer 20-year-old sister , Allyson , was severely burned and remained hospitalized in critical condition .\n0\t1865364\t1865251\tThe United States finally relented during President Bush 's visit to Africa earlier this month .\tDuring President Bush 's trip to Africa earlier this month , however , Washington said it would support the increase .\n1\t263690\t263819\t\" There is no conscious policy of the United States , I can assure you of this , to move the dollar at all , \" he said .\tHe also said there is no conscious policy by the United States to move the value of the dollar .\n1\t283751\t283290\tIt 's the first such drill since the September 11 terrorist attacks on New York and Washington .\tIt is the nation 's first large-scale counterterrorism exercise since the Sept . 11 terrorist attacks .\n1\t2517014\t2516995\tMyanmar 's pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital , her personal physician said .\tMyanmar 's pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery , her personal physician said Friday .\n1\t1330643\t1330622\tAccording to the Merchant Marine Ministry , the 37-year-old ship is registered to Alpha Shipping Inc. based in the Pacific Ocean nation of Marshall Islands .\tThe Baltic Sky is a 37-year-old ship registered to Alpha Shipping Inc. based in the Pacific Ocean nation of Marshall Islands .\n1\t3111452\t3111428\tIn an unusual move , the U.S. Patent and Trademark Office is reconsidering a patent affecting Internet pages that critics contend could disrupt millions of Web sites .\tIn an unusual move that critics contend could disrupt millions of Web sites , the U.S. Patent and Trademark Office is reconsidering a patent affecting Internet pages .\n0\t1167835\t1167651\tKansas Department of Health and Environment records show there were 88 abortions performed on girls age 14 and younger last year .\tStatistics from the Kansas Department of Health and Environment show that 11,844 abortions were performed in the state last year .\n0\t1423836\t1423708\tA European Union spokesman said the Commission was consulting EU member states \" with a view to taking appropriate action if necessary \" on the matter .\tLaos 's second most important export destination - said it was consulting EU member states ' ' with a view to taking appropriate action if necessary ' ' on the matter .\n1\t2090911\t2091154\tWaiting crowds filling the streets on both sides overwhelmed the peacekeepers soon after daylight , sweeping past the barbed wire barricades .\tBut waiting crowds filling the streets rushed the bridges soon after daylight , overrunning razor-wire barricades .\n1\t2265271\t2265152\tBarry Callebaut will be able to use Brach 's retail network to sell products made from its German subsidiary Stollwerck , which makes chocolate products not sold in the United States .\tBarry Callebaut will be able to use Brach 's retail network to sell products made from its German subsidiary Stollwerck , which makes chocolate products unknown to the American market .\n1\t3062202\t3062308\tBy skirting the FDA 's oversight , Eagan said , the quality of the imported drugs is \" less predictable \" than for those obtained in the United States .\tBy skirting the FDA 's oversight , Eagan said the quality of the imported drugs is \" less predictable \" than U.S. drugs .\n1\t2155514\t2155377\tHe said : \" For the first time there is an easy and affordable way of making this treasure trove of BBC content available to all . \"\t\" For the first time , there is an easy and affordable way of making this treasure trove of BBC content available to all , \" Dyke said .\n1\t1552068\t1551928\tThree such vigilante-style attacks forced the hacker organizer , who identified himself only as \" Eleonora [ 67 ] , \" to extend the contest until 7 p.m. EST Sunday .\tThree such vigilante-style attacks forced the hacker organiser , who identified himself only as \" Eleonora67 ] , \" to extend the contest until 8am ( AEST ) today .\n1\t936978\t937500\tEric Gagne pitched a perfect ninth for his 23rd save in as many opportunities .\tGagne struck out two in a perfect ninth inning for his 23rd save .\n0\t985015\t984975\tOne way or another , Harry Potter And The Order Of The Phoenix will be in your hands by Saturday .\tJust about everything about \" Harry Potter and the Order of the Phoenix \" will set records .\n1\t1430357\t1430425\t\" Allison just proves you don 't need to wait until August or September to have a disaster , \" said Josh Lichter , a meteorologist with the Houston-Galveston weather office .\t\" Allison just proves you don 't need to wait until August or September to have a disaster , \" Lichter said .\n1\t3039310\t3039413\tToday , analysts say , UN members can no longer ignore the shifts since the September 11 2001 attacks .\tOn Wednesday , analysts say , UN members can no longer ignore the shifts since the attacks in the US of September 11 2001 .\n1\t34513\t34742\tPolice say CIBA was involved in the importation of qat , a narcotic substance legal in Britain but banned in the United States .\tMr McKinlay said that CIBA was involved in the importation of qat , a narcotic substance legal in Britain but banned in the US .\n1\t368067\t368018\tChiron already has nearly 20 percent acceptances from PowderJect 's shareholders .\tChiron has acceptances from holders of nearly 20 percent of PowderJect shares .\n0\t611663\t611716\tErnst & Young has denied any wrongdoing and plans to fight the allegations .\tErnst & Young has denied the SEC 's claims , and called its recommendations \" irresponsible \" .\n1\t98432\t98657\tThe attack followed several days of disturbances in the city where American soldiers exchanged fire with an unknown number of attackers as civilians carried out demonstrations against the American presence .\tThe attack came after several days of disturbance in the city in which U.S. soldiers exchanged fire with an unknown number of attackers as civilians protested the American presence .\n1\t3039007\t3038845\tNo company employee has received an individual target letter at this time .\tShe said no company official had received \" an individual target letter at this time . \"\n1\t1708040\t1708062\tSecond-quarter results reflected a gain of 10 cents per diluted share , while the 2002 results included a loss of 19 cents per diluted share .\tThe second-quarter results had a non-operating gain of 10 cents a share while the 2002 second-quarter performance had a net non-operating loss of 19 cents a share .\n0\t1757264\t1757375\tHe allegedly told his ex-wife in an angry phone call that he had no intention of following their new custody agreement .\tThe two had battled over custody and he allegedly told her in an angry phone call that he had no intention of following their new custody agreement .\n1\t383417\t383558\tWorldwide , more than 50 million people have seen \" Les Miz , \" with gross receipts of $ 1.8 billion .\tWorldwide , Les Misérables has been seen by over 50 million people , with a total gross of over $ 2 billion .\n0\t2766112\t2766084\tIn fiction : Edward P. Jones ( \" The Known World \" ) and Scott Spencer ( \" A Ship Made of Paper \" ) .\tThe fifth nominee for fiction is Scott Spencer , for A Ship Made of Paper .\n1\t1261116\t1261234\t\" Overwhelmingly the Windows brand really resonated with them . \"\t\" Windows was the part of the experience that really resonated with people . \"\n1\t3028143\t3028234\tThe Centers for Medicare and Medicaid Services , the federal agency that runs Medicare , last year began a similar effort for nursing homes .\tThe Centers for Medicare and Medicaid launched a similar consumer tool for nursing homes last year .\n0\t249699\t249623\tVivace was founded in 1999 and has raised over $ 118 million in three rounds of venture financing .\tDuring difficult times for technology venture capital , Vivace raised over $ 118 million in three rounds of venture financing .\n0\t3448488\t3448449\tThe Dow Jones industrial average < .DJI > added 28 points , or 0.27 percent , at 10,557 , hitting its highest level in 21 months .\tThe Dow Jones industrial average < .DJI > rose 49 points , or 0.47 percent , to 10,578 .\n1\t2749322\t2749663\tThe Democratic candidates also began announcing their fund-raising totals before Wednesday 's deadline to file quarterly reports with the Federal Election Commission .\tThe Democratic candidates also began announcing their fund-raising totals in advance of the deadline today to file quarterly reports with the Federal Election Commission .\n0\t2204592\t2204588\tSun Microsystems Inc. on Thursday said it had added 100 new third-party systems and 100 new components to its Hardware Compatibility List for the Solaris x86 operating system Platform Edition .\tThe vendor has added 100 new third-party systems and 100 new components to the operating system 's Hardware Compatibility List ( HCL ) .\n1\t2889005\t2888954\tProsecutors said PW Marketing violated the state 's 1998 anti-spam law by sending unsolicited e-mail without a toll-free number for recipients to call to stop additional mailings .\tProsecutors said PW Marketing violated the 1998 anti-spam law because these unsolicited e-mails were sent without a free call number for recipients to phone to stop additional mailings .\n0\t1657632\t1657619\tThe Neighbours star and singer spent yesterday resting at her family home in Sydney and will have more tests today .\tGoodrem spent yesterday resting in her family home in Sydney and will have more tests today to determine her exact treatment .\n0\t555617\t555528\tThe 3 rd Armored Cavalry Regiment is 5,200 strong and the largest combat unit at Fort Carson .\tBroomhead , 34 , was assigned to the 2nd Squadron , 3rd Armored Cavalry Regiment .\n1\t2396937\t2396818\t\" The risk of inflation becoming undesirably low remains the predominant concern for the foreseeable future , \" the Fed said in a statement accompanying the unanimous decision .\t\" The risk of inflation becoming undesirably low remains the predominant concern for the foreseeable future , \" the policy-setting Federal Open Market Committee said .\n0\t2339738\t2339771\t\" It is bad for Symbian , \" said Per Lindberg , analyst at Dresdner Kleinwort Wasserstein .\t\" Motorola has displayed clear disloyalty \" to Symbian , said Per Lindberg , an analyst at Dresdner Kleinwort Wasserstein in London .\n0\t1616174\t1616206\tBob Richter , a spokesman for House Speaker Tom Craddick , had no comment about the ruling .\tBob Richter , spokesman for Craddick , R-Midland , said the speaker had not seen the ruling and could not comment .\n1\t635783\t635802\tBut Ms Ward said the headroom under its financial covenants was \" tight \" and that there could be another downgrade if Southcorp breached any of its banking covenants .\tBut Ms Ward said the headroom under its financial covenants was \" tight \" and that there could be a rating downgrade if Southcorp did breach any banking covenants .\n1\t3444633\t3444733\tHe added : ``I 've never heard of more reprehensiblebehaviour by a doctor .\tThe Harrisons ’ lawyer Paul LiCalsi said : “ I ’ ve never heard of more reprehensible behaviour by a doctor .\n1\t555553\t555528\tBroomhead was assigned to 2nd Squadron , 3rd Armor Cavalry Regiment , based at Fort Carson .\tBroomhead , 34 , was assigned to the 2nd Squadron , 3rd Armored Cavalry Regiment .\n1\t1112021\t1111925\tOther staff members , however , defended the document , saying it would still help policy-makers and the agency improve efforts to address the climate issue .\tSome E.P.A. staff members defended the document , saying that although pared down it would still help policy makers and the agency address the climate issue .\n0\t2749410\t2749625\tPresident Bush raised a record-breaking $ 49.5 million for his re-election campaign over the last three months , with contributions from 262,000 Americans , the president 's campaign chairman said Tuesday .\tPresident Bush has raised $ 83.9 million since beginning his re-election campaign in May , and has $ 70 million of that left to spend , his campaign said Tuesday .\n1\t1629064\t1629043\tAn episode is declared when the ozone reaches .20 parts per million parts of air for one hour .\tA Stage 1 episode is declared when ozone levels reach 0.20 parts per million .\n1\t789691\t789665\t\" He may not have been there , \" the defence official said on Thursday .\t\" He may not have been there , \" said a defence official speaking on condition of anonymity .\n1\t844421\t844679\tThe U.N. troops are in Congo to protect U.N. installations and personnel , and they can only fire in self defense and have been unable to stem the violence .\tThe troops - whose mandate is to protect U.N. installations and personnel - can only fire in self-defense and have been unable to stem the violence .\n1\t58540\t58567\tNorth American markets grabbed early gains Monday morning , as earnings season begins to slow and economic indicators take the spotlight .\tNorth American futures pointed to a strong start to the first trading session of the week Monday , as earnings season slows and economic indicators take the spotlight .\n1\t781439\t781461\tXerox itself paid a $ 10 million fine last year to settle similar SEC charges .\tXerox itself previously paid a $ 10-million penalty to settle the SEC accusations .\n1\t1909579\t1909408\t\" This deal makes sense for both companies , \" said National Chief Executive Brian Halla .\t\" This deal makes sense for both companies , \" Halla said in a prepared statement .\n0\t787432\t787464\tThe blasts killed two people and injured more than 150 others .\tThe Atlanta Olympic Games attack killed one woman and injured more than 100 other people .\n0\t52758\t52343\tMorrill 's wife , Ellie , sobbed and hugged Bondeson 's sister-in-law during the service .\tAt the service Morrill 's widow , Ellie , sobbed and hugged Bondeson 's sister-in-law as people consoled her .\n1\t1675025\t1675047\tSpansion products are to be available from both AMD and Fujitsu , AMD said .\tSpansion Flash memory solutions are available worldwide from AMD and Fujitsu .\n1\t2131318\t2131372\tAbout 1,500 police will be deployed for the visit .\tAround 1,500 police are to be deployed at Niigata for the ferry 's visit .\n1\t325763\t325928\tGamarekian told The News she remembers only the woman 's first name - and refused to reveal it .\tShe told the New York Daily News she remembers only the intern 's first name , which she refused to reveal .\n1\t2638975\t2638855\tOne of the FBI ’ s key operatives , who had a falling out with the bureau , provided an account of the operation at a friend ’ s closed immigration court proceeding .\tOne of the FBI 's key operatives , who has had a falling-out with the bureau , provided an account of the operation at a friend 's closed immigration court proceeding .\n1\t2198694\t2198937\tA nationally board certified teacher with a master 's degree , Kelley makes a salary of $ 65,000 in his 30th year .\tA nationally board certified teacher with a master 's degree , Kelley , in his 30th year teaching , makes $ 65,000 .\n1\t1825432\t1825301\tA man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $ 100,000 bail during an early morning court appearance Saturday .\tThe Queens man arrested for allegedly threatening to shoot City Councilman Hiram Monserrate was held on $ 100,000 bail Saturday , a spokesman for the Queens district attorney said .\n1\t2906104\t2906322\tThey were being held Sunday in the Camden County Jail on $ 100,000 bail .\tThey remained in Camden County Jail on Sunday on $ 100,000 bail .\n1\t722278\t722383\tMs Stewart , the chief executive , was not expected to attend .\tMs Stewart , 61 , its chief executive officer and chairwoman , did not attend .\n0\t101747\t101777\tChristina 's aunt , Shelley Riling , said the defense 's claims were preposterous .\tChristina 's aunt , Shelley Riling , said she will address the court .\n1\t2224884\t2224819\tThe Justice Department Aug. 19 gave pre-clearance for the Oct. 7 date for the election to recall Gov. Gray Davis , saying it would not affect minority voting rights .\tThe Justice Department on Aug. 19 sanctioned the Oct. 7 date for recall election , saying it would not affect voting rights .\n0\t977938\t978162\tLord Falconer hailed the changes as \" a new beginning as far as the courts , Crown Prosecution Service and police are concerned \" .\t\" It 's a new beginning as far as the courts , Crown Prosecution Service and police are concerned , making the criminal justice system work better . \"\n0\t1015010\t1014963\tGE stock closed at $ 30.65 a share , down about 42 cents , on the New York Stock Exchange .\tGE 's shares closed at $ 30.65 on Friday on the New York Stock Exchange .\n1\t1513190\t1513246\tAt least 27 US troops have been killed in hostile fire since Bush 's statement .\tAt least 26 American troops have been killed in hostile fire since major combat was officially declared over on May 1 .\n1\t2385348\t2385394\tA recent poll showed Edwards with a narrow lead in South Carolina , and he plans a rally there later on Tuesday .\tA recent poll showed Edwards in a virtual four-way tie at the top in South Carolina , and he plans a rally there later on Tuesday .\n1\t2317018\t2317252\tNovember 17 's last victim was British defence attache Stephen Saunders , who was shot on an Athens road in June 2000 .\tNovember 17 's last victim was British defense attache Stephen Saunders , who was shot and killed at point-blank range on a busy Athens road in June 2000 .\n0\t1831696\t1831660\tThe agency charged that one WD Energy worker discussed false reporting with traders at two other energy companies .\tThe agency found further that a WD Energy employee discussed false reporting with traders at two other energy companies , which the CFTC didn 't identify .\n1\t1528383\t1528083\tZulifquar Ali , a worshipper slightly wounded by shrapnel , said the assailants first targeted the mosque 's security guards .\tWitness Zulfiqar Ali , who was slightly wounded by shrapnel , said the attackers had focused on the mosque 's guards .\n1\t917965\t918315\tFor the second year in a row , rises in hospital costs accounted for much of the inflation , accounting for 51 percent of the overall cost increase .\tFor the second year in a row , rises in hospital costs dominated the increase , accounting for 51 percent of the overall cost spiral .\n0\t3218713\t3218830\tQ : Can I buy coverage for prescription drugs right away ?\tCongress has added a new benefit - an option to buy insurance coverage for prescription drugs .\n1\t221079\t221003\tThe airline also said it has the option to buy 380 more airplanes , orders that would be split evenly between the two manufacturers .\tThe airline has the option to buy 380 more , split evenly between the two manufacturers .\n1\t2546175\t2546198\tDr Mark McClean , Jonathan 's family doctor , said if the drug had been administered earlier Jonathan would have retained more of his brain functions .\tDr Mark McClean , the family 's GP , said had the drug been administered to Jonathan earlier , he would have retained more of his brain function .\n0\t799346\t799268\tThe chain operates more than 3,400 stores , and has annual revenue of about $ 15.8 billion .\tThe chain , which has been under new management since late 1999 , has more than 3,400 stores and $ 15.8 billion in annual revenue .\n0\t2673104\t2673130\tAll patients developed some or all of the symptoms of E. coli food poisoning : bloody diarrhea , vomiting , abdominal cramping and nausea .\tSymptoms of the E. coli infection include bloody diarrhea , nausea , vomiting and abdominal cramping .\n1\t1354501\t1354476\tFederal regulators have turned from sour to sweet on a proposed $ 2.8 billion merger of ice cream giants Nestle Holdings Inc. and Dreyer 's Grand Ice Cream Inc .\tFederal regulators have changed their minds on a proposed $ 2.8 billion merger of ice cream giants Nestle Holdings and Dreyer 's Grand Ice Cream .\n1\t3070979\t3070949\tEnvironmental campaigners are using this weekend ’ s lunar eclipse to highlight the huge increase in light pollution across the UK .\tEnvironmental campaigners used the eclipse to highlight the surge in light pollution across Britain .\n0\t1264509\t1264471\tAvailable July 7 , the software supports the Solaris , IBM AIX , Red Hat Linux and Windows operating systems .\tThe OpForce product currently works with Solaris , AIX , Red Hat Linux and Windows servers .\n1\t103280\t103431\tJustice Minister Martin Cauchon and Prime Minister Jean Chrétien have both said the Liberal government will introduce legislation soon to decriminalize possession of small amounts of pot for personal use .\tJustice Minister Martin Cauchon and Prime Minister Jean Chretien both have said the government will introduce legislation to decriminalize possession of small amounts of pot .\n0\t110731\t110648\tBut Chauncey Billups demonstrated he 's also capable of big games , scoring 77 points over the final two games against the Magic .\tBillups scored 77 points in the final two games of the first-round series against the Magic .\n1\t2274844\t2274714\tKelly killed himself after being exposed as the source for a BBC report which claimed the government had embellished evidence of Iraq 's banned weapons to justify the war .\tHe killed himself after being exposed as the source for a BBC report which claimed the government exaggerated the case for war against Iraq .\n0\t1050307\t1050144\tAnd it 's going to be a wild ride , \" said Allan Hoffenblum , a Republican consultant .\tNow the rest is just mechanical , \" said Allan Hoffenblum , a Republican consultant .\n1\t2810634\t2810670\tWhile the Ibrahims had one separation operation , Goodrich and Dr. David Staffenberg plan about three for the Aguirres , with several weeks between each .\tInstead of one long operation to separate the twins , Goodrich and Dr. David Staffenberg plan about three , with several weeks between each .\n1\t3073773\t3073779\tLay had contended that turning over the documents would violate his Fifth Amendment right against self-incrimination .\tLay had refused to turn over the papers , asserting his Fifth Amendment right against self-incrimination .\n0\t261202\t260995\tThe WHO experts didn 't say how many cases in Hebei were in rural areas .\tHebei has reported 191 cases and eight deaths , though the WHO experts did not say how many were in rural areas .\n1\t1824224\t1824209\tNearly 300 mutinous troops who seized a Manila shopping and apartment complex demanding the government resign gave up and retreated peacefully after some 19 hours .\tMutinous troops who seized a Manila shopping and apartment complex demanding the government resign ended a 19-hour standoff late Sunday and returned to barracks without a shot fired .\n1\t548867\t548785\tIn three years , Lend Lease has slipped from a top-five stock , when its share price was around $ 24 , to 37th .\tIn the space of three years , Lend Lease has slipped from a top-five 5 stock when its share price hovered around $ 24 to 37th on the list .\n0\t2796658\t2796682\tAbout two hours later , his body , wrapped in a blanket , was found dumped a few blocks away .\tThen his body was dumped a few blocks away , found in a driveway on Argyle Road .\n1\t1808166\t1808434\tColumbia broke up over Texas upon re-entry on Feb. 1 .\tColumbia broke apart in the skies above Texas on Feb. 1 .\n1\t853475\t853342\tA year or two later , 259 , or 10 per cent , of the youths reported that they had started to smoke , or had taken just a few puffs .\tWithin two years , 259 , or 10 percent , of the youths reported they had started to smoke or had at least taken a few puffs .\n0\t977772\t977804\tThe Lord Chancellor was guardian of the Great Seal , used to stamp all official documents from the sovereign .\tFalconer will hold on , for now , to the Lord Chancellor 's Great Seal , used to sign off instructions from the sovereign .\n1\t577854\t578500\tCindy Yeast , a 50-year-old Washington-area publicist , says she began taking supplements two years ago in part to avoid mild dementia that affects her elderly parents .\tShe started taking supplements two years ago - partly to stave off mild dementia that affects her elderly parents .\n1\t2829194\t2829229\tThe two are not related , but have referred to each other as father and son .\tHe 's not related to Malvo , but the two have referred to each other as father and son .\n1\t2074182\t2074668\tGibson said last month in a press statement that \" neither I nor my film are anti-Semitic .\tGibson said in a June statement that he and his film are not anti-Semitic .\n0\t2758265\t2758282\tThe world 's largest software company said it recognized the difficulty the multiple patches posed for companies , and set out to make it easier for them to apply the updates .\tThe world 's largest software company said it recognized the difficulty the multiple patches posed for companies trying to apply them .\n1\t1958079\t1958143\tThe Dow Jones industrial average .DJI ended up 64.64 points , or 0.71 percent , at 9,191.09 , according to the latest available data .\tThe blue-chip Dow Jones industrial average .DJI added 38 points , or 0.42 percent , to 9,165 .\n1\t544217\t544325\tThe vote came just two days after Kurds swept City Council elections , taking the largest single block of votes on the 30-seat council .\tThe vote for mayor followed City Council elections that gave Kurds the largest block of votes on the 30-seat council .\n1\t2385288\t2385256\tLarge swells and dangerous surf already were being felt along sections of the coast .\tAlready large swells and dangerous surf have arrived along the mid-Atlantic .\n0\t2324708\t2325028\tBased on a separate survey of households , the unemployment rate fell in August to 6.1 percent from 6.2 percent .\tLabor Department analysts discounted a slight improvement in the national unemployment rate , which fell in August to 6.1 percent from 6.2 percent .\n1\t2139506\t2139427\t\" We will work with the board to ensure a smooth transition . \"\tHe said federal regulators would work with the corporation to ensure a \" smooth transition . \"\n1\t2965576\t2965701\tGasps could be heard in the courtroom when the photo was displayed .\tGasps could be heard as the photo was projected onto the screen .\n1\t2931098\t2931144\tGilead had earnings of $ 73.1 million , or 33 cents a share , compared with $ 20.8 million , or 10 cents , in the year-ago quarter .\tQuarterly profit climbed to $ 73.1 million , or 33 cents a share , from $ 20.8 million , or 10 cents , a year earlier , the company said .\n0\t644788\t644816\t\" I had one bad stretch of holes that put me out of contention to win , \" Woods said .\t\" I had one bad stretch of holes that put me out of contention , \" Woods said , referring to his 42 on the front nine Saturday .\n0\t2551891\t2551563\tThe poll had a margin of error of plus or minus 2 percentage points .\tIt had a margin of sampling error of plus or minus four percentage points and was conducted Thursday through Saturday .\n1\t1089053\t1089297\tSen. Patrick Leahy of Vermont , the committee 's senior Democrat , later said the problem is serious but called Hatch 's suggestion too drastic .\tSen. Patrick Leahy , the committee 's senior Democrat , later said the problem is serious but called Hatch 's idea too drastic a remedy to be considered .\n1\t3435735\t3435717\tThe broad Standard & Poor 's 500 < .SPX > eased 0.37 of a point , or 0.03 percent , at 1,121 .\tThe Standard & Poor 's 500 Index < .SPX > slipped 0.26 point , or 0.02 percent , to 1,121.96 .\n0\t1954\t2142\tWatertown , Saugus and Framingham also are going smoke-free Monday , joining a growing number of cities around the country .\tAlong with Boston , Watertown , Saugus and Framingham also are going smoke-free Monday .\n1\t3400796\t3400822\tThat is evident from their failure , three times in a row , to get a big enough turnout to elect a president .\tThree times in a row , they failed to get a big _ enough turnout to elect a president .\n1\t1220668\t1220801\tWe firmly believe we have an absolute right to use the common word ' spike ' as the name of our network . \"\tWe firmly believe that we have an absolute right to use the common word ' spike ' to name our network .\n1\t1889954\t1889847\tSources who knew of the bidding said last week that cable TV company Comcast Corp. was also looking at VUE .\tLate last week , sources told Reuters cable TV company Comcast Corp. CMCSA.O also was looking at buying VUE assets .\n1\t315785\t315653\tBut MTA officials appropriated the money to the 2003 and 2004 budgets without notifying riders or even the MTA board members considering the 50-cent hike , Hevesi found .\tMTA officials appropriated the surplus money to later years ' budgets without notifying riders or the MTA board members when the 50-cent hike was being considered , he said .\n0\t1521034\t1520582\tWhite , who had suffered kidney failure from years of high blood pressure , died at Cedars-Sinai Medical Center around 9 : 30 a.m. , said manager Ned Shankman .\tWhite , who had kidney failure from years of high blood pressure , had been undergoing dialysis and had been hospitalized since a September stroke .\n1\t2083598\t2083810\tAbout 10 percent of high school and 16 percent of elementary students must be proficient at math .\tIn math , 16 percent of elementary and middle school students and 9.6 percent of high school students must be proficient .\n1\t1910610\t1910455\tThe legal ruling follows three days of intense speculation Hewlett-Packard Co. may be bidding for the company .\tThe legal ruling follows three days of wild volatility in RIM 's stock over speculation that PC giant Hewlett-Packard Co. may be bidding for the company .\n1\t3113791\t3113782\tThe European Commission , the EU 's antitrust enforcer , is expected to issue its decision next spring — unless a settlement is reached .\tThe European Commission is expected to issue its decision in the case next spring — unless a settlement is reached .\n1\t3214517\t3214483\t\" So Sebastian did his best to convincingly confess to a crime that he didn 't commit in order to survive , \" she told jurors .\t\" Sebastian did his best to confess convincingly to a crime he didn 't do in order to survive , \" Ms. Richardson declared .\n0\t2083612\t2083810\tTwenty percent of Latino students and 23 percent of black students performed at proficient or higher .\tIn math , 16 percent of elementary and middle school students and 9.6 percent of high school students must be proficient .\n1\t661390\t661218\tHe is charged in three bombings in Atlanta including a blast at the 1996 Olympics and one in Alabama .\tHe is charged in three bombings in Atlanta - including a blast at the 1996 Olympics - along with the bombing in Alabama .\n1\t1269572\t1269682\tThe men were remanded in custody and are due to appear again before court on July 8 .\tThey were remanded in custody and will appear in court again on July 8 .\n1\t1095780\t1095652\t\" No matter who becomes the sponsor for stock-car racing 's top series , NASCAR will need an all-star event , \" Wheeler said in a statement .\tNo matter who becomes the sponsor for stock-car racings top series , NASCAR will need an all-star event , Wheeler said Tuesday .\n1\t116294\t116332\tThe Phillies were upset that Counsell had stolen second in the sixth inning with Arizona leading 7-1 .\tThe Phillies were apparently upset when Counsell stole during the sixth with the Diamondbacks up 7-1 .\n1\t941617\t941673\tHe said his hatred for such people grew from these discussions and had helped convince him violence was the answer .\tHis hatred for these people had germinated from these discussions and helped cement his belief that violence was the panacea .\n1\t2640607\t2640576\t\" There is no need for one deadline for all to create the ASEAN Economic Community , \" Thaksin said .\tThus , he said , there did not have to one deadline to create the economic community .\n1\t3310210\t3310286\tThe announcement was made during the recording of a Christmas concert attended by top Vatican cardinals , bishops , and many elite from Italian society , witnesses said .\tThe broadside came during the recording on Saturday night of a Christmas concert attended by top Vatican cardinals , bishops and many elite of Italian society , witnesses said .\n1\t3376093\t3376101\tThe additional contribution brings total U.S. food aid to North Korea this year to 100,000 tonnes .\tThe donation of 60,000 tons brings the total of U.S. contributions for the year to 100,000 .\n1\t1549586\t1549609\tLeon Williams ' body was found inside his third-floor apartment at 196 Bay St. , in Tompkinsville .\tThe dead man , Leon Williams , was found in his third-floor apartment .\n1\t460211\t460445\tThe player 's eyes were bloodshot and a blood-alcohol test produced a reading of 0.18 - well above Tennessee 's level of presumed intoxication of 0.10 , the report said .\tHe failed a field sobriety test and a blood-alcohol test produced a reading of 0.18 – well above Tennessee 's level of presumed intoxication of 0.10 , the report said .\n1\t1196962\t1197061\tBut Virgin wants to operate Concorde on routes to New York , Barbados and Dubai .\tBranson said that his preference would be to operate a fully commercial service on routes to New York , Barbados and Dubai .\n0\t862804\t862715\tHe tried to fight off officers and was taken to a hospital after a police dog bit him but was later released .\tCruz tried to fight off officers and was hospitalized after a police dog bit him , Sgt. Steve Dixon said .\n1\t1726935\t1726879\tThe announcement , which economists said was not a surprise , may be bittersweet for the millions of Americans without jobs .\tEconomists said the announcement was not a surprise , and politicians said it offered little comfort to the millions of Americans without jobs .\n0\t331980\t332110\tAsked if the delegates could leave on Friday , police intelligence chief in Aceh , Surya Dharma , told reporters they could not because they did not have proper permission .\tAsked if the delegates could leave on Friday , police intelligence chief Surya Dharma told reporters : \" Of course they may not go .\n1\t173879\t173832\tDealers said the dollar also drew some downside support as Japanese investors are expected to keep snapping up foreign bonds amid the yen 's rise against the dollar .\tDealers said the dollar also drew some downside support as Japanese investors are expected to keep snapping up foreign bonds amid ever-falling domestic interest rates .\n0\t2834988\t2835026\tIran has until the end of the month to satisfy the agency it has no plans for nuclear weapons .\tThe Iranians have until the end of the month to answer all the agency 's questions about their past nuclear activities .\n1\t2587300\t2587243\tHer father , Florin Cioaba , the king of Transylvania 's Gypsies , had her brought back and she was married against her will .\tHer father , Roma King Florin Cioaba , had her brought back and she was promptly married against her will .\n0\t554905\t554627\tClaire had advanced to the third round of the 76th annual Scripps Howard National Spelling Bee .\tOne by one they strolled to the microphone , all 251 youngsters in the 76th Scripps Howard National Spelling Bee .\n1\t1912524\t1912648\tCitigroup Inc . C.N , the world 's largest financial services company , on Wednesday promoted Marjorie Magner to chairman and chief executive of its global consumer group .\tCitigroup ( C ) on Wednesday named Marjorie Magner chairman and chief executive of its colossal global consumer business .\n1\t3255597\t3255668\t\" They 've been in the stores for over six weeks , \" says Carney .\tThe quarterlies usually stay in stores for between six to eight weeks , \" Carney added .\n1\t629316\t629289\tLet me just say this : the evidence that we have of weapons of mass destruction was evidence drawn up and accepted by the joint intelligence community .\t\" The evidence that we had of weapons of mass destruction was drawn up and accepted by the Joint Intelligence Committee , \" he said .\n1\t54181\t53570\tRidge said no actual explosives or other harmful substances will be used .\tRidge said no real explosives or harmful devices will be used in the exercise .\n1\t723557\t724115\tThus far , Stewart 's company appears ready to stand behind her .\tFor now , the company 's management appears to be standing behind Stewart .\n0\t2607718\t2607708\tBut late Thursday night , the campaign issued a statement saying there would be no news conference and no big announcement .\tBut late yesterday , the campaign and the state Democratic Party said there would be no news conference .\n1\t753858\t753890\tThere 's also a flaw that results because IE does not implement an appropriate block on a file download dialog box .\tThe second vulnerability is a result of IE not implementing a block on a file download dialog box .\n1\t587009\t586969\tAnother $ 100-million in savings will come from management layoffs and pay cuts .\tThe airline expects to save another $ 100-million a year through management layoffs and pay cuts .\n1\t308567\t308525\tHe called on Prime Minister John Howard to establish a royal commission on child sex abuse .\tThe Senate motion also called on Prime Minister John Howard to hold a royal commission into child sex abuse .\n0\t665419\t665612\t\" We think that the United States of America should support the free speech of all groups , \" Mr. White said , objecting to Mr. Olson 's recommendation .\tWe think that the United States of America should support the free speech of all groups , he said .\n1\t2763517\t2763576\tTerri Schiavo , 39 , underwent the procedure at the Tampa Bay area hospice where she has been living for several years , said her father , Bob Schindler .\tThe tube was removed Wednesday from Terri Schiavo , 39 , at the Tampa Bay-area hospice where she has lived for several years .\n0\t3107118\t3107136\tAfter 18 months , Nissen found that Lipitor stopped plaque buildup in the patients ' arteries .\tAfter 18 months , the atorvastatin patients had no change in the plaque in their arteries .\n1\t780604\t780466\tToll , Australia 's second-largest transport company , last week offered NZ75 a share for Tranz Rail .\tToll last week offered to buy the company for NZ75c a share , or $ NZ158 million .\n0\t1989213\t1989116\t\" This child was literally neglected to death , \" Armstrong County District Attorney Scott Andreassi said .\tArmstrong County District Attorney Scott Andreassi said the many family photos in the home did not include Kristen .\n1\t1462409\t1462504\tWal-Mart , the nation 's largest private employer , has expanded its antidiscrimination policy to protect gay and lesbian employees , company officials said Tuesday .\tWal-Mart Stores Inc . , the nation 's largest private employer , will now include gays and lesbians in its anti-discrimination policy , company officials said Wednesday .\n1\t260952\t260924\tMetro , bus and local rail services in France 's four largest towns -- Paris , Lyon , Lille and Marseille -- were severely disrupted , Europe 1 radio reported .\tSubway , bus and suburban rail services in France 's four largest cities -- Paris , Lyon , Lille and Marseille -- were severely disrupted , transport authorities said .\n1\t1224743\t1225510\tIn the undergraduate case , Rehnquist said the use of race was not \" narrowly tailored \" to achieve the university 's asserted interest in diversity .\tRehnquist wrote that the system was not narrowly tailored to achieve the interest in educational diversity .\n0\t3329379\t3329416\tSP2 is basically about security enhancements to Windows , such as the improved Internet Connection Firewall ( ICF ) .\tThe firewall in the current Windows XP was known as the Internet Connection Firewall ( ICF ) .\n1\t2362761\t2362698\tA landslide in central Chungchong province derailed a Seoul-bound train and 28 passengers were injured , television said .\tIn central Chungchong province , a landslide caused a Seoul-bound Saemaeul Express train to derail , injuring 28 people , local television said .\n0\t1465073\t1464854\tThey will help draft a plan to attack obesity that Kraft will implement over three to four years .\tThe team will help draft a plan by the end of the year to attack obesity .\n1\t195728\t196099\tBut that amount would probably be impossible to pass in the Senate , where Republican moderates have refused to go above $ 350 billion .\tSuch an amount would probably be unable to summon a majority of the Senate , where Republican moderates have refused to go above $ 350 billion .\n1\t2587767\t2587673\tIn the clash with police , Lt. Mothana Ali said about 1,000 demonstrators had gone to the station demanding jobs .\tIn Baghdad , police Lieut . Mothana Ali said about 1,000 demonstrators arrived at the station demanding jobs .\n0\t1490044\t1489975\tCorixa shares rose 54 cents to $ 7.74 yesterday on the Nasdaq Stock Market .\tShares of Corixa rose 54 cents , or about 8 percent , to close at $ 7.74 .\n1\t958161\t957782\tCommittee approval , expected today , would set the stage for debate on the Senate floor beginning Monday .\tThat would clear the way for debate in the full Senate beginning on Monday .\n1\t1033204\t1033365\tO 'Brien was charged with leaving the scene of a fatal accident , a felony .\tBishop Thomas O 'Brien , 67 , was booked on a charge of leaving the scene of a fatal accident .\n0\t2996241\t2996734\tTom Hamilton said his daughter was conscious and alert and in stable condition after the attack Friday morning .\tBethany , who remained in stable condition after the attack Friday morning , talked of the attack Saturday .\n0\t2015389\t2015410\tThe Calgary woman , who is in her twenties , donated blood on Aug. 7 .\tThe woman -- who has no symptoms of illness -- donated blood Aug. 7 .\n1\t221515\t221509\tQuattrone lawyer John W. Keker said his client is innocent .\tIn a statement Monday , his lawyer John Keker said ``Frank Quattrone is innocent .\n0\t2283737\t2283794\tIn the weeks leading up to the execution , several Florida officials received anonymous threatening letters .\tSeveral Florida officials connected to the case have received threatening letters , accompanied by rifle bullets .\n1\t2826681\t2826474\tThe disagreement over online music sales was disclosed in documents filed last week with the judge and made available by the court yesterday .\tThe fight over online music sales was disclosed in documents made available Monday by the court .\n1\t2249237\t2249305\tParson was charged with intentionally causing and attempting to cause damage to protected computers .\tParson is charged with one count of intentionally causing damage to a protected computer .\n1\t389239\t389299\t\" The court and the public need to know much more of the details of the defendant 's seemingly massive fraud , \" the judge said .\t\" The court and the public need to know more of the defendants ' seemingly massive fraud , \" he said .\n1\t2652187\t2652218\tThe U.S. Supreme Court will hear arguments on Wednesday on whether companies can be sued under the Americans with Disabilities Act for refusing to rehire rehabilitated drug users .\tThe high court will hear arguments today on whether companies can be sued under the ADA for refusing to rehire rehabilitated drug users .\n1\t2945693\t2945847\tThe IRS said taxpayers can avoid undelivered checks by having refunds deposited directly into their checking or savings accounts .\tThe IRS said taxpayers can avoid problems with lost or stolen refunds by having refunds deposited directly into personal checking or savings accounts .\n1\t2065523\t2065836\t\" More than 70,000 men and women from bases in Southern California were deployed in Iraq .\tIn all , more than 70,000 troops based in Southern California were deployed to Iraq .\n1\t2222998\t2223097\tBP shares slipped 0.8 percent to 433.50 pence ( $ 6.85 ) each in afternoon trading on the London Stock Exchange .\tBP shares slipped 48 cents to $ 41.72 Friday in trading on the New York Stock Exchange .\n1\t2561999\t2561941\tBecause of the accounting charge , the company now says it lost $ 1.04 billion , or 32 cents a share , in the quarter ended June 30 .\tIncluding the charge , the Santa Clara , Calif.-based company said Monday it lost $ 1.04 billion , or 32 cents per share , in the period ending June 30 .\n0\t2324704\t2325023\tFriday 's report raised new worries that a weak job market could shackle the budding economic recovery despite a slight improvement in the overall unemployment rate .\tU.S. companies slashed payrolls for a seventh straight month in August , raising new worries that a weak jobs market could shackle the budding economic recovery .\n1\t2336453\t2336545\tFederal Emergency Management Administration designated $ 20 million to establish the registry .\tThe registry was launched with $ 20 million from the Federal Emergency Management Agency .\n1\t720572\t720486\tBREAST cancer cases in the UK have hit an all-time high with more than 40,000 women diagnosed with the disease each year , Cancer Re-search UK revealed yesterday .\tCases of breast cancer in Britain have reached a record high , with the number of women diagnosed with the disease passing the 40,000 mark for the first time .\n1\t1605818\t1605806\t\" It was never our intention to sell the product , \" said Health Minister Anne McClellan , a skeptic of medical marijuana use .\t\" It was never the intention of us to sell product , \" federal Health Minister Anne McLellan said yesterday in Edmonton .\n0\t2440680\t2440474\tGM , the world 's largest automaker , has 115,000 active UAW workers and another 340,000 retirees and spouses .\tThey cover more than 300,000 UAW workers and 500,000 retirees and spouses .\n0\t726399\t726078\tRosenthal is hereby sentenced to custody of the Federal Bureau of prisons for one day with credit for time served , \" Breyer said to tumultuous cheers in the courtroom .\t\" Rosenthal is hereby sentenced to custody of the Federal Bureau of Prisons for one day with credit for time served . \"\n1\t533903\t533818\t\" We are committed to helping the Iraqi people get on the path to a free society , \" Rumsfeld said in a speech to the Council on Foreign Relations .\t\" We are committed to helping the Iraqi people get on the path to a free society , \" he said .\n1\t1166473\t1166857\tMr. Young said he was disappointed that the government didn 't see the severe acute respiratory syndrome crisis as worthy of federal disaster-relief money .\tYoung said he was disappointed the government didn 't see the SARS crisis as worthy of federal disaster relief money .\n1\t144089\t143697\tThe 12-nation currency has risen by 33 percent against the dollar over the past 15 months .\tThe euro is up 9 percent against the dollar in the past six weeks .\n1\t3439854\t3439874\tIn February 2000 , the officers — Kenneth Boss , Sean Carroll , Edward McMellon and Richard Murphy — were acquitted of all charges in the killing .\tThe officers -- Kenneth Boss , Sean Carroll , Edward McMellon and Richard Murphy -- were acquitted in 2000 of state murder charges .\n1\t3464314\t3464302\tI was surprised it turned out me talking and the president just listening .\t\" I was surprised it turned out me talking and the president just listening . . . It was mostly a monologue . \"\n1\t2008984\t2009175\tThe state 's House delegation currently consists of 17 Democrats and 15 Republicans .\tDemocrats hold a 17-15 edge in the state 's U.S. House delegation .\n0\t816867\t816831\tFreddie also said Leland C. Brendsel will retire as chairman and chief executive and resign from the board .\tHe replaces Leland Brendsel , 61 , who retired as chairman and chief executive .\n1\t192285\t192327\tWe 'll be listening carefully to the [ IAEA ] director general 's report at the next board meeting .\t\" We 'll be listening carefully to the ( IAEA ) director-general 's report at the next board meeting . \"\n1\t2688145\t2688162\tIn that position , Elias will report to Joe Tucci , president and CEO of EMC .\tAs executive vice president of new ventures , Elias will report to Joe Tucci , EMC 's president and chief executive .\n1\t3294207\t3294290\tBut with the PM due to leave tomorrow afternoon for personal reasons there was a risk he might not be present when the final decision was made .\tBut with the Prime Minister due to leave tomorrow , a day early , he may not be present when the final decision is made .\n0\t205100\t205145\tA pro-independence radical , Miodrag Zivkovic , of the Liberal Alliance , came in second with 31 percent of the vote .\tMiodrag Zivkovic , of the Liberal Alliance of Montenegro , won 31 percent of the vote while the independent Dragan Hajdukovic got four percent .\n0\t3242051\t3241897\tMr. Kerkorian tried unsuccessfully to take over Chrysler in 1995 , but did win representation on its board .\tKerkorian and Tracinda had also tried to take over Chrysler in 1995 .\n0\t1076861\t1077018\tGlover spoke at a news conference that included about 20 relatives of the victims .\tAbout 20 family members of the victims were invited to the news conference .\n1\t2095803\t2095786\tDrax faced a financial crisis late last year after it lost its most lucrative sales contract , held with insolvent utility TXU Europe .\tDrax ’ s troubles began late last year when it lost its most lucrative sales contract , with the insolvent utility TXU Europe .\n1\t2112330\t2112376\tBut I would rather be talking about high standards than low standards . \"\t\" I would rather be talking about positive numbers rather than negative .\n1\t3389318\t3389271\tIt was not immediately known how many people were on flight UTA 141 , which could carry 141 passengers and crew .\tIt was still not known exactly how many people were on the plane , which could carry 141 passengers and crew .\n1\t698948\t698933\tThe market remains pinned in a narrow range after a powerful rally drove the broad Standard & Poor 's 500 index .SPX up more than 20 percent since mid-March .\tThe market remains pinned in a narrow range after a powerful rally pushed the broad S & P 500 index up more than 20 percent since mid-March .\n1\t539585\t539355\tWitnesses said they believed the man planned to crash the Launceston-bound Qantas flight 1737 , which was carrying 47 passengers and six crew .\tWitnesses believe he wanted to crash Flight 1737 , which had 47 passengers and six crew .\n1\t684848\t684557\tAs Samudra sat down to hear the indictment , he looked over to his nine lawyers and shouted ``God is Great ' ' three times .\tAs he sat down to hear the indictment , Samudra looked over to his nine lawyers and shouted \" Takbir ! \" , or \" Proclaim ! \" , a religious rallying cry .\n1\t347017\t347002\tIn hardest-hit Taipei , traffic has disappeared from once bustling streets , ubiquitous department stores stand mostly empty and restaurants are eerily quiet .\tIn hardest-hit Taipei , traffic has disappeared from once-bustling streets and department stores and restaurants are virtually empty .\n1\t1592037\t1592076\tIn a statement , Lee said he \" no longer believes that Viacom deliberately intended to trade on my name when naming Spike TV . \"\tSpike Lee no longer believes that Viacom deliberately intended to trade on his name by calling its own venture \" Spike TV , \" according to a statement read in court Tuesday .\n0\t3013483\t3013540\tSingapore Prime Minister Goh Chok Tong says China plays an important role in the integration of Asia , including managing the stresses and strains both within and between countries .\tHAINAN PROVINCE , China : Singapore Prime Minister Goh Chok Tong said China plays an important role in the integration of Asia .\n1\t2020252\t2020081\tThe worm attacks Windows computers via a hole in the operating system , an issue Microsoft on July 16 had warned about .\tThe worm attacks Windows computers via a hole in the operating system , which Microsoft warned of 16 July .\n0\t2614947\t2614904\tThe premium edition adds OfficeFront Page 2003 , Acceleration Server 2000 , and SQL Server 2000 .\tThe premium edition adds ISA Server , SQL Server and a specialized edition of BizTalk 2004 .\n0\t1744257\t1744378\tIn the year-ago quarter , the steelmaker recorded a profit of $ 16.2 million , or 15 cents per share , on sales of $ 1.14 billion .\tIn the second quarter last year , AK Steel reported a profit of $ 16.2 million , or 15 cents a share .\n0\t1119721\t1119714\tSony claimed that the reader 's capacitance sensing technology cannot be fooled by paper copies and does not require cleaning .\tIts capacitance sensing technology electronically reads a fingerprint ; Sony says it can 't be fooled by paper copies and doesn 't require cleaning .\n1\t1186754\t1187056\tAmazon.com shipped out more than a million copies of the new book , making Saturday the largest distribution day of a single item in e-commerce history .\tAmazon.com shipped more than a million copies by Saturday afternoon , making Saturday the largest distribution day of a single item in e-commerce history .\n1\t2842562\t2842582\tThe show 's closure affected third-quarter earnings per share by a penny .\tThe company said this impacted earnings by a penny a share .\n0\t431076\t431242\tAfter the two-hour meeting on May 14 , publisher Arthur O. Sulzberger Jr . , executive editor Howell Raines and managing editor Gerald Boyd pledged quick remedies to staff grievances .\tThe committee will make recommendations to Publisher Arthur Sulzberger , Executive Editor Howell Raines and Managing Editor Gerald Boyd .\n1\t1393764\t1393984\tIt 's been a busy couple of days for security gurus assigned to keep their companies safe and sound .\tIt 's been a busy couple of days for enterprise security gurus tasked with the job of keeping their companies safe and sound .\n0\t2916199\t2916164\tLu reclined in a soft chair wearing a woolly coat near the blackened capsule .\t\" It 's great to be back home , \" said Lu , dressed in a woolly coat near the blackened capsule .\n1\t2530671\t2530542\tGov. Bob Riley proposed the budget cuts after Alabama voters rejected his $ 1.2 billion tax plan Sept . 9 .\tAfter Alabama voters rejected his $ 1.2 billion tax plan Sept . 9 , Riley forecast significant cuts in state programs .\n1\t219064\t218969\t\" It is probably not the easiest time to come in and take over the shuttle program , but then again , I look forward to the challenge , \" he said .\t\" It 's probably not the easiest time to come in and take over the shuttle program , but I look forward to the challenge , \" Parsons told reporters at NASA headquarters .\n0\t2377289\t2377259\tEstonia 's place in the European mainstream and safeguard its independence regained in 1991 .\tEstonia was forcibly incorporated in the Soviet Union in 1940 and regained its independence only in 1991 .\n0\t2110220\t2110199\tFranklin County Judge-Executive Teresa Barton said a firefighter was struck by lightning and was taken to the Frankfort Regional Medical Center .\tA county firefighter , was struck by lightning and was in stable condition at Frankfort Regional Medical Center .\n0\t1864253\t1863810\tPolice suspected that Shaichat , 20 , had been abducted either by Palestinians or by Israeli Arabs .\tNobody claimed responsibility for Schaichat 's death , but police suspect that the 20-year-old soldier was abducted either by Palestinians or Israeli Arabs .\n0\t3150803\t3150839\tDuring this year 's August to October quarter , Lowe 's opened 38 new stores , including two relocations .\tDuring the third quarter , Lowe 's opened 38 new stores and now has 932 stores in 45 states .\n0\t969381\t969512\tThe technology-laced Nasdaq Composite Index < .IXIC > declined 25.78 points , or 1.56 percent , to 1,627.84 .\tThe broader Standard & Poor 's 500 Index .SPX gave up 11.91 points , or 1.19 percent , at 986.60 .\n1\t271891\t271839\tSony said the PSP would also feature a 4.5-inch LCD screen , Memory Stick expansion slots .\tIt also features a 4.5 in back-lit LCD screen and memory expansion facilities .\n0\t2829648\t2829613\tClinton did not mention that two Democratic senators , Charles Robb of Virginia and Wendell Ford of Kentucky , voted to shelve the McCain bill .\tTwo Democrats , Sen. Charles Robb of Virginia and Wendell Ford of Kentucky , voted with the 40 Republicans .\n1\t886904\t887158\tSome of the company 's software developers will join Microsoft , but details haven 't been finalized , said Mike Nash , corporate vice president of Microsoft 's security business unit .\tSome of the companys software developers will join Microsoft , but details havent been finalized , said Mike Nash , corporate vice president of Microsofts security business unit .\n0\t2632692\t2632767\tWal-Mart has said it plans to open at least 40 Supercenters in the state in the coming years ; analysts expect four or more to be in San Diego County .\tAt least 40 of the outlets will be in California , and analysts expect four or more to be in San Diego County .\n1\t2240399\t2240149\tCintas is battling efforts to unionize 17,000 of its workers and to let unions organize the workers by signing cards , rather than by a lengthy election process .\tCintas is battling efforts to unionize 17,000 of its workers and labor 's demands to let its workers organize by signing cards , rather than by a lengthy election process .\n1\t805457\t805985\tThe opposition would resort to rolling mass action \" at strategic times of our choice and without warning to the dictatorship , \" he said .\t\" From now onwards we will embark on rolling mass action at strategic times of our choice and without any warning to the dictatorship , \" he said .\n1\t2896308\t2896334\tFederal Agriculture Minister Warren Truss said the Government still did not know the real reason the sheep were rejected at the Saudi port of Jeddah on August 21 .\tHe said the Government still did not know the real reason the original Saudi buyer pulled out on August 21 .\n1\t2110775\t2110924\tTom Kraynak , manager of operations and resources for the Canton , Ohio-based East Central Area Reliability Council , said that scenario is one among many that investigators are considering .\tTom Kraynak , manager of operations and resources for the Canton , Ohio-based East Central Area Reliability Council , said investigators are considering the scenario .\n1\t1762569\t1762526\tHester said Sanmina was the best fit among several purchase offers the company received from electronics manufacturers and computer makers .\tHester said Sanmina 's offer was the best among several Newisys received from electronics manufacturers and computer makers .\n0\t2706154\t2706185\tThe other inmate fell but Selenski shimmed down the makeshift rope to a second-story roof and used the mattress to scale a razor-wire fence , Fischi said .\tAfter the other inmate fell , Selenski used the mattress to scale a 10-foot , razor-wire fence , Fischi said .\n1\t1057995\t1057778\tThe hearing , expected to last a week , will determine whether Akbar faces a court-martial .\tThe purpose of the hearing is to determine whether Akbar should be court-martialled .\n1\t1386884\t1386857\tHe said he has begun a court action to seize Beacon Hill 's assets and has frozen more than $ 13 million Beacon Hill had when it closed .\tHe said he has initiated a forfeiture action in court and frozen more than $ 13 million Beacon Hill had when it closed .\n1\t3093023\t3092996\tSpeaking for the first time yesterday , Brigitte 's maternal aunt said his family was unaware he had was in prison or that he had remarried .\tBrigitte 's maternal aunt said his family was unaware he had been sent to prison , or that he had remarried in Sydney .\n1\t1661381\t1661317\t\" Close co-operation between our law enforcement agencies , close co-operation between our intelligence services lie at the heart of the ongoing fight against terrorism . \"\tClose cooperation between regional law enforcement agencies and intelligence services was at the heart of the fight against terrorism , he said .\n0\t2926039\t2925982\tThe mother of a Briton held by Colombian guerrillasspoke of her relief yesterday after hearing that he might be freed in the next few weeks .\tThe parents of a Briton being held hostage by Colombian rebels spoke yesterday of their optimism that he would be freed in time for his birthday next month .\n0\t637168\t637447\tWe strongly disagree with Novell 's position and view it as a desperate measure to curry favor with the Linux community .\tMcBride characterized Novell 's move as \" a desperate measure to curry favor with the Linux community . \"\n1\t696677\t696932\tAfter more than two years ' detention under the State Security Bureau , the four were found guilty of subversion in Beijing 's No. 1 Intermediate Court last Wednesday .\tAfter more than two years in detention by the State Security Bureau , the four were found guilty last Wednesday of subversion .\n1\t3122429\t3122305\tMr Russell , 46 , a coal miner from Brisbane , said : \" They are obviously hurting , so we are basically going over there to help them . \"\t\" They are obviously hurting so we are basically going over there to help them , \" Russell , 46 , said .\n1\t1348909\t1348954\tThe New York Democrat and former first lady has said she will not run for the White House in 2004 , but has not ruled out a race in later years .\tThe former first lady has said she will not run for the White House in 2004 but has not ruled out a race later on .\n0\t162203\t162101\tIt does not affect the current Windows Media Player 9.0 Series .\tWindows Media Player has had security problems before .\n0\t71501\t71627\tThe seizure took place at 4 a.m. on March 18 , just hours before the first American air assault .\tThe time was about 4 a.m. on March 18 , just hours before the first pinpoint missiles rained down on the capital .\n1\t2907762\t2907649\tDonations stemming from the Sept . 11 attacks helped push up contributions to human service organizations and large branches of the United Way by 15 percent and 28.6 percent , respectively .\tDonations stemming from the Sept . 11 attacks helped push up contributions to human service organizations by 15 percent and to large branches of the United Way by 28.6 percent .\n1\t2167771\t2167744\tIn May , Mr. Hatfill said he was struck by a vehicle being driven by an FBI employee who was tailing him in Georgetown .\tLast May , Hatfill was struck by a vehicle being driven by an FBI employee who was tailing him in Washington 's Georgetown neighborhood .\n1\t3320577\t3320553\t\" I will support a constitutional amendment which would honor marriage between a man and a woman , codify that , \" he said .\t\" If necessary , I will support a constitutional amendment which would honour marriage between a man and a woman , codify that . \"\n1\t849291\t849442\tIBM of the US and Infineon Technologies of Germany will today announce a technological development that could threaten multi-billion dollar memory chip markets .\tIBMof the US andInfineon Technologies of Germany willon Tuesdayannounce a technological development that could threaten multi-billion dollar memory chip markets .\n0\t763948\t763991\tCosta 's semifinal opponent is Spaniard Juan Carlos Ferrero , whom he beat in last year 's final .\tCosta will play Juan Carlos Ferrero next in a rematch of last year 's final .\n1\t1908763\t1908744\tA former employee of a local power company pleaded guilty Wednesday to setting off a bomb that knocked out a power substation during the Winter Olympics last year .\tA former Utah Power meter reader pleaded guilty Wednesday to bombing a power substation during the 2002 Winter Olympics .\n0\t1876120\t1876059\tThyroid hormones are known to help in weight loss by stimulating metabolism - and cutting cholesterol - but come with the unwanted side effect of speeding up the heartbeat .\tThyroid hormones are known to help in weight loss by stimulating metabolism , and they can help cut cholesterol too .\n1\t518089\t518133\tJudge Craig Doran said it wasn 't his role to determine if Hovan was \" an evil man \" but maintained that \" he has committed an evil act . \"\tJudge Craig Doran said he couldn 't determine if Hovan was \" an evil man \" but said he \" has committed an evil act . \"\n0\t224932\t224868\tThe Hartford shares rose $ 2.88 , or 6.6 percent , to close Monday at $ 46.50 on the New York Stock Exchange .\tShares of Hartford rose $ 2.88 to $ 46.50 in New York Stock Exchange composite trading .\n1\t1771131\t1771091\tIt also offers a built-in NAND flash boot loader so that high-density NAND flash memory can be used without having to install an additional support chip .\tThe S3C2440 has a built-in NAND flash boot loader , for example , so that high-density NAND flash memory can be installed without an additional support chip .\n0\t2728425\t2728251\tIt decided instead to issue them before the stock market opened Monday after the downgrade of its debt late Friday by Moody 's , the credit rating agency .\tIt decided instead to issue them before the stock market opened Monday to counteract the downgrade of its debt late Friday by Moody 's to one step above junk status .\n0\t953733\t953537\tAltria shares fell 2.5 percent or $ 1.11 to $ 42.57 and were the Dow 's biggest percentage loser .\tIts shares fell $ 9.61 to $ 50.26 , ranking as the NYSE 's most-active issue and its biggest percentage loser .\n1\t349215\t349241\tIt will be followed in November by a third movie , \" The Matrix Revolutions . \"\tThe film is the second of a trilogy , which will wrap up in November with \" The Matrix Revolutions . \"\n1\t2919853\t2919804\tMassachusetts regulators and the Securities and Exchange Commission on Tuesday pressed securities fraud charges against Putnam Investments and two of its former portfolio managers for alleged improper mutual fund trading .\tState and federal securities regulators filed civil charges against Putnam Investments and two portfolio managers in the ever-expanding mutual fund trading scandal .\n1\t954526\t954607\tHe is blocking them until the Air Force assigns four additional C-130 cargo planes to Gowen Field , an Idaho Air National Guard base in Boise .\tHe is holding them up until the Air Force agrees to assign four additional C-130 cargo planes to the Idaho Air National Guard .\n1\t69773\t69792\tCisco pared spending to compensate for sluggish sales .\tIn response to sluggish sales , Cisco pared spending .\n0\t2823575\t2823513\tThe study , published Monday in the journal Molecular Brain Research , is likely to also apply to humans , its authors said .\tThe study , conducted on the brains of developing mice , was being published today in the journal Molecular Brain Research .\n1\t2455942\t2455978\tMy decision today is not based on any one event . \"\tGovernor Rowland said his decision was \" not based on any one event . \"\n1\t131979\t131957\tNelson , 27 , is being retried on civil-rights charges stemming from the disturbance which led to Rosenbaum 's death .\tNelson , 27 , is being retried on civil rights charges stemming from the disturbance that led to Rosenbaum 's death .\n0\t2010705\t2010779\t\" The government elements who have been causing trouble are still in place .\tThe government elements who have been causing trouble are still in place , they are attacking us . \"\n1\t54142\t53641\tNext Monday at about 2 p.m. ( CST ) , hospital officials in and near Chicago will notice a sudden increase in people complaining of flu-like symptoms .\tAround the same time , hospital officials in and near Chicago will notice a sudden increase in people complaining of flu-like symptoms .\n1\t1015249\t1015204\tWal-Mart Stores Inc . , Kohl 's Corp. , Family Dollar Stores Inc. and Big Lots Inc. were among the merchants posting May sales that fell below Wall Street 's modest expectations .\tWal- Mart , Kohl 's Corp. , Family Dollar Stores Inc . , and Big Lots Inc. posted May sales that fell below Wall Street 's modest expectations .\n0\t753928\t753890\tThe patch also fixes a vulnerability that results because IE does not implement an appropriate block on a file download dialog box .\tThe second vulnerability is a result of IE not implementing a block on a file download dialog box .\n1\t3022833\t3023029\tPeterson , a former fertilizer salesman , is charged with murder in the deaths of his 27-year-old wife and the baby boy she was carrying .\tPeterson , 31 , is now charged with murder in the deaths of his 27-year-old wife and their unborn son .\n0\t751520\t751373\tSPOT products run a Microsoft operating system and the company 's DirectBand radio technology developed with SCA Data Systems .\tThe DirectBand network was developed with the assistance of SCA Data Systems .\n0\t218848\t218851\tHe replaces Ron Dittemore , who announced his resignation in April .\tDittemore announced his plans to resign on April 23 .\n1\t3181118\t3181443\tDetectives told Deasean 's father , Stelly Chisolm , a college student , and mother , Kimberly Hill , of the arrest shortly after Perry was apprehended .\tShortly after his arrest , detectives told Deasean 's father , Stelly Chisolm , a college student , and mother , Kimberly Hill , a medical assistant , about the development .\n1\t515581\t515752\tThey were among about 40 people attending the traditional Jewish ceremony colored by some non-traditional touches .\tHe said about 40 people attended the traditional Jewish ceremony colored by some nontraditional touches .\n1\t347022\t347003\tTaiwan had been relatively free of the viral infection until a fiasco at a Taipei hospital in late April caused the number of infections to skyrocket .\tTaiwan had been relatively free of the viral infection until a severe outbreak at a Taipei hospital in late April .\n1\t3311600\t3311633\tMr. Rowland attended a party in South Windsor for the families of Connecticut National Guard soldiers called to active duty .\tRowland was making an appearance at a holiday party for families of Connecticut National Guard soldiers assigned to duty in Iraq and Afghanistan .\n0\t3439114\t3439084\tRoss Garber , Rowland 's lawyer , said Tuesday he would attend the meeting and would ask to speak on the issue .\tRoss Garber , Rowland 's legal counsel , said the governor would have no comment on the condo deal .\n0\t487951\t488007\tThe euro was at 1.5281 versus the Swiss franc EURCHF = , up 0.2 percent on the session , after hitting its highest since mid-2001 around 1.5292 earlier in the session .\tThe euro was steady versus the Swiss franc after hitting its highest since mid-2001 of 1.5261 earlier in the session .\n0\t314997\t315030\tOn the stand Wednesday , she said she was referring only to the kissing .\tOn the stand Wednesday , she testified that she was referring to the kissing before the alleged rape .\n0\t4733\t4557\tGarner said the group would probably be expanded to include , for example , a Christian and perhaps another Sunni leader .\tThe group has already met several times and Gen. Garner said it probably will be expanded to include a Christian and perhaps another Sunni Muslim leader .\n1\t2820371\t2820525\tBlair 's Foreign Secretary Jack Straw was to take his place on Monday to give a statement to parliament on the European Union .\tBlair 's office said his Foreign Secretary Jack Straw would take his place on Monday to give a statement to parliament on the EU meeting the prime minister attended last week .\n1\t801552\t801516\t\" There were more people surrounding the clubhouse than the Unabomber 's house up in the hills , \" Baker said .\t\" There are more people surrounding the clubhouse than surrounded the Unabomber 's home in the hills .\n1\t1704987\t1705268\tCharles O. Prince , 53 , was named as Mr. Weill 's successor .\tMr. Weill 's longtime confidant , Charles O. Prince , 53 , was named as his successor .\n1\t396041\t396188\tOfficials are also meeting with the International Organization for Epizootics ( OIE ) , which establishes animal-health standards for the world .\tCanadian officials were also expected to meet yesterday with the International Organization for Epizootics ( OIE ) , which establishes animal-health standards for the world .\n0\t1014983\t1014963\tGE stock closed Friday at $ 30.65 a share , down about 42 cents , on the New York Stock Exchange .\tGE 's shares closed at $ 30.65 on Friday on the New York Stock Exchange .\n1\t2320654\t2320666\tThe Midwestern research center will focus on the development of diagnostic , therapeutic and vaccine products for anthrax , botulism , tularemia , hemorrhagic fever viruses and plague .\tThe Midwestern center will focus on diagnosis , treatment and vaccines for anthrax , botulism , tularemia , hemorrhagic fever viruses and plague .\n1\t1057876\t1057778\tThe hearing is to determine whether there is enough evidence to order Akbar to a general court-martial proceeding .\tThe purpose of the hearing is to determine whether Akbar should be court-martialled .\n0\t2116843\t2116883\tIn the United States , heart attacks kill about 460,000 year , in Canada about 80,000 .\tIn the United States , heart attacks kill about 460,000 yearly , according to the National Institutes of Health .\n1\t1461629\t1461781\tNinety-five percent of international cargo to the United States is carried by ship .\tShips carry 95 percent of international cargo to the United States .\n0\t374015\t374162\t\" It 's a major victory for Maine , and it 's a major victory for other states .\tThe Maine program could be a model for other states .\n1\t2493369\t2493428\tNews that oil producers were lowering their output starting in November exacerbated a sell-off that was already under way on Wall Street .\tNews that the Organization of Petroleum Exporting Countries was lowering output starting in November exacerbated a stock sell-off already under way yesterday .\n1\t490355\t490378\tThey note that after several weeks of rallies on upbeat earnings , investors are looking for stronger evidence of a recovery before sending stocks higher .\tAfter several weeks of market rallies on upbeat earnings , many investors are looking for more concrete signs of an economic recovery .\n1\t2691044\t2691264\tMost economists had expected a more dire report , with many anticipating the fifth month of job losses in six months .\tMost economists had been expecting a far more dire report , with many expecting to see the fifth month of job losses in six months in September .\n1\t1831453\t1831491\tBut software license revenues , a measure financial analysts watch closely , decreased 21 percent to $ 107.6 million .\tLicense sales , a key measure of demand , fell 21 percent to $ 107.6 million .\n1\t2380695\t2380822\tKing , brand-name writer , master of the horror story and e-book pioneer , is receiving this year 's medal for Distinguished Contributions to American Letters .\tStephen King , master of the horror story and e-book pioneer , is receiving this year 's medal for Distinguished Contributions to American Letters from the National Book Foundation .\n1\t2577517\t2577531\tThe Denver-based natural gas producer and marketer said the inaccurate reporting was discovered after it received a subpoena from the U.S. Commodity Futures Trading Commission .\tThe natural gas producer and marketer said the inaccurate reporting was discovered in response to a subpoena from the U.S. Commodity Futures Trading Commission , or CFTC .\n1\t3267026\t3266930\tThe steel tariffs , which the U.S. president imposed in March 2002 , will officially end at midnight , instead of March 2005 as initially planned .\tThe U.S. steel tariffs , which Bush imposed in March 2002 , were to officially end at midnight Thursday ( 0500 GMT ) , instead of March 2005 as initially planned .\n1\t360875\t360943\tBusiness Week 's online edition reported on Friday that WorldCom and the SEC could announce a settlement as early as Monday .\tBusinessWeek Online has learned that the settlement could come as early as Monday , May 19 .\n1\t162632\t162653\tOnly one of the five buildings in the Baghdad compound of the United Nations Development Program escaped being burned , the UN said on its Web site .\tOnly one of the five buildings in the compound in Baghdad run by the UN Development Program , escaped being burned , the UN said on its Web site .\n1\t1128884\t1128865\tShares of Salix have rocketed 64 percent since Axcan made its first offer on April 10 .\tSince the initial takeover offer , Salix shares have risen about 35 percent .\n1\t3264732\t3264648\tThe jury verdict , reached Wednesday after less than four hours of deliberation , followed a 2 week trial , during which Waagner represented himself .\tThe quick conviction followed a 2 1 / 2 week trial , during which the Venango County man represented himself .\n1\t1721433\t1721267\tIt 's happened five times in the last 11 years : A disaster puts this Southwestern town in the headlines during the summer tourist season .\tIt 's happened five times in the last decade : A disaster puts this tourist town in the headlines during summer , its busiest season .\n0\t146112\t146127\tThe broader Standard & Poor 's 500 Index .SPX edged down 9 points , or 0.98 percent , to 921 .\tThe technology-laced Nasdaq Composite Index < .IXIC > shed 15 points , or 0.98 percent , to 1,492 .\n1\t389117\t389052\tThe company emphasized that McDonald 's USA does not import any raw beef or hamburger patties from Canada for McDonald 's use in the United States .\tMcDonald 's said in a statement that it does not import any raw beef or hamburger patties from Canada for use in the United States .\n1\t872784\t872834\tGregory Parseghian , a former investment banker , was appointed chief executive .\tGreg Parseghian was appointed the new chief executive .\n0\t2977500\t2977547\tTheir contract will expire at 12 : 01 a.m. Wednesday instead of 12 : 01 a.m. Sunday , said Rian Wathen , organizing director for United Food and Commercial Workers Local 700 .\t\" It has outraged the membership , \" said Rian Wathen , organizing director of United Food and Commercial Workers Local 700 .\n1\t3107137\t3107119\tBut plaque volume increased by 2.7 percent in pravastatin patients .\tThe volume of plaque in Pravachol patients ' arteries rose by 3 % .\n1\t1619244\t1619274\tToday in the US , the book - kept under wraps by its publishers , G. P. Putnam 's Sons , since its inception - will appear in bookstores .\tTomorrow the book , kept under wraps by G. P. Putnam 's Sons since its inception , will appear in bookstores .\n0\t3061836\t3062031\tThe S & P / TSX composite rose 87.74 points on the week , while the TSX Venture Exchange composite gained 44.49 points .\tOn the week , the Dow Jones industrial average rose 11.56 points , while the Nasdaq Stock Market gained 39.42 points .\n1\t485999\t486011\tEx-KGB agent Putin added that the Beatles were considered ' propaganda of an alien ideology ' .\tIn Soviet times the Beatles ' music \" was considered propaganda of an alien ideology ." }, { "path": "src/examples/pytorch/bert_tutorial/parallel.py", "content": "from concurrent import futures\nimport torch\nimport torch.neuron\nimport os\nfrom time import time\nfrom queue import Queue\nimport warnings\n\ndef consumer(model, input_queue):\n while True:\n inputs, input_id, callback_fn = input_queue.get()\n input_queue.task_done()\n # Stop execution if stopping condition is recieved\n if inputs == \"stop\":\n break\n start = time()\n results = model(*inputs)\n # Make the output iterable - if it is not already a tuple or list\n if not isinstance(results, tuple) or isinstance(results, list):\n results = [results]\n end = time()\n if callback_fn is not None:\n callback_fn(results, input_id, start, end)\n \nclass NeuronSimpleDataParallel():\n\n def __init__(self, model_file, num_neuron_cores, batch_size=1):\n self.num_neuron_cores = num_neuron_cores\n self.batch_size = batch_size\n \n os.environ['NEURON_RT_NUM_CORES'] = str(num_neuron_cores)\n \n # Construct a list of models\n self.models = [torch.jit.load(model_file)\n for i in range(num_neuron_cores)]\n \n # Create shared input queue\n self.input_queue = Queue(maxsize=num_neuron_cores*16)\n\n self.executor = futures.ThreadPoolExecutor(\n max_workers=num_neuron_cores)\n\n def eval(self):\n for model in self.models:\n model.eval()\n \n def train(self):\n for model in self.models:\n model.train()\n \n def start_continuous_inference(self):\n for model in self.models:\n self.executor.submit(consumer, model, self.input_queue)\n \n def infer(self, batch, input_id, callback_fn):\n self.input_queue.put((batch, input_id, callback_fn))\n \n def stop(self):\n for _ in range(self.num_neuron_cores):\n self.input_queue.put((\"stop\", -1, None))\n" }, { "path": "src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Compiling and Deploying HuggingFace Pretrained BERT\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Introduction\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy BERT-base version of HuggingFace 🤗 Transformers BERT for Inferentia. The full list of HuggingFace's pretrained BERT models can be found in the BERT section on this page https://huggingface.co/transformers/pretrained_models.html. \\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on an instance which is inf1.6xlarge or larger. The compile part of this tutorial requires inf1.6xlarge and not the inference itself. For simplicity we will run this tutorial on inf1.6xlarge but in real life scenario the compilation should be done on a compute instance and the deployment on inf1 instance to save costs.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [PyTorch Installation Guide](../../../../frameworks/torch/torch-neuron/setup/pytorch-install.html). You can select the kernel from the \\\"Kernel -> Change Kernel\\\" option on the top of this Jupyter notebook page.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuron`\\n\",\n \"- `neuron-cc[tensorflow]`\\n\",\n \"- `transformers`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Neuron PyTorch setup guide. The additional dependencies must be installed here.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"!pip install --upgrade \\\"transformers==4.6.0\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Compile the model into an AWS Neuron optimized TorchScript\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import tensorflow # to workaround a protobuf version conflict issue\\n\",\n \"import torch\\n\",\n \"import torch.neuron\\n\",\n \"from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig\\n\",\n \"import transformers\\n\",\n \"import os\\n\",\n \"import warnings\\n\",\n \"\\n\",\n \"# Setting up NeuronCore groups for inf1.6xlarge with 16 cores\\n\",\n \"num_cores = 16 # This value should be 4 on inf1.xlarge and inf1.2xlarge\\n\",\n \"os.environ['NEURON_RT_NUM_CORES'] = str(num_cores)\\n\",\n \"\\n\",\n \"# Build tokenizer and model\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"bert-base-cased-finetuned-mrpc\\\")\\n\",\n \"model = AutoModelForSequenceClassification.from_pretrained(\\\"bert-base-cased-finetuned-mrpc\\\", return_dict=False)\\n\",\n \"\\n\",\n \"# Setup some example inputs\\n\",\n \"sequence_0 = \\\"The company HuggingFace is based in New York City\\\"\\n\",\n \"sequence_1 = \\\"Apples are especially bad for your health\\\"\\n\",\n \"sequence_2 = \\\"HuggingFace's headquarters are situated in Manhattan\\\"\\n\",\n \"\\n\",\n \"max_length=128\\n\",\n \"paraphrase = tokenizer.encode_plus(sequence_0, sequence_2, max_length=max_length, padding='max_length', truncation=True, return_tensors=\\\"pt\\\")\\n\",\n \"not_paraphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\\\"pt\\\")\\n\",\n \"\\n\",\n \"# Run the original PyTorch model on compilation exaple\\n\",\n \"paraphrase_classification_logits = model(**paraphrase)[0]\\n\",\n \"\\n\",\n \"# Convert example inputs to a format that is compatible with TorchScript tracing\\n\",\n \"example_inputs_paraphrase = paraphrase['input_ids'], paraphrase['attention_mask'], paraphrase['token_type_ids']\\n\",\n \"example_inputs_not_paraphrase = not_paraphrase['input_ids'], not_paraphrase['attention_mask'], not_paraphrase['token_type_ids']\\n\",\n \"\\n\",\n \"# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\\n\",\n \"model_neuron = torch.neuron.trace(model, example_inputs_paraphrase)\\n\",\n \"\\n\",\n \"# Verify the TorchScript works on both example inputs\\n\",\n \"paraphrase_classification_logits_neuron = model_neuron(*example_inputs_paraphrase)\\n\",\n \"not_paraphrase_classification_logits_neuron = model_neuron(*example_inputs_not_paraphrase)\\n\",\n \"\\n\",\n \"# Save the TorchScript for later use\\n\",\n \"model_neuron.save('bert_neuron.pt')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You may inspect `model_neuron.graph` to see which part is running on CPU versus running on the accelerator. All native `aten` operators in the graph will be running on CPU.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print(model_neuron.graph)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"### Deploy the AWS Neuron optimized TorchScript\\n\",\n \"\\n\",\n \"To deploy the AWS Neuron optimized TorchScript, you may choose to load the saved TorchScript from disk and skip the slow compilation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Load TorchScript back\\n\",\n \"model_neuron = torch.jit.load('bert_neuron.pt')\\n\",\n \"# Verify the TorchScript works on both example inputs\\n\",\n \"paraphrase_classification_logits_neuron = model_neuron(*example_inputs_paraphrase)\\n\",\n \"not_paraphrase_classification_logits_neuron = model_neuron(*example_inputs_not_paraphrase)\\n\",\n \"classes = ['not paraphrase', 'paraphrase']\\n\",\n \"paraphrase_prediction = paraphrase_classification_logits_neuron[0][0].argmax().item()\\n\",\n \"not_paraphrase_prediction = not_paraphrase_classification_logits_neuron[0][0].argmax().item()\\n\",\n \"print('BERT says that \\\"{}\\\" and \\\"{}\\\" are {}'.format(sequence_0, sequence_2, classes[paraphrase_prediction]))\\n\",\n \"print('BERT says that \\\"{}\\\" and \\\"{}\\\" are {}'.format(sequence_0, sequence_1, classes[not_paraphrase_prediction]))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now let's run the model in parallel on four cores\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def get_input_with_padding(batch, batch_size, max_length):\\n\",\n \" ## Reformulate the batch into three batch tensors - default batch size batches the outer dimension\\n\",\n \" encoded = batch['encoded']\\n\",\n \" inputs = torch.squeeze(encoded['input_ids'], 1)\\n\",\n \" attention = torch.squeeze(encoded['attention_mask'], 1)\\n\",\n \" token_type = torch.squeeze(encoded['token_type_ids'], 1)\\n\",\n \" quality = list(map(int, batch['quality']))\\n\",\n \"\\n\",\n \" if inputs.size()[0] != batch_size:\\n\",\n \" print(\\\"Input size = {} - padding\\\".format(inputs.size()))\\n\",\n \" remainder = batch_size - inputs.size()[0]\\n\",\n \" zeros = torch.zeros( [remainder, max_length], dtype=torch.long )\\n\",\n \" inputs = torch.cat( [inputs, zeros] )\\n\",\n \" attention = torch.cat( [attention, zeros] )\\n\",\n \" token_type = torch.cat( [token_type, zeros] )\\n\",\n \"\\n\",\n \" assert(inputs.size()[0] == batch_size and inputs.size()[1] == max_length)\\n\",\n \" assert(attention.size()[0] == batch_size and attention.size()[1] == max_length)\\n\",\n \" assert(token_type.size()[0] == batch_size and token_type.size()[1] == max_length)\\n\",\n \"\\n\",\n \" return (inputs, attention, token_type), quality\\n\",\n \"\\n\",\n \"def count(output, quality):\\n\",\n \" assert output.size(0) >= len(quality)\\n\",\n \" correct_count = 0\\n\",\n \" count = len(quality)\\n\",\n \" \\n\",\n \" batch_predictions = [ row.argmax().item() for row in output ]\\n\",\n \"\\n\",\n \" for a, b in zip(batch_predictions, quality):\\n\",\n \" if int(a)==int(b):\\n\",\n \" correct_count += 1\\n\",\n \"\\n\",\n \" return correct_count, count\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Data parallel inference\\n\",\n \"In the below cell, we use the data parallel approach for inference. In this approach, we load multiple models, all of them running in parallel. Each model is loaded onto a single NeuronCore. In the below implementation, we launch 16 models, thereby utilizing all the 16 cores on an inf1.6xlarge.\\n\",\n \"\\n\",\n \"> Note: Now if you try to decrease the num_cores in the above cells, please restart the notebook and run `!sudo rmmod neuron; sudo modprobe neuron` step in cell 2 to clear the Neuron cores.\\n\",\n \"\\n\",\n \"Since, we can run more than 1 model concurrently, the throughput for the system goes up. To achieve maximum gain in throughput, we need to efficiently feed the models so as to keep them busy at all times. In the below setup, this is done by using a producer-consumer model. We maintain a common python queue shared across all the models. The common queue enables feeding data continuously to the models.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from parallel import NeuronSimpleDataParallel\\n\",\n \"from bert_benchmark_utils import BertTestDataset, BertResults\\n\",\n \"import time\\n\",\n \"import functools\\n\",\n \"\\n\",\n \"max_length = 128\\n\",\n \"num_cores = 16\\n\",\n \"batch_size = 1\\n\",\n \"\\n\",\n \"tsv_file=\\\"glue_mrpc_dev.tsv\\\"\\n\",\n \"\\n\",\n \"data_set = BertTestDataset( tsv_file=tsv_file, tokenizer=tokenizer, max_length=max_length )\\n\",\n \"data_loader = torch.utils.data.DataLoader(data_set, batch_size=batch_size, shuffle=True)\\n\",\n \"\\n\",\n \"#Result aggregation class (code in bert_benchmark_utils.py)\\n\",\n \"results = BertResults(batch_size, num_cores)\\n\",\n \"def result_handler(output, result_id, start, end, input_dict):\\n\",\n \" correct_count, inference_count = count(output[0], input_dict.pop(result_id))\\n\",\n \" elapsed = end - start\\n\",\n \" results.add_result(correct_count, inference_count, [elapsed], [end], [start])\\n\",\n \"\\n\",\n \"parallel_neuron_model = NeuronSimpleDataParallel('bert_neuron.pt', num_cores)\\n\",\n \"\\n\",\n \"#Starting the inference threads\\n\",\n \"parallel_neuron_model.start_continuous_inference()\\n\",\n \"\\n\",\n \"# Warm up the cores\\n\",\n \"z = torch.zeros( [batch_size, max_length], dtype=torch.long )\\n\",\n \"batch = (z, z, z)\\n\",\n \"for _ in range(num_cores*4):\\n\",\n \" parallel_neuron_model.infer(batch, -1, None)\\n\",\n \" \\n\",\n \"input_dict = {}\\n\",\n \"input_id = 0\\n\",\n \"for _ in range(30):\\n\",\n \" for batch in data_loader:\\n\",\n \" batch, quality = get_input_with_padding(batch, batch_size, max_length)\\n\",\n \" input_dict[input_id] = quality\\n\",\n \" callback_fn = functools.partial(result_handler, input_dict=input_dict)\\n\",\n \" parallel_neuron_model.infer(batch, input_id, callback_fn)\\n\",\n \" input_id+=1\\n\",\n \"\\n\",\n \"# Stop inference \\n\",\n \"parallel_neuron_model.stop()\\n\",\n \"\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"w\\\") as f:\\n\",\n \" results.report(f, window_size=1)\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"r\\\") as f:\\n\",\n \" for line in f:\\n\",\n \" print(line)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now recompile with a larger batch size of six sentence pairs\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"batch_size = 6\\n\",\n \"\\n\",\n \"example_inputs_paraphrase = (\\n\",\n \" torch.cat([paraphrase['input_ids']] * batch_size,0), \\n\",\n \" torch.cat([paraphrase['attention_mask']] * batch_size,0), \\n\",\n \" torch.cat([paraphrase['token_type_ids']] * batch_size,0)\\n\",\n \")\\n\",\n \"\\n\",\n \"# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\\n\",\n \"model_neuron_batch = torch.neuron.trace(model, example_inputs_paraphrase)\\n\",\n \"\\n\",\n \"## Save the batched model\\n\",\n \"model_neuron_batch.save('bert_neuron_b{}.pt'.format(batch_size))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Rerun inference with batch 6\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"from parallel import NeuronSimpleDataParallel\\n\",\n \"from bert_benchmark_utils import BertTestDataset, BertResults\\n\",\n \"import time\\n\",\n \"import functools\\n\",\n \"\\n\",\n \"max_length = 128\\n\",\n \"num_cores = 16\\n\",\n \"batch_size = 6\\n\",\n \"\\n\",\n \"data_set = BertTestDataset( tsv_file=tsv_file, tokenizer=tokenizer, max_length=max_length )\\n\",\n \"data_loader = torch.utils.data.DataLoader(data_set, batch_size=batch_size, shuffle=True)\\n\",\n \"\\n\",\n \"#Result aggregation class (code in bert_benchmark_utils.py)\\n\",\n \"results = BertResults(batch_size, num_cores)\\n\",\n \"def result_handler(output, result_id, start, end, input_dict):\\n\",\n \" correct_count, inference_count = count(output[0], input_dict.pop(result_id))\\n\",\n \" elapsed = end - start\\n\",\n \" results.add_result(correct_count, inference_count, [elapsed], [end], [start])\\n\",\n \"\\n\",\n \"parallel_neuron_model = NeuronSimpleDataParallel('bert_neuron_b{}.pt'.format(batch_size), num_cores)\\n\",\n \"\\n\",\n \"#Starting the inference threads\\n\",\n \"parallel_neuron_model.start_continuous_inference()\\n\",\n \"\\n\",\n \"# Adding to the input queue to warm all cores\\n\",\n \"z = torch.zeros( [batch_size, max_length], dtype=torch.long )\\n\",\n \"batch = (z, z, z)\\n\",\n \"for _ in range(num_cores*4):\\n\",\n \" parallel_neuron_model.infer(batch, -1, None)\\n\",\n \"\\n\",\n \"input_dict = {}\\n\",\n \"input_id = 0\\n\",\n \"for _ in range(30):\\n\",\n \" for batch in data_loader:\\n\",\n \" batch, quality = get_input_with_padding(batch, batch_size, max_length)\\n\",\n \" input_dict[input_id] = quality\\n\",\n \" callback_fn = functools.partial(result_handler, input_dict=input_dict)\\n\",\n \" parallel_neuron_model.infer(batch, input_id, callback_fn)\\n\",\n \" input_id+=1\\n\",\n \"\\n\",\n \"# Stop inference \\n\",\n \"parallel_neuron_model.stop()\\n\",\n \"\\n\",\n \"with open(\\\"benchmark_b{}.txt\\\".format(batch_size), \\\"w\\\") as f:\\n\",\n \" results.report(f, window_size=1)\\n\",\n \"\\n\",\n \"with open(\\\"benchmark_b{}.txt\\\".format(batch_size), \\\"r\\\") as f:\\n\",\n \" for line in f:\\n\",\n \" print(line)\\n\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert_shared_weights.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Data Parallel HuggingFace Pretrained BERT with Weight Sharing (Deduplication)\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Introduction\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy BERT-base version of HuggingFace 🤗 Transformers BERT for Inferentia, with additional demonstration of using Weight Sharing (Deduplication) feature.\\n\",\n \"\\n\",\n \"To use the [Weight Sharing (Deduplication) feature](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-configurable-parameters.html#shared-weights-neuron-rt-multi-instance-shared-weights), you must set the Neuron Runtime environmental variable NEURON_RT_MULTI_INSTANCE_SHARED_WEIGHTS to \\\"TRUE\\\" together with the [core placement API](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuron/api-core-placement.html) (``torch_neuron.experimental.neuron_cores_context()``).\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on an instance which is inf1.6xlarge or larger. The compile part of this tutorial requires inf1.6xlarge and not the inference itself. For simplicity we will run this tutorial on inf1.6xlarge but in real life scenario the compilation should be done on a compute instance and the deployment on inf1 instance to save costs.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [PyTorch Installation Guide](../../../../frameworks/torch/torch-neuron/setup/pytorch-install.html). You can select the kernel from the \\\"Kernel -> Change Kernel\\\" option on the top of this Jupyter notebook page.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuron`\\n\",\n \"- `neuron-cc[tensorflow]`\\n\",\n \"- `transformers`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Neuron PyTorch setup guide. The additional dependencies must be installed here.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"!pip install --upgrade \\\"transformers==4.6.0\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Compile the model into an AWS Neuron optimized TorchScript\\n\",\n \"\\n\",\n \"This step compiles the model into an AWS Neuron optimized TorchScript, and saves it in the filed ``bert_neuron.pt``. This step is the same as the pretrained BERT tutorial without Shared Weights feature. We use batch 1 for simplicity.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import tensorflow # to workaround a protobuf version conflict issue\\n\",\n \"import torch\\n\",\n \"import torch.neuron\\n\",\n \"from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig\\n\",\n \"import transformers\\n\",\n \"import os\\n\",\n \"import warnings\\n\",\n \"\\n\",\n \"\\n\",\n \"# Build tokenizer and model\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"bert-base-cased-finetuned-mrpc\\\")\\n\",\n \"model = AutoModelForSequenceClassification.from_pretrained(\\\"bert-base-cased-finetuned-mrpc\\\", return_dict=False)\\n\",\n \"\\n\",\n \"# Setup some example inputs\\n\",\n \"sequence_0 = \\\"The company HuggingFace is based in New York City\\\"\\n\",\n \"sequence_1 = \\\"Apples are especially bad for your health\\\"\\n\",\n \"sequence_2 = \\\"HuggingFace's headquarters are situated in Manhattan\\\"\\n\",\n \"\\n\",\n \"max_length=128\\n\",\n \"paraphrase = tokenizer.encode_plus(sequence_0, sequence_2, max_length=max_length, padding='max_length', truncation=True, return_tensors=\\\"pt\\\")\\n\",\n \"not_paraphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\\\"pt\\\")\\n\",\n \"\\n\",\n \"# Run the original PyTorch model on compilation exaple\\n\",\n \"paraphrase_classification_logits = model(**paraphrase)[0]\\n\",\n \"\\n\",\n \"# Convert example inputs to a format that is compatible with TorchScript tracing\\n\",\n \"example_inputs_paraphrase = paraphrase['input_ids'], paraphrase['attention_mask'], paraphrase['token_type_ids']\\n\",\n \"example_inputs_not_paraphrase = not_paraphrase['input_ids'], not_paraphrase['attention_mask'], not_paraphrase['token_type_ids']\\n\",\n \"\\n\",\n \"# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\\n\",\n \"model_neuron = torch.neuron.trace(model, example_inputs_paraphrase)\\n\",\n \"\\n\",\n \"# Verify the TorchScript works on both example inputs\\n\",\n \"paraphrase_classification_logits_neuron = model_neuron(*example_inputs_paraphrase)\\n\",\n \"not_paraphrase_classification_logits_neuron = model_neuron(*example_inputs_not_paraphrase)\\n\",\n \"\\n\",\n \"# Save the TorchScript for later use\\n\",\n \"model_neuron.save('bert_neuron.pt')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"### Deploy the AWS Neuron optimized TorchScript\\n\",\n \"\\n\",\n \"To deploy the AWS Neuron optimized TorchScript, you may choose to load the saved TorchScript from disk and skip the slow compilation. This step is the same as the pretrained BERT tutorial without Shared Weights feature\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Load TorchScript back\\n\",\n \"model_neuron = torch.jit.load('bert_neuron.pt')\\n\",\n \"# Verify the TorchScript works on both example inputs\\n\",\n \"paraphrase_classification_logits_neuron = model_neuron(*example_inputs_paraphrase)\\n\",\n \"not_paraphrase_classification_logits_neuron = model_neuron(*example_inputs_not_paraphrase)\\n\",\n \"classes = ['not paraphrase', 'paraphrase']\\n\",\n \"paraphrase_prediction = paraphrase_classification_logits_neuron[0][0].argmax().item()\\n\",\n \"not_paraphrase_prediction = not_paraphrase_classification_logits_neuron[0][0].argmax().item()\\n\",\n \"print('BERT says that \\\"{}\\\" and \\\"{}\\\" are {}'.format(sequence_0, sequence_2, classes[paraphrase_prediction]))\\n\",\n \"print('BERT says that \\\"{}\\\" and \\\"{}\\\" are {}'.format(sequence_0, sequence_1, classes[not_paraphrase_prediction]))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"We define two helper functions to pad input and to count correct results.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def get_input_with_padding(batch, batch_size, max_length):\\n\",\n \" ## Reformulate the batch into three batch tensors - default batch size batches the outer dimension\\n\",\n \" encoded = batch['encoded']\\n\",\n \" inputs = torch.squeeze(encoded['input_ids'], 1)\\n\",\n \" attention = torch.squeeze(encoded['attention_mask'], 1)\\n\",\n \" token_type = torch.squeeze(encoded['token_type_ids'], 1)\\n\",\n \" quality = list(map(int, batch['quality']))\\n\",\n \"\\n\",\n \" if inputs.size()[0] != batch_size:\\n\",\n \" print(\\\"Input size = {} - padding\\\".format(inputs.size()))\\n\",\n \" remainder = batch_size - inputs.size()[0]\\n\",\n \" zeros = torch.zeros( [remainder, max_length], dtype=torch.long )\\n\",\n \" inputs = torch.cat( [inputs, zeros] )\\n\",\n \" attention = torch.cat( [attention, zeros] )\\n\",\n \" token_type = torch.cat( [token_type, zeros] )\\n\",\n \"\\n\",\n \" assert(inputs.size()[0] == batch_size and inputs.size()[1] == max_length)\\n\",\n \" assert(attention.size()[0] == batch_size and attention.size()[1] == max_length)\\n\",\n \" assert(token_type.size()[0] == batch_size and token_type.size()[1] == max_length)\\n\",\n \"\\n\",\n \" return (inputs, attention, token_type), quality\\n\",\n \"\\n\",\n \"def count(output, quality):\\n\",\n \" assert output.size(0) >= len(quality)\\n\",\n \" correct_count = 0\\n\",\n \" count = len(quality)\\n\",\n \" \\n\",\n \" batch_predictions = [ row.argmax().item() for row in output ]\\n\",\n \"\\n\",\n \" for a, b in zip(batch_predictions, quality):\\n\",\n \" if int(a)==int(b):\\n\",\n \" correct_count += 1\\n\",\n \"\\n\",\n \" return correct_count, count\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Data parallel inference\\n\",\n \"In the below cell, we use the data parallel approach for inference. In this approach, we load multiple models, all of them running in parallel. Each model is loaded onto a single NeuronCore via the core placement API (``torch_neuron.experimental.neuron_cores_context()``). We also set Neuron Runtime environment variable ``NEURON_RT_MULTI_INSTANCE_SHARED_WEIGHTS`` to \\\"TRUE\\\" as required to use the Weight Sharing feature.\\n\",\n \"\\n\",\n \"In the below implementation, we launch 16 models, thereby utilizing all the 16 cores on an inf1.6xlarge.\\n\",\n \"\\n\",\n \"> Note: Now if you try to decrease the num_cores in the below cells, please restart the notebook and run `!sudo rmmod neuron; sudo modprobe neuron` step in cell 2 to clear the Neuron cores.\\n\",\n \"\\n\",\n \"Since, we can run more than 1 model concurrently, the throughput for the system goes up. To achieve maximum gain in throughput, we need to efficiently feed the models so as to keep them busy at all times. In the below setup, we use parallel threads to feed data continuously to the models.\\n\",\n \"\\n\",\n \"When running the cell below, you can monitor the Inferentia device activities by running ``neuron-top`` in another terminal. You will see that \\\"Device Used Memory\\\" is 1.6GB total, and the model instance loaded onto NeuronDevice 0 NeuronCore 0 uses the most device memory (272MB) while the other model instances loaded onto other NeuronCores use less device memory (92MB). This shows the effect of using Shared Weights as the device memory usage is lower. If you change ``NEURON_RT_MULTI_INSTANCE_SHARED_WEIGHTS`` to \\\"FALSE\\\" you will see that \\\"Device Used Memory\\\" is 3.2GB, and the model instances loaded onto NeuronDevice 0 NeuronCore 0 and 1 use the most device memory (360MB) while the other model instances now use 180MB each.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from bert_benchmark_utils import BertTestDataset, BertResults\\n\",\n \"import time\\n\",\n \"import functools\\n\",\n \"import os\\n\",\n \"import torch.neuron as torch_neuron\\n\",\n \"from concurrent import futures\\n\",\n \"\\n\",\n \"# Setting up NeuronCore groups for inf1.6xlarge with 16 cores\\n\",\n \"num_cores = 16 # This value should be 4 on inf1.xlarge and inf1.2xlarge\\n\",\n \"os.environ['NEURON_RT_NUM_CORES'] = str(num_cores)\\n\",\n \"os.environ['NEURON_RT_MULTI_INSTANCE_SHARED_WEIGHTS'] = 'TRUE'\\n\",\n \"#os.environ['NEURON_RT_MULTI_INSTANCE_SHARED_WEIGHTS'] = 'FALSE'\\n\",\n \"\\n\",\n \"max_length = 128\\n\",\n \"num_cores = 16\\n\",\n \"batch_size = 1\\n\",\n \"\\n\",\n \"tsv_file=\\\"glue_mrpc_dev.tsv\\\"\\n\",\n \"\\n\",\n \"data_set = BertTestDataset( tsv_file=tsv_file, tokenizer=tokenizer, max_length=max_length )\\n\",\n \"data_loader = torch.utils.data.DataLoader(data_set, batch_size=batch_size, shuffle=True)\\n\",\n \"\\n\",\n \"#Result aggregation class (code in bert_benchmark_utils.py)\\n\",\n \"results = BertResults(batch_size, num_cores)\\n\",\n \"def result_handler(output, result_id, start, end, input_dict):\\n\",\n \" correct_count, inference_count = count(output[0], input_dict.pop(result_id))\\n\",\n \" elapsed = end - start\\n\",\n \" results.add_result(correct_count, inference_count, [elapsed], [end], [start])\\n\",\n \"\\n\",\n \"with torch_neuron.experimental.neuron_cores_context(start_nc=0, nc_count=num_cores):\\n\",\n \" model = torch.jit.load('bert_neuron.pt')\\n\",\n \"\\n\",\n \"# Warm up the cores\\n\",\n \"z = torch.zeros( [batch_size, max_length], dtype=torch.long )\\n\",\n \"batch = (z, z, z)\\n\",\n \"for _ in range(num_cores*4):\\n\",\n \" model(*batch)\\n\",\n \"\\n\",\n \"# Prepare the input data\\n\",\n \"batch_list = []\\n\",\n \"for batch in data_loader:\\n\",\n \" batch, quality = get_input_with_padding(batch, batch_size, max_length)\\n\",\n \" batch_list.append((batch, quality))\\n\",\n \"\\n\",\n \"# One thread running a model on one core\\n\",\n \"def one_thread(feed_data, quality):\\n\",\n \" start = time.time()\\n\",\n \" result = model(*feed_data)\\n\",\n \" end = time.time() \\n\",\n \" return result[0], quality, start, end\\n\",\n \"\\n\",\n \"# Launch more threads than models/cores to keep them busy\\n\",\n \"processes = []\\n\",\n \"with futures.ThreadPoolExecutor(max_workers=num_cores*2) as executor:\\n\",\n \" # extra loops to help you see activities in neuron-top\\n\",\n \" for _ in range(10):\\n\",\n \" for input_id, (batch, quality) in enumerate(batch_list):\\n\",\n \" processes.append(executor.submit(one_thread, batch, quality))\\n\",\n \"\\n\",\n \"results = BertResults(batch_size, num_cores)\\n\",\n \"for _ in futures.as_completed(processes): \\n\",\n \" (output, quality, start, end) = _.result() \\n\",\n \" correct_count, inference_count = count(output, quality)\\n\",\n \" results.add_result(correct_count, inference_count, [start - end], [start], [end])\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"w\\\") as f:\\n\",\n \" results.report(f, window_size=1)\\n\",\n \"\\n\",\n \"with open(\\\"benchmark.txt\\\", \\\"r\\\") as f:\\n\",\n \" for line in f:\\n\",\n \" print(line)\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python (torch-neuron)\",\n \"language\": \"python\",\n \"name\": \"aws_neuron_venv_pytorch_inf1\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.10\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/pytorch/byoc_sm_bert_tutorial/code/inference.py", "content": "import os\nimport json\nimport tensorflow # to workaround a protobuf version conflict issue\nimport torch\nimport torch.neuron\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig\n\nJSON_CONTENT_TYPE = 'application/json'\n\n\ndef model_fn(model_dir):\n tokenizer_init = AutoTokenizer.from_pretrained(\"bert-base-cased-finetuned-mrpc\")\n model_file =os.path.join(model_dir, 'neuron_compiled_model.pt')\n model_neuron = torch.jit.load(model_file)\n# print(\"using {}\".format(model_file))\n\n return (model_neuron, tokenizer_init)\n\n\ndef input_fn(serialized_input_data, content_type=JSON_CONTENT_TYPE):\n if content_type == JSON_CONTENT_TYPE:\n input_data = json.loads(serialized_input_data)\n# print(input_data)\n return input_data\n\n else:\n raise Exception('Requested unsupported ContentType in Accept: ' + content_type)\n return\n\n\ndef predict_fn(input_data, models):\n# print('Got input Data: {}'.format(input_data))\n\n model_bert, tokenizer = models\n sequence_0 = input_data[0] \n sequence_1 = input_data[1]\n \n max_length=128\n paraphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\"pt\")\n # Convert example inputs to a format that is compatible with TorchScript tracing\n example_inputs_paraphrase = paraphrase['input_ids'], paraphrase['attention_mask'], paraphrase['token_type_ids'] \n\n # Verify the TorchScript works on example inputs\n paraphrase_classification_logits_neuron = model_bert(*example_inputs_paraphrase)\n classes = ['not paraphrase', 'paraphrase']\n paraphrase_prediction = paraphrase_classification_logits_neuron[0][0].argmax().item()\n out_str = 'BERT says that \"{}\" and \"{}\" are {}'.format(sequence_0, sequence_1, classes[paraphrase_prediction])\n \n return out_str\n\ndef output_fn(prediction_output, accept=JSON_CONTENT_TYPE):\n if accept == JSON_CONTENT_TYPE:\n return json.dumps(prediction_output), accept\n\n raise Exception('Requested unsupported ContentType in Accept: ' + accept)\n\n" }, { "path": "src/examples/pytorch/byoc_sm_bert_tutorial/container/Dockerfile", "content": "FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference-neuron:1.7.1-neuron-py36-ubuntu18.04\n\n# Install packages \nRUN pip install \"transformers==4.7.0\"\n# CMD [\"/usr/local/bin/entrypoint.sh\"]\n\n" }, { "path": "src/examples/pytorch/byoc_sm_bert_tutorial/sagemaker_container_neuron.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"4674f667\",\n \"metadata\": {},\n \"source\": [\n \"# Deploy a pretrained PyTorch BERT model from HuggingFace on Amazon SageMaker with Neuron container\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"b3e39838\",\n \"metadata\": {},\n \"source\": [\n \"## Overview\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"a92c454f\",\n \"metadata\": {},\n \"source\": [\n \"In this tutotial we will deploy on SageMaker a pretraine BERT Base model from HuggingFace Transformers, using the [AWS Deep Learning Containers](https://github.com/aws/deep-learning-containers). We will use the same model as shown in the [Neuron Tutorial \\\"PyTorch - HuggingFace Pretrained BERT Tutorial\\\"](../../../../frameworks/torch/torch-neuronx/tutorials/training/bert.html#). We will compile the model and build a custom AWS Deep Learning Container, to include the HuggingFace Transformers Library. \\n\",\n \"\\n\",\n \"This Jupyter Notebook should run on a ml.c5.4xlarge SageMaker Notebook instance. You can set up your SageMaker Notebook instance by following the [Get Started with Amazon SageMaker Notebook Instances](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-console.html) documentation. \\n\",\n \"\\n\",\n \"> We recommend increasing the size of the base root volume of you SM notebook instance, to accomodate the models and containers built locally. A root volume of 10Gb should suffice. \\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"37445ad2\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"3ecd765f\",\n \"metadata\": {},\n \"source\": [\n \"This tutorial requires the following pip packages:\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"cae3092c\",\n \"metadata\": {},\n \"source\": [\n \"- torch-neuron\\n\",\n \"- neuron-cc[tensorflow]\\n\",\n \"- transformers\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"066c3731\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"!pip install --upgrade --no-cache-dir torch-neuron neuron-cc[tensorflow] torchvision torch --extra-index-url=https://pip.repos.neuron.amazonaws.com\\n\",\n \"!pip install --upgrade --no-cache-dir 'transformers==4.6.0'\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"a4796d3a\",\n \"metadata\": {},\n \"source\": [\n \"## Compile the model into an AWS Neuron optimized TorchScript\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"6fe85f8e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuron\\n\",\n \"\\n\",\n \"from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"0c5c253a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Build tokenizer and model\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(\\\"bert-base-cased-finetuned-mrpc\\\")\\n\",\n \"model = AutoModelForSequenceClassification.from_pretrained(\\\"bert-base-cased-finetuned-mrpc\\\", return_dict=False)\\n\",\n \"\\n\",\n \"# Setup some example inputs\\n\",\n \"sequence_0 = \\\"The company HuggingFace is based in New York City\\\"\\n\",\n \"sequence_1 = \\\"Apples are especially bad for your health\\\"\\n\",\n \"sequence_2 = \\\"HuggingFace's headquarters are situated in Manhattan\\\"\\n\",\n \"\\n\",\n \"max_length=128\\n\",\n \"paraphrase = tokenizer.encode_plus(sequence_0, sequence_2, max_length=max_length, padding='max_length', truncation=True, return_tensors=\\\"pt\\\")\\n\",\n \"not_paraphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\\\"pt\\\")\\n\",\n \"\\n\",\n \"# Run the original PyTorch model on compilation exaple\\n\",\n \"paraphrase_classification_logits = model(**paraphrase)[0]\\n\",\n \"\\n\",\n \"# Convert example inputs to a format that is compatible with TorchScript tracing\\n\",\n \"example_inputs_paraphrase = paraphrase['input_ids'], paraphrase['attention_mask'], paraphrase['token_type_ids']\\n\",\n \"example_inputs_not_paraphrase = not_paraphrase['input_ids'], not_paraphrase['attention_mask'], not_paraphrase['token_type_ids']\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"44255ada\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\\n\",\n \"# This step may need 3-5 min\\n\",\n \"model_neuron = torch.neuron.trace(model, example_inputs_paraphrase, verbose=1, compiler_workdir='./compilation_artifacts')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5c4752ac\",\n \"metadata\": {},\n \"source\": [\n \"You may inspect **model_neuron.graph** to see which part is running on CPU versus running on the accelerator. All native **aten** operators in the graph will be running on CPU.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"dc00889e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# See which part is running on CPU versus running on the accelerator.\\n\",\n \"print(model_neuron.graph)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"775fb30d\",\n \"metadata\": {},\n \"source\": [\n \"Save the compiled model, so it can be packaged and sent to S3.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"027c4f53\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Save the TorchScript for later use\\n\",\n \"model_neuron.save('neuron_compiled_model.pt')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"d362c579\",\n \"metadata\": {},\n \"source\": [\n \"### Package the pre-trained model and upload it to S3\\n\",\n \"\\n\",\n \"To make the model available for the SageMaker deployment, you will TAR the serialized graph and upload it to the default Amazon S3 bucket for your SageMaker session. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"29c7f7b4\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Now you'll create a model.tar.gz file to be used by SageMaker endpoint\\n\",\n \"!tar -czvf model.tar.gz neuron_compiled_model.pt\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"1beadca0\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import boto3\\n\",\n \"import time\\n\",\n \"from sagemaker.utils import name_from_base\\n\",\n \"import sagemaker\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"06ad87d4\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# upload model to S3\\n\",\n \"role = sagemaker.get_execution_role()\\n\",\n \"sess=sagemaker.Session()\\n\",\n \"region=sess.boto_region_name\\n\",\n \"bucket=sess.default_bucket()\\n\",\n \"sm_client=boto3.client('sagemaker')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"5205ec55\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model_key = '{}/model/model.tar.gz'.format('inf1_compiled_model')\\n\",\n \"model_path = 's3://{}/{}'.format(bucket, model_key)\\n\",\n \"boto3.resource('s3').Bucket(bucket).upload_file('model.tar.gz', model_key)\\n\",\n \"print(\\\"Uploaded model to S3:\\\")\\n\",\n \"print(model_path)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e8b425d4\",\n \"metadata\": {},\n \"source\": [\n \"## Build and Push the container\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"430e6ed2\",\n \"metadata\": {},\n \"source\": [\n \"The following shell code shows how to build the container image using docker build and push the container image to ECR using docker push.\\n\",\n \"The Dockerfile in this example is available in the ***container*** folder.\\n\",\n \"Here's an example of the Dockerfile:\\n\",\n \"\\n\",\n \"```Dockerfile\\n\",\n \"FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference-neuron:1.7.1-neuron-py36-ubuntu18.04\\n\",\n \"\\n\",\n \"# Install packages \\n\",\n \"RUN pip install \\\"transformers==4.7.0\\\"\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"3970025d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!cat container/Dockerfile\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"62f78b0f\",\n \"metadata\": {},\n \"source\": [\n \"Before running the next cell, make sure your SageMaker IAM role has access to ECR. If not, you can attache the role `AmazonEC2ContainerRegistryPowerUser` to your IAM role ARN, which allows you to upload image layers to ECR. \\n\",\n \"\\n\",\n \"It takes 5 minutes to build docker images and upload image to ECR\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ecd51acf\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%sh\\n\",\n \"\\n\",\n \"# The name of our algorithm\\n\",\n \"algorithm_name=neuron-py36-inference\\n\",\n \"\\n\",\n \"cd container\\n\",\n \"\\n\",\n \"account=$(aws sts get-caller-identity --query Account --output text)\\n\",\n \"\\n\",\n \"# Get the region defined in the current configuration (default to us-west-2 if none defined)\\n\",\n \"region=$(aws configure get region)\\n\",\n \"region=${region:-us-west-2}\\n\",\n \"\\n\",\n \"fullname=\\\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\\\"\\n\",\n \"\\n\",\n \"# If the repository doesn't exist in ECR, create it.\\n\",\n \"\\n\",\n \"aws ecr describe-repositories --repository-names \\\"${algorithm_name}\\\" > /dev/null 2>&1\\n\",\n \"\\n\",\n \"if [ $? -ne 0 ]\\n\",\n \"then\\n\",\n \" aws ecr create-repository --repository-name \\\"${algorithm_name}\\\" > /dev/null\\n\",\n \"fi\\n\",\n \"\\n\",\n \"# Get the login command from ECR in order to pull down the SageMaker PyTorch image\\n\",\n \"aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-east-1.amazonaws.com\\n\",\n \"# Build the docker image locally with the image name and then push it to ECR\\n\",\n \"# with the full name.\\n\",\n \"docker build -t ${algorithm_name} . --build-arg REGION=${region}\\n\",\n \"docker tag ${algorithm_name} ${fullname}\\n\",\n \"\\n\",\n \"# Get the login command from ECR and execute it directly\\n\",\n \"aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin ${account}.dkr.ecr.${region}.amazonaws.com\\n\",\n \"docker push ${fullname}\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e4f6bbda\",\n \"metadata\": {},\n \"source\": [\n \"## Deploy Container and run inference based on the pretrained model\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"64e65e31\",\n \"metadata\": {},\n \"source\": [\n \"To deploy a pretrained PyTorch model, you'll need to use the PyTorch estimator object to create a PyTorchModel object and set a different entry_point.\\n\",\n \"\\n\",\n \"You'll use the PyTorchModel object to deploy a PyTorchPredictor. This creates a SageMaker Endpoint -- a hosted prediction service that we can use to perform inference.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"f343d3b1\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import sys\\n\",\n \"\\n\",\n \"!{sys.executable} -m pip install Transformers\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"2bd73b77\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import boto3\\n\",\n \"import sagemaker\\n\",\n \"\\n\",\n \"role = sagemaker.get_execution_role()\\n\",\n \"sess = sagemaker.Session()\\n\",\n \"\\n\",\n \"bucket = sess.default_bucket()\\n\",\n \"prefix = \\\"inf1_compiled_model/model\\\"\\n\",\n \"\\n\",\n \"# Get container name in ECR\\n\",\n \"client=boto3.client('sts')\\n\",\n \"account=client.get_caller_identity()['Account']\\n\",\n \"\\n\",\n \"my_session=boto3.session.Session()\\n\",\n \"region=my_session.region_name\\n\",\n \"\\n\",\n \"algorithm_name=\\\"neuron-py36-inference\\\"\\n\",\n \"ecr_image='{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(account, region, algorithm_name)\\n\",\n \"print(ecr_image)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"9298f2a7\",\n \"metadata\": {},\n \"source\": [\n \"An implementation of *model_fn* is required for inference script.\\n\",\n \"We are going to implement our own **model_fn** and **predict_fn** for Hugging Face Bert, and use default implementations of **input_fn** and **output_fn** defined in sagemaker-pytorch-containers.\\n\",\n \"\\n\",\n \"In this example, the inference script is put in ***code*** folder. Run the next cell to see it:\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"cfea75b6\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pygmentize code/inference.py\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"1b31a7b8\",\n \"metadata\": {},\n \"source\": [\n \"Path of compiled pretrained model in S3:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"61f3556e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"key = os.path.join(prefix, \\\"model.tar.gz\\\")\\n\",\n \"pretrained_model_data = \\\"s3://{}/{}\\\".format(bucket, key)\\n\",\n \"print(pretrained_model_data)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e7557a5f\",\n \"metadata\": {},\n \"source\": [\n \"The model object is defined by using the SageMaker Python SDK's PyTorchModel and pass in the model from the estimator and the entry_point. The endpoint's entry point for inference is defined by model_fn as seen in the previous code block that prints out **inference.py**. The model_fn function will load the model and required tokenizer.\\n\",\n \"\\n\",\n \"Note, **image_uri** must be user's own ECR images.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"0bd99768\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from sagemaker.pytorch.model import PyTorchModel\\n\",\n \"\\n\",\n \"pytorch_model = PyTorchModel(\\n\",\n \" model_data=pretrained_model_data,\\n\",\n \" role=role,\\n\",\n \" source_dir=\\\"code\\\",\\n\",\n \" framework_version=\\\"1.7.1\\\",\\n\",\n \" entry_point=\\\"inference.py\\\",\\n\",\n \" image_uri=ecr_image\\n\",\n \")\\n\",\n \"\\n\",\n \"# Let SageMaker know that we've already compiled the model via neuron-cc\\n\",\n \"pytorch_model._is_compiled_model = True\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"67439fe7\",\n \"metadata\": {},\n \"source\": [\n \"The arguments to the deploy function allow us to set the number and type of instances that will be used for the Endpoint.\\n\",\n \"\\n\",\n \"Here you will deploy the model to a single **ml.inf1.2xlarge** instance.\\n\",\n \"It may take 6-10 min to deploy.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"d771fc7c\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"\\n\",\n \"predictor = pytorch_model.deploy(initial_instance_count=1, instance_type=\\\"ml.inf1.2xlarge\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ab6342f3\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print(predictor.endpoint_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"059537d9\",\n \"metadata\": {},\n \"source\": [\n \"Since in the input_fn we declared that the incoming requests are json-encoded, we need to use a json serializer, to encode the incoming data into a json string. Also, we declared the return content type to be json string, we Need to use a json deserializer to parse the response.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"29e82f90\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"predictor.serializer = sagemaker.serializers.JSONSerializer()\\n\",\n \"predictor.deserializer = sagemaker.deserializers.JSONDeserializer()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"d006ea03\",\n \"metadata\": {},\n \"source\": [\n \"Using a list of sentences, now SageMaker endpoint is invoked to get predictions.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"325a87f8\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"result = predictor.predict(\\n\",\n \" [\\n\",\n \" \\\"Never allow the same bug to bite you twice.\\\",\\n\",\n \" \\\"The best part of Amazon SageMaker is that it makes machine learning easy.\\\",\\n\",\n \" ]\\n\",\n \")\\n\",\n \"print(result)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"4a12410d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"result = predictor.predict(\\n\",\n \" [\\n\",\n \" \\\"The company HuggingFace is based in New York City\\\",\\n\",\n \" \\\"HuggingFace's headquarters are situated in Manhattan\\\",\\n\",\n \" ]\\n\",\n \")\\n\",\n \"print(result)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"a72dfd16\",\n \"metadata\": {},\n \"source\": [\n \"## Benchmarking your endpoint\\n\",\n \"\\n\",\n \"The following cells create a load test for your endpoint. You first define some helper functions: `inference_latency` runs the endpoint request, collects cliend side latency and any errors, `random_sentence` builds random to be sent to the endpoint. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"088d0e75\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import numpy as np \\n\",\n \"import datetime\\n\",\n \"import math\\n\",\n \"import time\\n\",\n \"import boto3 \\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"from joblib import Parallel, delayed\\n\",\n \"import numpy as np\\n\",\n \"from tqdm import tqdm\\n\",\n \"import random\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"038d9953\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def inference_latency(model,*inputs):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" infetence_time is a simple method to return the latency of a model inference.\\n\",\n \"\\n\",\n \" Parameters:\\n\",\n \" model: torch model onbject loaded using torch.jit.load\\n\",\n \" inputs: model() args\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" latency in seconds\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" error = False\\n\",\n \" start = time.time()\\n\",\n \" try:\\n\",\n \" results = model(*inputs)\\n\",\n \" except:\\n\",\n \" error = True\\n\",\n \" results = []\\n\",\n \" return {'latency':time.time() - start, 'error': error, 'result': results}\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"d6b200ac\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def random_sentence():\\n\",\n \" \\n\",\n \" s_nouns = [\\\"A dude\\\", \\\"My mom\\\", \\\"The king\\\", \\\"Some guy\\\", \\\"A cat with rabies\\\", \\\"A sloth\\\", \\\"Your homie\\\", \\\"This cool guy my gardener met yesterday\\\", \\\"Superman\\\"]\\n\",\n \" p_nouns = [\\\"These dudes\\\", \\\"Both of my moms\\\", \\\"All the kings of the world\\\", \\\"Some guys\\\", \\\"All of a cattery's cats\\\", \\\"The multitude of sloths living under your bed\\\", \\\"Your homies\\\", \\\"Like, these, like, all these people\\\", \\\"Supermen\\\"]\\n\",\n \" s_verbs = [\\\"eats\\\", \\\"kicks\\\", \\\"gives\\\", \\\"treats\\\", \\\"meets with\\\", \\\"creates\\\", \\\"hacks\\\", \\\"configures\\\", \\\"spies on\\\", \\\"retards\\\", \\\"meows on\\\", \\\"flees from\\\", \\\"tries to automate\\\", \\\"explodes\\\"]\\n\",\n \" p_verbs = [\\\"eat\\\", \\\"kick\\\", \\\"give\\\", \\\"treat\\\", \\\"meet with\\\", \\\"create\\\", \\\"hack\\\", \\\"configure\\\", \\\"spy on\\\", \\\"retard\\\", \\\"meow on\\\", \\\"flee from\\\", \\\"try to automate\\\", \\\"explode\\\"]\\n\",\n \" infinitives = [\\\"to make a pie.\\\", \\\"for no apparent reason.\\\", \\\"because the sky is green.\\\", \\\"for a disease.\\\", \\\"to be able to make toast explode.\\\", \\\"to know more about archeology.\\\"]\\n\",\n \" \\n\",\n \" return (random.choice(s_nouns) + ' ' + random.choice(s_verbs) + ' ' + random.choice(s_nouns).lower() or random.choice(p_nouns).lower() + ' ' + random.choice(infinitives))\\n\",\n \"\\n\",\n \"print([random_sentence(), random_sentence()])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e2945dde\",\n \"metadata\": {},\n \"source\": [\n \"The following cell creates `number_of_clients` concurrent threads to run `number_of_runs` requests. Once completed, a `boto3` CloudWatch client will query for the server side latency metrics for comparison. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"69c047e3\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Defining Auxiliary variables\\n\",\n \"number_of_clients = 2\\n\",\n \"number_of_runs = 1000\\n\",\n \"t = tqdm(range(number_of_runs),position=0, leave=True)\\n\",\n \"\\n\",\n \"# Starting parallel clients\\n\",\n \"cw_start = datetime.datetime.utcnow()\\n\",\n \"\\n\",\n \"results = Parallel(n_jobs=number_of_clients,prefer=\\\"threads\\\")(delayed(inference_latency)(predictor.predict,[random_sentence(), random_sentence()]) for mod in t)\\n\",\n \"avg_throughput = t.total/t.format_dict['elapsed']\\n\",\n \"\\n\",\n \"cw_end = datetime.datetime.utcnow() \\n\",\n \"\\n\",\n \"# Computing metrics and print\\n\",\n \"latencies = [res['latency'] for res in results]\\n\",\n \"errors = [res['error'] for res in results]\\n\",\n \"error_p = sum(errors)/len(errors) *100\\n\",\n \"p50 = np.quantile(latencies[-1000:],0.50) * 1000\\n\",\n \"p90 = np.quantile(latencies[-1000:],0.95) * 1000\\n\",\n \"p95 = np.quantile(latencies[-1000:],0.99) * 1000\\n\",\n \"\\n\",\n \"print(f'Avg Throughput: :{avg_throughput:.1f}\\\\n')\\n\",\n \"print(f'50th Percentile Latency:{p50:.1f} ms')\\n\",\n \"print(f'90th Percentile Latency:{p90:.1f} ms')\\n\",\n \"print(f'95th Percentile Latency:{p95:.1f} ms\\\\n')\\n\",\n \"print(f'Errors percentage: {error_p:.1f} %\\\\n')\\n\",\n \"\\n\",\n \"# Querying CloudWatch\\n\",\n \"print('Getting Cloudwatch:')\\n\",\n \"cloudwatch = boto3.client('cloudwatch')\\n\",\n \"statistics=['SampleCount', 'Average', 'Minimum', 'Maximum']\\n\",\n \"extended=['p50', 'p90', 'p95', 'p100']\\n\",\n \"\\n\",\n \"# Give 5 minute buffer to end\\n\",\n \"cw_end += datetime.timedelta(minutes=5)\\n\",\n \"\\n\",\n \"# Period must be 1, 5, 10, 30, or multiple of 60\\n\",\n \"# Calculate closest multiple of 60 to the total elapsed time\\n\",\n \"factor = math.ceil((cw_end - cw_start).total_seconds() / 60)\\n\",\n \"period = factor * 60\\n\",\n \"print('Time elapsed: {} seconds'.format((cw_end - cw_start).total_seconds()))\\n\",\n \"print('Using period of {} seconds\\\\n'.format(period))\\n\",\n \"\\n\",\n \"cloudwatch_ready = False\\n\",\n \"# Keep polling CloudWatch metrics until datapoints are available\\n\",\n \"while not cloudwatch_ready:\\n\",\n \" time.sleep(30)\\n\",\n \" print('Waiting 30 seconds ...')\\n\",\n \" # Must use default units of microseconds\\n\",\n \" model_latency_metrics = cloudwatch.get_metric_statistics(MetricName='ModelLatency',\\n\",\n \" Dimensions=[{'Name': 'EndpointName',\\n\",\n \" 'Value': predictor.endpoint_name},\\n\",\n \" {'Name': 'VariantName',\\n\",\n \" 'Value': \\\"AllTraffic\\\"}],\\n\",\n \" Namespace=\\\"AWS/SageMaker\\\",\\n\",\n \" StartTime=cw_start,\\n\",\n \" EndTime=cw_end,\\n\",\n \" Period=period,\\n\",\n \" Statistics=statistics,\\n\",\n \" ExtendedStatistics=extended\\n\",\n \" )\\n\",\n \" # Should be 1000\\n\",\n \" if len(model_latency_metrics['Datapoints']) > 0:\\n\",\n \" print('{} latency datapoints ready'.format(model_latency_metrics['Datapoints'][0]['SampleCount']))\\n\",\n \" side_avg = model_latency_metrics['Datapoints'][0]['Average'] / number_of_runs\\n\",\n \" side_p50 = model_latency_metrics['Datapoints'][0]['ExtendedStatistics']['p50'] / number_of_runs\\n\",\n \" side_p90 = model_latency_metrics['Datapoints'][0]['ExtendedStatistics']['p90'] / number_of_runs\\n\",\n \" side_p95 = model_latency_metrics['Datapoints'][0]['ExtendedStatistics']['p95'] / number_of_runs\\n\",\n \" side_p100 = model_latency_metrics['Datapoints'][0]['ExtendedStatistics']['p100'] / number_of_runs\\n\",\n \" \\n\",\n \" print(f'50th Percentile Latency:{side_p50:.1f} ms')\\n\",\n \" print(f'90th Percentile Latency:{side_p90:.1f} ms')\\n\",\n \" print(f'95th Percentile Latency:{side_p95:.1f} ms\\\\n')\\n\",\n \"\\n\",\n \" cloudwatch_ready = True\\n\",\n \"\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"9035e681\",\n \"metadata\": {},\n \"source\": [\n \"### Cleanup\\n\",\n \"Endpoints should be deleted when no longer in use, to avoid costs.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"1284ef3f\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"predictor.delete_endpoint(predictor.endpoint)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"5af53873\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/pytorch/libtorch_demo/bert_neuronx/compile.py", "content": "import torch\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig\nimport transformers\nimport os\nimport warnings\n\nfrom detect_instance import get_instance_type, get_num_neuroncores\n\ninstance_type = get_instance_type() \n\nprint(f\"Detected instance type: {instance_type}\")\n\nif 'inf1' in instance_type:\n print(\" - using torch_neuron.trace\")\n from torch_neuron import trace\nelse:\n print(\" - using torch_neuronx.xla_impl.trace\")\n from torch_neuronx.xla_impl.trace import trace\nprint()\n\nos.environ['TOKENIZERS_PARALLELISM']='false'\nbatch_size = 6\n\n# Setting up NeuronCore groups for inf1.6xlarge with 16 cores\nnum_cores = get_num_neuroncores(instance_type)\nprint(f\"Number of cores = {num_cores}\")\nos.environ['NEURON_RT_NUM_CORES'] = str(num_cores)\n\n# Build tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased-finetuned-mrpc\")\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased-finetuned-mrpc\", return_dict=False)\n\n# Setup some example inputs\nsequence_0 = \"The company HuggingFace is based in New York City\"\nsequence_1 = \"Apples are especially bad for your health\"\nsequence_2 = \"HuggingFace's headquarters are situated in Manhattan\"\n\nmax_length=128\nparaphrase = tokenizer.encode_plus(sequence_0, sequence_2, max_length=max_length, padding='max_length', truncation=True, return_tensors=\"pt\")\nnot_paraphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\"pt\")\n\n# Convert example inputs to a format that is compatible with TorchScript tracing\nexample_inputs_paraphrase = (\n torch.cat([paraphrase['input_ids']] * batch_size,0),\n torch.cat([paraphrase['attention_mask']] * batch_size,0),\n torch.cat([paraphrase['token_type_ids']] * batch_size,0)\n)\n\n# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\ntry:\n model_neuron = trace(model, example_inputs_paraphrase)\nexcept Exception as e:\n print(e)\n print(\"libtorch_demo: Model tracing failed - check tutorial steps and preconditions\")\n print(\"libtorch_demo: If this does not resolve your issue - Report a bug at \")\n print(\"https://github.com/aws-neuron/aws-neuron-sdk/issues\")\n exit(1)\n\n# Verify the TorchScript works on both example inputs\ntry:\n paraphrase_classification_logits_neuron = model_neuron(*example_inputs_paraphrase)\nexcept:\n print(\"libtorch_demo: Neuron runtime failed - check tutorial steps and preconditions\")\n print(\"libtorch_demo: If this does not resolve your issue - Report a bug at \")\n print(\"https://github.com/aws-neuron/aws-neuron-sdk/issues\")\n exit(1)\n\n# Save the TorchScript for later use\nmodel_neuron.save(f'bert_neuron_b{batch_size}.pt')\n" }, { "path": "src/examples/pytorch/libtorch_demo/bert_neuronx/detect_instance.py", "content": "import torch\nimport torch_neuronx\nfrom typing import Optional\n\nINSTANCETYPE_TO_NEURONCORES = {\n \"inf1.xlarge\": 4,\n \"inf1.2xlarge\": 4,\n \"inf1.6xlarge\": 16,\n \"inf2.xlarge\": 2,\n \"inf2.8xlarge\": 2,\n \"inf2.24xlarge\": 12,\n \"inf2.48xlarge\": 24,\n \"inf1.24xlarge\": 64,\n \"trn1.2xlarge\": 2,\n \"trn1.32xlarge\": 32,\n}\n\ndef get_instance_type() -> str:\n \"\"\"Try to obtain the instance type.\"\"\"\n try:\n from urllib.request import Request, urlopen\n\n req = Request(\"http://169.254.169.254/latest/api/token\", method=\"PUT\")\n req.add_header(\"X-aws-ec2-metadata-token-ttl-seconds\", \"21600\")\n with urlopen(req) as response:\n token = response.read().decode(\"utf-8\")\n\n req = Request(\"http://169.254.169.254/latest/meta-data/instance-type\")\n req.add_header(\"X-aws-ec2-metadata-token\", token)\n with urlopen(req) as response:\n instance_type = response.read().decode(\"utf-8\")\n\n return instance_type\n except: # noqa: E722, there are various ways above code can fail and we don't care\n return None\n\n\ndef get_num_neuroncores(instance_type: Optional[str] = None) -> int:\n \"\"\"\n Try to obtain the maximum number of NeuronCores available on this instance.\n\n Args:\n instance_type: The Neuron instance type. Autodetermined from current instance\n if not provided.\n\n Returns:\n The number of NeuronCores (or 2 if the type is unknown).\n \"\"\"\n\n try:\n if not instance_type:\n instance_type = get_instance_type()\n return INSTANCETYPE_TO_NEURONCORES[instance_type]\n except KeyError:\n num_cores = get_num_neuroncores_v3()\n return num_cores\n\n\ndef get_num_neuroncores_v3() -> int:\n \"\"\"\n Retrieve the number of NeuronCores visible to this process.\n\n Returns:\n The number of visible neuron cores.\n\n Raises:\n RuntimeError: If the Neuron runtime cannot be initialized. This most\n commonly occurs when executing on an instance with no Neuron\n devices available or when no Neuron devices are visible to the\n process.\n \"\"\"\n runtime = torch.classes.neuron.Runtime()\n try:\n nc_count = runtime.get_visible_nc_count()\n except RuntimeError as e:\n raise RuntimeError(\n \"Neuron runtime cannot be initialized; cannot determine the number of available NeuronCores\" # noqa: E501\n ) from e\n return nc_count\n" }, { "path": "src/examples/pytorch/libtorch_demo/clean.sh", "content": "#!/bin/bash\n\necho \"Clean up constructed files\"\nrm -rf bert_neuron_b6.pt example-app tokenizers venv/ libtorch/ tokenizers_binding/lib/ tokenizers_binding/venv all_metrics.csv venv\n" }, { "path": "src/examples/pytorch/libtorch_demo/example_app/README.txt", "content": "AWS NEURON TORCHLIB DEMO FOR C++\n================================\n\nFor the full tutorial, please refer to:\nhttps://awsdocs-neuron.readthedocs-hosted.com\n" }, { "path": "src/examples/pytorch/libtorch_demo/example_app/build.sh", "content": "#!/bin/bash\n\n# Installation script to build with torch dependency from /usr/local\nset -x\n\n# Find paths for local packages\nPATH_TOKENIZERS_LIB=../tokenizers_binding/lib\nPATH_TORCH=../libtorch\nPATH_TORCH_INC=${PATH_TORCH}/include\nPATH_TORCH_LIB=${PATH_TORCH}/lib\nPATH_NEURON_LIB=${PATH_TORCH}/lib\n\nif [ ! -e \"${PATH_TORCH_LIB}/libnrt.so.1\" ] && [ -e \"/opt/aws/neuron/lib/libnrt.so.1\" ]\nthen\n PATH_NEURON_LIB=/opt/aws/neuron/lib/\nfi\n\ng++ utils.cpp example_app.cpp \\\n -o ../example-app \\\n -O2 \\\n -D_GLIBCXX_USE_CXX11_ABI=1 \\\n -I${PATH_TORCH_INC} \\\n -L${PATH_TOKENIZERS_LIB} \\\n -L${PATH_NEURON_LIB} \\\n -L${PATH_TORCH_LIB} \\\n -Wl,-rpath,libtorch/lib \\\n -Wl,-rpath,tokenizers_binding/lib \\\n -Wl,-rpath,$PATH_NEURON_LIB \\\n -Wl,-no-as-needed \\\n -ltokenizers \\\n -ltorchneuron \\\n -ltorch_cpu \\\n -lc10 \\\n -lpthread \\\n -lnrt \\\n -std=c++17\n" }, { "path": "src/examples/pytorch/libtorch_demo/example_app/core_count.hpp", "content": "#pragma once\n\n/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n \n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\ntypedef enum {\n NRT_SUCCESS = 0,\n NRT_FAILURE = 1, \n NRT_INVALID = 2, \n NRT_INVALID_HANDLE = 3, \n NRT_RESOURCE = 4, \n NRT_TIMEOUT = 5, \n NRT_HW_ERROR = 6, \n NRT_QUEUE_FULL = 7, \n NRT_LOAD_NOT_ENOUGH_NC = 9, \n NRT_UNSUPPORTED_NEFF_VERSION = 10, \n NRT_FAIL_HOST_MEM_ALLOC = 11, \n NRT_EXEC_BAD_INPUT = 1002, \n NRT_EXEC_COMPLETED_WITH_NUM_ERR = 1003, \n NRT_EXEC_COMPLETED_WITH_ERR = 1004, \n NRT_EXEC_NC_BUSY = 1005, \n NRT_COLL_PENDING = 1100, \n} NRT_STATUS;\n\nNRT_STATUS nrt_get_total_nc_count(uint32_t *nc_count);\n\n#ifdef __cplusplus\n}\n#endif" }, { "path": "src/examples/pytorch/libtorch_demo/example_app/example_app.cpp", "content": "#include \n#include \n#include \n#include \n\n#include \"utils.hpp\"\n#include \"core_count.hpp\"\n#include \"../tokenizers_binding/remote_rust_tokenizer.h\"\n\ntypedef std::vector> Input;\n\nnamespace\n{\n // some hardcoded parameters that could be read from a config file\n const size_t seq_len = 128;\n const size_t batch_size = 6;\n uint32_t num_neuron_cores = 0;\n const size_t cores_per_model = 1;\n const size_t num_runs_per_neuron_core = 2000;\n\n // these token ids are particular to a vocabulary, could be parsed from vocab file\n const long start_token = 101;\n const long end_token = 102;\n}\n\n// construct a single input: input_ids, attention_mask, and token_type_ids from two input sentences\nInput get_input(const std::string& sentence_1, const std::string& sentence_2)\n{\n // ensure the concatenated sentences + separator tokens do not exceed the compiled sequence length\n assert(sentence_1.size() + sentence_2.size() + 3 <= seq_len);\n\n // tokenize the input sentence using the HuggingFace Tokenizers library\n std::vector input_ids(seq_len, 0);\n input_ids[0] = start_token;\n size_t pos = 1; // current write position in input_ids\n\n // tokenize sentence_1 and copy to output buffer\n std::vector buffer(seq_len, 0);\n remote_rust_encode(sentence_1.c_str(), buffer.data(), buffer.size());\n for (size_t i = 0; i < seq_len && buffer[i]; i++, pos++) {\n input_ids[pos] = buffer[i];\n }\n\n // mark end of sentence_1\n input_ids[pos++] = end_token;\n const size_t sentence_2_start = pos;\n\n // tokenize sentence_2 and copy to output buffer\n std::fill(buffer.begin(), buffer.end(), 0);\n remote_rust_encode(sentence_2.c_str(), buffer.data(), buffer.size());\n for (size_t i = 0; i < seq_len && buffer[i]; i++, pos++) {\n input_ids[pos] = buffer[i];\n }\n\n // mark end of sentence_2\n input_ids[pos++] = end_token;\n\n // construct attention mask\n std::vector attention_mask(seq_len, 0);\n for (size_t i = 0; i < seq_len; ++i) attention_mask[i] = input_ids[i] ? 1 : 0;\n\n // token type ids are 0s for sentence_1 (incl. separators), 1s for sentence_2\n std::vector token_type_ids(seq_len, 0);\n for (size_t i = sentence_2_start; i < seq_len; i++) {\n if (!attention_mask[i]) break;\n token_type_ids[i] = 1;\n }\n\n return {input_ids, attention_mask, token_type_ids};\n}\n\n// reshape a vector of inputs into a proper batch\nstd::vector get_batch(const std::vector& inputs)\n{\n // must be given a full batch\n assert(inputs.size() == batch_size);\n\n torch::Tensor input_ids_tensor = torch::zeros({batch_size, seq_len}, at::kLong);\n torch::Tensor attention_mask_tensor = torch::zeros({batch_size, seq_len}, at::kLong);\n torch::Tensor token_type_ids_tensor = torch::zeros({batch_size, seq_len}, at::kLong);\n\n const auto opts = torch::TensorOptions().dtype(torch::kLong);\n for (size_t i = 0; i < batch_size; i++) {\n input_ids_tensor.slice(0, i, i+1) = torch::from_blob((void*)inputs[i][0].data(), {seq_len}, opts);\n attention_mask_tensor.slice(0, i, i+1) = torch::from_blob((void*)inputs[i][1].data(), {seq_len}, opts);\n token_type_ids_tensor.slice(0, i, i+1) = torch::from_blob((void*)inputs[i][2].data(), {seq_len}, opts);\n }\n\n return {input_ids_tensor, attention_mask_tensor, token_type_ids_tensor};\n}\n\nint sanity_check(const std::string& model_filename)\n{\n // load the model\n auto model = get_model(model_filename);\n\n // construct some example inputs\n const std::string sentence_1 = \"The company HuggingFace is based in New York City\";\n const std::string sentence_2 = \"Apples are especially bad for your health\";\n const std::string sentence_3 = \"HuggingFace's headquarters are situated in Manhattan\";\n const auto paraphrase = get_input(sentence_1, sentence_3);\n const auto not_paraphrase = get_input(sentence_1, sentence_2);\n\n // batch the inputs 50/50 positive/negative\n std::vector inputs(batch_size);\n for (size_t i = 0; i < batch_size; ++i) {\n if (i < batch_size / 2) {\n inputs[i] = paraphrase;\n } else {\n inputs[i] = not_paraphrase;\n }\n }\n const auto batch = get_batch(inputs);\n\n // forward pass\n const auto output = model.forward(batch);\n\n // interpret output\n const auto output_tensor = output.toTuple()->elements()[0].toTensor();\n const auto paraphrase_probabilities = torch::softmax(output_tensor[0], 0);\n const auto not_paraphrase_probabilities = torch::softmax(output_tensor[batch_size-1], 0);\n const auto paraphrase_0 = std::round(paraphrase_probabilities[0].item() * 100);\n const auto paraphrase_1 = std::round(paraphrase_probabilities[1].item() * 100);\n const auto not_paraphrase_0 = std::round(not_paraphrase_probabilities[0].item() * 100);\n const auto not_paraphrase_1 = std::round(not_paraphrase_probabilities[1].item() * 100);\n\n std::cout << sentence_1 << std::endl << sentence_3 << std::endl;\n std::cout << \"not paraphrase: \" << paraphrase_0 << \"%\" << std::endl;\n std::cout << \"paraphrase: \" << paraphrase_1 << \"%\" << std::endl;\n if (paraphrase_0 >= paraphrase_1) return -1;\n\n std::cout << std::endl;\n\n std::cout << sentence_1 << std::endl << sentence_2 << std::endl;\n std::cout << \"not paraphrase: \" << not_paraphrase_0 << \"%\" << std::endl;\n std::cout << \"paraphrase: \" << not_paraphrase_1 << \"%\" << std::endl;\n if (not_paraphrase_0 <= not_paraphrase_1) return -2;\n\n return 0;\n}\n\nvoid benchmark(const std::string& model_filename, const std::vector& batch,\n std::condition_variable& warmup_cv, std::atomic_size_t& warmup_count,\n std::condition_variable& ready_cv)\n{\n // load model and warmup\n auto model = get_model(model_filename);\n model.forward(batch);\n std::cout << \".\" << std::flush;\n --warmup_count;\n warmup_cv.notify_one();\n\n // wait for ready signal\n std::mutex ready_mutex;\n std::unique_lock lk(ready_mutex);\n ready_cv.wait(lk);\n\n // benchmark\n for (size_t i = 0; i < num_runs_per_neuron_core; i++) {\n if (i == num_runs_per_neuron_core/2) std::cout << \".\" << std::flush;\n model.forward(batch);\n }\n}\n\nint main(int argc, char *argv[])\n{\n if (argc < 2) {\n std::cerr << \"Usage: ./example_app neuron_traced_model.pt [--sanity]\" << std::endl;\n return -1;\n }\n\n if( nrt_get_total_nc_count( &num_neuron_cores ) != NRT_SUCCESS ) {\n std::cerr << \"Could not determine number of cores - aborting!\" << std::endl;\n return -1;\n }\n\n // let runtime know we want M models / core for N cores (e.g. \"1,1,1,1\")\n setenv(\"NEURON_RT_VISIBLE_CORES\", get_visible_cores_str(num_neuron_cores, cores_per_model).c_str(), true);\n\n if (argc >= 3 && std::string(\"--sanity\") == argv[2]) {\n return sanity_check(argv[1]);\n }\n\n /*************************************************************************/\n // prepare inputs, prepare models, and perform warmup inference\n\n std::cout << \"Getting ready\" << std::flush;\n\n const auto input = get_input(\"This sentence is for benchmarking.\", \"For benchmarking, use this sentence.\");\n const auto batch = get_batch(std::vector(batch_size, input));\n\n std::condition_variable warmup_cv, ready_cv;\n std::atomic_size_t warmup_count(num_neuron_cores);\n std::vector threads(num_neuron_cores);\n for (size_t i = 0; i < threads.size(); i++) {\n threads[i] = std::move(std::thread(benchmark, argv[1], batch, std::ref(warmup_cv),\n std::ref(warmup_count), std::ref(ready_cv)));\n }\n\n // wait for warmup to complete\n auto is_warmup_complete = [](std::atomic_size_t& warmup_count) { return warmup_count.load() == 0; };\n std::mutex warmup_mutex;\n std::unique_lock lk(warmup_mutex);\n warmup_cv.wait(lk, std::bind(is_warmup_complete, std::ref(warmup_count)));\n std::cout << std::endl;\n\n /*************************************************************************/\n // begin timed benchmarking\n\n std::cout << \"Benchmarking\" << std::flush;\n\n // signal workers to begin benchmarking and wait for completion\n const auto start_time = std::chrono::high_resolution_clock::now();\n ready_cv.notify_all();\n for (auto& thread : threads) thread.join();\n const auto end_time = std::chrono::high_resolution_clock::now();\n std::cout << std::endl;\n\n // report statistics\n const float elapsed = (end_time - start_time) / std::chrono::seconds(1);\n const size_t num_inferences = num_neuron_cores * num_runs_per_neuron_core;\n const float throughput = (float)(num_inferences * batch_size) / elapsed;\n std::cout << \"Completed \" << num_inferences << \" operations in \" << elapsed << \" seconds => \" << throughput << \" pairs / second\" << std::endl;\n\n std::cout << std::endl;\n std::cout << \"====================\" << std::endl;\n std::cout << \"Summary information:\" << std::endl;\n std::cout << \"====================\" << std::endl;\n std::cout << \"Batch size = \" << batch_size << std::endl;\n std::cout << \"Num neuron cores = \" << num_neuron_cores << std::endl;\n std::cout << \"Num runs per neuron core = \" << num_runs_per_neuron_core << std::endl;\n\n return 0;\n}\n" }, { "path": "src/examples/pytorch/libtorch_demo/example_app/utils.cpp", "content": "#include \"utils.hpp\"\n#include \"../tokenizers_binding/remote_rust_tokenizer.h\"\n\n#include \n#include \n\n#include \n#include \n\nstd::string get_visible_cores_str(size_t num_neuron_cores, size_t cores_per_model)\n{\n std::ostringstream oss;\n oss << \"0-\" << ((num_neuron_cores * cores_per_model) - 1);\n return oss.str();\n}\n\nstd::string get_uuid()\n{\n // xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx\n // M = version = 4, (4 bits, 0100 = 0x4)\n // N = variant = 1, (2 bits, 10XX = 0x{8, 9, A, B})\n\n static const char *chars = \"0123456789abcdef\";\n static std::random_device rd;\n static std::mt19937 mt(rd());\n static std::uniform_int_distribution<> dist(0, 15);\n\n std::stringstream ss;\n for (size_t i = 0; i < 37; i++) {\n const int index = dist(mt);\n ss << chars[index];\n }\n\n // variant bits are 10XX\n std::stringstream variant_ss;\n size_t variant;\n variant_ss << std::hex << chars[dist(mt)];\n variant_ss >> variant;\n variant = 0x8 | (0x3 & variant);\n\n ss.seekp(9); ss << \"-\";\n ss.seekp(14); ss << \"-4\";\n ss.seekp(19); ss << \"-\" << std::hex << variant;\n ss.seekp(24); ss << \"-\";\n return ss.str();\n}\n\ntorch::jit::script::Module get_model(const std::string& filename)\n{\n torch::jit::script::Module model = torch::jit::load(filename);\n\n // If you're using a model traced with torch-neuron >= 1.8, \n // the section below is no longer necessary. It was a workaround \n // for a runtime issue when loading identical copies of a model.\n\n // This is redundant in the new flow, but left to provide future \n // pointer on torchscript graph manipulation if needed\n\n // this next section adds a unique uuid to the graph, so that the neuron runtime\n // will load the graph multiple times instead of reusing a previously loaded copy\n\n /*\n auto fwd = model.get_method(\"forward\");\n auto& fn = static_cast(fwd.function());\n auto graph = fn.graph();\n\n torch::jit::Inline(*graph);\n for (auto node : graph->nodes()) {\n if (std::string(node->kind().toQualString()).rfind(\"neuron::forward\") == 0) {\n auto uuid_input_tensor = node->inputs()[1];\n if (std::string(uuid_input_tensor->node()->kind().toQualString()).rfind(\"prim::Constant\") == 0) {\n // we clone the tensor to retain ownership of \"the blob\" after it goes out of scope\n const std::string uuid = get_uuid();\n torch::Tensor t = torch::from_blob((void*)uuid.c_str(), {36}, torch::kUInt8).clone();\n\n // if we don't move the insertion point so that the copy of the constant appears after the operator,\n // the inference will crash\n graph->setInsertPoint(node);\n torch::jit::Value *val = graph->insertConstant(t);\n node->replaceInputWith(uuid_input_tensor, val);\n\n // ensure a valid graph\n graph->lint();\n }\n }\n }\n */\n\n return model;\n}\n" }, { "path": "src/examples/pytorch/libtorch_demo/example_app/utils.hpp", "content": "#ifndef __UTILS_HPP__\n#define __UTILS_HPP__\n\n#include \n\nstd::string get_visible_cores_str(size_t num_neuron_cores, size_t cores_per_model);\nstd::string get_uuid();\ntorch::jit::script::Module get_model(const std::string& filename);\n\n#endif // __UTILS_HPP__\n" }, { "path": "src/examples/pytorch/libtorch_demo/neuron.patch", "content": "\nFrom 3f126613c47e4261d0e86520cb6e85c5713e2b15 Mon Sep 17 00:00:00 2001\nFrom: Stephen Dunn \nDate: Tue, 26 Jan 2021 22:55:40 +0000\nSubject: [PATCH] Adds AWS Neuron native C++ interface\n\n---\ndiff --git a/tokenizers/Cargo.toml b/tokenizers/Cargo.toml\nindex c0f1aff..9767da7 100644\n--- a/tokenizers/Cargo.toml\n+++ b/tokenizers/Cargo.toml\n@@ -19,6 +19,7 @@ exclude = [ \"rust-toolchain\", \"target/*\", \"Cargo.lock\", \"benches/*.txt\", \"benche\n name = \"tokenizers\"\n path = \"src/lib.rs\"\n bench = false\n+crate-type = [\"rlib\", \"cdylib\"]\n \n [[bench]]\n name = \"bpe_benchmark\"\ndiff --git a/tokenizers/src/lib.rs b/tokenizers/src/lib.rs\nindex eb89b93..2392f28 100644\n--- a/tokenizers/src/lib.rs\n+++ b/tokenizers/src/lib.rs\n@@ -145,6 +145,8 @@ pub mod tokenizer;\n // Re-export from tokenizer\n pub use tokenizer::*;\n \n+mod neuron;\n+\n // Re-export also parallelism utils\n pub use utils::parallelism;\n \ndiff --git a/b_tokenizers/tokenizers/src/neuron.rs b/tokenizers/src/neuron.rs\nnew file mode 100644\nindex 0000000..af4a679\n--- /dev/null\n+++ b/tokenizers/src/neuron.rs\n@@ -0,0 +1,25 @@\n+use crate::tokenizer::Tokenizer;\n+use std::ffi::CStr;\n+use std::os::raw::c_char;\n+\n+// cached tokenizer\n+static mut TOKENIZER: Option = None;\n+\n+#[no_mangle]\n+pub unsafe extern \"C\" fn remote_rust_encode(input_arr: *const c_char, output_arr: *mut u32, output_arr_len: u32) {\n+ // load the pretrained tokenizer up if we haven't already\n+ let tokenizer = TOKENIZER.get_or_insert_with(|| Tokenizer::from_file(\"./tokenizer.json\").unwrap());\n+\n+ // convert input from C -> Rust\n+ let cstr = CStr::from_ptr(input_arr);\n+ let input = cstr.to_str().unwrap();\n+\n+ // tokenize raw text\n+ let encoding = tokenizer.encode(input, false).unwrap();\n+\n+ // hand the output back to C across shared memory\n+ let output = std::slice::from_raw_parts_mut(output_arr, output_arr_len as usize);\n+ for (i, token) in &mut encoding.get_ids().to_vec().iter().enumerate() {\n+ output[i] = *token;\n+ }\n+}\n\\ No newline at end of file\n" }, { "path": "src/examples/pytorch/libtorch_demo/run_tests.sh", "content": "#!/bin/bash\n\nset -e\n\nif [ \"$#\" -ne 1 ]; then\n echo \"usage: ./run_tests.sh model_filename.pt\"\n exit 1\nfi\n\necho -e \"\\nRunning tokenization sanity checks.\\n\"\npushd tokenizers_binding 2>&1 >/dev/null\nchmod +x run_python.sh run.sh\n(./run_python.sh && ./run.sh) || { echo \"Sanity checks failed.\"; exit 2; }\npopd 2>&1 >/dev/null\necho -e \"\\nTokenization sanity checks passed.\"\n\necho -e \"Running end-to-end sanity check.\\n\"\n(./example-app $1 --sanity) || { echo \"Sanity check failed.\"; exit 3; }\necho -e \"\\nSanity check passed.\\n\"" }, { "path": "src/examples/pytorch/libtorch_demo/setup.sh", "content": "#!/bin/bash\n\nset -eEx\n\n# Fail on error\nset -e\n\nTORCH_VERSION=$(python -c \"import torch; v=torch.__version__.split('+')[0]; print(f'{v}')\")\n\n#Parse cli\nwhile [ \"$1\" != \"\" ]; do\n case $1 in\n --torch-version ) shift\n TORCH_VERSION=$1\n ;;\n esac\n shift\ndone\n\necho \"Using PyTorch version ${TORCH_VERSION}\"\n\n# Python setup\nPYTHON=python3\nPYTHON_VERSION=$($PYTHON --version | cut -f2 -d' ' | cut -f1,2 -d'.')\necho \"Python version is '$PYTHON_VERSION'\"\n\nOLD_TOOL_CHAIN=$($PYTHON -c \\\n \"from bert_neuronx.detect_instance import get_instance_type; print('inf1' in get_instance_type())\")\n\nif [ \"$OLD_TOOL_CHAIN\" == \"True\" ]; then\n TORCH_VERSION=\"1.13\"\n echo \"- Detected inf1 - using version ${TORCH_VERSION}\"\nelse\n echo \"- Detected inf2 or trn1 - using version ${TORCH_VERSION}\"\nfi\n\n# checkout tokenizers and apply neuron patch\nif [ ! -e \"tokenizers\" ]; then\n git clone https://github.com/huggingface/tokenizers.git\n cp neuron.patch tokenizers/neuron.patch\n pushd tokenizers\n git checkout d8c4388166cad8f0216dfc485efd6207a3275af2\n git apply neuron.patch\n rm neuron.patch\n popd\nfi\n\n# build tests\npushd tokenizers_binding\nchmod +x build.sh\n./build.sh\npopd\ncp -f tokenizers_binding/tokenizer.json .\n\n# setup torch\nif [ ! -e \"libtorch\" ]; then\n # Use different download paths based on PyTorch version\n MAJOR_VERSION=$(echo \"${TORCH_VERSION}\" | cut -d. -f1)\n MINOR_VERSION=$(echo \"${TORCH_VERSION}\" | cut -d. -f2)\n \n if [ \"$MAJOR_VERSION\" -gt 2 ] || ([ \"$MAJOR_VERSION\" -eq 2 ] && [ \"$MINOR_VERSION\" -ge 8 ]); then\n wget -q https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-${TORCH_VERSION}%2Bcpu.zip\n unzip -q libtorch-shared-with-deps-${TORCH_VERSION}+cpu.zip\n rm -f libtorch-shared-with-deps-${TORCH_VERSION}+cpu.zip\n else\n wget -q https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-${TORCH_VERSION}%2Bcpu.zip\n unzip -q libtorch-cxx11-abi-shared-with-deps-${TORCH_VERSION}+cpu.zip\n rm -f libtorch-cxx11-abi-shared-with-deps-${TORCH_VERSION}+cpu.zip\n fi\nfi\n\n# get libneuron_op.so and install into libtorch\n$PYTHON -m pip install --upgrade \"transformers==4.40.0\"\n$PYTHON bert_neuronx/compile.py\n\nsite_pkgs_dir=$($PYTHON -c \"import site; print(site.getsitepackages()[0])\")\nif [ \"$OLD_TOOL_CHAIN\" == \"True\" ]\n then\n cp -f $(find $site_pkgs_dir -exec find {} -type f -name 'libtorchneuron.so' \\; -quit | grep torch_neuron) libtorch/lib/\n cp -f $(find $site_pkgs_dir -exec find {} -type f -name 'libnrt.so' \\; -quit ) libtorch/lib/\n cp -f $(find $site_pkgs_dir -exec find {} -type f -name 'libnrt.so.1' \\; -quit ) libtorch/lib/\n else\n cp -f $(find $site_pkgs_dir -exec find {} -type f -name 'libtorchneuron.so' \\; -quit | grep torch_neuronx) libtorch/lib/\nfi\n\n# compile example app\npushd example_app\nchmod +x build.sh\n./build.sh\npopd\n\nchmod +x run_tests.sh\necho \"Successfully completed setup\"\n" }, { "path": "src/examples/pytorch/libtorch_demo/tokenizers_binding/build.sh", "content": "#!/bin/bash\n\n# clean old artifacts\nrm tokenizer_test 2>&1 >/dev/null\nrm -rf lib 2>&1 >/dev/null\n\n# build shared library\nif [ $# -eq 0 ]; then\n pushd ../tokenizers/tokenizers\n echo \"Building release test...\"\n cargo build --release\n popd\n cp -r ../tokenizers/tokenizers/target/release lib\n g++ -O3 -o tokenizer_test tokenizer_test.cpp -L./lib -ltokenizers\nelse\n pushd ../tokenizers/tokenizers\n echo \"Building debug test...\"\n cargo build\n popd\n cp -r ../tokenizers/tokenizers/target/debug lib\n g++ -O0 -o tokenizer_test tokenizer_test.cpp -L./lib -ltokenizers\nfi\n\nif [ ! -e \"tokenizer.json\" ]; then\n wget https://huggingface.co/bert-base-cased-finetuned-mrpc/raw/main/tokenizer.json\nfi\n" }, { "path": "src/examples/pytorch/libtorch_demo/tokenizers_binding/remote_rust_tokenizer.h", "content": "#ifndef __REMOTE_RUST_TOKENIZER_H__\n#define __REMOTE_RUST_TOKENIZER_H__\n\n#include \n\nextern \"C\" {\n extern void remote_rust_encode(const char *input_arr, uint32_t* output_arr, uint32_t output_arr_len);\n}\n\n#endif // __REMOTE_RUST_TOKENIZER_H__\n" }, { "path": "src/examples/pytorch/libtorch_demo/tokenizers_binding/run.sh", "content": "#!/bin/bash\n\nset -e\n\nLD_LIBRARY_PATH=./lib ./tokenizer_test\n" }, { "path": "src/examples/pytorch/libtorch_demo/tokenizers_binding/run_python.sh", "content": "#!/bin/bash\n\nset -e\n\npython tokenizer_test.py\n" }, { "path": "src/examples/pytorch/libtorch_demo/tokenizers_binding/tokenizer_test.cpp", "content": "#include \n#include // timing\n#include // rust interface\n#include // std::setprecision\n#include // parse args\n#include \n\n#include \"remote_rust_tokenizer.h\"\n\n#define DEFAULT_NUM_TESTS 10000u\n\nint main(int argc, char *argv[]) {\n // prepare some input to tokenize\n const uint32_t seq_len = 128;\n const std::vector ground_truth = { 1409, 1917, 2947, 16193, 117, 1142, 3087, 1209, 1129, 22559, 2200, 1656, 155, 8954, 119 };\n const char *input_arr = \"If everything goes smoothly, this text will be tokenized inside Rust.\";\n uint32_t* output_arr = new uint32_t[seq_len];\n std::memset(output_arr, 0, sizeof(uint32_t) * seq_len);\n\n // call rust tokenizer\n remote_rust_encode(input_arr, output_arr, seq_len);\n\n // check output\n std::cout << \"Sanity check \";\n for (auto i = 0; i < ground_truth.size(); ++i) {\n if (output_arr[i] != ground_truth[i]) {\n std::cerr << \"failed at: \" << i << \", \" << output_arr[i] << \" != \" << ground_truth[i] << std::endl;\n return -1;\n }\n }\n std::cout << \"passed.\" << std::endl;\n\n // run timed test\n uint32_t num_tests = DEFAULT_NUM_TESTS;\n if (argc >= 3 && !strcmp(\"--num_tests\", argv[1])) {\n std::istringstream iss(argv[2]);\n iss >> num_tests;\n }\n\n const uint32_t ten_percent = uint32_t(0.1 * num_tests);\n std::cout << \"Begin \" << num_tests << \" timed tests.\" << std::endl;\n auto start = std::chrono::high_resolution_clock::now();\n\n for (auto test_num = 0; test_num < num_tests; ++test_num) {\n if (test_num % ten_percent == 0) {\n std::cout << \".\" << std::flush;\n }\n remote_rust_encode(input_arr, output_arr, seq_len);\n }\n\n auto end = std::chrono::high_resolution_clock::now();\n auto duration = std::chrono::duration(end - start);\n std::cout << std::endl << \"End timed tests.\" << std::endl << \"C++ took \"\n << std::setprecision(3) << duration.count()\n << \" seconds.\" << std::endl;\n\n return 0;\n}\n" }, { "path": "src/examples/pytorch/libtorch_demo/tokenizers_binding/tokenizer_test.py", "content": "from transformers import AutoTokenizer\nimport argparse\nimport time\nfrom tqdm import tqdm\n\nparser = argparse.ArgumentParser()\nparser.add_argument('--num_tests', type=int, default=10_000)\nargs = parser.parse_args()\n\ntokenizer = AutoTokenizer.from_pretrained('bert-base-cased-finetuned-mrpc')\n\nstart = time.time()\nfor _ in tqdm(range(args.num_tests), desc='Tokenizing'):\n tokenizer.encode(\"If everything goes smoothly, this text will be tokenized inside Rust.\")\nend = time.time()\nprint('Python took {:.2f} seconds.'.format(end - start))\n" }, { "path": "src/examples/pytorch/libtorch_demo/trace_bert_neuron.py", "content": "import torch\nimport torch_neuron\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Build tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased-finetuned-mrpc\")\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased-finetuned-mrpc\", return_dict=False)\n\n# Setup some example inputs\nsequence_0 = \"The company HuggingFace is based in New York City\"\nsequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n\nmax_length = 128\nbatch_size = 6\n\nparaphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\"pt\")\n\nexample_inputs_paraphrase = (\n torch.cat([paraphrase['input_ids']] * batch_size, 0),\n torch.cat([paraphrase['attention_mask']] * batch_size, 0),\n torch.cat([paraphrase['token_type_ids']] * batch_size, 0)\n)\n\n# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\nmodel_neuron_batch = torch_neuron.trace(model, example_inputs_paraphrase)\n\n# Save the batched model\nmodel_neuron_batch.save('bert_neuron_b{}.pt'.format(batch_size))" }, { "path": "src/examples/pytorch/mnist_mlp/train_monitor.py", "content": "import os\nimport time\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom torchvision.datasets import mnist\nfrom torch.optim import SGD\nfrom torch.utils.data import DataLoader\nfrom torchvision.transforms import ToTensor\n\n# XLA imports\nimport torch_xla.core.xla_model as xm\n\n# Declare 3-layer MLP for MNIST dataset\nclass MLP(nn.Module):\n def __init__(self, input_size = 28 * 28, output_size = 10, layers = [120, 84]):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return F.log_softmax(x, dim=1)\n\n# Load MNIST train dataset\ntrain_dataset = mnist.MNIST(root='./MNIST_DATA_train', \\\n train=True, download=True, transform=ToTensor())\n\ndef main():\n # Prepare data loader\n train_loader = DataLoader(train_dataset, batch_size=32)\n\n # Fix the random number generator seeds for reproducibility\n torch.manual_seed(0)\n\n # XLA: Specify XLA device (defaults to a NeuronCore on Trn1 instance)\n device = 'xla'\n\n # Move model to device and declare optimizer and loss function\n model = MLP().to(device)\n optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n loss_fn = torch.nn.NLLLoss()\n\n # Run the training loop\n print('----------Training ---------------')\n for run in range(0, 1000):\n print(f'Run {run}')\n model.train()\n for idx, (train_x, train_label) in enumerate(train_loader):\n optimizer.zero_grad()\n train_x = train_x.view(train_x.size(0), -1)\n train_x = train_x.to(device)\n train_label = train_label.to(device)\n output = model(train_x)\n loss = loss_fn(output, train_label)\n loss.backward()\n optimizer.step()\n xm.mark_step() # XLA: collect ops and run them in XLA runtime\n if idx < 2: # skip warmup iterations\n start = time.time()\n\n # Save checkpoint for evaluation\n os.makedirs(\"checkpoints\", exist_ok=True)\n checkpoint = {'state_dict': model.state_dict()}\n # XLA: use xm.save instead of torch.save to ensure states are moved back to cpu\n # This can prevent \"XRT memory handle not found\" at end of test.py execution\n xm.save(checkpoint,'checkpoints/checkpoint.pt')\n\n print('----------End Training ---------------')" }, { "path": "src/examples/pytorch/mnist_mlp/train_tb.py", "content": "import os\nimport time\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom torchvision.datasets import mnist\nfrom torch.optim import SGD\nfrom torch.utils.data import DataLoader\nfrom torchvision.transforms import ToTensor\n\n# XLA imports\nimport torch_xla.core.xla_model as xm\n\nfrom torch.utils.tensorboard import SummaryWriter\n\n# Declare 3-layer MLP for MNIST dataset\nclass MLP(nn.Module):\n def __init__(self, input_size = 28 * 28, output_size = 10, layers = [120, 84]):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return F.log_softmax(x, dim=1)\n\n# Load MNIST train dataset\ntrain_dataset = mnist.MNIST(root='./MNIST_DATA_train', \\\n train=True, download=True, transform=ToTensor())\n\ndef main():\n # Prepare data loader\n train_loader = DataLoader(train_dataset, batch_size=32)\n\n # Fix the random number generator seeds for reproducibility\n torch.manual_seed(0)\n \n # XLA: Specify XLA device (defaults to a NeuronCore on Trn1 instance)\n device = 'xla'\n \n # Move model to device and declare optimizer and loss function\n model = MLP().to(device)\n optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n loss_fn = torch.nn.NLLLoss()\n\n # Use SummaryWriter to generate logs for TensorBoard\n writer = SummaryWriter('./output')\n\n # Run the training loop\n print('----------Training ---------------')\n model.train()\n start = time.time()\n for idx, (train_x, train_label) in enumerate(train_loader):\n optimizer.zero_grad()\n train_x = train_x.view(train_x.size(0), -1)\n train_x = train_x.to(device)\n train_label = train_label.to(device)\n output = model(train_x)\n loss = loss_fn(output, train_label)\n writer.add_scalar(\"step loss\", loss, idx) # add the step loss to the TensorBoard logs\n loss.backward()\n optimizer.step()\n xm.mark_step() # XLA: collect ops and run them in XLA runtime\n if idx < 2: # skip warmup iterations\n start = time.time()\n \n # Compute statistics\n interval = idx - 2 # skip warmup iterations\n throughput = interval / (time.time() - start)\n print(\"Train throughput (iter/sec): {}\".format(throughput))\n print(\"Final loss is {:0.4f}\".format(loss.detach().to('cpu')))\n \n # Ensure TensorBoard logs are all written\n writer.flush()\n\n # Save checkpoint for evaluation\n os.makedirs(\"checkpoints\", exist_ok=True)\n checkpoint = {'state_dict': model.state_dict()}\n # XLA: use xm.save instead of torch.save to ensure states are moved back to cpu\n # This can prevent \"XRT memory handle not found\" at end of test.py execution\n xm.save(checkpoint,'checkpoints/checkpoint.pt')\n \n print('----------End Training ---------------')\n \nif __name__ == '__main__':\n main()" }, { "path": "src/examples/pytorch/neuronx_distributed/t5-inference/t5-inference-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# [Broken] T5 inference with Tensor Parallelism\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"This is an extension to the [t5 inference tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/torch-neuronx/t5-inference-tutorial.html). Here we will use NeuronxDistributed to improve the inference performance using tensor parallelism.\\n\",\n \"\\n\",\n \"This tutorial has the following main sections:\\n\",\n \"\\n\",\n \"1. Install dependencies\\n\",\n \"1. Plug in `NeuronxDistributed` layers into T5\\n\",\n \"1. Compile the T5 model\\n\",\n \"1. Run distributed inference with beam search \\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on a Inf2 instance (`inf2.24xlarge`) or Trn1 isntance (`trn1.32xlarge`)\\n\",\n \"\\n\",\n \"> The tutorial works for t5 and flan-t5 models. In this notebook we will run distributed inference with flan-t5-xl.\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"The code in this tutorial is written for Jupyter Notebooks. To use Jupyter Notebook on the Neuron instance, you\\n\",\n \"can use this [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html).\\n\",\n \"\\n\",\n \"Run the notebook by cloning aws-neuron-sdk\\n\",\n \"```\\n\",\n \"git clone https://github.com/aws-neuron/aws-neuron-sdk.git\\n\",\n \"cd aws-neuron-sdk/src/examples/pytorch/neuronx_distributed/t5-inference/\\n\",\n \"```\\n\",\n \"\\n\",\n \"Once done execute `t5-inference-tutorial.ipynb`\\n\",\n \"\\n\",\n \"It is recommended to go through the [t5 inference tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/torch-neuronx/t5-inference-tutorial.html) before you start this tutorial. \\n\",\n \"In addition to the dependencies in the [t5 inference tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/torch-neuronx/t5-inference-tutorial.html), we need to install neuronx-distributed. \\n\",\n \"\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuronx`\\n\",\n \"- `neuronx-cc`\\n\",\n \"- `transformers`\\n\",\n \"- `optimum-neuron`\\n\",\n \"- `neuronx-distributed`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Trn1/Inf2 [ setup guide ](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu20.html#setup-torch-neuronx-ubuntu20). The additional dependencies must be installed here:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"! pip install --upgrade transformers==4.33.1 optimum-neuron neuronx_distributed --extra-index-url https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Pull the latest version of the compiler\\n\",\n \"! pip install --upgrade neuronx-cc>=2.11 --no-deps\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Lets update numpy to a newer version \\n\",\n \"! pip install --upgrade \\\"numpy>=1.22.2,<2\\\" --no-deps\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": []\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Plug in NeuronxDistributed layers into T5\\n\",\n \"\\n\",\n \"We extend the huggingface's T5 model to use the `NeuronxDistributed` parallel layers. To do so, we simply swap linear layers in `T5LayerSelfAttention`, `T5LayerCrossAttention`, and `T5LayerFF` definitions with `ColumnParallelLinear` and `RowParallelLinear`. We also need to swap the `Embedding` layer with `ParallelEmbedding`.\\n\",\n \"\\n\",\n \"Let us take the example of T5Attention. The [attention block](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#L363-L366) has q, k, v, and o linear layers. \\n\",\n \"The multi-head attention block uses q, k and v to compute the attention scores. The attention scores are then passed through o to compute the attention block output. \\n\",\n \"So let us swap q, k and v layers with `ColumnParallelLinear` and o with `RowParallelLinear`. Having `RowParallelLinear` following a `ColumnParallelLinear` is a performance optimization. The attention scores computed with q, k and v are already split across Neuron devices. The row parallel layer can use this shared output directly. \\n\",\n \"The embedding layer is simply swapped with the `ParallelEmbedding`.\\n\",\n \"\\n\",\n \"```\\n\",\n \"class ParallelAttention(T5Attention):\\n\",\n \" def __init__(self, config: T5Config, has_relative_attention_bias=False):\\n\",\n \" super().__init__(config, has_relative_attention_bias)\\n\",\n \" # Per attention head and per partition values\\n\",\n \" world_size = parallel_state.get_tensor_model_parallel_size()\\n\",\n \" self.num_attention_heads_per_partition = divide(self.n_heads, world_size)\\n\",\n \" self.hidden_size_per_partition = self.num_attention_heads_per_partition * self.key_value_proj_dim\\n\",\n \"\\n\",\n \" # Mesh TensorFlow initialization to avoid scaling before softmax\\n\",\n \" self.q = ColumnParallelLinear(self.d_model,\\n\",\n \" self.inner_dim,\\n\",\n \" bias=False,\\n\",\n \" gather_output=False)\\n\",\n \" self.k = ColumnParallelLinear(self.d_model,\\n\",\n \" self.inner_dim,\\n\",\n \" bias=False,\\n\",\n \" gather_output=False)\\n\",\n \" self.v = ColumnParallelLinear(self.d_model,\\n\",\n \" self.inner_dim,\\n\",\n \" bias=False,\\n\",\n \" gather_output=False)\\n\",\n \" self.o = RowParallelLinear(self.inner_dim,\\n\",\n \" self.d_model,\\n\",\n \" bias=False,\\n\",\n \" input_is_parallel=True)\\n\",\n \"\\n\",\n \" if self.has_relative_attention_bias:\\n\",\n \" self.relative_attention_bias = ParallelEmbedding(self.relative_attention_num_buckets, self.n_heads)\\n\",\n \" self.n_heads = self.num_attention_heads_per_partition\\n\",\n \"...\\n\",\n \"```\\n\",\n \"\\n\",\n \"You can find the all modified T5 layers defined in [t5_model_layers.py](https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/examples/pytorch/neuronx_distributed/t5-inference/t5_model_layers.py). \\n\",\n \"\\n\",\n \"\\n\",\n \"Once we have the modified T5 layers, we can plug in the T5Attention and T5LayerFF into the pretrained model. Here is how you do that. \\n\",\n \"\\n\",\n \"```\\n\",\n \"def load_pretrained_with_parallel_attn(model_name):\\n\",\n \" \\n\",\n \" model = T5ForConditionalGeneration.from_pretrained(model_name, torch_dtype=\\\"auto\\\")\\n\",\n \"\\n\",\n \" # Parallel implementation of Attention modules.\\n\",\n \" from t5_model_layers import ParallelSelfAttention, ParallelFF, ParallelCrossAttention\\n\",\n \"\\n\",\n \" for index, block in enumerate(model.decoder.block):\\n\",\n \" if index == 0:\\n\",\n \" block.layer[0] = ParallelSelfAttention(model.config,\\n\",\n \" has_relative_attention_bias=True)\\n\",\n \" else:\\n\",\n \" block.layer[0] = ParallelSelfAttention(model.config)\\n\",\n \" block.layer[1] = ParallelCrossAttention(model.config)\\n\",\n \" block.layer[2] = ParallelFF(model.config)\\n\",\n \" # Load the weights into the parallel layers \\n\",\n \" neuronx_distributed.parallel_layers.load(model_name + \\\".pt\\\", model, sharded=False)\\n\",\n \"\\n\",\n \" return model\\n\",\n \"\\n\",\n \"```\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Compile the parallel T5 model\\n\",\n \"\\n\",\n \"Let us set some model parameters.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model_name = \\\"google/flan-t5-xl\\\" \\n\",\n \"max_length = 128\\n\",\n \"num_beams = 4\\n\",\n \"tp_degree = 8 # tensor parallelism degree\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Download and save the model that we want to trace. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"from transformers import T5ForConditionalGeneration\\n\",\n \"\\n\",\n \"model = T5ForConditionalGeneration.from_pretrained(model_name, torch_dtype=\\\"auto\\\")\\n\",\n \"torch.save({\\\"model\\\":model.state_dict()}, model_name.split(\\\"/\\\")[-1] + \\\".pt\\\")\\n\",\n \"model.config.use_cache = True\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"To run HuggingFace T5 models on Neuron, we need to make a couple of changes. Let us reuse the code from the [t5 inference tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/torch-neuronx/t5-inference-tutorial.html) which makes T5 compatible with Neuron. For your convenience, the code copied into [wrapper.py](https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/examples/pytorch/neuronx_distributed/t5-inference/wrapper.py) and [t5_models.py](https://github.com/aws-neuron/aws-neuron-sdk/tree/master/src/examples/pytorch/neuronx_distributed/t5-inference/t5_models.py). This notebook will import these files. \\n\",\n \"\\n\",\n \"The only change made to this code is that we use `neuronx_distributed.trace` instead of `torch_neuronx.trace`. \\n\",\n \"\\n\",\n \"Let us trace the encoder and decoder. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import t5_models \\n\",\n \"import neuronx_distributed\\n\",\n \"import time \\n\",\n \"\\n\",\n \"# This can take up to 20 minutes\\n\",\n \"encoder_compile_start_time = time.time()\\n\",\n \"traced_encoder = t5_models.parallel_trace_encoder(model_name, max_length, num_beams, tp_degree)\\n\",\n \"print(\\\"Encoder compilation time {}\\\".format(time.time() - encoder_compile_start_time))\\n\",\n \"\\n\",\n \"neuronx_distributed.trace.parallel_model_save(traced_encoder, \\\"TracedParallelEncoder.pt\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# This can take up to 15 minutes\\n\",\n \"decoder_compile_start_time = time.time()\\n\",\n \"traced_decoder = t5_models.parallel_trace_decoder(model, model_name, num_beams, max_length, tp_degree)\\n\",\n \"print(\\\"Decoder compilation time {}\\\".format(time.time() - decoder_compile_start_time))\\n\",\n \"\\n\",\n \"neuronx_distributed.trace.parallel_model_save(traced_decoder, \\\"TracedParallelDecoder.pt\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Inference with the traced parallel T5 model\\n\",\n \"\\n\",\n \"With the traced model, let us try using beam search for inference.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Results:\\n\",\n \"1 Lassen Sie uns gutes Essen essen.\\n\",\n \"2 Lassen Sie uns gut essen.\\n\",\n \"3 Lassen Sie uns gutes Essen zu essen.\\n\",\n \"4 Lassen Sie uns gutes Essen zu sich nehmen.\\n\"\n ]\n }\n ],\n \"source\": [\n \"import neuronx_distributed\\n\",\n \"from wrapper import T5Wrapper\\n\",\n \"from transformers import T5Tokenizer\\n\",\n \"\\n\",\n \"\\n\",\n \"num_return_sequences = 4\\n\",\n \"\\n\",\n \"traced_encoder = neuronx_distributed.trace.parallel_model_load(\\\"TracedParallelEncoder.pt\\\")\\n\",\n \"traced_decoder = neuronx_distributed.trace.parallel_model_load(\\\"TracedParallelDecoder.pt\\\")\\n\",\n \"\\n\",\n \"tokenizer = T5Tokenizer.from_pretrained(model_name)\\n\",\n \"model = T5Wrapper.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"model.encoder = traced_encoder\\n\",\n \"model.decoder = traced_decoder\\n\",\n \"setattr(model.encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\\n\",\n \"\\n\",\n \"output = model.parallel_infer(tokenizer=tokenizer,\\n\",\n \" prompt=\\\"translate English to German: Lets eat good food.\\\",\\n\",\n \" max_length=max_length,\\n\",\n \" num_beams=num_beams,\\n\",\n \" num_return_sequences=num_return_sequences,\\n\",\n \" device=\\\"xla\\\")\\n\",\n \"\\n\",\n \"results = [tokenizer.decode(t, skip_special_tokens=True) for t in output]\\n\",\n \"\\n\",\n \"print('Results:')\\n\",\n \"for i, summary in enumerate(results):\\n\",\n \" print(i + 1, summary)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Benchmarking\\n\",\n \"\\n\",\n \"Let us benchmark the per token decoder latency\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Let us install NeuronPerf. We will use it to measure the performance.\\n\",\n \"! pip install neuronperf --extra-index-url=https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os \\n\",\n \"import neuronperf as npf\\n\",\n \"\\n\",\n \"d_model = model.config.d_model\\n\",\n \"model_dir = \\\"TracedParallelDecoder.pt\\\"\\n\",\n \"decoder_run_count = 128\\n\",\n \"\\n\",\n \"def load_fn(model_path, **kwargs):\\n\",\n \" return neuronx_distributed.trace.parallel_model_load(model_path)\\n\",\n \" \\n\",\n \"# NeuronPerf can't see tp_degree at the moment, so just expose all cores\\n\",\n \"def env_setup_fn(*_):\\n\",\n \" del os.environ[\\\"NEURON_RT_VISIBLE_CORES\\\"]\\n\",\n \"\\n\",\n \"def benchmark():\\n\",\n \"\\n\",\n \" # Create some sample inputs for the decoder\\n\",\n \" decoder_input_ids = torch.ones((num_beams, 1), dtype=torch.int64)\\n\",\n \" decoder_attention_mask = torch.ones((num_beams, max_length), dtype=torch.int32)\\n\",\n \" encoder_attention_mask = torch.ones((num_beams, max_length), dtype=torch.int64)\\n\",\n \" encoder_hidden_states = torch.ones((num_beams, max_length, d_model), dtype=torch.float32)\\n\",\n \" beam_idx = torch.arange(0, num_beams, dtype=torch.int64)\\n\",\n \" beam_scores = torch.zeros((num_beams,), dtype=torch.float)\\n\",\n \"\\n\",\n \" inputs = (decoder_input_ids,\\n\",\n \" decoder_attention_mask,\\n\",\n \" encoder_hidden_states,\\n\",\n \" encoder_attention_mask,\\n\",\n \" beam_idx,\\n\",\n \" beam_scores)\\n\",\n \"\\n\",\n \" reports = npf.benchmark(\\n\",\n \" load_fn,\\n\",\n \" model_dir,\\n\",\n \" [inputs], \\n\",\n \" batch_sizes=1,\\n\",\n \" n_models=1,\\n\",\n \" max_infers=decoder_run_count,\\n\",\n \" workers_per_model=1, # no bottleneck on model inputs, so 1 is fine\\n\",\n \" env_setup_fn=env_setup_fn,\\n\",\n \" multiprocess=False,\\n\",\n \" )\\n\",\n \" \\n\",\n \" report = reports[0]\\n\",\n \"\\n\",\n \" # let's update throughput to be tokens / second and add a new recor\\n\",\n \" latency_in_s = report[\\\"latency_ms_avg\\\"] / 1000\\n\",\n \" tokens_per_s = decoder_run_count / latency_in_s\\n\",\n \" report[\\\"throughput_avg\\\"] = tokens_per_s\\n\",\n \" \\n\",\n \" # display and save results\\n\",\n \" npf.print_reports(reports, cols=[\\\"throughput_avg\\\", \\\"latency_ms_p50\\\", \\\"latency_ms_p99\\\"])\\n\",\n \" print(f\\\"Results saved to: {npf.write_json(reports[0])}\\\")\\n\",\n \"\\n\",\n \"benchmark()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now lets benchmark inference as a whole including sampling. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import torch\\n\",\n \"import neuronx_distributed\\n\",\n \"import neuronperf as npf\\n\",\n \"\\n\",\n \"from transformers import T5Tokenizer\\n\",\n \"from wrapper import T5Wrapper\\n\",\n \"\\n\",\n \"tokenizer = T5Tokenizer.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"generated_token_count = 0\\n\",\n \"\\n\",\n \"class Wrapper(torch.nn.Module):\\n\",\n \" def __init__(self, \\n\",\n \" traced_encoder,\\n\",\n \" traced_decoder):\\n\",\n \" super().__init__()\\n\",\n \" self.model = T5Wrapper.from_pretrained(model_name)\\n\",\n \" self.model.encoder = traced_encoder\\n\",\n \" self.model.decoder = traced_decoder\\n\",\n \" setattr(self.model.encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\\n\",\n \"\\n\",\n \" def forward(self, *inputs):\\n\",\n \" input_ids = inputs[0]['input_ids']\\n\",\n \" attention_mask = inputs[0]['attention_mask']\\n\",\n \" return self.model.parallel_infer(input_ids=input_ids,\\n\",\n \" attention_mask=attention_mask,\\n\",\n \" max_length=max_length,\\n\",\n \" num_beams=num_beams,\\n\",\n \" num_return_sequences=num_return_sequences)\\n\",\n \"\\n\",\n \"def load_fn(filename, **kwargs):\\n\",\n \" traced_encoder = neuronx_distributed.trace.parallel_model_load(filename + \\\"TracedParallelEncoder.pt\\\")\\n\",\n \" traced_decoder = neuronx_distributed.trace.parallel_model_load(filename + \\\"TracedParallelDecoder.pt\\\")\\n\",\n \" return Wrapper(traced_encoder, traced_decoder)\\n\",\n \"\\n\",\n \"# NeuronPerf can't see tp_degree at the moment, so just expose all cores\\n\",\n \"def env_setup_fn(*_):\\n\",\n \" del os.environ[\\\"NEURON_RT_VISIBLE_CORES\\\"]\\n\",\n \"\\n\",\n \"def preprocess_fn(inputs):\\n\",\n \" \\n\",\n \" encoding = []\\n\",\n \" for text in inputs:\\n\",\n \" batch_encoding = tokenizer(text, \\n\",\n \" max_length=max_length, \\n\",\n \" truncation=True, \\n\",\n \" padding='max_length',\\n\",\n \" return_tensors=\\\"pt\\\")\\n\",\n \" input_ids = batch_encoding['input_ids']\\n\",\n \" attention_mask = batch_encoding['attention_mask']\\n\",\n \" encoding.append({\\\"input_ids\\\": input_ids,\\n\",\n \" \\\"attention_mask\\\": attention_mask})\\n\",\n \" return encoding\\n\",\n \"\\n\",\n \"def postprocess_fn(outputs):\\n\",\n \" output = [tokenizer.decode(seq) for seq in outputs]\\n\",\n \" global generated_token_count \\n\",\n \" generated_token_count = len(outputs[0])\\n\",\n \" return output\\n\",\n \"\\n\",\n \"def benchmark():\\n\",\n \" inputs = [\\\"summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes.\\\"]\\n\",\n \" reports = npf.benchmark(\\n\",\n \" load_fn,\\n\",\n \" \\\"\\\", # Model dir\\n\",\n \" [inputs], \\n\",\n \" batch_sizes=1,\\n\",\n \" n_models=1,\\n\",\n \" max_infers=5,\\n\",\n \" max_duration=0, # sampling can take a while, so let's not timeout\\n\",\n \" workers_per_model=1, \\n\",\n \" env_setup_fn=env_setup_fn,\\n\",\n \" preprocess_fn=preprocess_fn,\\n\",\n \" postprocess_fn=postprocess_fn,\\n\",\n \" multiprocess=False,\\n\",\n \" )\\n\",\n \" \\n\",\n \" report = reports[0]\\n\",\n \"\\n\",\n \" report[\\\"throughput_avg\\\"] = round(generated_token_count / (report[\\\"latency_ms_avg\\\"] / 1000), 2)\\n\",\n \" report[\\\"latency_per_token_ms_p50\\\"] = round((report[\\\"latency_ms_p50\\\"])/generated_token_count, 2)\\n\",\n \" report[\\\"latency_per_token_ms_p99\\\"] = round((report[\\\"latency_ms_p99\\\"])/generated_token_count, 2)\\n\",\n \"\\n\",\n \" # display and save results\\n\",\n \" npf.print_reports(reports, cols=[\\\"throughput_avg\\\", \\\"latency_per_token_ms_p50\\\", \\\"latency_per_token_ms_p99\\\"])\\n\",\n \" print(f\\\"Results saved to: {npf.write_json(report)}\\\")\\n\",\n \"\\n\",\n \"benchmark()\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"aws_neuron_venv_pytorch\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.10\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n" }, { "path": "src/examples/pytorch/neuronx_distributed/t5-inference/t5_model_layers.py", "content": "from neuronx_distributed.parallel_layers import parallel_state\nfrom neuronx_distributed.parallel_layers.layers import BaseParallelLinear, ColumnParallelLinear, RowParallelLinear, ParallelEmbedding\nfrom neuronx_distributed.parallel_layers.utils import divide\n\nimport torch\nfrom torch import nn\nfrom torch.nn.parameter import Parameter\nfrom transformers import T5Config\nfrom transformers.activations import ACT2FN\nfrom transformers.pytorch_utils import find_pruneable_heads_and_indices\nfrom transformers.models.t5.modeling_t5 import T5Attention, T5LayerSelfAttention, T5LayerNorm,\\\n T5LayerCrossAttention, T5LayerFF, T5DenseGatedActDense, T5DenseActDense\n\nfrom transformers import T5ForConditionalGeneration\nimport neuronx_distributed\n\ndef prune_linear_layer(layer: BaseParallelLinear, index: torch.LongTensor,\n dim: int = 0) -> BaseParallelLinear:\n \"\"\"\n Prune a linear layer to keep only entries in index.\n\n Used to remove heads.\n\n Args:\n layer (`BaseParallelLinear`): The layer to prune.\n index (`torch.LongTensor`): The indices to keep in the layer.\n dim (`int`, *optional*, defaults to 0): The dimension on which to keep the indices.\n\n Returns:\n `BaseParallelLinear`: The pruned layer as a new layer with `requires_grad=True`.\n \"\"\"\n index = index.to(layer.weight.device)\n W = layer.weight.index_select(dim, index).clone().detach()\n if layer.bias is not None:\n if dim == 1:\n b = layer.bias.clone().detach()\n else:\n b = layer.bias[index].clone().detach()\n new_size = list(layer.weight.size())\n new_size[dim] = len(index)\n new_layer = ColumnParallelLinear(new_size[1],\n new_size[0],\n bias=layer.bias is not None,\n gather_output=False).to(layer.weight.device)\n new_layer.weight.requires_grad = False\n new_layer.weight.copy_(W.contiguous())\n new_layer.weight.requires_grad = True\n if layer.bias is not None:\n new_layer.bias.requires_grad = False\n new_layer.bias.copy_(b.contiguous())\n new_layer.bias.requires_grad = True\n return new_layer\n\n\nclass ParallelAttention(T5Attention):\n def __init__(self, config: T5Config, has_relative_attention_bias=False):\n super().__init__(config, has_relative_attention_bias)\n # Per attention head and per partition values\n world_size = parallel_state.get_tensor_model_parallel_size()\n self.num_attention_heads_per_partition = divide(\n self.n_heads, world_size)\n self.hidden_size_per_partition = self.num_attention_heads_per_partition * self.key_value_proj_dim\n\n # Mesh TensorFlow initialization to avoid scaling before softmax\n self.q = ColumnParallelLinear(self.d_model,\n self.inner_dim,\n bias=False,\n gather_output=False)\n self.k = ColumnParallelLinear(self.d_model,\n self.inner_dim,\n bias=False,\n gather_output=False)\n self.v = ColumnParallelLinear(self.d_model,\n self.inner_dim,\n bias=False,\n gather_output=False)\n self.o = RowParallelLinear(self.inner_dim,\n self.d_model,\n bias=False,\n input_is_parallel=True)\n\n if self.has_relative_attention_bias:\n self.relative_attention_bias = ParallelEmbedding(self.relative_attention_num_buckets, self.n_heads)\n self.n_heads = self.num_attention_heads_per_partition\n\n def prune_heads(self, heads):\n if len(heads) == 0:\n return\n heads, index = find_pruneable_heads_and_indices(\n heads, self.num_attention_heads_per_partition, self.key_value_proj_dim, self.pruned_heads\n )\n # Prune linear layers\n self.q = prune_linear_layer(self.q, index)\n self.k = prune_linear_layer(self.k, index)\n self.v = prune_linear_layer(self.v, index)\n self.o = prune_linear_layer(self.o, index, dim=1)\n # Update hyper params\n self.num_attention_heads_per_partition = self.num_attention_heads_per_partition - len(heads)\n self.hidden_size_per_partition = self.key_value_proj_dim * self.num_attention_heads_per_partition\n self.pruned_heads = self.pruned_heads.union(heads)\n\n def compute_bias(self, query_length, key_length, device=None):\n \"\"\"Compute binned relative position bias\"\"\"\n if device is None:\n device = self.relative_attention_bias.weight.device\n context_position = torch.arange(query_length, dtype=torch.long, device=device)[:, None]\n memory_position = torch.arange(key_length, dtype=torch.long, device=device)[None, :]\n relative_position = memory_position - context_position # shape (query_length, key_length)\n relative_position_bucket = self._relative_position_bucket(\n relative_position, # shape (query_length, key_length)\n bidirectional=(not self.is_decoder),\n num_buckets=self.relative_attention_num_buckets,\n max_distance=self.relative_attention_max_distance,\n )\n values = self.relative_attention_bias(\n relative_position_bucket)\n tp_rank = parallel_state.get_tensor_model_parallel_rank()\n values = values[:, :, tp_rank * self.num_attention_heads_per_partition:(tp_rank + 1)\n * self.num_attention_heads_per_partition]\n\n # values = self.relative_attention_bias(\n # relative_position_bucket) # shape (query_length, key_length, num_heads)\n values = values.permute([2, 0, 1]).unsqueeze(\n 0) # shape (1, num_heads, query_length, key_length)\n # print(\"Values shape is: \", values.shape)\n return values\n\n def forward(\n self,\n hidden_states,\n mask=None,\n key_value_states=None,\n position_bias=None,\n past_key_value=None,\n layer_head_mask=None,\n query_length=None,\n use_cache=False,\n output_attentions=False,\n ):\n \"\"\"\n Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states).\n \"\"\"\n # Input is (batch_size, seq_length, dim)\n # Mask is (batch_size, key_length) (non-causal) or (batch_size, key_length, key_length)\n # past_key_value[0] is (batch_size, n_heads, q_len - 1, dim_per_head)\n self.is_decoder = True\n batch_size, seq_length = hidden_states.shape[:2]\n\n real_seq_length = seq_length\n\n if past_key_value is not None:\n assert (\n len(past_key_value) == 2\n ), f\"past_key_value should have 2 past states: keys and values. Got {len(past_key_value)} past states\"\n real_seq_length += past_key_value[0].shape[2] if query_length is None else query_length\n\n key_length = real_seq_length if key_value_states is None else key_value_states.shape[1]\n\n def shape(states):\n \"\"\"projection\"\"\"\n return states.view(batch_size, -1, self.num_attention_heads_per_partition,\n self.key_value_proj_dim).transpose(1, 2)\n\n def unshape(states):\n \"\"\"reshape\"\"\"\n return states.transpose(1, 2).contiguous().view(batch_size, -1,\n self.hidden_size_per_partition)\n\n def project(hidden_states, proj_layer, key_value_states, past_key_value):\n \"\"\"projects hidden states correctly to key/query states\"\"\"\n if key_value_states is None:\n # self-attn\n # (batch_size, n_heads, seq_length, dim_per_head)\n hidden_states = shape(proj_layer(hidden_states))\n elif past_key_value is None:\n # cross-attn\n # (batch_size, n_heads, seq_length, dim_per_head)\n hidden_states = shape(proj_layer(key_value_states))\n\n if past_key_value is not None:\n # import pdb; pdb.set_trace()\n if key_value_states is None:\n # self-attn\n # (batch_size, n_heads, key_length, dim_per_head)\n hidden_states = torch.cat([past_key_value, hidden_states], dim=2)\n elif past_key_value.shape[2] != key_value_states.shape[1]:\n # checking that the `sequence_length` of the `past_key_value` is the same as\n # the provided `key_value_states` to support prefix tuning\n # cross-attn\n # (batch_size, n_heads, seq_length, dim_per_head)\n hidden_states = shape(proj_layer(key_value_states))\n else:\n # cross-attn\n hidden_states = past_key_value\n return hidden_states\n\n # get query states\n query_states = shape(\n self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head)\n\n # get key/value states\n key_states = project(\n hidden_states, self.k, key_value_states,\n past_key_value[0] if past_key_value is not None else None\n )\n value_states = project(\n hidden_states, self.v, key_value_states,\n past_key_value[1] if past_key_value is not None else None\n )\n\n # compute scores\n scores = torch.matmul(\n query_states, key_states.transpose(3, 2)\n ) # equivalent of torch.einsum(\"bnqd,bnkd->bnqk\", query_states, key_states), compatible with onnx op>9\n\n if position_bias is None:\n if not self.has_relative_attention_bias:\n position_bias = torch.zeros(\n (1, self.num_attention_heads_per_partition, real_seq_length, key_length),\n device=scores.device,\n dtype=scores.dtype\n )\n if self.gradient_checkpointing and self.training:\n position_bias.requires_grad = True\n else:\n position_bias = self.compute_bias(real_seq_length, key_length, device=scores.device)\n\n # if key and values are already calculated\n # we want only the last query position bias\n if past_key_value is not None:\n position_bias = position_bias[:, :, -hidden_states.size(1):, :]\n\n if mask is not None:\n print(position_bias.shape, mask.shape, flush=True)\n position_bias = position_bias + mask # (batch_size, n_heads, seq_length, key_length)\n\n if self.pruned_heads:\n mask = torch.ones(position_bias.shape[1])\n mask[list(self.pruned_heads)] = 0\n position_bias_masked = position_bias[:, mask.bool()]\n else:\n position_bias_masked = position_bias\n\n # print(\"Scores is: \", scores.shape)\n # print(\"position_bias_masked: \", position_bias_masked.shape)\n # print(scores.dtype, position_bias_masked.dtype)\n\n scores += position_bias_masked\n attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(\n scores\n ) # (batch_size, n_heads, seq_length, key_length)\n attn_weights = nn.functional.dropout(\n attn_weights, p=self.dropout, training=self.training\n ) # (batch_size, n_heads, seq_length, key_length)\n\n # Mask heads if we want to\n if layer_head_mask is not None:\n attn_weights = attn_weights * layer_head_mask\n\n attn_output = unshape(\n torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim)\n attn_output = self.o(attn_output)\n\n print(self.is_decoder,use_cache, flush=True)\n present_key_value_state = (key_states, value_states) if (\n self.is_decoder and use_cache) else None\n outputs = (attn_output,) + (present_key_value_state,) + (position_bias,)\n\n if output_attentions:\n outputs = outputs + (attn_weights,)\n return outputs\n\n\nclass ParallelSelfAttention(T5LayerSelfAttention):\n def __init__(self, config, has_relative_attention_bias=False):\n super().__init__(config, has_relative_attention_bias=False)\n self.SelfAttention = ParallelAttention(config,\n has_relative_attention_bias=has_relative_attention_bias)\n self.layer_norm = T5LayerNorm(config.d_model, eps=config.layer_norm_epsilon)\n self.dropout = nn.Dropout(config.dropout_rate)\n\n\nclass ParallelCrossAttention(T5LayerCrossAttention):\n def __init__(self, config):\n super().__init__(config)\n self.EncDecAttention = ParallelAttention(config, has_relative_attention_bias=False)\n self.layer_norm = T5LayerNorm(config.d_model, eps=config.layer_norm_epsilon)\n self.dropout = nn.Dropout(config.dropout_rate)\n\n\nclass ParallelDenseActDense(T5DenseActDense):\n def __init__(self, config: T5Config):\n super().__init__(config)\n self.wi = ColumnParallelLinear(config.d_model, config.d_ff, gather_output=False, bias=False)\n self.wo = RowParallelLinear(config.d_ff, config.d_model, input_is_parallel=True, bias=False)\n self.dropout = nn.Dropout(config.dropout_rate)\n self.act = ACT2FN[config.dense_act_fn]\n\n\nclass ParallelDenseGatedActDense(T5DenseGatedActDense):\n def __init__(self, config: T5Config):\n super().__init__(config)\n self.wi_0 = ColumnParallelLinear(config.d_model,\n config.d_ff,\n gather_output=False,\n bias=False)\n self.wi_1 = ColumnParallelLinear(config.d_model,\n config.d_ff,\n gather_output=False,\n bias=False)\n self.wo = RowParallelLinear(config.d_ff,\n config.d_model,\n input_is_parallel=True,\n bias=False)\n self.dropout = nn.Dropout(config.dropout_rate)\n self.act = ACT2FN[config.dense_act_fn]\n\n\nclass ParallelFF(T5LayerFF):\n def __init__(self, config: T5Config):\n super().__init__(config)\n if config.is_gated_act:\n self.DenseReluDense = ParallelDenseGatedActDense(config)\n else:\n self.DenseReluDense = ParallelDenseActDense(config)\n\n self.layer_norm = T5LayerNorm(config.d_model, eps=config.layer_norm_epsilon)\n self.dropout = nn.Dropout(config.dropout_rate)\n\n\ndef load_pretrained_with_parallel_attn(model_name):\n \n model = T5ForConditionalGeneration.from_pretrained(model_name, torch_dtype=\"auto\")\n\n # Parallel implementation of Attention modules.\n from t5_model_layers import ParallelSelfAttention, ParallelFF, ParallelCrossAttention\n\n for index, block in enumerate(model.decoder.block):\n if index == 0:\n block.layer[0] = ParallelSelfAttention(model.config,\n has_relative_attention_bias=True)\n else:\n block.layer[0] = ParallelSelfAttention(model.config)\n block.layer[1] = ParallelCrossAttention(model.config)\n block.layer[2] = ParallelFF(model.config)\n # Load the weights into the parallel layers \n neuronx_distributed.parallel_layers.load(model_name.split(\"/\")[-1] + \".pt\", model, sharded=False)\n\n return model\n" }, { "path": "src/examples/pytorch/neuronx_distributed/t5-inference/t5_models.py", "content": "import torch\nimport neuronx_distributed\n\nfrom functools import partial\nfrom transformers import T5Tokenizer, T5ForConditionalGeneration\n\nfrom wrapper import EncoderWrapper, DecoderWrapper\nfrom t5_model_layers import load_pretrained_with_parallel_attn\n\ndef get_wrapped_encoder(max_length, num_beams, tp_degree, model_name):\n \n model = load_pretrained_with_parallel_attn(model_name)\n\n encoder = EncoderWrapper(model.encoder, model.decoder, model.config, num_beams, max_length, \"xla\", num_beams, tp_degree=tp_degree)\n encoder.eval()\n \n # We are alaising the cache, so that way we keep the cache always on device.\n aliases = {}\n for i in range(len(encoder.past_key_values_sa)):\n aliases[encoder.past_key_values_sa[i]] = i\n \n for i in range(len(encoder.past_key_values_ca)):\n aliases[encoder.past_key_values_ca[i]] = len(encoder.past_key_values_sa) + i\n\n return encoder, aliases\n\n\ndef get_wrapped_decoder(max_length, num_beams, tp_degree, model_name):\n \n model = load_pretrained_with_parallel_attn(model_name)\n\n decoder = DecoderWrapper(decoder=model.decoder,\n lm_head=model.lm_head,\n model_config=model.config,\n num_beams=num_beams,\n max_length=max_length,\n device=\"xla\",\n tp_degree=tp_degree)\n \n decoder.eval()\n num_outputs_from_trace = 3 if num_beams > 1 else 1\n aliases = {}\n for i in range(len(decoder.past_key_values_sa)):\n aliases[decoder.past_key_values_sa[i]] = i + num_outputs_from_trace\n for i in range(len(decoder.past_key_values_ca)):\n aliases[decoder.past_key_values_ca[i]] = len(decoder.past_key_values_sa) + i + num_outputs_from_trace\n\n return decoder, aliases\n\ndef parallel_trace_encoder(model_name: str,\n max_length: int,\n num_beams: int,\n tp_degree: int):\n \n print(\"starting encoder parallel trace\")\n \n tokenizer = T5Tokenizer.from_pretrained(model_name)\n get_encoder_callable = partial(get_wrapped_encoder, max_length, num_beams, tp_degree, model_name)\n\n # Trace encoder\n batch_encoding = tokenizer(\"translate English to German: Lets go home now\",\n max_length=max_length, truncation=True, padding='max_length', return_tensors=\"pt\")\n input_ids = batch_encoding['input_ids']\n attention_mask = batch_encoding['attention_mask']\n\n # Here we are tracing the encoder and cache together. Cache is marked as state and we are aliasing.\n traced_encoder = neuronx_distributed.trace.parallel_model_trace(get_encoder_callable, (\n input_ids,\n attention_mask,\n ), \n tp_degree=tp_degree, \n compiler_workdir=\"/tmp/encoder/\",\n )\n setattr(traced_encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\n\n print(\"completed encoder parallel trace\")\n\n return traced_encoder\n\n\ndef parallel_trace_decoder(model: T5ForConditionalGeneration,\n model_name: str,\n num_beams: int,\n max_length: int,\n tp_degree: int):\n\n print(\"starting decoder trace\")\n\n get_decoder_callable = partial(get_wrapped_decoder, max_length, num_beams, tp_degree, model_name)\n \n # We create mock inputs so we can trace the decoder\n decoder_input_ids = torch.ones((num_beams, 1), dtype=torch.int64)\n decoder_attention_mask = torch.ones((num_beams, max_length), dtype=torch.int32)\n encoder_attention_mask = torch.ones((num_beams, max_length), dtype=torch.int64)\n encoder_hidden_states = torch.ones((num_beams, max_length, model.config.d_model), dtype=torch.float32)\n\n beam_idx = torch.arange(0, num_beams, dtype=torch.int64)\n beam_scores = torch.zeros((num_beams,), dtype=torch.float)\n\n traced_decoder = neuronx_distributed.trace.parallel_model_trace(get_decoder_callable, (\n decoder_input_ids,\n decoder_attention_mask,\n encoder_hidden_states,\n encoder_attention_mask,\n beam_idx,\n beam_scores\n ), \n tp_degree=tp_degree,\n compiler_workdir=\"/tmp/decoder/\",\n )\n\n print(\"complete decoder trace\")\n\n return traced_decoder\n" }, { "path": "src/examples/pytorch/neuronx_distributed/t5-inference/wrapper.py", "content": "import torch\nimport neuronx_distributed\nimport torch_xla.core.xla_model as xm\n\nfrom transformers import T5Tokenizer, T5ForConditionalGeneration\nfrom transformers.modeling_outputs import BaseModelOutput, Seq2SeqLMOutput\nfrom transformers.models.t5.modeling_t5 import T5Stack, T5LayerCrossAttention\nfrom transformers.generation.utils import ModelOutput\nfrom typing import Any, Dict, List, Optional, Tuple, Union\nfrom transformers.generation.beam_search import BeamScorer, BeamSearchScorer\n\nfrom optimum.neuron.generation import NeuronGenerationMixin\n\nfrom transformers.generation.logits_process import (\n LogitsProcessorList,\n)\nfrom transformers.generation.stopping_criteria import (\n MaxLengthCriteria,\n MaxTimeCriteria,\n StoppingCriteriaList,\n validate_stopping_criteria,\n)\n\nfrom transformers.generation.utils import (\n BeamSearchDecoderOnlyOutput,\n BeamSearchEncoderDecoderOutput,\n BeamSearchOutput,\n GreedySearchOutput,\n)\n\nclass T5Wrapper(T5ForConditionalGeneration, NeuronGenerationMixin):\n\n def _prepare_encoder_decoder_kwargs_for_generation(\n self, \n inputs_tensor: torch.Tensor, \n model_kwargs, \n model_input_name: Optional[str] = None\n ) -> Dict[str, Any]:\n encoder = self.get_encoder()\n model_kwargs[\"encoder_outputs\"]: ModelOutput = encoder(inputs_tensor, model_kwargs[\"attention_mask\"])\n return model_kwargs\n\n # Override to cut the input_ids to just last token\n def prepare_inputs_for_generation(\n self,\n input_ids,\n past_key_values=None,\n attention_mask=None,\n head_mask=None,\n decoder_head_mask=None,\n decoder_attention_mask=None,\n cross_attn_head_mask=None,\n use_cache=None,\n encoder_outputs=None,\n **kwargs,\n ):\n # cut decoder_input_ids as past is cached\n input_ids = input_ids[:, -1:]\n\n return {\n \"decoder_input_ids\": input_ids,\n \"past_key_values\": past_key_values,\n \"encoder_outputs\": encoder_outputs,\n \"attention_mask\": attention_mask,\n \"head_mask\": head_mask,\n \"decoder_head_mask\": decoder_head_mask,\n \"decoder_attention_mask\": decoder_attention_mask,\n \"cross_attn_head_mask\": cross_attn_head_mask,\n \"use_cache\": use_cache,\n }\n \n '''\n We update the cache in the decoder trace, so lets override the _update_model_kwargs_for_xla_generation in NeuronGenerationMixin\n '''\n def _update_model_kwargs_for_xla_generation(\n self,\n model_kwargs: Dict[str, Any],\n batch_size: int,\n is_encoder_decoder: bool = False,\n standardize_cache_format: bool = False,\n max_length: Optional[int] = None,\n seq_length: Optional[int] = None,\n use_cache: bool = True,\n ) -> Dict[str, Any]:\n\n def _update_attention(model_kwargs, is_encoder_decoder):\n \"\"\"Updates the appropriate attention mask -- encoder-decoder models use `decoder_attention_mask`\"\"\"\n\n attention_mask_name = \"decoder_attention_mask\" if is_encoder_decoder else \"attention_mask\"\n attention_mask = model_kwargs.pop(attention_mask_name)\n attention_mask_update_slice = torch.ones(\n (batch_size, 1), dtype=attention_mask.dtype, device=attention_mask.device\n )\n attention_mask = torch.cat([attention_mask[:, 1:], attention_mask_update_slice], dim=-1)\n mask = {attention_mask_name: attention_mask}\n return mask\n\n mask = _update_attention(model_kwargs, is_encoder_decoder)\n # sets the updated variables (mask and past_key_values)\n model_kwargs.update(mask)\n\n # Set a mock cache tensor\n model_kwargs[\"past_key_values\"] = torch.tensor([])\n\n return model_kwargs\n \n def _reorder_cache(self, past_key_values, beam_idx):\n '''\n This is needed for beam search and not greedy sampling\n We reorder the cache within the trace so we can skip it in modelling_t5.py. So we override the _reorder_cache\n '''\n self.beam_idx = beam_idx\n return past_key_values\n\n def infer(self,\n tokenizer: T5Tokenizer,\n prompt: str,\n max_length: int,\n num_beams: int,\n num_return_sequences: int,\n device: str):\n\n batch_encoding = tokenizer(prompt, max_length=max_length, truncation=True, padding='max_length',\n return_tensors=\"pt\")\n\n past_key_values = self.encoder(batch_encoding['input_ids'],batch_encoding['attention_mask'])\n \n decoder_attention_mask = torch.cat([torch.zeros((1, max_length-1), dtype=torch.int32),\n torch.ones((1, 1), dtype=torch.int32)], axis=1)\n\n # copy the new cache state to the decoder\n if device == \"xla\":\n for state, tensor in zip(self.decoder.parameters(), past_key_values):\n state.copy_(tensor)\n else:\n # First half of the cache is self attention and the rest is cross attention\n self.decoder.past_key_values_sa = past_key_values[:len(past_key_values)//2]\n self.decoder.past_key_values_ca = past_key_values[len(past_key_values)//2:]\n \n output = self.generate(**batch_encoding,\n max_length=max_length,\n num_beams=num_beams,\n num_return_sequences=num_return_sequences,\n do_sample=False,\n use_cache=True,\n decoder_attention_mask=decoder_attention_mask, \n encoder_outputs={\"last_hidden_state\": torch.ones((1,128,1))}) # Pass fake encoder_outputs so the transfomers code will not invoke the encoder\n return output\n\n def parallel_infer(self,\n max_length: int,\n num_beams: int,\n num_return_sequences: int,\n device: str = None,\n tokenizer: T5Tokenizer = None,\n prompt: str = None, \n input_ids: torch.Tensor = None,\n attention_mask: torch.Tensor = None):\n\n if input_ids is None or attention_mask is None: \n batch_encoding = tokenizer(prompt, \n max_length=max_length, \n truncation=True, \n padding='max_length',\n return_tensors=\"pt\")\n else: \n batch_encoding = {\n 'input_ids' : input_ids,\n 'attention_mask': attention_mask\n }\n\n \n past_key_values = self.encoder(batch_encoding['input_ids'],batch_encoding['attention_mask'])\n \n decoder_attention_mask = torch.cat([torch.zeros((1, max_length-1), dtype=torch.int32),\n torch.ones((1, 1), dtype=torch.int32)], axis=1)\n\n # Here the encoder is now returning cache which is device tensor, so we directly assign\n # the cache device tensor to decoder's cache (which is also a device tensor). \n # We thereby avoid the copy and always use a pre-allocated memory\n for model_tp_decoder, model_tp_encoder in zip(self.decoder.models, self.encoder.models):\n model_tp_decoder.load_state_dict(model_tp_encoder.state_dict(), strict=True)\n \n # Pass fake encoder_outputs so the transfomers code will not invoke the encoder\n output = self.generate(**batch_encoding,\n max_length=max_length,\n num_beams=num_beams,\n num_return_sequences=num_return_sequences,\n do_sample=False,\n use_cache=True,\n decoder_attention_mask=decoder_attention_mask, \n encoder_outputs={\"last_hidden_state\": torch.ones((1,128,1))}) \n return output\n\n def forward(\n self,\n attention_mask: Optional[torch.FloatTensor] = None,\n decoder_input_ids: Optional[torch.LongTensor] = None,\n decoder_attention_mask: Optional[torch.BoolTensor] = None,\n encoder_outputs: Optional[Tuple[Tuple[torch.Tensor]]] = None,\n beam_scores = None,\n **kwargs\n ) -> Union[Tuple[torch.FloatTensor], Seq2SeqLMOutput]:\n\n hidden_states = encoder_outputs[\"last_hidden_state\"]\n\n if not hasattr(self, 'beam_idx'):\n # Infering the number of beams from the attention mask\n num_beams = attention_mask.shape[0]\n self.beam_idx = torch.arange(0, num_beams, dtype=torch.int64)\n\n decoder_outputs = self.decoder(\n decoder_input_ids,\n decoder_attention_mask,\n hidden_states,\n attention_mask,\n self.beam_idx,\n beam_scores\n )\n\n # lm_logits = decoder_outputs[0]\n next_token_scores = decoder_outputs[0]\n next_tokens = decoder_outputs[1]\n next_indices = decoder_outputs[2]\n\n return next_token_scores, next_tokens, next_indices\n\n def beam_search(\n self,\n input_ids: torch.LongTensor,\n beam_scorer: BeamScorer,\n logits_processor: Optional[LogitsProcessorList] = None,\n stopping_criteria: Optional[StoppingCriteriaList] = None,\n max_length: Optional[int] = None,\n pad_token_id: Optional[int] = None,\n eos_token_id: Optional[Union[int, List[int]]] = None,\n output_attentions: Optional[bool] = None,\n output_hidden_states: Optional[bool] = None,\n output_scores: Optional[bool] = None,\n return_dict_in_generate: Optional[bool] = None,\n synced_gpus: Optional[bool] = False,\n seq_length: Optional[int] = None,\n **model_kwargs,\n ) -> Union[BeamSearchOutput, torch.LongTensor]:\n\n logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()\n stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()\n pad_token_id = pad_token_id if pad_token_id is not None else self.generation_config.pad_token_id\n eos_token_id = eos_token_id if eos_token_id is not None else self.generation_config.eos_token_id\n if isinstance(eos_token_id, int):\n eos_token_id = [eos_token_id]\n output_scores = output_scores if output_scores is not None else self.generation_config.output_scores\n output_attentions = (\n output_attentions if output_attentions is not None else self.generation_config.output_attentions\n )\n output_hidden_states = (\n output_hidden_states if output_hidden_states is not None else self.generation_config.output_hidden_states\n )\n\n batch_size = len(beam_scorer._beam_hyps)\n num_beams = beam_scorer.num_beams\n\n batch_beam_size, cur_len = input_ids.shape\n\n # Overwrite cur_len\n cur_len = seq_length\n\n if num_beams * batch_size != batch_beam_size:\n raise ValueError(\n f\"Batch dimension of `input_ids` should be {num_beams * batch_size}, but is {batch_beam_size}.\"\n )\n\n # init attention / hidden states / scores tuples\n scores = () if (return_dict_in_generate and output_scores) else None\n beam_indices = (\n tuple(() for _ in range(batch_beam_size)) if (return_dict_in_generate and output_scores) else None\n )\n\n # initialise score of first beam with 0 and the rest with -1e9. This makes sure that only tokens\n # of the first beam are considered to avoid sampling the exact same tokens across all beams.\n # beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=input_ids.device)\n beam_scores_device = \"cpu\"\n beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=beam_scores_device)\n beam_scores[:, 1:] = -1e9\n beam_scores = beam_scores.view((batch_size * num_beams,))\n\n while True:\n # prepare model inputs\n # From max_length-sized input_ids, select first\n # cur_len - 1 values.\n update_indices = torch.stack(\n [torch.arange(input_ids.size(0)), torch.tensor(cur_len - 1).repeat(input_ids.size(0))], dim=-1\n )\n input_ids_ = input_ids[update_indices[:, 0], update_indices[:, 1], None]\n model_inputs = self.prepare_inputs_for_generation(input_ids_, **model_kwargs)\n\n next_token_scores, next_tokens, next_indices = self(\n **model_inputs,\n return_dict=True,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n beam_scores=beam_scores\n )\n\n # stateless\n beam_outputs = beam_scorer.process(\n input_ids.to(\"cpu\")[:, :cur_len],\n next_token_scores.to(\"cpu\"),\n next_tokens.to(\"cpu\"),\n next_indices.to(\"cpu\"),\n pad_token_id=pad_token_id,\n eos_token_id=eos_token_id,\n beam_indices=beam_indices,\n )\n\n beam_scores = beam_outputs[\"next_beam_scores\"]\n beam_next_tokens = beam_outputs[\"next_beam_tokens\"]\n beam_idx = beam_outputs[\"next_beam_indices\"]\n\n update_indices = torch.stack(\n [torch.arange(batch_beam_size), torch.tensor(cur_len - 1).repeat(batch_beam_size)], dim=-1\n )\n update_indices_2 = torch.stack(\n [torch.arange(batch_beam_size), torch.tensor(cur_len).repeat(batch_beam_size)], dim=-1\n )\n # First select beam_indices\n device = input_ids.device\n beam_idx_device = beam_idx.to(device=input_ids.device)\n input_ids[:, :] = input_ids[beam_idx_device.long(), :]\n\n # Then append new tokens\n input_ids[update_indices_2[:, 0], update_indices_2[:, 1], None] = beam_next_tokens.unsqueeze(-1).to(device).to(torch.long)\n input_ids = input_ids * 1 # Hack to materialize tensor\n\n # update generated ids, model inputs, and length for next step\n model_kwargs = self._update_model_kwargs_for_xla_generation(\n model_kwargs,\n batch_size=batch_beam_size,\n is_encoder_decoder=self.config.is_encoder_decoder,\n max_length=stopping_criteria.max_length,\n seq_length=cur_len,\n use_cache=model_kwargs[\"use_cache\"],\n )\n if model_kwargs[\"past_key_values\"] is not None:\n model_kwargs[\"past_key_values\"] = self._reorder_cache(model_kwargs[\"past_key_values\"], beam_idx.to(torch.int64))\n\n if return_dict_in_generate and output_scores:\n beam_indices = tuple((beam_indices[beam_idx[i]] + (beam_idx[i],) for i in range(len(beam_indices))))\n\n # increase cur_len\n cur_len = cur_len + 1\n\n # stop when each sentence is finished, or if we exceed the maximum length\n stop_criterion_1 = beam_scorer.is_done\n if isinstance(stopping_criteria, list):\n if len(stopping_criteria) == 1:\n stopping_criteria = stopping_criteria[0]\n\n # Cases that can be handled in XLA without requiring\n # non-padded input_ids\n if isinstance(stopping_criteria, MaxLengthCriteria):\n stop_criterion_2 = cur_len >= stopping_criteria.max_length\n elif isinstance(stopping_criteria, MaxTimeCriteria):\n stop_criterion_2 = stopping_criteria(input_ids, scores)\n else:\n # Other cases will be handled on CPU\n batch_size, _ = input_ids.shape\n input_ids_cpu = input_ids.to(\"cpu\")\n mask = torch.cat(\n [torch.ones(batch_size, cur_len), torch.zeros(batch_size, input_ids.shape[1] - cur_len)], dim=1\n ).bool()\n input_ids_cpu = torch.masked_select(input_ids_cpu, mask).reshape((batch_size, cur_len))\n scores_cpu = scores.to(\"cpu\") if torch.is_tensor(scores) else scores\n stop_criterion_2 = stopping_criteria(input_ids_cpu, scores_cpu)\n\n if stop_criterion_1 or stop_criterion_2:\n if not synced_gpus:\n break\n else:\n this_peer_finished = True\n\n sequence_outputs = beam_scorer.finalize(\n input_ids.to(\"cpu\"),\n beam_scores.to(\"cpu\"),\n next_tokens.to(\"cpu\"),\n next_indices.to(\"cpu\"),\n pad_token_id=pad_token_id,\n eos_token_id=eos_token_id,\n max_length=stopping_criteria.max_length,\n beam_indices=beam_indices,\n )\n\n for k, v in sequence_outputs.items():\n if type(v) == torch.Tensor:\n sequence_outputs[k] = sequence_outputs[k].to(input_ids.device)\n\n return sequence_outputs[\"sequences\"]\n\n\n def greedy_search(\n self,\n input_ids: torch.LongTensor,\n logits_processor: Optional[LogitsProcessorList] = None,\n stopping_criteria: Optional[StoppingCriteriaList] = None,\n max_length: Optional[int] = None,\n pad_token_id: Optional[int] = None,\n eos_token_id: Optional[Union[int, List[int]]] = None,\n output_attentions: Optional[bool] = None,\n output_hidden_states: Optional[bool] = None,\n output_scores: Optional[bool] = None,\n return_dict_in_generate: Optional[bool] = None,\n seq_length: Optional[int] = int,\n streamer: Optional[\"BaseStreamer\"] = None,\n **model_kwargs,\n ) -> Union[GreedySearchOutput, torch.LongTensor]:\n \"\"\"\n Overriding greedy sampling to use next tokens returned from neuron device instead of logits.\n \"\"\"\n # init values\n logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()\n use_cache = model_kwargs[\"use_cache\"] if \"use_cache\" in model_kwargs else False\n stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()\n pad_token_id = pad_token_id if pad_token_id is not None else self.generation_config.pad_token_id\n eos_token_id = eos_token_id if eos_token_id is not None else self.generation_config.eos_token_id\n if isinstance(eos_token_id, int):\n eos_token_id = [eos_token_id]\n eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None\n output_scores = output_scores if output_scores is not None else self.generation_config.output_scores\n output_attentions = (\n output_attentions if output_attentions is not None else self.generation_config.output_attentions\n )\n output_hidden_states = (\n output_hidden_states if output_hidden_states is not None else self.generation_config.output_hidden_states\n )\n\n # init attention / hidden states / scores tuples\n scores = () if (return_dict_in_generate and output_scores) else None\n decoder_attentions = () if (return_dict_in_generate and output_attentions) else None\n cross_attentions = () if (return_dict_in_generate and output_attentions) else None\n decoder_hidden_states = () if (return_dict_in_generate and output_hidden_states) else None\n\n\n # keep track of which sequences are already finished\n unfinished_sequences = torch.ones(input_ids.shape[0], dtype=torch.long, device=input_ids.device)\n\n this_peer_finished = False # used by synced_gpus only\n while True:\n\n # prepare model inputs\n # From max_length-sized input_ids, select first\n # seq_length - 1 values.\n\n if model_kwargs.get(\"past_key_values\") is None:\n input_ids_ = input_ids[:, :seq_length]\n else:\n update_indices = torch.stack(\n [torch.arange(input_ids.size(0)), torch.tensor(seq_length - 1).repeat(input_ids.size(0))],\n dim=-1,\n )\n input_ids_ = input_ids[update_indices[:, 0], update_indices[:, 1], None]\n\n model_inputs = self.prepare_inputs_for_generation(input_ids_, **model_kwargs)\n \n # forward pass to get next token\n output = self(\n **model_inputs,\n return_dict=True,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n )\n next_tokens = output[0]\n\n # finished sentences should have their next token be a padding token\n if eos_token_id is not None:\n if pad_token_id is None:\n raise ValueError(\"If `eos_token_id` is defined, make sure that `pad_token_id` is defined.\")\n next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)\n\n # update generated ids, model inputs, and length for next step\n\n batch_size, _ = input_ids.shape\n update_indices = torch.stack(\n [torch.arange(batch_size), torch.tensor(seq_length).repeat(batch_size)], dim=-1\n )\n input_ids[update_indices[:, 0], update_indices[:, 1]] = next_tokens[:]\n model_kwargs = self._update_model_kwargs_for_xla_generation(\n model_kwargs,\n batch_size=batch_size,\n is_encoder_decoder=self.config.is_encoder_decoder,\n max_length=stopping_criteria.max_length,\n seq_length=seq_length,\n use_cache=use_cache,\n )\n\n seq_length += 1\n\n # if eos_token was found in one sentence, set sentence to finished\n if eos_token_id_tensor is not None:\n unfinished_sequences = unfinished_sequences.mul(\n next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_tensor.unsqueeze(1)).prod(dim=0)\n )\n\n # stop when each sentence is finished, or if we exceed the maximum length\n stop_criterion_1 = unfinished_sequences.max() == 0\n\n if isinstance(stopping_criteria, list):\n if len(stopping_criteria) == 1:\n stopping_criteria = stopping_criteria[0]\n\n # Cases that can be handled in XLA without requiring\n # non-padded input_ids\n if isinstance(stopping_criteria, MaxLengthCriteria):\n stop_criterion_2 = seq_length >= stopping_criteria.max_length\n elif isinstance(stopping_criteria, MaxTimeCriteria):\n stop_criterion_2 = stopping_criteria(input_ids, scores)\n else:\n # Other cases will be handled on CPU\n batch_size, _ = input_ids.shape\n mask = torch.cat(\n [torch.ones(batch_size, seq_length), torch.zeros(batch_size, input_ids.shape[1] - seq_length)],\n dim=1,\n ).bool()\n input_ids_cpu = torch.masked_select(input_ids, mask).reshape((batch_size, seq_length)).to(\"cpu\")\n scores_cpu = scores.to(\"cpu\") if torch.is_tensor(scores) else scores\n stop_criterion_2 = stopping_criteria(input_ids_cpu, scores_cpu)\n\n if stop_criterion_1 or stop_criterion_2:\n this_peer_finished = True\n\n if this_peer_finished:\n break\n\n if streamer is not None:\n streamer.end()\n\n return input_ids\n \nclass EncoderWrapper(torch.nn.Module):\n '''\n This wrapper converts positional args to kwargs\n '''\n\n def __init__(self, \n encoder,\n decoder, \n model_config, \n batch_size, \n max_length, \n device, \n num_beams,\n tp_degree=None):\n \n super().__init__()\n self.encoder = encoder\n self.decoder = decoder\n self.batch_size = batch_size\n self.max_length = max_length\n self.model_config = model_config\n self.device = device\n self.num_beams = num_beams\n self.num_attention_heads_per_partition = model_config.num_heads\n self.tp_degree = tp_degree\n if self.tp_degree is not None:\n self.num_attention_heads_per_partition = model_config.num_heads // neuronx_distributed.parallel_layers.parallel_state.get_tensor_model_parallel_size()\n self.past_key_values_sa = torch.nn.ParameterList([torch.nn.Parameter(torch.ones((self.num_beams,self.num_attention_heads_per_partition,self.max_length-1,model_config.d_kv), dtype=torch.float32), requires_grad=False) for _ in range(model_config.num_decoder_layers * 2)])\n self.past_key_values_ca = torch.nn.ParameterList([torch.nn.Parameter(torch.ones((self.num_beams,self.num_attention_heads_per_partition,self.max_length,model_config.d_kv), dtype=torch.float32), requires_grad=False) for _ in range(model_config.num_decoder_layers * 2)])\n\n def forward(self, input_ids, attention_mask):\n '''\n This is the core functionality we want to trace. \n '''\n encoder_output = self.encoder(input_ids=input_ids,\n attention_mask=attention_mask,\n output_attentions=False,\n output_hidden_states=False)\n\n last_hidden_state = encoder_output[\"last_hidden_state\"]\n encoder_hidden_states = torch.concat([tensor.unsqueeze(0).repeat(self.num_beams, 1, 1) for tensor in last_hidden_state])\n\n decoder_blocks = self.decoder.block\n present_key_value_states_sa = []\n present_key_value_states_ca = []\n\n for i, block in enumerate(decoder_blocks):\n\n # Cross attention has to be initialized with the encoder hidden state\n cross_attention: T5LayerCrossAttention = block.layer[1]\n attention = cross_attention.EncDecAttention\n\n def shape(states):\n \"\"\"projection\"\"\"\n return states.view(self.batch_size, -1, self.num_attention_heads_per_partition, attention.key_value_proj_dim).transpose(1, 2)\n\n key_states = shape(attention.k(encoder_hidden_states))\n value_states = shape(attention.v(encoder_hidden_states))\n\n if self.tp_degree is None:\n # cross_attn_kv_state\n present_key_value_states_ca.append(key_states) \n present_key_value_states_ca.append(value_states) \n \n # Self attention kv states are initialized to zeros.\n present_key_value_states_sa.append(torch.zeros((self.batch_size, # key states\n self.model_config.num_heads, \n self.max_length-1, \n self.model_config.d_kv), dtype=torch.float32, device=self.device)) \n present_key_value_states_sa.append(torch.zeros((self.batch_size, # value states\n self.model_config.num_heads, \n self.max_length-1, \n self.model_config.d_kv), dtype=torch.float32, device=self.device))\n else:\n # We want to copy the cross attention states (key_states and value_states) into the decoder trace. \n # One way of doing it is to get the encoder trace to return the kv states as an output and then we can pass it to the decoder trace \n # as an output. But this requires a copy from device to cpu and back. \n #\n # There is no good way to keep the output within the device yet. Until we build that feature, we use this workaround. \n # The work around uses input_output_aliasing to map the output kv state to an input parameter. The output present_key_value_states_ca\n # represents the cross attention kv states and is aliased to a similarly named parameter. \n # \n # Why are we multiplying past_key_values_ca with 0 and adding to the key or value state? \n # The trace api will remove any variables that are not used to compute the output tensor. As the past_key_values parameter is not \n # being used and to compute the kv cache, it would be removed. To avoid that, we use it in an operation that computes the output \n # but at the same time does not effect the output. \n present_key_value_states_ca.append((self.past_key_values_ca[i*2] * 0) + key_states)\n present_key_value_states_ca.append((self.past_key_values_ca[i*2+1] * 0) + value_states)\n present_key_value_states_sa.append(self.past_key_values_sa[i*2]*torch.zeros((self.batch_size, self.num_attention_heads_per_partition, self.max_length-1, self.model_config.d_kv), dtype=torch.float32, device=\"xla\"))\n present_key_value_states_sa.append(self.past_key_values_sa[i*2+1]*torch.zeros((self.batch_size, self.num_attention_heads_per_partition, self.max_length-1, self.model_config.d_kv), dtype=torch.float32, device=\"xla\"))\n\n return present_key_value_states_sa + present_key_value_states_ca\n\nclass DecoderWrapper(torch.nn.Module):\n\n def __init__(self, \n decoder: T5Stack, \n lm_head: torch.nn.Linear,\n model_config,\n num_beams: int, \n max_length: int,\n device: str,\n tp_degree=None):\n super().__init__() \n self.decoder = decoder\n self.lm_head = lm_head\n self.model_dim=model_config.d_model\n self.device = device\n self.num_beams = num_beams\n self.batch_size = 1\n self.config = model_config\n\n num_heads=model_config.num_heads\n num_decoder_layers=model_config.num_decoder_layers\n\n self.num_attention_heads_per_partition = num_heads\n if tp_degree is not None:\n self.num_attention_heads_per_partition = num_heads // neuronx_distributed.parallel_layers.parallel_state.get_tensor_model_parallel_size()\n\n # (num_beams, n_heads, seq_length, dim_per_head)\n if device == \"cpu\":\n self.past_key_values_sa = [torch.ones((num_beams,num_heads,max_length-1,model_config.d_kv), dtype=torch.float32) for _ in range(num_decoder_layers * 2)]\n self.past_key_values_ca = [torch.ones((num_beams,num_heads,max_length,model_config.d_kv), dtype=torch.float32) for _ in range(num_decoder_layers * 2)]\n elif device == \"xla\":\n self.past_key_values_sa = torch.nn.ParameterList([torch.nn.Parameter(torch.ones((num_beams,self.num_attention_heads_per_partition,max_length-1,model_config.d_kv), dtype=torch.float32), requires_grad=False) for _ in range(num_decoder_layers * 2)])\n self.past_key_values_ca = torch.nn.ParameterList([torch.nn.Parameter(torch.ones((num_beams,self.num_attention_heads_per_partition,max_length,model_config.d_kv), dtype=torch.float32), requires_grad=False) for _ in range(num_decoder_layers * 2)])\n\n def update_past(self, past_key_values):\n new_past_sa = []\n new_past_ca = []\n for past_layer in past_key_values:\n new_past_layer = list(past_layer)\n for i in range(len(new_past_layer[:2])):\n new_past_layer[i] = past_layer[i][:, :, 1:]\n new_past_sa += [new_past_layer[:2],]\n new_past_ca += [new_past_layer[2:],]\n return new_past_sa, new_past_ca\n \n def reorder_cache(self, past_key_values, beam_idx):\n for i in range(len(past_key_values)):\n past_key_values[i] = torch.index_select(past_key_values[i], 0, beam_idx)\n return past_key_values\n\n def forward(self,\n input_ids,\n decoder_attention_mask,\n encoder_hidden_states,\n encoder_attention_mask,\n beam_idx,\n beam_scores,\n **kwargs):\n\n if self.num_beams > 1:\n # We reorder the cache based on the beams selected in each iteration. Required step for beam search.\n past_key_values_sa = self.reorder_cache(self.past_key_values_sa, beam_idx)\n past_key_values_ca = self.reorder_cache(self.past_key_values_ca, beam_idx)\n else:\n # We do not need to reorder for greedy sampling\n past_key_values_sa = self.past_key_values_sa\n past_key_values_ca = self.past_key_values_ca\n\n # The cache is stored in a flatten form. We order the cache per layer before passing it to the decoder. \n # Each layer has 4 tensors, so we group by 4. \n past_key_values = [[*past_key_values_sa[i*2:i*2+2], *past_key_values_ca[i*2:i*2+2]] for i in range(0, int(len(past_key_values_ca)/2))]\n\n decoder_output = self.decoder(\n input_ids=input_ids,\n attention_mask=decoder_attention_mask,\n past_key_values=past_key_values,\n encoder_hidden_states=encoder_hidden_states,\n encoder_attention_mask=encoder_attention_mask,\n use_cache=True,\n output_attentions=False,\n output_hidden_states=False)\n\n last_hidden_state = decoder_output['last_hidden_state']\n past_key_values = decoder_output['past_key_values']\n\n if self.config.tie_word_embeddings:\n last_hidden_state = last_hidden_state * (self.model_dim**-0.5)\n\n lm_logits = self.lm_head(last_hidden_state)\n\n past_key_values_sa, past_key_values_ca = self.update_past(past_key_values)\n\n # We flatten the cache to a single array. This is required for the input output aliasing to work\n past_key_values_sa = [vec for kv_per_layer in past_key_values_sa for vec in kv_per_layer]\n past_key_values_ca = [vec for kv_per_layer in past_key_values_ca for vec in kv_per_layer]\n\n if self.device == \"cpu\":\n self.past_key_values_sa = past_key_values_sa\n self.past_key_values_ca = past_key_values_ca\n\n # Moving the topk inside \n next_token_logits = lm_logits[:, -1, :]\n\n if self.num_beams > 1:\n logit_max, _ = torch.max(next_token_logits, dim=-1, keepdim=True)\n logsumexp = torch.log(torch.exp(next_token_logits - logit_max).sum(dim=-1, keepdim=True))\n next_token_scores = next_token_logits - logit_max - logsumexp\n next_token_scores = next_token_scores + beam_scores[:, None].expand_as(next_token_scores)\n\n # reshape for beam search\n vocab_size = next_token_scores.shape[-1]\n next_token_scores = next_token_scores.view(self.batch_size, self.num_beams * vocab_size)\n next_token_scores = next_token_scores * 1\n\n # Sample 2 next tokens for each beam (so we have some spare tokens and match output of beam search)\n next_token_scores, next_tokens = torch.topk(\n next_token_scores, 2 * self.num_beams, dim=1, largest=True, sorted=True\n ) \n\n next_indices = torch.div(next_tokens, vocab_size, rounding_mode=\"floor\")\n next_tokens = next_tokens % vocab_size\n\n return [next_token_scores, next_tokens, next_indices] + past_key_values_sa + past_key_values_ca\n else:\n # Greedy \n next_tokens = torch.argmax(next_token_logits, dim=-1)\n return [next_tokens] + past_key_values_sa + past_key_values_ca\n \n" }, { "path": "src/examples/pytorch/pipeline_tutorial/neuroncore_pipeline_pytorch.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"variable-character\",\n \"metadata\": {},\n \"source\": [\n \"# Using NeuronCore Pipeline with PyTorch\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"valued-economics\",\n \"metadata\": {},\n \"source\": [\n \"In this tutorial you compile a pretrained BERT base model from HuggingFace 🤗 Transformers, using the NeuronCore Pipeline feature of the AWS Neuron SDK. You benchmark model latency of the pipeline parallel mode and compare with the usual data parallel (multi-worker) deployment.\\n\",\n \"\\n\",\n \"This tutorial is intended to run in an inf1.6xlarge, running the latest AWS Deep Learning AMI (DLAMI). The inf1.6xlarge instance size has AWS Inferentia chips for a total of 16 NeuronCores.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python or Conda kernel environment that was set up according to the [PyTorch Installation Guide](../../../../frameworks/torch/torch-neuron/setup/pytorch-install.html). You can select the kernel from the \\\"Kernel -> Change Kernel\\\" option on the top of this Jupyter notebook page.\\n\",\n \"\\n\",\n \"> __Note:__ Do not execute this tutorial using \\\"Run -> Run all cells\\\" option. \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"private-authentication\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuron`\\n\",\n \"- `neuron-cc[tensorflow]`\\n\",\n \"- `transformers`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Neuron PyTorch setup guide. The additional HuggingFace 🤗 Transformers dependency must be installed here.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"romantic-accident\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"!pip install --upgrade \\\"transformers==4.6.0\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"prompt-australian\",\n \"metadata\": {},\n \"source\": [\n \"## Compiling a BERT base model for a single NeuronCore\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"aging-biodiversity\",\n \"metadata\": {},\n \"source\": [\n \"To run a HuggingFace [BERTModel](https://huggingface.co/transformers/model_doc/bert.html#bertmodel) on Inferentia, you only need to add a single extra line of code to the usual 🤗 Transformers PyTorch implementation, after importing the torch_neuron framework. \\n\",\n \"\\n\",\n \"Add the argument `return_dict=False` to the BERT transformers model so it can be traced with [TorchScript](https://pytorch.org/docs/stable/jit.html). TorchScript is a way to create serializable and optimizable models from PyTorch code. \\n\",\n \"\\n\",\n \"Enable padding to a maximum sequence length of 128, to test the model's performance with a realistic payload size. You can adapt this sequence length to your application's requirement. \\n\",\n \"\\n\",\n \"You can adapt the original example on the [BertModel forward pass docstring](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel.forward) according to the following cell\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"stretch-preview\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuron\\n\",\n \"from transformers import BertTokenizer, BertModel\\n\",\n \"\\n\",\n \"from joblib import Parallel, delayed \\n\",\n \"import numpy as np\\n\",\n \"from tqdm import tqdm\\n\",\n \"\\n\",\n \"import os\\n\",\n \"import time \\n\",\n \"\\n\",\n \"\\n\",\n \"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\\n\",\n \"model = BertModel.from_pretrained('bert-base-uncased',return_dict=False)\\n\",\n \"\\n\",\n \"inputs = tokenizer(\\\"Hello, my dog is cute\\\",return_tensors=\\\"pt\\\",max_length=128,padding='max_length',truncation=True)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"conceptual-aberdeen\",\n \"metadata\": {},\n \"source\": [\n \"The one extra line required is the call to torch.neuron.trace() method. This call compiles the model and returns the forwad method of the torch `nn.Model` method, which you can use to run inference. \\n\",\n \"\\n\",\n \"The compiled graph can be saved using the `torch.jit.save` function and restored using `torch.jit.load` function for inference on Inf1 instances. During inference, the previously compiled artifacts will be loaded into the Neuron Runtime for inference execution.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"secondary-exclusive\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"neuron_model = torch.neuron.trace(model, \\n\",\n \" example_inputs = (inputs['input_ids'],inputs['attention_mask']),\\n\",\n \" verbose=1)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"atmospheric-stewart\",\n \"metadata\": {},\n \"source\": [\n \"## Running the BERT base model on a single NeuronCore\\n\",\n \"With the model already available in memory, you can time one execution and check for the latency on the single inference call. You will load the model into Inferentia with a single inference call. A large \\\"wall time\\\" is expected when you first run the next cell, running the cell twice will show the actual inference latency:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"approved-reputation\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"# The following line tests inference and should be executed on Inf1 instance family. \\n\",\n \"outputs = neuron_model(*(inputs['input_ids'],inputs['attention_mask']))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"great-collective\",\n \"metadata\": {},\n \"source\": [\n \"You can also check for the throughput of the single model running on a single NeuronCore.\\n\",\n \"\\n\",\n \"The sequential inference test (for loop) does not measure all the performance one can achieve in an instance with multiple NeuronCores. To improve hardwar utilization you can run parallel inference requests over multiple model workers, which you'll test in the Data Parallel Bonus Section below.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"framed-reference\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"for _ in tqdm(range(100)):\\n\",\n \" outputs = neuron_model(*(inputs['input_ids'],inputs['attention_mask'])) \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"super-innocent\",\n \"metadata\": {},\n \"source\": [\n \"Save the compiled model for later use:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"express-greensboro\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"neuron_model.save('bert-base-uncased-neuron.pt')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"modified-government\",\n \"metadata\": {},\n \"source\": [\n \"## Compiling a BERT base model for 16 NeuronCores\\n\",\n \"\\n\",\n \"Our next step is to compile the same model for all 16 NeuronCores available in the inf1.6xlarge and check the performance difference when running pipeline parallel inferences.. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"compound-initial\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuron\\n\",\n \"from transformers import BertTokenizer, BertModel\\n\",\n \"\\n\",\n \"from joblib import Parallel, delayed \\n\",\n \"import numpy as np\\n\",\n \"from tqdm import tqdm\\n\",\n \"\\n\",\n \"import os\\n\",\n \"import time \\n\",\n \"\\n\",\n \"\\n\",\n \"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\\n\",\n \"model = BertModel.from_pretrained('bert-base-uncased',return_dict=False)\\n\",\n \"\\n\",\n \"inputs = tokenizer(\\\"Hello, my dog is cute\\\",return_tensors=\\\"pt\\\",max_length=128,padding='max_length',truncation=True)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"universal-desperate\",\n \"metadata\": {},\n \"source\": [\n \"To enable pipeline mode during compilation, you need only to add the compiler flag `--neuroncore-pipeline-cores` and set the number of desired cores. The cell below sets up a `neuroncore_pipeline_cores` string, which you can set for the available number of NeuronCores on the instance: _inf1.6xlarge_ has 16 NeuronCores in 4 Inferentia chips. \\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"passing-masters\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Number of Cores in the Pipeline Mode\\n\",\n \"neuroncore_pipeline_cores = 16 # This string should be '4' on an inf1.xlarge\\n\",\n \"\\n\",\n \"# Compiling for neuroncore-pipeline-cores='16'\\n\",\n \"neuron_pipeline_model = torch.neuron.trace(model,\\n\",\n \" example_inputs = (inputs['input_ids'],inputs['attention_mask']),\\n\",\n \" verbose=1,\\n\",\n \" compiler_args = ['--neuroncore-pipeline-cores', str(neuroncore_pipeline_cores)]\\n\",\n \" )\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"enhanced-swedish\",\n \"metadata\": {},\n \"source\": [\n \"## Running the BERT base model on 16 NeuronCores\\n\",\n \"Next, time one execution and check for the latency on the single inference call over 16 cores. You will load the model into Inferentia with a single inference call. A large \\\"wall time\\\" is expected when you first run the next cell, running the cell twice will show the actual inference latency:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"expressed-trinity\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"# The following line tests inference and should be executed on Inf1 instance family. \\n\",\n \"outputs = neuron_pipeline_model(*(inputs['input_ids'],inputs['attention_mask']))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"located-graphic\",\n \"metadata\": {},\n \"source\": [\n \"Check also for the throughput of the single model running over a 16 NeuronCores. \\n\",\n \"\\n\",\n \"The sequential inference test (for loop) does not measure all the performance one can achieve with Pipeline mode. As the inference runs in streaming fashion, at least 15 cores are waiting for a new call until the last one processes the first call. This results in low NeuronCore utilization. To improve hardware utilization you will require parallel inference requests, which you'll test in the next section.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"hydraulic-calcium\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"for _ in tqdm(range(100)):\\n\",\n \" outputs = neuron_pipeline_model(*(inputs['input_ids'],inputs['attention_mask']))\\n\",\n \" \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"patent-victoria\",\n \"metadata\": {},\n \"source\": [\n \"## Load Testing the Pipeline Parallel Mode\\n\",\n \"\\n\",\n \"To put the 16 NeuronCores group to test, a client has to run concurrent requests to the model. In this Notebook setup you achieve it by creating a thread pool with `Joblib.Parallel`, with all workers on the pool runing one inference call. \\n\",\n \"\\n\",\n \"You can define a new method called `inference_latency()` so that you measure the amount of time each inference calls take.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"appointed-adventure\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def inference_latency(model,*inputs):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" infetence_time is a simple method to return the latency of a model inference.\\n\",\n \" \\n\",\n \" Parameters:\\n\",\n \" model: torch model onbject loaded using torch.jit.load\\n\",\n \" inputs: model() args\\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" latency in seconds\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" start = time.time()\\n\",\n \" _ = model(*inputs)\\n\",\n \" return time.time() - start\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"environmental-guinea\",\n \"metadata\": {},\n \"source\": [\n \"Use `tqdm` to measure total throughput of your experiment, with a nice side-effect of \\\"cool progress bar!\\\". The total throughput is expected to be high, so set your experiment range to a large number, here 30k inferences. \\n\",\n \"\\n\",\n \"To calculate the latency statistics over the returned 30k list of latencies use `numpy.qunatile()` method.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"played-catch\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"t = tqdm(range(30000), position=0, leave=True)\\n\",\n \"latency = Parallel(n_jobs=12,prefer=\\\"threads\\\")(delayed(inference_latency)(neuron_pipeline_model,*(inputs['input_ids'],inputs['attention_mask'])) for i in t)\\n\",\n \"\\n\",\n \"p50 = np.quantile(latency[-10000:],0.50) * 1000\\n\",\n \"p95 = np.quantile(latency[-10000:],0.95) * 1000\\n\",\n \"p99 = np.quantile(latency[-10000:],0.99) * 1000\\n\",\n \"avg_throughput = t.total/t.format_dict['elapsed']\\n\",\n \"print(f'Avg Throughput: :{avg_throughput:.1f}')\\n\",\n \"print(f'50th Percentile Latency:{p50:.1f} ms')\\n\",\n \"print(f'95th Percentile Latency:{p95:.1f} ms')\\n\",\n \"print(f'99th Percentile Latency:{p99:.1f} ms')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"exposed-northern\",\n \"metadata\": {},\n \"source\": [\n \"Save compile model for later use:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"imperial-complex\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Save the TorchScript graph\\n\",\n \"neuron_pipeline_model.save('bert-base-uncased-neuron-pipeline.pt')\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"abroad-earthquake\",\n \"metadata\": {},\n \"source\": [\n \"## Bonus Section - Load Testing Data Parallel Mode\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"therapeutic-detector\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuron\\n\",\n \"from transformers import BertTokenizer \\n\",\n \"\\n\",\n \"from joblib import Parallel, delayed \\n\",\n \"import numpy as np\\n\",\n \"from tqdm import tqdm\\n\",\n \"\\n\",\n \"import os\\n\",\n \"import time \\n\",\n \"\\n\",\n \"def inference_latency(model,*inputs):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" infetence_time is a simple method to return the latency of a model inference.\\n\",\n \" \\n\",\n \" Parameters:\\n\",\n \" model: torch model onbject loaded using torch.jit.load\\n\",\n \" inputs: model() args\\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" latency in seconds\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" start = time.time()\\n\",\n \" _ = model(*inputs)\\n\",\n \" return time.time() - start\\n\",\n \"\\n\",\n \"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\\n\",\n \"\\n\",\n \"inputs = tokenizer(\\\"Hello, my dog is cute\\\",return_tensors=\\\"pt\\\",max_length=128,padding='max_length',truncation=True)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"legal-terrorist\",\n \"metadata\": {},\n \"source\": [\n \"You use the `'NEURON_RT_NUM_CORES'` environment variable to define how many Neuron cores to be used. Set the environment variable to the number of individual workers you want to test in parallel.\\n\",\n \"\\n\",\n \"`torch_neuron` will load one model per NeuronCore group until it runs out of cores. At that point, if the Python process continues to spawn more model objest using `torch.jit.load`, `torch_neuron` will start stacking more than one model per core, until the Inferentia chip memory is full. \\n\",\n \"\\n\",\n \"Inferentia is able to run inference over all the loaded models, but only one at a time. The Neuron Runtime takes care of dynamically switching the model context as requests come in, no extra worker process management required. Use 1 model per NeuronCore to achieve maximum performance.\\n\",\n \"\\n\",\n \"The following cell creates a list with as many models as NeuronCore Groups and execute one single dummy inference to load the models into Inferentia. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"current-mechanics\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import warnings\\n\",\n \"# Number of data parallel workers\\n\",\n \"number_of_workers=16 # This number should be 4 on an inf1.xlarge\\n\",\n \"\\n\",\n \"# Setting up a data parallel group\\n\",\n \"os.environ['NEURON_RT_NUM_CORES'] = str(number_of_workers)\\n\",\n \"\\n\",\n \"# Loading 'number_of_workers' amount of models in Python memory\\n\",\n \"model_list = [torch.jit.load('bert-base-uncased-neuron.pt') for _ in range(number_of_workers)]\\n\",\n \"\\n\",\n \"# Dummy inference to load models to Inferentia\\n\",\n \"_ = [mod(*(inputs['input_ids'],inputs['attention_mask'])) for mod in model_list]\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"threatened-swaziland\",\n \"metadata\": {},\n \"source\": [\n \"Adapt the call to `joblib.Parallel()` iterating over a concatenated version of the `model_list`, to run 'round-robin' calls to each of the model workers. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"fleet-month\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"t = tqdm(model_list*1500,position=0, leave=True)\\n\",\n \"latency = Parallel(n_jobs=number_of_workers,prefer=\\\"threads\\\")(delayed(inference_latency)(mod,*(inputs['input_ids'],inputs['attention_mask'])) for mod in t)\\n\",\n \"\\n\",\n \"p50 = np.quantile(latency[-10000:],0.50) * 1000\\n\",\n \"p95 = np.quantile(latency[-10000:],0.95) * 1000\\n\",\n \"p99 = np.quantile(latency[-10000:],0.99) * 1000\\n\",\n \"avg_throughput = t.total/t.format_dict['elapsed']\\n\",\n \"print(f'Avg Throughput: :{avg_throughput:.1f}')\\n\",\n \"print(f'50th Percentile Latency:{p50:.1f} ms')\\n\",\n \"print(f'95th Percentile Latency:{p95:.1f} ms')\\n\",\n \"print(f'99th Percentile Latency:{p99:.1f} ms')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"aggressive-stevens\",\n \"metadata\": {},\n \"source\": [\n \"For this model, despite the larger number of workers, the per-worker latency increases when running a single model per core, which in turn reduces the total throughput. \\n\",\n \"\\n\",\n \"This behavior may not repeat if the model memory footprint or the input payload size changes, i.e batch size > 1. We encourage you to experiment with the data parallel and pipeline parallel modes to optimize your application performance. \"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Environment (conda_aws_neuron_pytorch_p36)\",\n \"language\": \"python\",\n \"name\": \"conda_aws_neuron_pytorch_p36\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/pytorch/resnet50.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# ResNet50 model for Inferentia\\n\",\n \"\\n\",\n \"\\n\",\n \"## Introduction:\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy a ResNet50 model for inference on Inferentia. \\n\",\n \"\\n\",\n \"This Jupyter notebook should run on an inf1.6xlarge instance. The inference part of this tutorial requires an inf1 instance, not the compilation stage. For simplicity we will run this tutorial on an inf1.6xlarge, but in real life scenarios the compilation should be done on a compute instance and the deployment on an inf1 instance to save costs. \\n\",\n \"\\n\",\n \"In this tutorial we provide three main sections:\\n\",\n \"\\n\",\n \"1. Compile the ResNet50 model and infer with a batch size of 1\\n\",\n \"\\n\",\n \"2. Run the same compiled model on multiple NeuronCores using `torch.neuron.DataParallel` and dynamic batching\\n\",\n \"\\n\",\n \"3. Compile the ResNet50 model with a batch size of 5 and run it on multiple NeuronCores using `torch.neuron.DataParallel` for optimal performance on Inferentia\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [PyTorch Installation Guide](../../../frameworks/torch/torch-neuron/setup/pytorch-install.html). You can select the kernel from the \\\"Kernel -> Change Kernel\\\" option on the top of this Jupyter notebook page.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch>=1.8`\\n\",\n \"- `torch-neuron`\\n\",\n \"- `torchvision`\\n\",\n \"- `neuron-cc[tensorflow]`\\n\",\n \"\\n\",\n \"These will be installed by default when configuring your environment using the Neuron PyTorch setup guide.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Compile model for Neuron\\n\",\n \"\\n\",\n \"The following step will compile the ResNet50 model for Inferentia. This will take a few minutes. At the end of script execution, the compiled model is saved as `resnet50_neuron.pt` in your local directory\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"from torchvision import models, transforms, datasets\\n\",\n \"import torch_neuron\\n\",\n \"\\n\",\n \"# Create an example input for compilation\\n\",\n \"image = torch.zeros([1, 3, 224, 224], dtype=torch.float32)\\n\",\n \"\\n\",\n \"# Load a pretrained ResNet50 model\\n\",\n \"model = models.resnet50(pretrained=True)\\n\",\n \"\\n\",\n \"# Tell the model we are using it for evaluation (not training)\\n\",\n \"model.eval()\\n\",\n \"\\n\",\n \"# Analyze the model - this will show operator support and operator count\\n\",\n \"torch.neuron.analyze_model(model, example_inputs=[image])\\n\",\n \"\\n\",\n \"# Compile the model using torch.neuron.trace to create a Neuron model\\n\",\n \"# that that is optimized for the Inferentia hardware\\n\",\n \"model_neuron = torch.neuron.trace(model, example_inputs=[image])\\n\",\n \"\\n\",\n \"# The output of the compilation step will report the percentage of operators that \\n\",\n \"# are compiled to Neuron, for example:\\n\",\n \"#\\n\",\n \"# INFO:Neuron:The neuron partitioner created 1 sub-graphs\\n\",\n \"# INFO:Neuron:Neuron successfully compiled 1 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 100.0%\\n\",\n \"# \\n\",\n \"# We will also be warned if there are operators that are not placed on the Inferentia hardware\\n\",\n \"\\n\",\n \"# Save the compiled model\\n\",\n \"model_neuron.save(\\\"resnet50_neuron.pt\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference on Inferentia\\n\",\n \"\\n\",\n \"We can use the compiled Neuron model to run inference on Inferentia.\\n\",\n \"\\n\",\n \"In the following example, we preprocess a sample image for inference using the CPU model and Neuron model. We compare the predicted labels from the CPU model and Neuron model to verify that they are the same.\\n\",\n \"\\n\",\n \"Important: Do not perform inference with a Neuron traced model on a non-Neuron supported instance, as the results will not be calculated properly.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Define a preprocessing function\\n\",\n \"\\n\",\n \"We define a basic image preprocessing function that loads a sample image and labels, normalizes and batches the image, and transforms the image into a tensor for inference using the compiled Neuron model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"import os\\n\",\n \"from urllib import request\\n\",\n \"\\n\",\n \"# Create an image directory containing a sample image of a small kitten\\n\",\n \"os.makedirs(\\\"./torch_neuron_test/images\\\", exist_ok=True)\\n\",\n \"request.urlretrieve(\\\"https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg\\\",\\n\",\n \" \\\"./torch_neuron_test/images/kitten_small.jpg\\\")\\n\",\n \"\\n\",\n \"# Fetch labels to output the top classifications\\n\",\n \"request.urlretrieve(\\\"https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json\\\",\\\"imagenet_class_index.json\\\")\\n\",\n \"idx2label = []\\n\",\n \"\\n\",\n \"# Read the labels and create a list to hold them for classification \\n\",\n \"with open(\\\"imagenet_class_index.json\\\", \\\"r\\\") as read_file:\\n\",\n \" class_idx = json.load(read_file)\\n\",\n \" idx2label = [class_idx[str(k)][1] for k in range(len(class_idx))]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import numpy as np\\n\",\n \"\\n\",\n \"def preprocess(batch_size=1, num_neuron_cores=1):\\n\",\n \" # Define a normalization function using the ImageNet mean and standard deviation\\n\",\n \" normalize = transforms.Normalize(\\n\",\n \" mean=[0.485, 0.456, 0.406],\\n\",\n \" std=[0.229, 0.224, 0.225])\\n\",\n \"\\n\",\n \" # Resize the sample image to [1, 3, 224, 224], normalize it, and turn it into a tensor\\n\",\n \" eval_dataset = datasets.ImageFolder(\\n\",\n \" os.path.dirname(\\\"./torch_neuron_test/\\\"),\\n\",\n \" transforms.Compose([\\n\",\n \" transforms.Resize([224, 224]),\\n\",\n \" transforms.ToTensor(),\\n\",\n \" normalize,\\n\",\n \" ])\\n\",\n \" )\\n\",\n \" image, _ = eval_dataset[0]\\n\",\n \" image = torch.tensor(image.numpy()[np.newaxis, ...])\\n\",\n \"\\n\",\n \" # Create a \\\"batched\\\" image with enough images to go on each of the available NeuronCores\\n\",\n \" # batch_size is the per-core batch size\\n\",\n \" # num_neuron_cores is the number of NeuronCores being used\\n\",\n \" batch_image = image\\n\",\n \" for i in range(batch_size * num_neuron_cores - 1):\\n\",\n \" batch_image = torch.cat([batch_image, image], 0)\\n\",\n \" \\n\",\n \" return batch_image\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Run inference using the Neuron model\\n\",\n \"\\n\",\n \"We import the necessary python modules, load the torch-neuron compiled model, and run inference on Inferentia. \\n\",\n \"\\n\",\n \"By default, the Neuron model will run on a single NeuronCore. In the next section, we will see how to run the Neuron model on multiple NeuronCores to fully saturate our hardware for optimal performance on Inferentia. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"from torchvision import models, transforms, datasets\\n\",\n \"import torch_neuron\\n\",\n \"\\n\",\n \"# Get a sample image\\n\",\n \"image = preprocess()\\n\",\n \"\\n\",\n \"# Run inference using the CPU model\\n\",\n \"output_cpu = model(image)\\n\",\n \"\\n\",\n \"# Load the compiled Neuron model\\n\",\n \"model_neuron = torch.jit.load('resnet50_neuron.pt')\\n\",\n \"\\n\",\n \"# Run inference using the Neuron model\\n\",\n \"output_neuron = model_neuron(image)\\n\",\n \"\\n\",\n \"# Verify that the CPU and Neuron predictions are the same by comparing\\n\",\n \"# the top-5 results\\n\",\n \"top5_cpu = output_cpu[0].sort()[1][-5:]\\n\",\n \"top5_neuron = output_neuron[0].sort()[1][-5:]\\n\",\n \"\\n\",\n \"# Lookup and print the top-5 labels\\n\",\n \"top5_labels_cpu = [idx2label[idx] for idx in top5_cpu]\\n\",\n \"top5_labels_neuron = [idx2label[idx] for idx in top5_neuron]\\n\",\n \"print(\\\"CPU top-5 labels: {}\\\".format(top5_labels_cpu))\\n\",\n \"print(\\\"Neuron top-5 labels: {}\\\".format(top5_labels_neuron))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run Inference using torch.neuron.DataParallel\\n\",\n \"\\n\",\n \"To fully leverage the Inferentia hardware we want to use all avaialable NeuronCores. An inf1.xlarge and inf1.2xlarge have four NeuronCores, an inf1.6xlarge has 16 NeuronCores, and an inf1.24xlarge has 64 NeuronCores. For maximum performance on Inferentia hardware, we can use `torch.neuron.DataParallel` to utilize all available NeuronCores.\\n\",\n \"\\n\",\n \"`torch.neuron.DataParallel` implements data parallelism at the module level by duplicating the Neuron model on all available NeuronCores and distributing data across the different cores for parallelized inference.\\n\",\n \"\\n\",\n \"In the following section, we will run inference using the `torch.neuron.DataParallel` module to fully saturate the Inferentia hardware. We benchmark the model to collect throughput and latency statistics.\\n\",\n \"\\n\",\n \"Note: `torch.neuron.DataParallel` is new with Neuron 1.16.0. Please ensure you are using the latest Neuron package to run the following sections. \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Define a benchmarking function\\n\",\n \"\\n\",\n \"We create a function that handles benchmarking the Neuron model to collect throughput and latency metrics. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from time import time\\n\",\n \"\\n\",\n \"def benchmark(model, image):\\n\",\n \" print('Input image shape is {}'.format(list(image.shape)))\\n\",\n \" \\n\",\n \" # The first inference loads the model so exclude it from timing \\n\",\n \" results = model(image)\\n\",\n \" \\n\",\n \" # Collect throughput and latency metrics\\n\",\n \" latency = []\\n\",\n \" throughput = []\\n\",\n \"\\n\",\n \" # Run inference for 100 iterations and calculate metrics\\n\",\n \" num_infers = 100\\n\",\n \" for _ in range(num_infers):\\n\",\n \" delta_start = time()\\n\",\n \" results = model(image)\\n\",\n \" delta = time() - delta_start\\n\",\n \" latency.append(delta)\\n\",\n \" throughput.append(image.size(0)/delta)\\n\",\n \" \\n\",\n \" # Calculate and print the model throughput and latency\\n\",\n \" print(\\\"Avg. Throughput: {:.0f}, Max Throughput: {:.0f}\\\".format(np.mean(throughput), np.max(throughput)))\\n\",\n \" print(\\\"Latency P50: {:.0f}\\\".format(np.percentile(latency, 50)*1000.0))\\n\",\n \" print(\\\"Latency P90: {:.0f}\\\".format(np.percentile(latency, 90)*1000.0))\\n\",\n \" print(\\\"Latency P95: {:.0f}\\\".format(np.percentile(latency, 95)*1000.0))\\n\",\n \" print(\\\"Latency P99: {:.0f}\\\\n\\\".format(np.percentile(latency, 99)*1000.0))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Run Inference using torch.neuron.DataParallel\\n\",\n \"\\n\",\n \"We create the `torch.neuron.DataParallel` module using the compiled Neuron model, get a sample image, and benchmark the parallelized model on Neuron.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a torch.neuron.DataParallel module using the compiled Neuron model\\n\",\n \"# By default, torch.neuron.DataParallel will use four cores on an inf1.xlarge\\n\",\n \"# or inf1.2xlarge, 16 cores on an inf1.6xlarge, and 24 cores on an inf1.24xlarge\\n\",\n \"model_neuron_parallel = torch.neuron.DataParallel(model_neuron)\\n\",\n \"\\n\",\n \"# Get sample image with batch size=1 per NeuronCore\\n\",\n \"batch_size = 1\\n\",\n \"\\n\",\n \"# For an inf1.xlarge or inf1.2xlarge, set num_neuron_cores = 4\\n\",\n \"num_neuron_cores = 16\\n\",\n \"\\n\",\n \"image = preprocess(batch_size=batch_size, num_neuron_cores=num_neuron_cores)\\n\",\n \"\\n\",\n \"# Benchmark the model\\n\",\n \"benchmark(model_neuron_parallel, image)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference with dynamic batch sizes\\n\",\n \"\\n\",\n \"Batch size has a direct impact on model performance. The Inferentia chip is optimized to run with small batch sizes. This means that a Neuron compiled model can outperform a GPU model, even if running single digit batch sizes.\\n\",\n \"\\n\",\n \"As a general best practice, we recommend optimizing your model's throughput by compiling the model with a small batch size and gradually increasing it to find the peak throughput on Inferentia.\\n\",\n \"\\n\",\n \"Dynamic batching is a feature that allows you to use tensor batch sizes that the Neuron model was not originally compiled against. This is necessary because the underlying Inferentia hardware will always execute inferences with the batch size used during compilation. Fixed batch size execution allows tuning the input batch size for optimal performance. For example, batch size 1 may be best suited for an ultra-low latency on-demand inference application, while batch size > 1 can be used to maximize throughput for offline inferencing. Dynamic batching is implemented by slicing large input tensors into chunks that match the batch size used during the `torch.neuron.trace` compilation call. \\n\",\n \"\\n\",\n \"The `torch.neuron.DataParallel` class automatically enables dynamic batching on eligible models. This allows us to run inference in applications that have inputs with a variable batch size without needing to recompile the model.\\n\",\n \"\\n\",\n \"In the following example, we use the same `torch.neuron.DataParallel` module to run inference using several different batch sizes. Notice that latency increases consistently as the batch size increases. Throughput increases as well, up until a certain point where the input size becomes too large to be efficient.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# using the same DataParallel model_neuron_parallel model, we can run\\n\",\n \"# inference on inputs with a variable batch size without recompiling\\n\",\n \"batch_sizes = [2, 3, 4, 5, 6, 7]\\n\",\n \"for batch_size in batch_sizes:\\n\",\n \" print('Batch size: {}'.format(batch_size))\\n\",\n \" image = preprocess(batch_size=batch_size, num_neuron_cores=num_neuron_cores)\\n\",\n \" \\n\",\n \" # Benchmark the model for each input batch size\\n\",\n \" benchmark(model_neuron_parallel, image)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Compile and Infer with different batch sizes on multiple NeuronCores\\n\",\n \"\\n\",\n \"Dynamic batching using small batch sizes can result in sub-optimal throughput because it involves slicing tensors into chunks and iteratively sending data to the hardware. Using a larger batch size at compilation time can use the Inferentia hardware more efficiently in order to maximize throughput. You can test the tradeoff between individual request latency and total throughput by fine-tuning the input batch size.\\n\",\n \"\\n\",\n \"In the following example, we recompile our model using a batch size of 5 and run the model using `torch.neuron.DataParallel` to fully saturate our Inferentia hardware for optimal performance.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create an input with batch size 5 for compilation\\n\",\n \"batch_size = 5\\n\",\n \"image = torch.zeros([batch_size, 3, 224, 224], dtype=torch.float32)\\n\",\n \"\\n\",\n \"# Recompile the ResNet50 model for inference with batch size 5\\n\",\n \"model_neuron = torch.neuron.trace(model, example_inputs=[image])\\n\",\n \"\\n\",\n \"# Export to saved model\\n\",\n \"model_neuron.save(\\\"resnet50_neuron_b{}.pt\\\".format(batch_size))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Run inference with batch size of 5 using the Neuron model compiled for a batch size of 5.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"batch_size = 5\\n\",\n \"\\n\",\n \"# Load compiled Neuron model\\n\",\n \"model_neuron = torch.jit.load(\\\"resnet50_neuron_b{}.pt\\\".format(batch_size))\\n\",\n \"\\n\",\n \"# Create DataParallel model\\n\",\n \"model_neuron_parallel = torch.neuron.DataParallel(model_neuron)\\n\",\n \"\\n\",\n \"# Get sample image with batch size=5\\n\",\n \"image = preprocess(batch_size=batch_size, num_neuron_cores=num_neuron_cores)\\n\",\n \"\\n\",\n \"# Benchmark the model\\n\",\n \"benchmark(model_neuron_parallel, image)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can experiment with different batch size values to see what gives the best overall throughput on Inferentia.\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/pytorch/resnet50_partition.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Manual Partitioning Tutorial\\n\",\n \"\\n\",\n \"In this tutorial we will run through how to manually partition a graph. There are six steps:\\n\",\n \"\\n\",\n \"1. Import ResNet50 code from torchvision and set to evaluation mode\\n\",\n \"1. Download a test image and preprocess it\\n\",\n \"1. Run inference on CPU as a baseline\\n\",\n \"1. Manually partition the graph using Neuron\\n\",\n \"1. Save the model to be loaded on another instance\\n\",\n \"1. Inspect the graph to deepen our understanding\\n\",\n \"\\n\",\n \"The following is a ResNet50 implementation copied from `torchvision.models.resnet`.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**STEP 1:** Import torchvision ResNet50 and run the model on CPU\\n\",\n \"\\n\",\n \"Note that training code can be inserted before `model.eval()` if retraining/fine-tuning is necessary.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch.nn as nn\\n\",\n \"\\n\",\n \"\\n\",\n \"def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1):\\n\",\n \" \\\"\\\"\\\"3x3 convolution with padding\\\"\\\"\\\"\\n\",\n \" return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,\\n\",\n \" padding=dilation, groups=groups, bias=False, dilation=dilation)\\n\",\n \"\\n\",\n \"\\n\",\n \"def conv1x1(in_planes, out_planes, stride=1):\\n\",\n \" \\\"\\\"\\\"1x1 convolution\\\"\\\"\\\"\\n\",\n \" return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)\\n\",\n \"\\n\",\n \"\\n\",\n \"class Bottleneck(nn.Module):\\n\",\n \" expansion = 4\\n\",\n \"\\n\",\n \" def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,\\n\",\n \" base_width=64, dilation=1, norm_layer=None):\\n\",\n \" super(Bottleneck, self).__init__()\\n\",\n \" if norm_layer is None:\\n\",\n \" norm_layer = nn.BatchNorm2d\\n\",\n \" width = int(planes * (base_width / 64.)) * groups\\n\",\n \" # Both self.conv2 and self.downsample layers downsample the input when stride != 1\\n\",\n \" self.conv1 = conv1x1(inplanes, width)\\n\",\n \" self.bn1 = norm_layer(width)\\n\",\n \" self.conv2 = conv3x3(width, width, stride, groups, dilation)\\n\",\n \" self.bn2 = norm_layer(width)\\n\",\n \" self.conv3 = conv1x1(width, planes * self.expansion)\\n\",\n \" self.bn3 = norm_layer(planes * self.expansion)\\n\",\n \" self.relu = nn.ReLU(inplace=True)\\n\",\n \" self.downsample = downsample\\n\",\n \" self.stride = stride\\n\",\n \"\\n\",\n \" def forward(self, x):\\n\",\n \" identity = x\\n\",\n \"\\n\",\n \" out = self.conv1(x)\\n\",\n \" out = self.bn1(out)\\n\",\n \" out = self.relu(out)\\n\",\n \"\\n\",\n \" out = self.conv2(out)\\n\",\n \" out = self.bn2(out)\\n\",\n \" out = self.relu(out)\\n\",\n \"\\n\",\n \" out = self.conv3(out)\\n\",\n \" out = self.bn3(out)\\n\",\n \"\\n\",\n \" if self.downsample is not None:\\n\",\n \" identity = self.downsample(x)\\n\",\n \"\\n\",\n \" out += identity\\n\",\n \" out = self.relu(out)\\n\",\n \"\\n\",\n \" return out\\n\",\n \"\\n\",\n \"\\n\",\n \"class ResNet(nn.Module):\\n\",\n \"\\n\",\n \" def __init__(self, block, layers, num_classes=1000, zero_init_residual=False,\\n\",\n \" groups=1, width_per_group=64, replace_stride_with_dilation=None,\\n\",\n \" norm_layer=None):\\n\",\n \" super(ResNet, self).__init__()\\n\",\n \" if norm_layer is None:\\n\",\n \" norm_layer = nn.BatchNorm2d\\n\",\n \" self._norm_layer = norm_layer\\n\",\n \"\\n\",\n \" self.inplanes = 64\\n\",\n \" self.dilation = 1\\n\",\n \" if replace_stride_with_dilation is None:\\n\",\n \" # each element in the tuple indicates if we should replace\\n\",\n \" # the 2x2 stride with a dilated convolution instead\\n\",\n \" replace_stride_with_dilation = [False, False, False]\\n\",\n \" if len(replace_stride_with_dilation) != 3:\\n\",\n \" raise ValueError(\\\"replace_stride_with_dilation should be None \\\"\\n\",\n \" \\\"or a 3-element tuple, got {}\\\".format(replace_stride_with_dilation))\\n\",\n \" self.groups = groups\\n\",\n \" self.base_width = width_per_group\\n\",\n \" self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3,\\n\",\n \" bias=False)\\n\",\n \" self.bn1 = norm_layer(self.inplanes)\\n\",\n \" self.relu = nn.ReLU(inplace=True)\\n\",\n \" self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)\\n\",\n \" self.layer1 = self._make_layer(block, 64, layers[0])\\n\",\n \" self.layer2 = self._make_layer(block, 128, layers[1], stride=2,\\n\",\n \" dilate=replace_stride_with_dilation[0])\\n\",\n \" self.layer3 = self._make_layer(block, 256, layers[2], stride=2,\\n\",\n \" dilate=replace_stride_with_dilation[1])\\n\",\n \" self.layer4 = self._make_layer(block, 512, layers[3], stride=2,\\n\",\n \" dilate=replace_stride_with_dilation[2])\\n\",\n \" self.avgpool = nn.AdaptiveAvgPool2d((1, 1))\\n\",\n \" self.fc = nn.Linear(512 * block.expansion, num_classes)\\n\",\n \"\\n\",\n \" for m in self.modules():\\n\",\n \" if isinstance(m, nn.Conv2d):\\n\",\n \" nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')\\n\",\n \" elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):\\n\",\n \" nn.init.constant_(m.weight, 1)\\n\",\n \" nn.init.constant_(m.bias, 0)\\n\",\n \"\\n\",\n \" # Zero-initialize the last BN in each residual branch,\\n\",\n \" # so that the residual branch starts with zeros, and each residual block behaves like an identity.\\n\",\n \" # This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677\\n\",\n \" if zero_init_residual:\\n\",\n \" for m in self.modules():\\n\",\n \" if isinstance(m, Bottleneck):\\n\",\n \" nn.init.constant_(m.bn3.weight, 0)\\n\",\n \" elif isinstance(m, BasicBlock):\\n\",\n \" nn.init.constant_(m.bn2.weight, 0)\\n\",\n \"\\n\",\n \" def _make_layer(self, block, planes, blocks, stride=1, dilate=False):\\n\",\n \" norm_layer = self._norm_layer\\n\",\n \" downsample = None\\n\",\n \" previous_dilation = self.dilation\\n\",\n \" if dilate:\\n\",\n \" self.dilation *= stride\\n\",\n \" stride = 1\\n\",\n \" if stride != 1 or self.inplanes != planes * block.expansion:\\n\",\n \" downsample = nn.Sequential(\\n\",\n \" conv1x1(self.inplanes, planes * block.expansion, stride),\\n\",\n \" norm_layer(planes * block.expansion),\\n\",\n \" )\\n\",\n \"\\n\",\n \" layers = []\\n\",\n \" layers.append(block(self.inplanes, planes, stride, downsample, self.groups,\\n\",\n \" self.base_width, previous_dilation, norm_layer))\\n\",\n \" self.inplanes = planes * block.expansion\\n\",\n \" for _ in range(1, blocks):\\n\",\n \" layers.append(block(self.inplanes, planes, groups=self.groups,\\n\",\n \" base_width=self.base_width, dilation=self.dilation,\\n\",\n \" norm_layer=norm_layer))\\n\",\n \"\\n\",\n \" return nn.Sequential(*layers)\\n\",\n \"\\n\",\n \" def forward(self, x):\\n\",\n \" x = self.conv1(x)\\n\",\n \" x = self.bn1(x)\\n\",\n \" x = self.relu(x)\\n\",\n \" x = self.maxpool(x)\\n\",\n \"\\n\",\n \" x = self.layer1(x)\\n\",\n \" x = self.layer2(x)\\n\",\n \" x = self.layer3(x)\\n\",\n \" x = self.layer4(x)\\n\",\n \"\\n\",\n \" x = self.avgpool(x)\\n\",\n \" x = torch.flatten(x, 1)\\n\",\n \" x = self.fc(x)\\n\",\n \"\\n\",\n \" return x\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from torch.utils.model_zoo import load_url as load_state_dict_from_url\\n\",\n \"\\n\",\n \"model = ResNet(Bottleneck, [3, 4, 6, 3])\\n\",\n \"state_dict = load_state_dict_from_url('https://download.pytorch.org/models/resnet50-19c8e357.pth', progress=True)\\n\",\n \"model.load_state_dict(state_dict)\\n\",\n \"# you can do some training here, before calling model.eval()\\n\",\n \"model.eval()\\n\",\n \"print('ResNet50 model is turned into inference mode')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**STEP 2:** Download a cat image and preprocess it\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import numpy as np\\n\",\n \"from torchvision import transforms, datasets\\n\",\n \"from tensorflow.keras.applications import resnet50\\n\",\n \"import urllib.request\\n\",\n \"\\n\",\n \"imagedir = './images'\\n\",\n \"os.makedirs(imagedir, exist_ok=True)\\n\",\n \"urllib.request.urlretrieve(\\n\",\n \" 'https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg',\\n\",\n \" os.path.join(imagedir, 'kitten_small.jpg'),\\n\",\n \")\\n\",\n \"normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\\n\",\n \" std=[0.229, 0.224, 0.225])\\n\",\n \"eval_dataset = datasets.ImageFolder(\\n\",\n \" '.',\\n\",\n \" transforms.Compose([\\n\",\n \" transforms.Resize([224, 224]),\\n\",\n \" transforms.ToTensor(),\\n\",\n \" normalize,\\n\",\n \" ])\\n\",\n \")\\n\",\n \"image, label = eval_dataset[0]\\n\",\n \"image = torch.tensor(image.numpy()[np.newaxis, ...])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**STEP 3:** Run inference without neuron for comparison\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"print('model inference result:')\\n\",\n \"print(resnet50.decode_predictions(model(image).detach().numpy(), top=5)[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"STEP 4: Run the same inference using torch.jit.trace - then we can save and load the model\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"jit_trace = torch.jit.trace(model, example_inputs=image)\\n\",\n \"print('jit.trace inference result:')\\n\",\n \"print(resnet50.decode_predictions(jit_trace(image).detach().numpy(), top=5)[0])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"jit_trace_filename = 'resnet50_jit_trace.pt'\\n\",\n \"jit_trace.save(jit_trace_filename)\\n\",\n \"jit_trace_loaded = torch.jit.load(jit_trace_filename)\\n\",\n \"print('loaded jit.trace inferenced result:')\\n\",\n \"print(resnet50.decode_predictions(jit_trace_loaded(image).detach().numpy(), top=5)[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**STEP 4:** Manually partition the ResNet50 model and execute it\\n\",\n \"\\n\",\n \"To generate a Neuron-optimized TorchScript with only layers 1~4 placed on Neuron runtime, we first define a new module class `ResNetNeuron` inheriting from `ResNet`. We add 'torch.neuron.trace' calls in the forward function of this module in order to turn layer submodules into Neuron-optimized ones.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch.neuron\\n\",\n \"\\n\",\n \"class ResNetNeuron(ResNet):\\n\",\n \"\\n\",\n \" def trace(self, x):\\n\",\n \" x = self.conv1(x)\\n\",\n \" x = self.bn1(x)\\n\",\n \" x = self.relu(x)\\n\",\n \" x = self.maxpool(x)\\n\",\n \"\\n\",\n \" self.layer1 = torch.neuron.trace(self.layer1, x, fallback=False)\\n\",\n \" x = self.layer1(x)\\n\",\n \" \\n\",\n \" self.layer2 = torch.neuron.trace(self.layer2, x, fallback=False)\\n\",\n \" x = self.layer2(x)\\n\",\n \" \\n\",\n \" self.layer3 = torch.neuron.trace(self.layer3, x, fallback=False)\\n\",\n \" x = self.layer3(x)\\n\",\n \" \\n\",\n \" self.layer4 = torch.neuron.trace(self.layer4, x, fallback=False)\\n\",\n \" \\n\",\n \" def forward(self, x):\\n\",\n \" x = self.conv1(x)\\n\",\n \" x = self.bn1(x)\\n\",\n \" x = self.relu(x)\\n\",\n \" x = self.maxpool(x)\\n\",\n \" \\n\",\n \" # After running ResNetNeuron::trace, these layers will be placed on Neuron\\n\",\n \" x = self.layer1(x)\\n\",\n \" x = self.layer2(x)\\n\",\n \" x = self.layer3(x)\\n\",\n \" x = self.layer4(x)\\n\",\n \" \\n\",\n \" x = self.avgpool(x)\\n\",\n \" x = torch.flatten(x, 1)\\n\",\n \" x = self.fc(x)\\n\",\n \"\\n\",\n \" return x\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now construct the class and runn an inference to trigger the `neuron-cc` compiler. Watch for the [ \\\\* ] icon to the left of this cell to disappear and show a number - this will take a minute or two to run\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"model_neuron = ResNetNeuron(Bottleneck, [3, 4, 6, 3])\\n\",\n \"model_neuron.load_state_dict(state_dict)\\n\",\n \"model_neuron.eval()\\n\",\n \"model_neuron.trace(image) # this line triggers neuron-cc compiler\\n\",\n \"result = model_neuron(image)\\n\",\n \"print('Neuron optimized model inference result:')\\n\",\n \"print(resnet50.decode_predictions(result.detach().numpy(), top=5)[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**STEP 5:** Save the model as TorchScript ready to deploy\\n\",\n \"\\n\",\n \"To deploy the Neuron-optimized as TorchScript, we use `torch.jit.trace` again to generate TorchScript for the entire mode, including the Neuron-optimized `ScriptModule`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"neuron_trace = torch.jit.trace(model_neuron, example_inputs=image)\\n\",\n \"print('neuron.trace inference result:')\\n\",\n \"print(resnet50.decode_predictions(neuron_trace(image).detach().numpy(), top=5)[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"This Neuron-optimized `ScriptModule` can be saved/loaded easily and be deployed on inf1 instances.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"neuron_trace_filename = 'resnet50_neuron_trace.pt'\\n\",\n \"neuron_trace.save(neuron_trace_filename)\\n\",\n \"neuron_trace_loaded = torch.jit.load(neuron_trace_filename)\\n\",\n \"print('loaded neuron.trace inference result:')\\n\",\n \"print(resnet50.decode_predictions(neuron_trace_loaded(image).detach().numpy(), top=5)[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**STEP 6:** Understanding the neuron graph\\n\",\n \"\\n\",\n \"We can inspect the graph property of the Neuron-optimized `ScriptModule` to get an idea of how Neuron-optimization is performed. Each `torch.neuron.trace` call fuses a submodule (layer) into a `neuron::forward`/`NeuronModule` operator.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"neuron_trace.graph\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.2\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n" }, { "path": "src/examples/pytorch/torch-neuronx/bert-base-cased-finetuned-mrpc-inference-on-trn1-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"e11b2ce1\",\n \"metadata\": {},\n \"source\": [\n \"# Compiling and Deploying HuggingFace Pretrained BERT on Trn1 or Inf2\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"59a44364\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy a HuggingFace 🤗 Transformers BERT model for accelerated inference on Neuron. In this tutorial, we will be deploying directly on Trn1/Inf2 instances. If you are looking to deploy this model through SageMaker on Inf2 instance, please visit the [Sagemaker samples repository](https://github.com/aws-neuron/aws-neuron-sagemaker-samples/tree/master/inference/inf2-bert-on-sagemaker). \\n\",\n \"\\n\",\n \"This tutorial will use the [bert-base-cased-finetuned-mrpc](https://huggingface.co/bert-base-cased-finetuned-mrpc) model. This model has 12 layers, 768 hidden dimensions, 12 attention heads, and 110M total parameters. The final layer is a binary classification head that has been trained on the Microsoft Research Paraphrase Corpus (`mrpc`). The input to the model is two sentences and the output of the model is whether or not those sentences are a paraphrase of each other. \\n\",\n \"\\n\",\n \"This tutorial has the following main sections:\\n\",\n \"\\n\",\n \"1. Install dependencies\\n\",\n \"1. Compile the BERT model\\n\",\n \"1. Run inference on Neuron and compare results to CPU\\n\",\n \"1. Benchmark the model using multicore inference\\n\",\n \"1. Finding the optimal batch size\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on a Trn1 instance (`trn1.2xlarge` or larger.) or Inf2 instance (`inf2.xlarge` or larger.)\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"9ceecb92\",\n \"metadata\": {},\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"The code in this tutorial is written for Jupyter Notebooks. To use Jupyter Notebook on the Neuron instance, you\\n\",\n \"can use this [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html).\\n\",\n \"\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuronx`\\n\",\n \"- `neuronx-cc`\\n\",\n \"- `transformers`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Trn1/Inf2 setup guide. The additional dependencies must be installed here:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"66392b0b\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"%env HF_HUB_DISABLE_PROGRESS_BARS=1 # Avoids xet progress bar model download error\\n\",\n \"!pip install --upgrade transformers\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"82533d8e\",\n \"metadata\": {},\n \"source\": [\n \"## Compile the model into an AWS Neuron optimized TorchScript\\n\",\n \"\\n\",\n \"In the following section, we load the BERT model and tokenizer, get a sample input, run inference on CPU, compile the model for Neuron using `torch_neuronx.trace()`, and save the optimized model as `TorchScript`.\\n\",\n \"\\n\",\n \"`torch_neuronx.trace()` expects a tensor or tuple of tensor inputs to use for tracing, so we unpack the tokenizer output using the `encode` function. \\n\",\n \"\\n\",\n \"The result of the trace stage will be a static executable where the operations to be run upon inference are determined during compilation. This means that when inferring, the resulting Neuron model must be executed with tensors that are the exact same shape as those provided at compilation time. If a model is given a tensor at inference time whose shape does not match the tensor given at compilation time, an error will occur.\\n\",\n \"\\n\",\n \"For language models, the shape of the tokenizer tensors can vary based on the length of input sentence. We can satisfy the Neuron restriction of using a fixed shape input by padding all varying input tensors to a specified length. In a deployment scenario, the padding size should be chosen based on the maximum token length that is expected to occur for the application.\\n\",\n \"\\n\",\n \"In the following section we will assume that we will receive a maximum of 128 tokens at inference time. We will pad our example inputs by using `padding='max_length'` and to avoid potential errors caused by creating a tensor that is larger than `max_length=128`, we will always tokenize using `truncation=True`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"0c9aac5e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuronx\\n\",\n \"from transformers import AutoTokenizer, AutoModelForSequenceClassification\\n\",\n \"import transformers\\n\",\n \"\\n\",\n \"\\n\",\n \"def encode(tokenizer, *inputs, max_length=128, batch_size=1):\\n\",\n \" tokens = tokenizer.encode_plus(\\n\",\n \" *inputs,\\n\",\n \" max_length=max_length,\\n\",\n \" padding='max_length',\\n\",\n \" truncation=True,\\n\",\n \" return_tensors=\\\"pt\\\"\\n\",\n \" )\\n\",\n \" return (\\n\",\n \" torch.repeat_interleave(tokens['input_ids'], batch_size, 0),\\n\",\n \" torch.repeat_interleave(tokens['attention_mask'], batch_size, 0),\\n\",\n \" torch.repeat_interleave(tokens['token_type_ids'], batch_size, 0),\\n\",\n \" )\\n\",\n \"\\n\",\n \"\\n\",\n \"# Create the tokenizer and model\\n\",\n \"name = \\\"bert-base-cased-finetuned-mrpc\\\"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(name)\\n\",\n \"model = AutoModelForSequenceClassification.from_pretrained(name, torchscript=True)\\n\",\n \"\\n\",\n \"# Set up some example inputs\\n\",\n \"sequence_0 = \\\"The company HuggingFace is based in New York City\\\"\\n\",\n \"sequence_1 = \\\"Apples are especially bad for your health\\\"\\n\",\n \"sequence_2 = \\\"HuggingFace's headquarters are situated in Manhattan\\\"\\n\",\n \"\\n\",\n \"paraphrase = encode(tokenizer, sequence_0, sequence_2)\\n\",\n \"not_paraphrase = encode(tokenizer, sequence_0, sequence_1)\\n\",\n \"\\n\",\n \"# Run the original PyTorch BERT model on CPU\\n\",\n \"cpu_paraphrase_logits = model(*paraphrase)[0]\\n\",\n \"cpu_not_paraphrase_logits = model(*not_paraphrase)[0]\\n\",\n \"\\n\",\n \"# Compile the model for Neuron\\n\",\n \"model_neuron = torch_neuronx.trace(model, paraphrase)\\n\",\n \"\\n\",\n \"# Save the TorchScript for inference deployment\\n\",\n \"filename = 'model.pt'\\n\",\n \"torch.jit.save(model_neuron, filename)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"53e9605d\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference and compare results\\n\",\n \"\\n\",\n \"In this section we load the compiled model, run inference on Neuron, and compare the CPU and Neuron outputs.\\n\",\n \"\\n\",\n \"NOTE: Although this tutorial section uses one NeuronCore (and the next section uses two NeuronCores), by default each Jupyter notebook Python process will attempt to take ownership of all NeuronCores visible on the instance. For multi-process applications where each process should only use a subset of the NeuronCores on the instance you can use NEURON_RT_NUM_CORES=N or NEURON_RT_VISIBLE_CORES=< list of NeuronCore IDs > when starting the Jupyter notebook as described in [NeuronCore Allocation and Model Placement for Inference](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/inference/core-placement.html).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"a8d509aa\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Load the TorchScript compiled model\\n\",\n \"model_neuron = torch.jit.load(filename)\\n\",\n \"\\n\",\n \"# Verify the TorchScript works on both example inputs\\n\",\n \"neuron_paraphrase_logits = model_neuron(*paraphrase)[0]\\n\",\n \"neuron_not_paraphrase_logits = model_neuron(*not_paraphrase)[0]\\n\",\n \"\\n\",\n \"# Compare the results\\n\",\n \"print('CPU paraphrase logits: ', cpu_paraphrase_logits.detach().numpy())\\n\",\n \"print('Neuron paraphrase logits: ', neuron_paraphrase_logits.detach().numpy())\\n\",\n \"print('CPU not-paraphrase logits: ', cpu_not_paraphrase_logits.detach().numpy())\\n\",\n \"print('Neuron not-paraphrase logits: ', neuron_not_paraphrase_logits.detach().numpy())\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"a4553cc9\",\n \"metadata\": {},\n \"source\": [\n \"## Benchmarking\\n\",\n \"\\n\",\n \"In this section we benchmark the performance of the BERT model on Neuron. By default, models compiled with `torch_neuronx` will always execute on a *single* NeuronCore. When loading *multiple* models, the default behavior of the Neuron runtime is to evenly distribute models across all available NeuronCores. The runtime places models on the NeuronCore that has the fewest models loaded to it first. In the following section, we will `torch.jit.load` multiple instances of the model which should each be loaded onto their own NeuronCore. It is not useful to load more copies of a model than the number of NeuronCores on the instance since an individual NeuronCore can only execute one model at a time.\\n\",\n \"\\n\",\n \"To ensure that we are maximizing hardware utilization, we must run inferences using multiple threads in parallel. It is nearly always recommended to use some form of threading/multiprocessing and some form of model replication since even the smallest Neuron EC2 instance has 2 NeuronCores available. Applications with no form of threading are only capable of `1 / num_neuron_cores` hardware utilization which becomes especially problematic on large instances.\\n\",\n \"\\n\",\n \"One way to view the hardware utilization is by executing the `neuron-top` application in the terminal while the benchmark is executing. If the monitor shows >90% utilization on all NeuronCores, this is a good indication that the hardware is being utilized effectively.\\n\",\n \"\\n\",\n \"In this example we load two models, which utilizes all NeuronCores (2) on a `trn1.2xlarge` or `inf2.xlarge` instance. Additional models can be loaded and run in parallel on larger Trn1 or Inf2 instance sizes to increase throughput.\\n\",\n \"\\n\",\n \"We define a benchmarking function that loads two optimized BERT models onto two separate NeuronCores, runs multithreaded inference, and calculates the corresponding latency and throughput.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"c9e14b0d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import time\\n\",\n \"import concurrent.futures\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"\\n\",\n \"def benchmark(filename, example, n_models=2, n_threads=2, batches_per_thread=1000):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Record performance statistics for a serialized model and its input example.\\n\",\n \"\\n\",\n \" Arguments:\\n\",\n \" filename: The serialized torchscript model to load for benchmarking.\\n\",\n \" example: An example model input.\\n\",\n \" n_models: The number of models to load.\\n\",\n \" n_threads: The number of simultaneous threads to execute inferences on.\\n\",\n \" batches_per_thread: The number of example batches to run per thread.\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" A dictionary of performance statistics.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" # Load models\\n\",\n \" models = [torch.jit.load(filename) for _ in range(n_models)]\\n\",\n \"\\n\",\n \" # Warmup\\n\",\n \" for _ in range(8):\\n\",\n \" for model in models:\\n\",\n \" model(*example)\\n\",\n \"\\n\",\n \" latencies = []\\n\",\n \"\\n\",\n \" # Thread task\\n\",\n \" def task(model):\\n\",\n \" for _ in range(batches_per_thread):\\n\",\n \" start = time.time()\\n\",\n \" model(*example)\\n\",\n \" finish = time.time()\\n\",\n \" latencies.append((finish - start) * 1000)\\n\",\n \"\\n\",\n \" # Submit tasks\\n\",\n \" begin = time.time()\\n\",\n \" with concurrent.futures.ThreadPoolExecutor(max_workers=n_threads) as pool:\\n\",\n \" for i in range(n_threads):\\n\",\n \" pool.submit(task, models[i % len(models)])\\n\",\n \" end = time.time()\\n\",\n \"\\n\",\n \" # Compute metrics\\n\",\n \" boundaries = [50, 95, 99]\\n\",\n \" percentiles = {}\\n\",\n \"\\n\",\n \" for boundary in boundaries:\\n\",\n \" name = f'latency_p{boundary}'\\n\",\n \" percentiles[name] = np.percentile(latencies, boundary)\\n\",\n \" duration = end - begin\\n\",\n \" batch_size = 0\\n\",\n \" for tensor in example:\\n\",\n \" if batch_size == 0:\\n\",\n \" batch_size = tensor.shape[0]\\n\",\n \" inferences = len(latencies) * batch_size\\n\",\n \" throughput = inferences / duration\\n\",\n \"\\n\",\n \" # Metrics\\n\",\n \" metrics = {\\n\",\n \" 'filename': str(filename),\\n\",\n \" 'batch_size': batch_size,\\n\",\n \" 'batches': len(latencies),\\n\",\n \" 'inferences': inferences,\\n\",\n \" 'threads': n_threads,\\n\",\n \" 'models': n_models,\\n\",\n \" 'duration': duration,\\n\",\n \" 'throughput': throughput,\\n\",\n \" **percentiles,\\n\",\n \" }\\n\",\n \"\\n\",\n \" display(metrics)\\n\",\n \"\\n\",\n \"\\n\",\n \"def display(metrics):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Display the metrics produced by `benchmark` function.\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" metrics: A dictionary of performance statistics.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" pad = max(map(len, metrics)) + 1\\n\",\n \" for key, value in metrics.items():\\n\",\n \"\\n\",\n \" parts = key.split('_')\\n\",\n \" parts = list(map(str.title, parts))\\n\",\n \" title = ' '.join(parts) + \\\":\\\"\\n\",\n \"\\n\",\n \" if isinstance(value, float):\\n\",\n \" value = f'{value:0.3f}'\\n\",\n \"\\n\",\n \" print(f'{title :<{pad}} {value}')\\n\",\n \"\\n\",\n \"\\n\",\n \"# Benchmark BERT on Neuron\\n\",\n \"benchmark(filename, paraphrase)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"fc374b12\",\n \"metadata\": {},\n \"source\": [\n \"## Finding the optimal batch size\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"113acb55\",\n \"metadata\": {},\n \"source\": [\n \"Batch size has a direct impact on model performance. The NeuronCore architecture is optimized to maximize throughput with relatively small batch sizes. This means that a Neuron compiled model can outperform a GPU model, even if running single digit batch sizes.\\n\",\n \"\\n\",\n \"As a general best practice, we recommend optimizing your model’s throughput by compiling the model with a small batch size and gradually increasing it to find the peak throughput on Neuron. To minimize latency, using `batch size = 1` will nearly always be optimal. This batch size configuration is typically used for on-demand inference applications. To maximize throughput, *usually* `1 < batch_size < 10` is optimal. A configuration which uses a larger batch size is generally ideal for batched on-demand inference or offline batch processing.\\n\",\n \"\\n\",\n \"In the following section, we compile BERT for multiple batch size inputs. We then run inference on each batch size and benchmark the performance. Notice that latency increases consistently as the batch size increases. Throughput increases as well, up until a certain point where the input size becomes too large to be efficient.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"be26aafc\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Compile BERT for different batch sizes\\n\",\n \"for batch_size in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:\\n\",\n \" tokenizer = AutoTokenizer.from_pretrained(name)\\n\",\n \" model = AutoModelForSequenceClassification.from_pretrained(name, torchscript=True)\\n\",\n \" example = encode(tokenizer, sequence_0, sequence_2, batch_size=batch_size)\\n\",\n \" model_neuron = torch_neuronx.trace(model, example)\\n\",\n \" filename = f'model_batch_size_{batch_size}.pt'\\n\",\n \" torch.jit.save(model_neuron, filename)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"8f0f6ed2\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Benchmark BERT for different batch sizes\\n\",\n \"for batch_size in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:\\n\",\n \" print('-'*50)\\n\",\n \" example = encode(tokenizer, sequence_0, sequence_2, batch_size=batch_size)\\n\",\n \" filename = f'model_batch_size_{batch_size}.pt'\\n\",\n \" benchmark(filename, example)\\n\",\n \" print()\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python (Neuron PyTorch)\",\n \"language\": \"python\",\n \"name\": \"pytorch_venv\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.7.16\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/pytorch/torch-neuronx/resnet50-inference-on-trn1-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"6a30ffd9\",\n \"metadata\": {},\n \"source\": [\n \"# Compiling and Deploying ResNet50 on Trn1 or Inf2\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"ea682fbe\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy a TorchVision ResNet50 model for accelerated inference on Neuron. To get started with\\n\",\n \"Jupyter Notebook on Neuron Instance you launched, please use this [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html).\\n\",\n \"\\n\",\n \"This tutorial will use the [resnet50](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html) model, which is primarily used for arbitrary image classification tasks.\\n\",\n \"\\n\",\n \"This tutorial has the following main sections:\\n\",\n \"\\n\",\n \"1. Install dependencies\\n\",\n \"1. Compile the ResNet model\\n\",\n \"1. Run inference on Neuron and compare results to CPU\\n\",\n \"1. Benchmark the model using multicore inference\\n\",\n \"1. Finding the optimal batch size\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on a Trn1 instance (`trn1.2xlarge` or larger.) or Inf2 instance (`inf2.xlarge` or larger.)\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"5f60760a\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies\\n\",\n \"The code in this tutorial is written for Jupyter Notebooks. To use Jupyter Notebook on the Neuron instance, you\\n\",\n \"can use this [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html).\\n\",\n \"\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuronx`\\n\",\n \"- `neuronx-cc`\\n\",\n \"- `torchvision`\\n\",\n \"- `Pillow`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Trn1 setup guide. The additional dependencies must be installed here:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"c44c5df5\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env HF_HUB_DISABLE_PROGRESS_BARS=1 # Avoids xet progress bar model download error\\n\",\n \"!pip install Pillow\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"de2efba5\",\n \"metadata\": {},\n \"source\": [\n \"## Compile the model into an AWS Neuron optimized TorchScript\\n\",\n \"\\n\",\n \"In the following section, we load the model, get a sample input, run inference on CPU, compile the model for Neuron using `torch_neuronx.trace()`, and save the optimized model as `TorchScript`.\\n\",\n \"\\n\",\n \"`torch_neuronx.trace()` expects a tensor or tuple of tensor inputs to use for tracing, so we convert the input image into a tensor using the `get_image` function.\\n\",\n \"\\n\",\n \"The result of the trace stage will be a static executable where the operations to be run upon inference are determined during compilation. This means that when inferring, the resulting Neuron model must be executed with tensors that are the exact same shape as those provided at compilation time. If a model is given a tensor at inference time whose shape does not match the tensor given at compilation time, an error will occur. \\n\",\n \"\\n\",\n \"In the following section, we assume that we will receive an image shape of `[1, 3, 224, 224]` at inference time.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"1650de1f\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import urllib\\n\",\n \"from PIL import Image\\n\",\n \"\\n\",\n \"import torch\\n\",\n \"import torch_neuronx\\n\",\n \"from torchvision import models\\n\",\n \"from torchvision.transforms import functional\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_image(batch_size=1, image_shape=(224, 224)):\\n\",\n \" # Get an example input\\n\",\n \" filename = \\\"000000039769.jpg\\\"\\n\",\n \" if not os.path.exists(filename):\\n\",\n \" url = \\\"http://images.cocodataset.org/val2017/000000039769.jpg\\\"\\n\",\n \" urllib.request.urlretrieve(url, filename)\\n\",\n \" image = Image.open(filename).convert('RGB')\\n\",\n \" image = functional.resize(image, (image_shape))\\n\",\n \" image = functional.to_tensor(image)\\n\",\n \" image = torch.unsqueeze(image, 0)\\n\",\n \" image = torch.repeat_interleave(image, batch_size, 0)\\n\",\n \" return (image, )\\n\",\n \"\\n\",\n \"\\n\",\n \"# Create the model\\n\",\n \"model = models.resnet50(pretrained=True)\\n\",\n \"model.eval()\\n\",\n \"\\n\",\n \"# Get an example input\\n\",\n \"image = get_image()\\n\",\n \"\\n\",\n \"# Run inference on CPU\\n\",\n \"output_cpu = model(*image)\\n\",\n \"\\n\",\n \"# Compile the model\\n\",\n \"model_neuron = torch_neuronx.trace(model, image)\\n\",\n \"\\n\",\n \"# Save the TorchScript for inference deployment\\n\",\n \"filename = 'model.pt'\\n\",\n \"torch.jit.save(model_neuron, filename)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"25f453f8\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference and compare results\\n\",\n \"\\n\",\n \"In this section we load the compiled model, run inference on Neuron, and compare the CPU and Neuron outputs using the ImageNet classes.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"b4a203aa\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"\\n\",\n \"# Load the TorchScript compiled model\\n\",\n \"model_neuron = torch.jit.load(filename)\\n\",\n \"\\n\",\n \"# Run inference using the Neuron model\\n\",\n \"output_neuron = model_neuron(*image)\\n\",\n \"\\n\",\n \"# Compare the results\\n\",\n \"print(f\\\"CPU tensor: {output_cpu[0][0:10]}\\\")\\n\",\n \"print(f\\\"Neuron tensor: {output_neuron[0][0:10]}\\\")\\n\",\n \"\\n\",\n \"# Download and read the ImageNet classes\\n\",\n \"urllib.request.urlretrieve(\\\"https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json\\\",\\\"imagenet_class_index.json\\\")\\n\",\n \"with open(\\\"imagenet_class_index.json\\\", \\\"r\\\") as file:\\n\",\n \" class_id = json.load(file)\\n\",\n \" id2label = [class_id[str(i)][1] for i in range(len(class_id))]\\n\",\n \"\\n\",\n \"# Lookup and print the top-5 labels\\n\",\n \"top5_cpu = output_cpu[0].sort()[1][-5:]\\n\",\n \"top5_neuron = output_neuron[0].sort()[1][-5:]\\n\",\n \"top5_labels_cpu = [id2label[idx] for idx in top5_cpu]\\n\",\n \"top5_labels_neuron = [id2label[idx] for idx in top5_neuron]\\n\",\n \"print(f\\\"CPU top-5 labels: {top5_labels_cpu}\\\")\\n\",\n \"print(f\\\"Neuron top-5 labels: {top5_labels_neuron}\\\")\"\n ]\n },\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"c96389ae\",\n \"metadata\": {},\n \"source\": [\n \"## Benchmarking\\n\",\n \"\\n\",\n \"In this section we benchmark the performance of the ResNet model on Neuron. By default, models compiled with `torch_neuronx` will always execute on a *single* NeuronCore. When loading *multiple* models, the default behavior of the Neuron runtime is to evenly distribute models across all available NeuronCores. The runtime places models on the NeuronCore that has the fewest models loaded to it first. In the following section, we will `torch.jit.load` multiple instances of the model which should each be loaded onto their own NeuronCore. It is not useful to load more copies of a model than the number of NeuronCores on the instance since an individual NeuronCore can only execute one model at a time.\\n\",\n \"\\n\",\n \"To ensure that we are maximizing hardware utilization, we must run inferences using multiple threads in parallel. It is nearly always recommended to use some form of threading/multiprocessing and some form of model replication since even the smallest Neuron EC2 instance has 2 NeuronCores available. Applications with no form of threading are only capable of `1 / num_neuron_cores` hardware utilization which becomes especially problematic on large instances.\\n\",\n \"\\n\",\n \"One way to view the hardware utilization is by executing the `neuron-top` application in the terminal while the benchmark is executing. If the monitor shows >90% utilization on all NeuronCores, this is a good indication that the hardware is being utilized effectively.\\n\",\n \"\\n\",\n \"In this example we load two models, which utilizes all NeuronCores (2) on a `trn1.2xlarge` or `inf2.xlarge` instance. Additional models can be loaded and run in parallel on larger Trn1 or Inf2 instance sizes to increase throughput.\\n\",\n \"\\n\",\n \"We define a benchmarking function that loads two optimized ResNet models onto two separate NeuronCores, runs multithreaded inference, and calculates the corresponding latency and throughput.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"9657ae4f\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import time\\n\",\n \"import concurrent.futures\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"\\n\",\n \"def benchmark(filename, example, n_models=2, n_threads=2, batches_per_thread=1000):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Record performance statistics for a serialized model and its input example.\\n\",\n \"\\n\",\n \" Arguments:\\n\",\n \" filename: The serialized torchscript model to load for benchmarking.\\n\",\n \" example: An example model input.\\n\",\n \" n_models: The number of models to load.\\n\",\n \" n_threads: The number of simultaneous threads to execute inferences on.\\n\",\n \" batches_per_thread: The number of example batches to run per thread.\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" A dictionary of performance statistics.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" # Load models\\n\",\n \" models = [torch.jit.load(filename) for _ in range(n_models)]\\n\",\n \"\\n\",\n \" # Warmup\\n\",\n \" for _ in range(8):\\n\",\n \" for model in models:\\n\",\n \" model(*example)\\n\",\n \"\\n\",\n \" latencies = []\\n\",\n \"\\n\",\n \" # Thread task\\n\",\n \" def task(model):\\n\",\n \" for _ in range(batches_per_thread):\\n\",\n \" start = time.time()\\n\",\n \" model(*example)\\n\",\n \" finish = time.time()\\n\",\n \" latencies.append((finish - start) * 1000)\\n\",\n \"\\n\",\n \" # Submit tasks\\n\",\n \" begin = time.time()\\n\",\n \" with concurrent.futures.ThreadPoolExecutor(max_workers=n_threads) as pool:\\n\",\n \" for i in range(n_threads):\\n\",\n \" pool.submit(task, models[i % len(models)])\\n\",\n \" end = time.time()\\n\",\n \"\\n\",\n \" # Compute metrics\\n\",\n \" boundaries = [50, 95, 99]\\n\",\n \" percentiles = {}\\n\",\n \"\\n\",\n \" for boundary in boundaries:\\n\",\n \" name = f'latency_p{boundary}'\\n\",\n \" percentiles[name] = np.percentile(latencies, boundary)\\n\",\n \" duration = end - begin\\n\",\n \" batch_size = 0\\n\",\n \" for tensor in example:\\n\",\n \" if batch_size == 0:\\n\",\n \" batch_size = tensor.shape[0]\\n\",\n \" inferences = len(latencies) * batch_size\\n\",\n \" throughput = inferences / duration\\n\",\n \"\\n\",\n \" # Metrics\\n\",\n \" metrics = {\\n\",\n \" 'filename': str(filename),\\n\",\n \" 'batch_size': batch_size,\\n\",\n \" 'batches': len(latencies),\\n\",\n \" 'inferences': inferences,\\n\",\n \" 'threads': n_threads,\\n\",\n \" 'models': n_models,\\n\",\n \" 'duration': duration,\\n\",\n \" 'throughput': throughput,\\n\",\n \" **percentiles,\\n\",\n \" }\\n\",\n \"\\n\",\n \" display(metrics)\\n\",\n \"\\n\",\n \"\\n\",\n \"def display(metrics):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Display the metrics produced by `benchmark` function.\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" metrics: A dictionary of performance statistics.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" pad = max(map(len, metrics)) + 1\\n\",\n \" for key, value in metrics.items():\\n\",\n \"\\n\",\n \" parts = key.split('_')\\n\",\n \" parts = list(map(str.title, parts))\\n\",\n \" title = ' '.join(parts) + \\\":\\\"\\n\",\n \"\\n\",\n \" if isinstance(value, float):\\n\",\n \" value = f'{value:0.3f}'\\n\",\n \"\\n\",\n \" print(f'{title :<{pad}} {value}')\\n\",\n \"\\n\",\n \"\\n\",\n \"# Benchmark ResNet on Neuron\\n\",\n \"benchmark(filename, image)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"795d2fca\",\n \"metadata\": {},\n \"source\": [\n \"## Finding the optimal batch size\\n\",\n \"\\n\",\n \"Batch size has a direct impact on model performance. The NeuronCore architecture is optimized to maximize throughput with relatively small batch sizes. This means that a Neuron compiled model can outperform a GPU model, even if running single digit batch sizes.\\n\",\n \"\\n\",\n \"As a general best practice, we recommend optimizing your model’s throughput by compiling the model with a small batch size and gradually increasing it to find the peak throughput on Neuron. To minimize latency, using `batch size = 1` will nearly always be optimal. This batch size configuration is typically used for on-demand inference applications. To maximize throughput, *usually* `1 < batch_size < 10` is optimal. A configuration which uses a larger batch size is generally ideal for batched on-demand inference or offline batch processing.\\n\",\n \"\\n\",\n \"In the following section, we compile ResNet for multiple batch size inputs. We then run inference on each batch size and benchmark the performance. Notice that latency increases consistently as the batch size increases. Throughput increases as well, up until a certain point where the input size becomes too large to be efficient.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"fdef1805\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Compile ResNet for different batch sizes\\n\",\n \"for batch_size in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:\\n\",\n \" model = models.resnet50(pretrained=True)\\n\",\n \" model.eval()\\n\",\n \" example = get_image(batch_size=batch_size)\\n\",\n \" model_neuron = torch_neuronx.trace(model, example)\\n\",\n \" filename = f'model_batch_size_{batch_size}.pt'\\n\",\n \" torch.jit.save(model_neuron, filename)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ec244d4e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Benchmark ResNet for different batch sizes\\n\",\n \"for batch_size in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:\\n\",\n \" print('-'*50)\\n\",\n \" example = get_image(batch_size=batch_size)\\n\",\n \" filename = f'model_batch_size_{batch_size}.pt'\\n\",\n \" benchmark(filename, example)\\n\",\n \" print()\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python (Neuron PyTorch)\",\n \"language\": \"python\",\n \"name\": \"pytorch_venv\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.7.16\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/pytorch/torch-neuronx/t5-inference-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# T5 model inference on Trn1 or Inf2\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy a pretrained T5 model for accelerated inference on Neuron. \\n\",\n \"\\n\",\n \"This tutorial will use the [t5-large](https://huggingface.co/t5-large) model. The T5 model can be used for machine translation, document summarization, question answering, and classification tasks. \\n\",\n \"\\n\",\n \"This tutorial has the following main sections:\\n\",\n \"\\n\",\n \"1. Install dependencies\\n\",\n \"1. Compile the T5 model\\n\",\n \"1. Run inference with greedy decoding on Neuron\\n\",\n \"1. Run infernece with beam search on Neuron\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on a Trn1 instance (`trn1.2xlarge` or larger.) or Inf2 instance (`inf2.xlarge` or larger.)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"The code in this tutorial is written for Jupyter Notebooks. To use Jupyter Notebook on the Neuron instance, you\\n\",\n \"can use this [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html).\\n\",\n \"\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuronx`\\n\",\n \"- `neuronx-cc`\\n\",\n \"- `transformers`\\n\",\n \"- `optimum-neuron`\\n\",\n \"\\n\",\n \"Most of these packages will be installed when configuring your environment using the Trn1/Inf2 setup guide. The additional dependencies must be installed here:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env HF_HUB_DISABLE_PROGRESS_BARS=1 # Avoids xet progress bar model download error\\n\",\n \"!pip install --upgrade transformers==4.31.0 optimum-neuron==0.0.8 sentencepiece\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"🤗 Optimum Neuron is the interface between the 🤗 Transformers library and AWS Accelerators including AWS Trainium and AWS Inferentia. It provides a set of tools enabling easy model loading, training and inference on single- and multi-Accelerator settings for different downstream tasks. In this tutorial we use 🤗 HuggingFace Optimum Neuron's generate() method instead of 🤗 [transformers's generate()](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.generate) to perform greedy decoding. Optimum Neuron takes care of padding the inputs which is necessary to infer on Neuron.\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Compile the model into an AWS Neuron optimized TorchScript\\n\",\n \"\\n\",\n \"In the following section, we load the T5 model, compile the model's encoder and decoder for Neuron using `torch_neuronx.trace()`, and save the optimized encoder and decoder as `TorchScript`. \\n\",\n \"\\n\",\n \"Before we trace the model, we need to make a couple of changes. \\n\",\n \"\\n\",\n \"1. We need to write encoder and decoder wrappers - `torch_neuronx` can only trace functions with positional arguments. But the T5 encoder and decoder both use keyword arguments. So, in order to trace them, we have to write wrappers that convert keyword arguments to positional arguments \\n\",\n \"2. We modify the t5 code to maximize the computation on the neuron device - Having sections of code running on cpu will reduce the performance. Moreover, we do not want to move data berween the neuron device and cpu during inference. The code we trace with `torch_neuronx` is the code that runs on the neuron device, so we refactor the t5 code to run computationally heavy operations within the wrapper. \\n\",\n \"\\n\",\n \"Let us start with the EncoderWrapper. \\n\",\n \"\\n\",\n \"In the huggingface t5 implementation, the encoder block takes in the input ids and returns the encoder hidden states. This hidden states are then used to initialize the KV cache in the decoder blocks during the first decoder invocation. We could trace both the encoder and the cache initialization step separately. But there is a better way, we could just compute the initial KV cache state within the encoder wrapper. This way, we remove the overhead of moving the hidden states from neuron device to cpu and back. This also allows neuron's compiler to optimize execution across both the encoder and cache initialization. \\n\",\n \"\\n\",\n \"*Why don't we just initalize the cache on the first decoder run?* \\n\",\n \"\\n\",\n \"This is harder to do on Neuron. Similar to `torch.jit.trace()`, `torch_neuronx.trace()` produces a function that has a fixed control flow, i.e. there are no conditional executions. So we cannot choose to conditionally initialize the cache in the first decoder iteration. Instead, we can compute the initial cache state outside the generation flow and pass the cache to it. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"\\n\",\n \"from transformers.models.t5.modeling_t5 import T5Stack, T5LayerCrossAttention\\n\",\n \"\\n\",\n \"class EncoderWrapper(torch.nn.Module):\\n\",\n \" '''\\n\",\n \" We will trace an instance of the EncoderWrapper. \\n\",\n \" This wrapper just converts positional args to kwargs. \\n\",\n \" '''\\n\",\n \"\\n\",\n \" def __init__(self, \\n\",\n \" encoder,\\n\",\n \" decoder, \\n\",\n \" model_config, \\n\",\n \" batch_size, \\n\",\n \" max_length, \\n\",\n \" device, \\n\",\n \" num_beams,\\n\",\n \" tp_degree=None):\\n\",\n \" \\n\",\n \" super().__init__()\\n\",\n \" self.encoder = encoder\\n\",\n \" self.decoder = decoder\\n\",\n \" self.batch_size = batch_size\\n\",\n \" self.max_length = max_length\\n\",\n \" self.model_config = model_config\\n\",\n \" self.device = device\\n\",\n \" self.num_beams = num_beams\\n\",\n \" self.num_attention_heads_per_partition = model_config.num_heads\\n\",\n \" self.tp_degree = tp_degree\\n\",\n \"\\n\",\n \" def forward(self, input_ids, attention_mask):\\n\",\n \" '''\\n\",\n \" This is the core functionality we want to trace. \\n\",\n \" '''\\n\",\n \" encoder_output = self.encoder(input_ids=input_ids,\\n\",\n \" attention_mask=attention_mask,\\n\",\n \" output_attentions=False,\\n\",\n \" output_hidden_states=False)\\n\",\n \"\\n\",\n \" last_hidden_state = encoder_output[\\\"last_hidden_state\\\"]\\n\",\n \" encoder_hidden_states = torch.concat([tensor.unsqueeze(0).repeat(self.num_beams, 1, 1) for tensor in last_hidden_state])\\n\",\n \"\\n\",\n \" decoder_blocks = self.decoder.block\\n\",\n \" present_key_value_states_sa = []\\n\",\n \" present_key_value_states_ca = []\\n\",\n \"\\n\",\n \" for i, block in enumerate(decoder_blocks):\\n\",\n \"\\n\",\n \" # Cross attention has to be initialized with the encoder hidden state\\n\",\n \" cross_attention: T5LayerCrossAttention = block.layer[1]\\n\",\n \" attention = cross_attention.EncDecAttention\\n\",\n \"\\n\",\n \" def shape(states):\\n\",\n \" \\\"\\\"\\\"projection\\\"\\\"\\\"\\n\",\n \" return states.view(self.batch_size, -1, self.num_attention_heads_per_partition, attention.key_value_proj_dim).transpose(1, 2)\\n\",\n \"\\n\",\n \" key_states = shape(attention.k(encoder_hidden_states))\\n\",\n \" value_states = shape(attention.v(encoder_hidden_states))\\n\",\n \"\\n\",\n \" # cross_attn_kv_state\\n\",\n \" present_key_value_states_ca.append(key_states) \\n\",\n \" present_key_value_states_ca.append(value_states) \\n\",\n \" \\n\",\n \" # Self attention kv states are initialized to zeros. This is done to keep the size of the kv cache tensor constant. \\n\",\n \" # The kv cache will be an input to the decoder trace. Any traced function will have a fixed control flow. What this means \\n\",\n \" # is that the trace performs the exact same computations on inputs of the same shape in each invocation. So the attention \\n\",\n \" # kv cache is padded here to keep a fixed shape. \\n\",\n \" present_key_value_states_sa.append(torch.zeros((self.batch_size, # key states\\n\",\n \" self.model_config.num_heads, \\n\",\n \" self.max_length-1, \\n\",\n \" self.model_config.d_kv), dtype=torch.float32, device=self.device)) \\n\",\n \" present_key_value_states_sa.append(torch.zeros((self.batch_size, # value states\\n\",\n \" self.model_config.num_heads, \\n\",\n \" self.max_length-1, \\n\",\n \" self.model_config.d_kv), dtype=torch.float32, device=self.device))\\n\",\n \"\\n\",\n \" return present_key_value_states_sa + present_key_value_states_ca\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"In the decoder wrapper, in addition to converting keyword arguments to positional arguments we add support for attention caching. Generating text from the encoder decoder models is an autoregressive process. For each invocation, we have to compute the key and value states of the attention heads repeatedly. To improve the performance, we cache the key and value states. This cache is what HuggingFace transformers code refers to as `past_key_values`.\\n\",\n \"\\n\",\n \"In HuggingFace transformers, the `past_key_values` are updated outside the decoder. This works for training and evaluation but for inference we want to perform them within a single trace. This way, we can optimize across both the decoder execution and cache update. So, we move the cache update within the decoder wrapper.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"class DecoderWrapper(torch.nn.Module):\\n\",\n \"\\n\",\n \" def __init__(self, \\n\",\n \" decoder: T5Stack, \\n\",\n \" lm_head: torch.nn.Linear,\\n\",\n \" model_config,\\n\",\n \" num_beams: int, \\n\",\n \" max_length: int,\\n\",\n \" device: str,\\n\",\n \" tp_degree=None):\\n\",\n \" super().__init__() \\n\",\n \" self.decoder = decoder\\n\",\n \" self.lm_head = lm_head\\n\",\n \" self.model_dim=model_config.d_model\\n\",\n \" self.device = device\\n\",\n \" self.num_beams = num_beams\\n\",\n \" self.batch_size = 1\\n\",\n \" self.config = model_config\\n\",\n \" \\n\",\n \" num_heads=model_config.num_heads\\n\",\n \" num_decoder_layers=model_config.num_decoder_layers\\n\",\n \"\\n\",\n \" self.num_attention_heads_per_partition = num_heads\\n\",\n \"\\n\",\n \" # (num_beams, n_heads, seq_length, dim_per_head)\\n\",\n \" if device == \\\"cpu\\\":\\n\",\n \" self.past_key_values_sa = [torch.ones((num_beams,num_heads,max_length-1,model_config.d_kv), dtype=torch.float32) for _ in range(num_decoder_layers * 2)]\\n\",\n \" self.past_key_values_ca = [torch.ones((num_beams,num_heads,max_length,model_config.d_kv), dtype=torch.float32) for _ in range(num_decoder_layers * 2)]\\n\",\n \" elif device == \\\"xla\\\":\\n\",\n \" self.past_key_values_sa = torch.nn.ParameterList([torch.nn.Parameter(torch.ones((num_beams,self.num_attention_heads_per_partition,max_length-1,model_config.d_kv), dtype=torch.float32), requires_grad=False) for _ in range(num_decoder_layers * 2)])\\n\",\n \" self.past_key_values_ca = torch.nn.ParameterList([torch.nn.Parameter(torch.ones((num_beams,self.num_attention_heads_per_partition,max_length,model_config.d_kv), dtype=torch.float32), requires_grad=False) for _ in range(num_decoder_layers * 2)])\\n\",\n \"\\n\",\n \" def update_past(self, past_key_values):\\n\",\n \" new_past_sa = []\\n\",\n \" new_past_ca = []\\n\",\n \" for past_layer in past_key_values:\\n\",\n \" new_past_layer = list(past_layer)\\n\",\n \" for i in range(len(new_past_layer[:2])):\\n\",\n \" new_past_layer[i] = past_layer[i][:, :, 1:]\\n\",\n \" new_past_sa += [new_past_layer[:2],]\\n\",\n \" new_past_ca += [new_past_layer[2:],]\\n\",\n \" return new_past_sa, new_past_ca\\n\",\n \" \\n\",\n \" def reorder_cache(self, past_key_values, beam_idx):\\n\",\n \" for i in range(len(past_key_values)):\\n\",\n \" gather_index = beam_idx.view([beam_idx.shape[0],1,1,1]).expand_as(past_key_values[i])\\n\",\n \" past_key_values[i] = torch.gather(past_key_values[i], dim = 0, index=gather_index)\\n\",\n \" return past_key_values\\n\",\n \"\\n\",\n \" def forward(self,\\n\",\n \" input_ids,\\n\",\n \" decoder_attention_mask,\\n\",\n \" encoder_hidden_states,\\n\",\n \" encoder_attention_mask,\\n\",\n \" beam_idx,\\n\",\n \" beam_scores,\\n\",\n \" **kwargs):\\n\",\n \"\\n\",\n \" if self.num_beams > 1:\\n\",\n \" # We reorder the cache based on the beams selected in each iteration. Required step for beam search.\\n\",\n \" past_key_values_sa = self.reorder_cache(self.past_key_values_sa, beam_idx)\\n\",\n \" past_key_values_ca = self.reorder_cache(self.past_key_values_ca, beam_idx)\\n\",\n \" else:\\n\",\n \" # We do not need to reorder for greedy sampling\\n\",\n \" past_key_values_sa = self.past_key_values_sa\\n\",\n \" past_key_values_ca = self.past_key_values_ca\\n\",\n \"\\n\",\n \" # The cache is stored in a flatten form. We order the cache per layer before passing it to the decoder. \\n\",\n \" # Each layer has 4 tensors, so we group by 4. \\n\",\n \" past_key_values = [[*past_key_values_sa[i*2:i*2+2], *past_key_values_ca[i*2:i*2+2]] for i in range(0, int(len(past_key_values_ca)/2))]\\n\",\n \"\\n\",\n \" decoder_output = self.decoder(\\n\",\n \" input_ids=input_ids,\\n\",\n \" attention_mask=decoder_attention_mask,\\n\",\n \" past_key_values=past_key_values,\\n\",\n \" encoder_hidden_states=encoder_hidden_states,\\n\",\n \" encoder_attention_mask=encoder_attention_mask,\\n\",\n \" use_cache=True,\\n\",\n \" output_attentions=False,\\n\",\n \" output_hidden_states=False)\\n\",\n \"\\n\",\n \" last_hidden_state = decoder_output['last_hidden_state']\\n\",\n \" past_key_values = decoder_output['past_key_values']\\n\",\n \"\\n\",\n \" if self.config.tie_word_embeddings:\\n\",\n \" # Rescale output before projecting on vocab\\n\",\n \" # See https://github.com/tensorflow/mesh/blob/fa19d69eafc9a482aff0b59ddd96b025c0cb207d/mesh_tensorflow/transformer/transformer.py#L586\\n\",\n \" last_hidden_state = last_hidden_state * (self.model_dim**-0.5)\\n\",\n \" \\n\",\n \" lm_logits = self.lm_head(last_hidden_state)\\n\",\n \"\\n\",\n \" past_key_values_sa, past_key_values_ca = self.update_past(past_key_values)\\n\",\n \"\\n\",\n \" # We flatten the cache to a single array. This is required for the input output aliasing to work\\n\",\n \" past_key_values_sa = [vec for kv_per_layer in past_key_values_sa for vec in kv_per_layer]\\n\",\n \" past_key_values_ca = [vec for kv_per_layer in past_key_values_ca for vec in kv_per_layer]\\n\",\n \"\\n\",\n \" if self.device == \\\"cpu\\\":\\n\",\n \" self.past_key_values_sa = past_key_values_sa\\n\",\n \" self.past_key_values_ca = past_key_values_ca\\n\",\n \"\\n\",\n \" # We calculate topk inside the wrapper\\n\",\n \" next_token_logits = lm_logits[:, -1, :]\\n\",\n \"\\n\",\n \" if self.num_beams > 1:\\n\",\n \" # This section of beam search is run outside the decoder in the huggingface t5 implementation. \\n\",\n \" # To maximize the computation within the neuron device, we move this within the wrapper\\n\",\n \" logit_max, _ = torch.max(next_token_logits, dim=-1, keepdim=True)\\n\",\n \" logsumexp = torch.log(torch.exp(next_token_logits - logit_max).sum(dim=-1, keepdim=True))\\n\",\n \" next_token_scores = next_token_logits - logit_max - logsumexp\\n\",\n \" next_token_scores = next_token_scores + beam_scores[:, None].expand_as(next_token_scores)\\n\",\n \"\\n\",\n \" # reshape for beam search\\n\",\n \" vocab_size = next_token_scores.shape[-1]\\n\",\n \" next_token_scores = next_token_scores.view(self.batch_size, self.num_beams * vocab_size)\\n\",\n \" next_token_scores = next_token_scores * 1\\n\",\n \"\\n\",\n \" # Sample 2 next tokens for each beam (so we have some spare tokens and match output of beam search)\\n\",\n \" next_token_scores, next_tokens = torch.topk(\\n\",\n \" next_token_scores, 2 * self.num_beams, dim=1, largest=True, sorted=True\\n\",\n \" ) \\n\",\n \"\\n\",\n \" next_indices = torch.div(next_tokens, vocab_size, rounding_mode=\\\"floor\\\")\\n\",\n \" next_tokens = next_tokens % vocab_size\\n\",\n \"\\n\",\n \" return [next_token_scores, next_tokens, next_indices] + past_key_values_sa + past_key_values_ca\\n\",\n \" else:\\n\",\n \" # Greedy \\n\",\n \" next_tokens = torch.argmax(next_token_logits, dim=-1)\\n\",\n \" return [next_tokens] + past_key_values_sa + past_key_values_ca\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now let's create a T5 model wrapper to make it compatible with our traced encoder and decoder. \\n\",\n \"\\n\",\n \"There are two reasons for having this wrapper, \\n\",\n \"\\n\",\n \"1. The encoder and decoder traces can only be invoked with positional arguments. But the HuggingFace transformers code is written with keyword arguments. So we override the functions that invoke encoder and decoder to call with positional arguments. \\n\",\n \"1. The generate() function in the NeuronGenerationMixin performs cache update within the CPU. As we are handling the cache within the DecoderWrapper, we disable the cache update on CPU. \\n\",\n \"1. The topK computation to determine the next tokens for beam search was moved into the decoder wrapper. So, we need to override the huggingface's beam search implementation to accept the next tokens and the beam scores from the decoder. \\n\",\n \"\\n\",\n \"Let's also override the `generate()` function so that it will intialize the cache using the cache initalizer before starting the greedy decoding.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_xla.core.xla_model as xm\\n\",\n \"\\n\",\n \"from transformers import T5Tokenizer, T5ForConditionalGeneration\\n\",\n \"from transformers.modeling_outputs import BaseModelOutput, Seq2SeqLMOutput\\n\",\n \"from transformers.models.t5.modeling_t5 import T5Stack, T5LayerCrossAttention\\n\",\n \"from transformers.generation.utils import ModelOutput\\n\",\n \"from typing import Any, Dict, List, Optional, Tuple, Union\\n\",\n \"from transformers.generation.beam_search import BeamScorer, BeamSearchScorer\\n\",\n \"\\n\",\n \"from optimum.neuron.generation import NeuronGenerationMixin\\n\",\n \"\\n\",\n \"from transformers.generation.logits_process import (\\n\",\n \" LogitsProcessorList,\\n\",\n \")\\n\",\n \"from transformers.generation.stopping_criteria import (\\n\",\n \" MaxLengthCriteria,\\n\",\n \" MaxTimeCriteria,\\n\",\n \" StoppingCriteriaList,\\n\",\n \" validate_stopping_criteria,\\n\",\n \")\\n\",\n \"\\n\",\n \"from transformers.generation.utils import (\\n\",\n \" BeamSearchOutput,\\n\",\n \" GreedySearchOutput,\\n\",\n \")\\n\",\n \"\\n\",\n \"class T5Wrapper(T5ForConditionalGeneration, NeuronGenerationMixin):\\n\",\n \"\\n\",\n \" def _prepare_encoder_decoder_kwargs_for_generation(\\n\",\n \" self, \\n\",\n \" inputs_tensor: torch.Tensor, \\n\",\n \" model_kwargs, \\n\",\n \" model_input_name: Optional[str] = None\\n\",\n \" ) -> Dict[str, Any]:\\n\",\n \" encoder = self.get_encoder()\\n\",\n \" model_kwargs[\\\"encoder_outputs\\\"]: ModelOutput = encoder(inputs_tensor, model_kwargs[\\\"attention_mask\\\"])\\n\",\n \" return model_kwargs\\n\",\n \"\\n\",\n \" # Override to cut the input_ids to just last token\\n\",\n \" def prepare_inputs_for_generation(\\n\",\n \" self,\\n\",\n \" input_ids,\\n\",\n \" past_key_values=None,\\n\",\n \" attention_mask=None,\\n\",\n \" head_mask=None,\\n\",\n \" decoder_head_mask=None,\\n\",\n \" decoder_attention_mask=None,\\n\",\n \" cross_attn_head_mask=None,\\n\",\n \" use_cache=None,\\n\",\n \" encoder_outputs=None,\\n\",\n \" **kwargs,\\n\",\n \" ):\\n\",\n \" # cut decoder_input_ids as past is cached\\n\",\n \" input_ids = input_ids[:, -1:]\\n\",\n \"\\n\",\n \" return {\\n\",\n \" \\\"decoder_input_ids\\\": input_ids,\\n\",\n \" \\\"past_key_values\\\": past_key_values,\\n\",\n \" \\\"encoder_outputs\\\": encoder_outputs,\\n\",\n \" \\\"attention_mask\\\": attention_mask,\\n\",\n \" \\\"head_mask\\\": head_mask,\\n\",\n \" \\\"decoder_head_mask\\\": decoder_head_mask,\\n\",\n \" \\\"decoder_attention_mask\\\": decoder_attention_mask,\\n\",\n \" \\\"cross_attn_head_mask\\\": cross_attn_head_mask,\\n\",\n \" \\\"use_cache\\\": use_cache,\\n\",\n \" }\\n\",\n \" \\n\",\n \" '''\\n\",\n \" We update the cache in the decoder trace, so lets override the _update_model_kwargs_for_xla_generation in NeuronGenerationMixin\\n\",\n \" '''\\n\",\n \" def _update_model_kwargs_for_xla_generation(\\n\",\n \" self,\\n\",\n \" model_kwargs: Dict[str, Any],\\n\",\n \" batch_size: int,\\n\",\n \" is_encoder_decoder: bool = False,\\n\",\n \" standardize_cache_format: bool = False,\\n\",\n \" max_length: Optional[int] = None,\\n\",\n \" seq_length: Optional[int] = None,\\n\",\n \" use_cache: bool = True,\\n\",\n \" ) -> Dict[str, Any]:\\n\",\n \"\\n\",\n \" def _update_attention(model_kwargs, is_encoder_decoder):\\n\",\n \" \\\"\\\"\\\"Updates the appropriate attention mask -- encoder-decoder models use `decoder_attention_mask`\\\"\\\"\\\"\\n\",\n \"\\n\",\n \" attention_mask_name = \\\"decoder_attention_mask\\\" if is_encoder_decoder else \\\"attention_mask\\\"\\n\",\n \" attention_mask = model_kwargs.pop(attention_mask_name)\\n\",\n \" attention_mask_update_slice = torch.ones(\\n\",\n \" (batch_size, 1), dtype=attention_mask.dtype, device=attention_mask.device\\n\",\n \" )\\n\",\n \" attention_mask = torch.cat([attention_mask[:, 1:], attention_mask_update_slice], dim=-1)\\n\",\n \" mask = {attention_mask_name: attention_mask}\\n\",\n \" return mask\\n\",\n \"\\n\",\n \" mask = _update_attention(model_kwargs, is_encoder_decoder)\\n\",\n \" # sets the updated variables (mask and past_key_values)\\n\",\n \" model_kwargs.update(mask)\\n\",\n \"\\n\",\n \" # Set a mock cache tensor\\n\",\n \" model_kwargs[\\\"past_key_values\\\"] = torch.tensor([])\\n\",\n \"\\n\",\n \" return model_kwargs\\n\",\n \" \\n\",\n \" def _reorder_cache(self, past_key_values, beam_idx):\\n\",\n \" '''\\n\",\n \" This is needed for beam search and not greedy sampling\\n\",\n \" We reorder the cache within the trace so we can skip it in modelling_t5.py. So we override the _reorder_cache\\n\",\n \" '''\\n\",\n \" self.beam_idx = beam_idx\\n\",\n \" return past_key_values\\n\",\n \"\\n\",\n \" def generate(self,\\n\",\n \" tokenizer: T5Tokenizer,\\n\",\n \" prompt: str,\\n\",\n \" max_length: int,\\n\",\n \" num_beams: int,\\n\",\n \" num_return_sequences: int,\\n\",\n \" device: str):\\n\",\n \"\\n\",\n \" batch_encoding = tokenizer(prompt, max_length=max_length, truncation=True, padding='max_length',\\n\",\n \" return_tensors=\\\"pt\\\")\\n\",\n \"\\n\",\n \" past_key_values = self.encoder(batch_encoding['input_ids'],batch_encoding['attention_mask'])\\n\",\n \" \\n\",\n \" decoder_attention_mask = torch.cat([torch.zeros((1, max_length-1), dtype=torch.int32),\\n\",\n \" torch.ones((1, 1), dtype=torch.int32)], axis=1)\\n\",\n \"\\n\",\n \" # copy the new cache state to the decoder\\n\",\n \" if device == \\\"xla\\\":\\n\",\n \" for state, tensor in zip(self.decoder.parameters(), past_key_values):\\n\",\n \" state.copy_(tensor)\\n\",\n \" else:\\n\",\n \" # First half of the cache is self attention and the rest is cross attention\\n\",\n \" self.decoder.past_key_values_sa = past_key_values[:len(past_key_values)//2]\\n\",\n \" self.decoder.past_key_values_ca = past_key_values[len(past_key_values)//2:]\\n\",\n \" \\n\",\n \" output = super().generate(**batch_encoding,\\n\",\n \" max_length=max_length,\\n\",\n \" num_beams=num_beams,\\n\",\n \" num_return_sequences=num_return_sequences,\\n\",\n \" do_sample=False,\\n\",\n \" use_cache=True,\\n\",\n \" decoder_attention_mask=decoder_attention_mask, \\n\",\n \" encoder_outputs={\\\"last_hidden_state\\\": torch.ones((1,128,1))}) # Pass fake encoder_outputs so the transfomers code will not invoke the encoder\\n\",\n \" return output\\n\",\n \"\\n\",\n \" def forward(\\n\",\n \" self,\\n\",\n \" attention_mask: Optional[torch.FloatTensor] = None,\\n\",\n \" decoder_input_ids: Optional[torch.LongTensor] = None,\\n\",\n \" decoder_attention_mask: Optional[torch.BoolTensor] = None,\\n\",\n \" encoder_outputs: Optional[Tuple[Tuple[torch.Tensor]]] = None,\\n\",\n \" beam_scores = None,\\n\",\n \" **kwargs\\n\",\n \" ) -> Union[Tuple[torch.FloatTensor], Seq2SeqLMOutput]:\\n\",\n \"\\n\",\n \" hidden_states = encoder_outputs[\\\"last_hidden_state\\\"]\\n\",\n \"\\n\",\n \" if not hasattr(self, 'beam_idx'):\\n\",\n \" # Infering the number of beams from the attention mask\\n\",\n \" num_beams = attention_mask.shape[0]\\n\",\n \" self.beam_idx = torch.arange(0, num_beams, dtype=torch.int64)\\n\",\n \"\\n\",\n \" decoder_outputs = self.decoder(\\n\",\n \" decoder_input_ids,\\n\",\n \" decoder_attention_mask,\\n\",\n \" hidden_states,\\n\",\n \" attention_mask,\\n\",\n \" self.beam_idx,\\n\",\n \" beam_scores\\n\",\n \" )\\n\",\n \"\\n\",\n \" # lm_logits = decoder_outputs[0]\\n\",\n \" next_token_scores = decoder_outputs[0]\\n\",\n \" next_tokens = decoder_outputs[1]\\n\",\n \" next_indices = decoder_outputs[2]\\n\",\n \"\\n\",\n \" return next_token_scores, next_tokens, next_indices\\n\",\n \"\\n\",\n \" def beam_search(\\n\",\n \" self,\\n\",\n \" input_ids: torch.LongTensor,\\n\",\n \" beam_scorer: BeamScorer,\\n\",\n \" logits_processor: Optional[LogitsProcessorList] = None,\\n\",\n \" stopping_criteria: Optional[StoppingCriteriaList] = None,\\n\",\n \" max_length: Optional[int] = None,\\n\",\n \" pad_token_id: Optional[int] = None,\\n\",\n \" eos_token_id: Optional[Union[int, List[int]]] = None,\\n\",\n \" output_attentions: Optional[bool] = None,\\n\",\n \" output_hidden_states: Optional[bool] = None,\\n\",\n \" output_scores: Optional[bool] = None,\\n\",\n \" return_dict_in_generate: Optional[bool] = None,\\n\",\n \" synced_gpus: Optional[bool] = False,\\n\",\n \" seq_length: Optional[int] = None,\\n\",\n \" **model_kwargs,\\n\",\n \" ) -> Union[BeamSearchOutput, torch.LongTensor]:\\n\",\n \"\\n\",\n \" logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()\\n\",\n \" stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()\\n\",\n \" pad_token_id = pad_token_id if pad_token_id is not None else self.generation_config.pad_token_id\\n\",\n \" eos_token_id = eos_token_id if eos_token_id is not None else self.generation_config.eos_token_id\\n\",\n \" if isinstance(eos_token_id, int):\\n\",\n \" eos_token_id = [eos_token_id]\\n\",\n \" output_scores = output_scores if output_scores is not None else self.generation_config.output_scores\\n\",\n \" output_attentions = (\\n\",\n \" output_attentions if output_attentions is not None else self.generation_config.output_attentions\\n\",\n \" )\\n\",\n \" output_hidden_states = (\\n\",\n \" output_hidden_states if output_hidden_states is not None else self.generation_config.output_hidden_states\\n\",\n \" )\\n\",\n \"\\n\",\n \" batch_size = len(beam_scorer._beam_hyps)\\n\",\n \" num_beams = beam_scorer.num_beams\\n\",\n \"\\n\",\n \" batch_beam_size, cur_len = input_ids.shape\\n\",\n \"\\n\",\n \" # Overwrite cur_len\\n\",\n \" cur_len = seq_length\\n\",\n \"\\n\",\n \" if num_beams * batch_size != batch_beam_size:\\n\",\n \" raise ValueError(\\n\",\n \" f\\\"Batch dimension of `input_ids` should be {num_beams * batch_size}, but is {batch_beam_size}.\\\"\\n\",\n \" )\\n\",\n \"\\n\",\n \" # init attention / hidden states / scores tuples\\n\",\n \" scores = () if (return_dict_in_generate and output_scores) else None\\n\",\n \" beam_indices = (\\n\",\n \" tuple(() for _ in range(batch_beam_size)) if (return_dict_in_generate and output_scores) else None\\n\",\n \" )\\n\",\n \"\\n\",\n \" # initialise score of first beam with 0 and the rest with -1e9. This makes sure that only tokens\\n\",\n \" # of the first beam are considered to avoid sampling the exact same tokens across all beams.\\n\",\n \" # beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=input_ids.device)\\n\",\n \" beam_scores_device = \\\"cpu\\\"\\n\",\n \" beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=beam_scores_device)\\n\",\n \" beam_scores[:, 1:] = -1e9\\n\",\n \" beam_scores = beam_scores.view((batch_size * num_beams,))\\n\",\n \"\\n\",\n \" while True:\\n\",\n \" # prepare model inputs\\n\",\n \" # From max_length-sized input_ids, select first\\n\",\n \" # cur_len - 1 values.\\n\",\n \" update_indices = torch.stack(\\n\",\n \" [torch.arange(input_ids.size(0)), torch.tensor(cur_len - 1).repeat(input_ids.size(0))], dim=-1\\n\",\n \" )\\n\",\n \" input_ids_ = input_ids[update_indices[:, 0], update_indices[:, 1], None]\\n\",\n \" model_inputs = self.prepare_inputs_for_generation(input_ids_, **model_kwargs)\\n\",\n \"\\n\",\n \" next_token_scores, next_tokens, next_indices = self(\\n\",\n \" **model_inputs,\\n\",\n \" return_dict=True,\\n\",\n \" output_attentions=output_attentions,\\n\",\n \" output_hidden_states=output_hidden_states,\\n\",\n \" beam_scores=beam_scores\\n\",\n \" )\\n\",\n \"\\n\",\n \" # stateless\\n\",\n \" beam_outputs = beam_scorer.process(\\n\",\n \" input_ids.to(\\\"cpu\\\")[:, :cur_len],\\n\",\n \" next_token_scores.to(\\\"cpu\\\"),\\n\",\n \" next_tokens.to(\\\"cpu\\\"),\\n\",\n \" next_indices.to(\\\"cpu\\\"),\\n\",\n \" pad_token_id=pad_token_id,\\n\",\n \" eos_token_id=eos_token_id,\\n\",\n \" beam_indices=beam_indices,\\n\",\n \" )\\n\",\n \"\\n\",\n \" beam_scores = beam_outputs[\\\"next_beam_scores\\\"]\\n\",\n \" beam_next_tokens = beam_outputs[\\\"next_beam_tokens\\\"]\\n\",\n \" beam_idx = beam_outputs[\\\"next_beam_indices\\\"]\\n\",\n \"\\n\",\n \" update_indices = torch.stack(\\n\",\n \" [torch.arange(batch_beam_size), torch.tensor(cur_len - 1).repeat(batch_beam_size)], dim=-1\\n\",\n \" )\\n\",\n \" update_indices_2 = torch.stack(\\n\",\n \" [torch.arange(batch_beam_size), torch.tensor(cur_len).repeat(batch_beam_size)], dim=-1\\n\",\n \" )\\n\",\n \" # First select beam_indices\\n\",\n \" device = input_ids.device\\n\",\n \" beam_idx_device = beam_idx.to(device=input_ids.device)\\n\",\n \" input_ids[:, :] = input_ids[beam_idx_device.long(), :]\\n\",\n \"\\n\",\n \" # Then append new tokens\\n\",\n \" input_ids[update_indices_2[:, 0], update_indices_2[:, 1], None] = beam_next_tokens.unsqueeze(-1).to(device).to(torch.long)\\n\",\n \" input_ids = input_ids * 1 # Hack to materialize tensor\\n\",\n \"\\n\",\n \" # update generated ids, model inputs, and length for next step\\n\",\n \" model_kwargs = self._update_model_kwargs_for_xla_generation(\\n\",\n \" model_kwargs,\\n\",\n \" batch_size=batch_beam_size,\\n\",\n \" is_encoder_decoder=self.config.is_encoder_decoder,\\n\",\n \" max_length=stopping_criteria.max_length,\\n\",\n \" seq_length=cur_len,\\n\",\n \" use_cache=model_kwargs[\\\"use_cache\\\"],\\n\",\n \" )\\n\",\n \" if model_kwargs[\\\"past_key_values\\\"] is not None:\\n\",\n \" model_kwargs[\\\"past_key_values\\\"] = self._reorder_cache(model_kwargs[\\\"past_key_values\\\"], beam_idx.to(torch.int64))\\n\",\n \"\\n\",\n \" if return_dict_in_generate and output_scores:\\n\",\n \" beam_indices = tuple((beam_indices[beam_idx[i]] + (beam_idx[i],) for i in range(len(beam_indices))))\\n\",\n \"\\n\",\n \" # increase cur_len\\n\",\n \" cur_len = cur_len + 1\\n\",\n \"\\n\",\n \" # stop when each sentence is finished, or if we exceed the maximum length\\n\",\n \" stop_criterion_1 = beam_scorer.is_done\\n\",\n \" if isinstance(stopping_criteria, list):\\n\",\n \" if len(stopping_criteria) == 1:\\n\",\n \" stopping_criteria = stopping_criteria[0]\\n\",\n \"\\n\",\n \" # Cases that can be handled in XLA without requiring\\n\",\n \" # non-padded input_ids\\n\",\n \" if isinstance(stopping_criteria, MaxLengthCriteria):\\n\",\n \" stop_criterion_2 = cur_len >= stopping_criteria.max_length\\n\",\n \" elif isinstance(stopping_criteria, MaxTimeCriteria):\\n\",\n \" stop_criterion_2 = stopping_criteria(input_ids, scores)\\n\",\n \" else:\\n\",\n \" # Other cases will be handled on CPU\\n\",\n \" batch_size, _ = input_ids.shape\\n\",\n \" input_ids_cpu = input_ids.to(\\\"cpu\\\")\\n\",\n \" mask = torch.cat(\\n\",\n \" [torch.ones(batch_size, cur_len), torch.zeros(batch_size, input_ids.shape[1] - cur_len)], dim=1\\n\",\n \" ).bool()\\n\",\n \" input_ids_cpu = torch.masked_select(input_ids_cpu, mask).reshape((batch_size, cur_len))\\n\",\n \" scores_cpu = scores.to(\\\"cpu\\\") if torch.is_tensor(scores) else scores\\n\",\n \" stop_criterion_2 = stopping_criteria(input_ids_cpu, scores_cpu)\\n\",\n \"\\n\",\n \" if stop_criterion_1 or stop_criterion_2:\\n\",\n \" if not synced_gpus:\\n\",\n \" break\\n\",\n \" else:\\n\",\n \" this_peer_finished = True\\n\",\n \"\\n\",\n \" sequence_outputs = beam_scorer.finalize(\\n\",\n \" input_ids.to(\\\"cpu\\\"),\\n\",\n \" beam_scores.to(\\\"cpu\\\"),\\n\",\n \" next_tokens.to(\\\"cpu\\\"),\\n\",\n \" next_indices.to(\\\"cpu\\\"),\\n\",\n \" pad_token_id=pad_token_id,\\n\",\n \" eos_token_id=eos_token_id,\\n\",\n \" max_length=stopping_criteria.max_length,\\n\",\n \" beam_indices=beam_indices,\\n\",\n \" )\\n\",\n \"\\n\",\n \" for k, v in sequence_outputs.items():\\n\",\n \" if type(v) == torch.Tensor:\\n\",\n \" sequence_outputs[k] = sequence_outputs[k].to(input_ids.device)\\n\",\n \"\\n\",\n \" return sequence_outputs[\\\"sequences\\\"]\\n\",\n \"\\n\",\n \"\\n\",\n \" def greedy_search(\\n\",\n \" self,\\n\",\n \" input_ids: torch.LongTensor,\\n\",\n \" logits_processor: Optional[LogitsProcessorList] = None,\\n\",\n \" stopping_criteria: Optional[StoppingCriteriaList] = None,\\n\",\n \" max_length: Optional[int] = None,\\n\",\n \" pad_token_id: Optional[int] = None,\\n\",\n \" eos_token_id: Optional[Union[int, List[int]]] = None,\\n\",\n \" output_attentions: Optional[bool] = None,\\n\",\n \" output_hidden_states: Optional[bool] = None,\\n\",\n \" output_scores: Optional[bool] = None,\\n\",\n \" return_dict_in_generate: Optional[bool] = None,\\n\",\n \" seq_length: Optional[int] = int,\\n\",\n \" streamer: Optional[\\\"BaseStreamer\\\"] = None,\\n\",\n \" **model_kwargs,\\n\",\n \" ) -> Union[GreedySearchOutput, torch.LongTensor]:\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Overriding greedy sampling to use next tokens returned from neuron device instead of logits.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" # init values\\n\",\n \" logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()\\n\",\n \" use_cache = model_kwargs[\\\"use_cache\\\"] if \\\"use_cache\\\" in model_kwargs else False\\n\",\n \" stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()\\n\",\n \" pad_token_id = pad_token_id if pad_token_id is not None else self.generation_config.pad_token_id\\n\",\n \" eos_token_id = eos_token_id if eos_token_id is not None else self.generation_config.eos_token_id\\n\",\n \" if isinstance(eos_token_id, int):\\n\",\n \" eos_token_id = [eos_token_id]\\n\",\n \" eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None\\n\",\n \" output_scores = output_scores if output_scores is not None else self.generation_config.output_scores\\n\",\n \" output_attentions = (\\n\",\n \" output_attentions if output_attentions is not None else self.generation_config.output_attentions\\n\",\n \" )\\n\",\n \" output_hidden_states = (\\n\",\n \" output_hidden_states if output_hidden_states is not None else self.generation_config.output_hidden_states\\n\",\n \" )\\n\",\n \"\\n\",\n \" # init attention / hidden states / scores tuples\\n\",\n \" scores = () if (return_dict_in_generate and output_scores) else None\\n\",\n \" decoder_attentions = () if (return_dict_in_generate and output_attentions) else None\\n\",\n \" cross_attentions = () if (return_dict_in_generate and output_attentions) else None\\n\",\n \" decoder_hidden_states = () if (return_dict_in_generate and output_hidden_states) else None\\n\",\n \"\\n\",\n \"\\n\",\n \" # keep track of which sequences are already finished\\n\",\n \" unfinished_sequences = torch.ones(input_ids.shape[0], dtype=torch.long, device=input_ids.device)\\n\",\n \"\\n\",\n \" this_peer_finished = False # used by synced_gpus only\\n\",\n \" while True:\\n\",\n \"\\n\",\n \" # prepare model inputs\\n\",\n \" # From max_length-sized input_ids, select first\\n\",\n \" # seq_length - 1 values.\\n\",\n \"\\n\",\n \" if model_kwargs.get(\\\"past_key_values\\\") is None:\\n\",\n \" input_ids_ = input_ids[:, :seq_length]\\n\",\n \" else:\\n\",\n \" update_indices = torch.stack(\\n\",\n \" [torch.arange(input_ids.size(0)), torch.tensor(seq_length - 1).repeat(input_ids.size(0))],\\n\",\n \" dim=-1,\\n\",\n \" )\\n\",\n \" input_ids_ = input_ids[update_indices[:, 0], update_indices[:, 1], None]\\n\",\n \"\\n\",\n \" model_inputs = self.prepare_inputs_for_generation(input_ids_, **model_kwargs)\\n\",\n \" \\n\",\n \" # forward pass to get next token\\n\",\n \" output = self(\\n\",\n \" **model_inputs,\\n\",\n \" return_dict=True,\\n\",\n \" output_attentions=output_attentions,\\n\",\n \" output_hidden_states=output_hidden_states,\\n\",\n \" )\\n\",\n \" next_tokens = output[0]\\n\",\n \"\\n\",\n \" # finished sentences should have their next token be a padding token\\n\",\n \" if eos_token_id is not None:\\n\",\n \" if pad_token_id is None:\\n\",\n \" raise ValueError(\\\"If `eos_token_id` is defined, make sure that `pad_token_id` is defined.\\\")\\n\",\n \" next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)\\n\",\n \"\\n\",\n \" # update generated ids, model inputs, and length for next step\\n\",\n \"\\n\",\n \" batch_size, _ = input_ids.shape\\n\",\n \" update_indices = torch.stack(\\n\",\n \" [torch.arange(batch_size), torch.tensor(seq_length).repeat(batch_size)], dim=-1\\n\",\n \" )\\n\",\n \" input_ids[update_indices[:, 0], update_indices[:, 1]] = next_tokens[:]\\n\",\n \" model_kwargs = self._update_model_kwargs_for_xla_generation(\\n\",\n \" model_kwargs,\\n\",\n \" batch_size=batch_size,\\n\",\n \" is_encoder_decoder=self.config.is_encoder_decoder,\\n\",\n \" max_length=stopping_criteria.max_length,\\n\",\n \" seq_length=seq_length,\\n\",\n \" use_cache=use_cache,\\n\",\n \" )\\n\",\n \"\\n\",\n \" seq_length += 1\\n\",\n \"\\n\",\n \" # if eos_token was found in one sentence, set sentence to finished\\n\",\n \" if eos_token_id_tensor is not None:\\n\",\n \" unfinished_sequences = unfinished_sequences.mul(\\n\",\n \" next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_tensor.unsqueeze(1)).prod(dim=0)\\n\",\n \" )\\n\",\n \"\\n\",\n \" # stop when each sentence is finished, or if we exceed the maximum length\\n\",\n \" stop_criterion_1 = unfinished_sequences.max() == 0\\n\",\n \"\\n\",\n \" if isinstance(stopping_criteria, list):\\n\",\n \" if len(stopping_criteria) == 1:\\n\",\n \" stopping_criteria = stopping_criteria[0]\\n\",\n \"\\n\",\n \" # Cases that can be handled in XLA without requiring\\n\",\n \" # non-padded input_ids\\n\",\n \" if isinstance(stopping_criteria, MaxLengthCriteria):\\n\",\n \" stop_criterion_2 = seq_length >= stopping_criteria.max_length\\n\",\n \" elif isinstance(stopping_criteria, MaxTimeCriteria):\\n\",\n \" stop_criterion_2 = stopping_criteria(input_ids, scores)\\n\",\n \" else:\\n\",\n \" # Other cases will be handled on CPU\\n\",\n \" batch_size, _ = input_ids.shape\\n\",\n \" mask = torch.cat(\\n\",\n \" [torch.ones(batch_size, seq_length), torch.zeros(batch_size, input_ids.shape[1] - seq_length)],\\n\",\n \" dim=1,\\n\",\n \" ).bool()\\n\",\n \" input_ids_cpu = torch.masked_select(input_ids, mask).reshape((batch_size, seq_length)).to(\\\"cpu\\\")\\n\",\n \" scores_cpu = scores.to(\\\"cpu\\\") if torch.is_tensor(scores) else scores\\n\",\n \" stop_criterion_2 = stopping_criteria(input_ids_cpu, scores_cpu)\\n\",\n \"\\n\",\n \" if stop_criterion_1 or stop_criterion_2:\\n\",\n \" this_peer_finished = True\\n\",\n \"\\n\",\n \" if this_peer_finished:\\n\",\n \" break\\n\",\n \"\\n\",\n \" if streamer is not None:\\n\",\n \" streamer.end()\\n\",\n \"\\n\",\n \" return input_ids\\n\",\n \" \\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now let's test inference on CPU with all the wrappers before tracing.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Let's set some run parameters\\n\",\n \"\\n\",\n \"model_name = \\\"t5-large\\\"\\n\",\n \"num_beams = 1\\n\",\n \"num_return_sequences = 1\\n\",\n \"max_length = 128\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Results:\\n\",\n \"1 Lassen Sie uns gutes Essen essen.\\n\"\n ]\n }\n ],\n \"source\": [\n \"from transformers import T5Tokenizer\\n\",\n \"\\n\",\n \"\\n\",\n \"prompt=\\\"translate English to German: Lets eat good food.\\\"\\n\",\n \" \\n\",\n \"tokenizer = T5Tokenizer.from_pretrained(model_name, model_max_length=max_length)\\n\",\n \"model = T5Wrapper.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"model.encoder = EncoderWrapper(model.encoder, model.decoder, model.config, num_beams, max_length, \\\"cpu\\\", num_beams)\\n\",\n \"setattr(model.encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\\n\",\n \"\\n\",\n \"model.decoder = DecoderWrapper(decoder=model.decoder,\\n\",\n \" lm_head=model.lm_head,\\n\",\n \" model_config=model.config,\\n\",\n \" num_beams=num_beams,\\n\",\n \" max_length=max_length,\\n\",\n \" device=\\\"cpu\\\")\\n\",\n \"\\n\",\n \"output = model.generate(tokenizer=tokenizer,\\n\",\n \" prompt=prompt,\\n\",\n \" max_length=max_length,\\n\",\n \" num_beams=num_beams,\\n\",\n \" num_return_sequences=num_return_sequences,\\n\",\n \" device=\\\"cpu\\\")\\n\",\n \"\\n\",\n \"results = [tokenizer.decode(t, skip_special_tokens=True) for t in output]\\n\",\n \"\\n\",\n \"print('Results:')\\n\",\n \"for i, summary in enumerate(results):\\n\",\n \" print(i + 1, summary)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now that the wrappers are running as expected, let's trace the encoder, and decoder. To trace these functions, we pass the function and a sample input to the trace function. The result of the trace stage will be a static executable where the operations to be run upon inference are determined during compilation. This means that when inferring, the resulting Neuron model must be executed with tensors that are the exact same shape as those provided at compilation time. If a model is given a tensor at inference time whose shape does not match the tensor given at compilation time, an error will occur.\\n\",\n \"\\n\",\n \"The decoder wrapper returns the new state of the cache as an output which is copied back to the CPU. As the cache is a large tensor, copying it to and from the XLA device for each decoder invocation will significantly slow down the inference. Instead, we can use input output aliasing, a feature of `torch_neuronx` to keep these tensors on device rather than copying back to the CPU. To use input output aliasing, we need to map the outputs to input parameters while tracing. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuronx\\n\",\n \"\\n\",\n \"from transformers import T5Tokenizer, T5ForConditionalGeneration\\n\",\n \"\\n\",\n \"def trace_encoder(model: T5ForConditionalGeneration,\\n\",\n \" tokenizer: T5Tokenizer,\\n\",\n \" max_length: int,\\n\",\n \" num_beams: int):\\n\",\n \" \\n\",\n \" # Trace encoder\\n\",\n \" batch_encoding = tokenizer(\\\"translate English to German: Lets go home now\\\",\\n\",\n \" max_length=max_length, truncation=True, padding='max_length', return_tensors=\\\"pt\\\")\\n\",\n \" input_ids = batch_encoding['input_ids']\\n\",\n \" attention_mask = batch_encoding['attention_mask']\\n\",\n \"\\n\",\n \" encoder = EncoderWrapper(model.encoder, model.decoder, model.config, num_beams, max_length, \\\"xla\\\", num_beams)\\n\",\n \" traced_encoder = torch_neuronx.trace(encoder, (input_ids, attention_mask), compiler_workdir=\\\"/tmp/encoder/\\\")\\n\",\n \" setattr(traced_encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\\n\",\n \"\\n\",\n \" return traced_encoder\\n\",\n \"\\n\",\n \"def trace_decoder(model: T5ForConditionalGeneration,\\n\",\n \" num_beams: int,\\n\",\n \" max_length: int):\\n\",\n \"\\n\",\n \" decoder = DecoderWrapper(decoder=model.decoder,\\n\",\n \" lm_head=model.lm_head,\\n\",\n \" model_config=model.config,\\n\",\n \" num_beams=num_beams,\\n\",\n \" max_length=max_length,\\n\",\n \" device=\\\"xla\\\")\\n\",\n \"\\n\",\n \" # We create mock inputs so we can trace the decoder\\n\",\n \" decoder_input_ids = torch.ones((num_beams, 1), dtype=torch.int64)\\n\",\n \" decoder_attention_mask = torch.ones((num_beams, max_length), dtype=torch.int32)\\n\",\n \" encoder_attention_mask = torch.ones((num_beams, max_length), dtype=torch.int64)\\n\",\n \" encoder_hidden_states = torch.ones((num_beams, max_length, model.config.d_model), dtype=torch.float32)\\n\",\n \"\\n\",\n \" beam_idx = torch.arange(0, num_beams, dtype=torch.int64)\\n\",\n \" beam_scores = torch.zeros((num_beams,), dtype=torch.float)\\n\",\n \"\\n\",\n \" num_outputs_from_trace = 3 if num_beams > 1 else 1\\n\",\n \"\\n\",\n \" aliases = {}\\n\",\n \" for i in range(len(decoder.past_key_values_sa)):\\n\",\n \" aliases[decoder.past_key_values_sa[i]] = i + num_outputs_from_trace\\n\",\n \" for i in range(len(decoder.past_key_values_ca)):\\n\",\n \" aliases[decoder.past_key_values_ca[i]] = len(decoder.past_key_values_sa) + i + num_outputs_from_trace\\n\",\n \"\\n\",\n \" traced_decoder = torch_neuronx.trace(decoder, (\\n\",\n \" decoder_input_ids,\\n\",\n \" decoder_attention_mask,\\n\",\n \" encoder_hidden_states,\\n\",\n \" encoder_attention_mask,\\n\",\n \" beam_idx,\\n\",\n \" beam_scores,\\n\",\n \" ), input_output_aliases=aliases, compiler_workdir=\\\"/tmp/decoder/\\\")\\n\",\n \"\\n\",\n \" return traced_decoder\\n\",\n \"\\n\",\n \"\\n\",\n \"tokenizer = T5Tokenizer.from_pretrained(model_name, model_max_length=max_length)\\n\",\n \"model = T5ForConditionalGeneration.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"# We enable this flag to ensure model uses attention key value caching\\n\",\n \"model.config.use_cache = True\\n\",\n \"\\n\",\n \"traced_encoder = trace_encoder(model, tokenizer, max_length, num_beams)\\n\",\n \"traced_decoder = trace_decoder(model, num_beams, max_length)\\n\",\n \"\\n\",\n \"torch.jit.save(traced_encoder, \\\"TracedEncoder.pt\\\")\\n\",\n \"torch.jit.save(traced_decoder, \\\"TracedDecoder.pt\\\")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference with greedy decoding\\n\",\n \"Now that we have the traced model, let's use it for inference. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Results:\\n\",\n \"1 Lassen Sie uns gutes Essen essen.\\n\"\n ]\n }\n ],\n \"source\": [\n \"runtime = torch.classes.neuron.Runtime()\\n\",\n \"runtime.initialize()\\n\",\n \"runtime.set_default_neuron_cores(0, 1)\\n\",\n \"\\n\",\n \"tokenizer = T5Tokenizer.from_pretrained(model_name)\\n\",\n \"model = T5Wrapper.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"model.encoder = torch.jit.load(\\\"TracedEncoder.pt\\\")\\n\",\n \"# Attribute required by beam search\\n\",\n \"setattr(model.encoder, 'main_input_name', 'input_ids') \\n\",\n \"\\n\",\n \"model.decoder = torch.jit.load(\\\"TracedDecoder.pt\\\")\\n\",\n \"torch_neuronx.move_trace_to_device(model.decoder, 0)\\n\",\n \"\\n\",\n \"\\n\",\n \"output = model.generate(tokenizer=tokenizer,\\n\",\n \" prompt=\\\"translate English to German: Lets eat good food.\\\",\\n\",\n \" max_length=max_length,\\n\",\n \" num_beams=num_beams,\\n\",\n \" num_return_sequences=num_return_sequences,\\n\",\n \" device=\\\"xla\\\")\\n\",\n \"\\n\",\n \"results = [tokenizer.decode(t, skip_special_tokens=True) for t in output]\\n\",\n \"\\n\",\n \"print('Results:')\\n\",\n \"for i, summary in enumerate(results):\\n\",\n \" print(i + 1, summary)\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Run inference with beam search\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Let's set some run parameters for beam search\\n\",\n \"\\n\",\n \"model_name = \\\"t5-large\\\"\\n\",\n \"num_beams = 4\\n\",\n \"num_return_sequences = 4\\n\",\n \"max_length = 128\\n\",\n \"\\n\",\n \"tokenizer = T5Tokenizer.from_pretrained(model_name, model_max_length=max_length)\\n\",\n \"model = T5ForConditionalGeneration.from_pretrained(model_name)\\n\",\n \"model.config.use_cache = True\\n\",\n \"\\n\",\n \"traced_encoder = trace_encoder(model, tokenizer, max_length, num_beams)\\n\",\n \"traced_decoder = trace_decoder(model, num_beams, max_length)\\n\",\n \"\\n\",\n \"torch.jit.save(traced_encoder, \\\"TracedEncoder.pt\\\")\\n\",\n \"torch.jit.save(traced_decoder, \\\"TracedDecoder.pt\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Results:\\n\",\n \"1 Lassen Sie uns gutes Essen essen.\\n\",\n \"2 Lassen Sie uns gutes Essen zu essen.\\n\",\n \"3 Lassen Sie uns essen gutes Essen.\\n\",\n \"4 Lassen Sie uns gutes Essen.\\n\"\n ]\n }\n ],\n \"source\": [\n \"tokenizer = T5Tokenizer.from_pretrained(model_name)\\n\",\n \"model = T5Wrapper.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"model.encoder = torch.jit.load(\\\"TracedEncoder.pt\\\")\\n\",\n \"# Attribute required by beam search\\n\",\n \"setattr(model.encoder, 'main_input_name', 'input_ids') \\n\",\n \"\\n\",\n \"model.decoder = torch.jit.load(\\\"TracedDecoder.pt\\\")\\n\",\n \"torch_neuronx.move_trace_to_device(model.decoder, 0)\\n\",\n \"\\n\",\n \"\\n\",\n \"output = model.generate(tokenizer=tokenizer,\\n\",\n \" prompt=\\\"translate English to German: Lets eat good food.\\\",\\n\",\n \" max_length=max_length,\\n\",\n \" num_beams=num_beams,\\n\",\n \" num_return_sequences=num_return_sequences,\\n\",\n \" device=\\\"xla\\\")\\n\",\n \"\\n\",\n \"results = [tokenizer.decode(t, skip_special_tokens=True) for t in output]\\n\",\n \"\\n\",\n \"print('Results:')\\n\",\n \"for i, summary in enumerate(results):\\n\",\n \" print(i + 1, summary)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"venv\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.10\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n" }, { "path": "src/examples/pytorch/torchserve/benchmark_bert.py", "content": "import os\nimport argparse\nimport time\nimport numpy as np\nimport requests\nimport sys\nfrom concurrent import futures\n\nimport torch\n\n\nparser = argparse.ArgumentParser()\nparser.add_argument('--url', help='Torchserve model URL', type=str, default=f'http://127.0.0.1:8080/predictions/bert-max_length128-batch_size6')\nparser.add_argument('--num_thread', type=int, default=64, help='Number of threads invoking the model URL')\nparser.add_argument('--batch_size', type=int, default=6)\nparser.add_argument('--sequence_length', type=int, default=128)\nparser.add_argument('--latency_window_size', type=int, default=1000)\nparser.add_argument('--throughput_time', type=int, default=300)\nparser.add_argument('--throughput_interval', type=int, default=10)\nargs = parser.parse_args()\n\ndata = { 'seq_0': 'A completely made up sentence.',\n 'seq_1': 'Well, I suppose they are all made up.' }\nlive = True\nnum_infer = 0\nlatency_list = []\n\n\ndef one_thread(pred, feed_data):\n global latency_list\n global num_infer\n global live\n session = requests.Session()\n while True:\n start = time.time()\n result = session.post(pred, data=feed_data)\n latency = time.time() - start\n latency_list.append(latency)\n num_infer += 1\n if not live:\n break\n\n\ndef current_performance():\n last_num_infer = num_infer\n for _ in range(args.throughput_time // args.throughput_interval):\n current_num_infer = num_infer\n throughput = (current_num_infer - last_num_infer) / args.throughput_interval\n p50 = 0.0\n p90 = 0.0\n if latency_list:\n p50 = np.percentile(latency_list[-args.latency_window_size:], 50)\n p90 = np.percentile(latency_list[-args.latency_window_size:], 90)\n print('pid {}: current throughput {}, latency p50={:.3f} p90={:.3f}'.format(os.getpid(), throughput, p50, p90))\n sys.stdout.flush()\n last_num_infer = current_num_infer\n time.sleep(args.throughput_interval)\n global live\n live = False\n\n\nwith futures.ThreadPoolExecutor(max_workers=args.num_thread+1) as executor:\n executor.submit(current_performance)\n for _ in range(args.num_thread):\n executor.submit(one_thread, args.url, data)" }, { "path": "src/examples/pytorch/torchserve/config.json", "content": "{\n \"model_name\": \"bert-base-cased-finetuned-mrpc\",\n \"max_length\": 128,\n \"batch_size\": 6\n}" }, { "path": "src/examples/pytorch/torchserve/handler_bert.py", "content": "import os\nimport json\nimport sys\nimport logging\nfrom abc import ABC\n\nimport torch\nimport torch_neuron\n\nfrom transformers import AutoTokenizer\nfrom ts.torch_handler.base_handler import BaseHandler\n\n\n# one core per worker\nos.environ['NEURON_RT_NUM_CORES'] = '1'\n\nlogger = logging.getLogger(__name__)\n\nclass BertEmbeddingHandler(BaseHandler, ABC):\n \"\"\"\n Handler class for Bert Embedding computations.\n \"\"\"\n def __init__(self):\n super(BertEmbeddingHandler, self).__init__()\n self.initialized = False\n\n def initialize(self, ctx):\n self.manifest = ctx.manifest\n properties = ctx.system_properties\n self.device = 'cpu'\n model_dir = properties.get('model_dir')\n serialized_file = self.manifest['model']['serializedFile']\n model_pt_path = os.path.join(model_dir, serialized_file)\n\n # point sys.path to our config file\n with open('config.json') as fp:\n config = json.load(fp)\n self.max_length = config['max_length']\n self.batch_size = config['batch_size']\n self.classes = ['not paraphrase', 'paraphrase']\n\n self.model = torch.jit.load(model_pt_path)\n logger.debug(f'Model loaded from {model_dir}')\n self.model.to(self.device)\n self.model.eval()\n\n self.tokenizer = AutoTokenizer.from_pretrained(config['model_name'])\n self.initialized = True\n\n def preprocess(self, input_data):\n \"\"\"\n Tokenization pre-processing\n \"\"\"\n\n input_ids = []\n attention_masks = []\n token_type_ids = []\n for row in input_data:\n seq_0 = row['seq_0'].decode('utf-8')\n seq_1 = row['seq_1'].decode('utf-8')\n logger.debug(f'Received text: \"{seq_0}\", \"{seq_1}\"')\n\n inputs = self.tokenizer.encode_plus(\n seq_0,\n seq_1,\n max_length=self.max_length,\n padding='max_length',\n truncation=True,\n return_tensors='pt'\n )\n\n input_ids.append(inputs['input_ids'])\n attention_masks.append(inputs['attention_mask'])\n token_type_ids.append(inputs['token_type_ids'])\n\n batch = (torch.cat(input_ids, 0),\n torch.cat(attention_masks, 0),\n torch.cat(token_type_ids, 0))\n\n return batch\n\n def inference(self, inputs):\n \"\"\"\n Predict the class of a text using a trained transformer model.\n \"\"\"\n\n # sanity check dimensions\n assert(len(inputs) == 3)\n num_inferences = len(inputs[0])\n assert(num_inferences <= self.batch_size)\n\n # insert padding if we received a partial batch\n padding = self.batch_size - num_inferences\n if padding > 0:\n pad = torch.nn.ConstantPad1d((0, 0, 0, padding), value=0)\n inputs = [pad(x) for x in inputs]\n\n outputs = self.model(*inputs)[0]\n predictions = []\n for i in range(num_inferences):\n prediction = self.classes[outputs[i].argmax().item()]\n predictions.append([prediction])\n logger.debug(\"Model predicted: '%s'\", prediction)\n return predictions\n\n def postprocess(self, inference_output):\n return inference_output\n" }, { "path": "src/examples/pytorch/torchserve/handler_bert_neuronx.py", "content": "import os\nimport json\nimport sys\nimport logging\nfrom abc import ABC\n\nimport torch\nimport torch_neuronx\n\nfrom transformers import AutoTokenizer\nfrom ts.torch_handler.base_handler import BaseHandler\n\n\n# one core per worker\nos.environ['NEURON_RT_NUM_CORES'] = '1'\n\nlogger = logging.getLogger(__name__)\n\nclass BertEmbeddingHandler(BaseHandler, ABC):\n \"\"\"\n Handler class for Bert Embedding computations.\n \"\"\"\n def __init__(self):\n super(BertEmbeddingHandler, self).__init__()\n self.initialized = False\n\n def initialize(self, ctx):\n self.manifest = ctx.manifest\n properties = ctx.system_properties\n self.device = 'cpu'\n model_dir = properties.get('model_dir')\n serialized_file = self.manifest['model']['serializedFile']\n model_pt_path = os.path.join(model_dir, serialized_file)\n\n # point sys.path to our config file\n with open('config.json') as fp:\n config = json.load(fp)\n self.max_length = config['max_length']\n self.batch_size = config['batch_size']\n self.classes = ['not paraphrase', 'paraphrase']\n\n self.model = torch.jit.load(model_pt_path)\n logger.debug(f'Model loaded from {model_dir}')\n self.model.to(self.device)\n self.model.eval()\n\n self.tokenizer = AutoTokenizer.from_pretrained(config['model_name'])\n self.initialized = True\n\n def preprocess(self, input_data):\n \"\"\"\n Tokenization pre-processing\n \"\"\"\n\n input_ids = []\n attention_masks = []\n token_type_ids = []\n for row in input_data:\n seq_0 = row['seq_0'].decode('utf-8')\n seq_1 = row['seq_1'].decode('utf-8')\n logger.debug(f'Received text: \"{seq_0}\", \"{seq_1}\"')\n\n inputs = self.tokenizer.encode_plus(\n seq_0,\n seq_1,\n max_length=self.max_length,\n padding='max_length',\n truncation=True,\n return_tensors='pt'\n )\n\n input_ids.append(inputs['input_ids'])\n attention_masks.append(inputs['attention_mask'])\n token_type_ids.append(inputs['token_type_ids'])\n\n batch = (torch.cat(input_ids, 0),\n torch.cat(attention_masks, 0),\n torch.cat(token_type_ids, 0))\n\n return batch\n\n def inference(self, inputs):\n \"\"\"\n Predict the class of a text using a trained transformer model.\n \"\"\"\n\n # sanity check dimensions\n assert(len(inputs) == 3)\n num_inferences = len(inputs[0])\n assert(num_inferences <= self.batch_size)\n\n # insert padding if we received a partial batch\n padding = self.batch_size - num_inferences\n if padding > 0:\n pad = torch.nn.ConstantPad1d((0, 0, 0, padding), value=0)\n inputs = [pad(x) for x in inputs]\n\n outputs = self.model(*inputs)[0]\n predictions = []\n for i in range(num_inferences):\n prediction = self.classes[outputs[i].argmax(dim=-1).item()]\n predictions.append([prediction])\n logger.debug(\"Model predicted: '%s'\", prediction)\n return predictions\n\n def postprocess(self, inference_output):\n return inference_output\n" }, { "path": "src/examples/pytorch/torchserve/infer_bert.py", "content": "import json\nimport concurrent.futures\nimport requests\n\nwith open('config.json') as fp:\n config = json.load(fp)\nmax_length = config['max_length']\nbatch_size = config['batch_size']\nname = f'bert-max_length{max_length}-batch_size{batch_size}'\n\n# dispatch requests in parallel\nurl = f'http://localhost:8080/predictions/{name}'\nparaphrase = {'seq_0': \"HuggingFace's headquarters are situated in Manhattan\",\n 'seq_1': \"The company HuggingFace is based in New York City\"}\nnot_paraphrase = {'seq_0': paraphrase['seq_0'], 'seq_1': 'This is total nonsense.'}\n\nwith concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as executor:\n def worker_thread(worker_index):\n # we'll send half the requests as not_paraphrase examples for sanity\n data = paraphrase if worker_index < batch_size//2 else not_paraphrase\n try:\n response = requests.post(url, data=data)\n\n # Check if the response status code indicates success\n if response.status_code == 200:\n print(worker_index, response.json())\n else:\n # If the response is not successful, raise an exception with the status code and error message\n error_message = response.json().get('message', 'Unknown Error')\n raise Exception(f\"Failed request with status code {response.status_code}: {error_message}\")\n except Exception as e:\n # Catch all other exceptions that may be raised\n print(f\"An unexpected error occurred: {e}\")\n raise\n\n for worker_index in range(batch_size):\n executor.submit(worker_thread, worker_index)\n" }, { "path": "src/examples/pytorch/torchserve/torchserve.config", "content": "# bind inference API to all network interfaces with SSL enabled\ninference_address=http://0.0.0.0:8080\ndefault_workers_per_model=1" }, { "path": "src/examples/pytorch/torchserve/trace_bert_neuron.py", "content": "import torch\nimport torch_neuron\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Build tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased-finetuned-mrpc\")\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased-finetuned-mrpc\", return_dict=False)\n\n# Setup some example inputs\nsequence_0 = \"The company HuggingFace is based in New York City\"\nsequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n\nmax_length = 128\nbatch_size = 6\n\nparaphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\"pt\")\n\nexample_inputs_paraphrase = (\n torch.cat([paraphrase['input_ids']] * batch_size, 0),\n torch.cat([paraphrase['attention_mask']] * batch_size, 0),\n torch.cat([paraphrase['token_type_ids']] * batch_size, 0)\n)\n\n# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\nmodel_neuron_batch = torch_neuron.trace(model, example_inputs_paraphrase)\n\n# Save the batched model\nmodel_neuron_batch.save('bert_neuron_b{}.pt'.format(batch_size))\n" }, { "path": "src/examples/pytorch/torchserve/trace_bert_neuronx.py", "content": "import torch\nimport torch_neuronx\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n\n# Build tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased-finetuned-mrpc\")\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased-finetuned-mrpc\", return_dict=False)\n\n# Setup some example inputs\nsequence_0 = \"The company HuggingFace is based in New York City\"\nsequence_1 = \"HuggingFace's headquarters are situated in Manhattan\"\n\nmax_length = 128\nbatch_size = 6\n\nparaphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors=\"pt\")\n\nexample_inputs_paraphrase = (\n torch.cat([paraphrase['input_ids']] * batch_size, 0),\n torch.cat([paraphrase['attention_mask']] * batch_size, 0),\n torch.cat([paraphrase['token_type_ids']] * batch_size, 0)\n)\n\n# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuron\nmodel_neuron_batch = torch_neuronx.trace(model, example_inputs_paraphrase)\n\n# Save the batched model\nmodel_neuron_batch.save('bert_neuron_b{}.pt'.format(batch_size))\n" }, { "path": "src/examples/pytorch/transformers-marianmt.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Transformers MarianMT Tutorial\\n\",\n \"\\n\",\n \"In this tutorial, you will deploy the [HuggingFace MarianMT](https://huggingface.co/transformers/v4.0.1/model_doc/marian.html) model for text translation.\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on an inf1.6xlarge instance since you will be loading and compiling several large models.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [PyTorch Installation Guide](../../../frameworks/torch/torch-neuron/setup/pytorch-install.html). You can select the kernel from the \\\"Kernel -> Change Kernel\\\" option on the top of this Jupyter notebook page.\\n\",\n \"\\n\",\n \"To generate text, you will be using the beam search algorithm to incrementally generate token candidates until the full output text has been created. Unlike simple single-pass models, this algorithm divides the work into two distinct phases:\\n\",\n \"\\n\",\n \"- **Encoder**: Convert the input text into an encoded representation. (Executed once)\\n\",\n \"- **Decoder**: Use the encoded representation of the input text and the current output tokens to incrementally generate the set of next best candidate tokens. (Executed many times)\\n\",\n \"\\n\",\n \"In this tutorial you will perform the following steps:\\n\",\n \"\\n\",\n \"- **Compile**: Compile both the Encoder and Decoder for Neuron using simplified interfaces for inference.\\n\",\n \"- **Infer**: Run on CPU and Neuron and compare results.\\n\",\n \"\\n\",\n \"Finally, a completely unrolled decoder will be built which simplifies the implementation at the cost of performing fixed-length inferences.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\\n\",\n \"\\n\",\n \"This tutorial has the following dependencies:\\n\",\n \"\\n\",\n \"- `transformers==4.25.1`\\n\",\n \"- `torch-neuron`\\n\",\n \"- `sentencepiece`\\n\",\n \"- `neuron-cc[tensorflow]`\\n\",\n \"\\n\",\n \"The following will install the required `transformers` version. Note that encoder/decoder API changes across different minor versions requires that you are specific about the version used. Also note that the `torch-neuron` version is pinned due to `transformer` compatibility issues.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install sentencepiece transformers==4.26.1\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Parameters\\n\",\n \"\\n\",\n \"The parameters of a generative model can be tuned for different use-cases. In this example, you'll tailor the parameters to a single inference beam search for an on-demand inference use-case. See the [MarianConfig](https://huggingface.co/transformers/v4.0.1/model_doc/marian.html#marianconfig) for parameter details.\\n\",\n \"\\n\",\n \"Rather than varying the encoder/decoder token sizes at runtime, you must define these parameters prior to compilation. The encoder/decoder token sizes are important tunable parameters as a large token sequence will offer greater sentence length flexibility but perform worse than a small token sequence.\\n\",\n \"\\n\",\n \"To maximize performance on Neuron, the `num_beams`, `max_encode_length` and `max_decoder_length` should be made as small as possible for the use-case.\\n\",\n \"\\n\",\n \"For this tutorial you will use a model that translates sentences of up to 32 token from English to German.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"model_name = \\\"Helsinki-NLP/opus-mt-en-de\\\" # English -> German model\\n\",\n \"num_texts = 1 # Number of input texts to decode\\n\",\n \"num_beams = 4 # Number of beams per input text\\n\",\n \"max_encoder_length = 32 # Maximum input token length\\n\",\n \"max_decoder_length = 32 # Maximum output token length\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## CPU Model Inference\\n\",\n \"\\n\",\n \"Start by executing the model on CPU to test its execution.\\n\",\n \"\\n\",\n \"The following defines the inference function which will be used to compare the Neuron and CPU output. In this example you will display all beam search sequences that were generated. For a real on-demand use case, set the `num_beams` to `1` to return only the top result.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def infer(model, tokenizer, text):\\n\",\n \"\\n\",\n \" # Truncate and pad the max length to ensure that the token size is compatible with fixed-sized encoder (Not necessary for pure CPU execution)\\n\",\n \" batch = tokenizer(text, max_length=max_decoder_length, truncation=True, padding='max_length', return_tensors=\\\"pt\\\")\\n\",\n \" output = model.generate(**batch, max_length=max_decoder_length, num_beams=num_beams, num_return_sequences=num_beams)\\n\",\n \" results = [tokenizer.decode(t, skip_special_tokens=True) for t in output]\\n\",\n \"\\n\",\n \" print('Texts:')\\n\",\n \" for i, summary in enumerate(results):\\n\",\n \" print(i + 1, summary)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Note that after loading the model, we also set the maximum length. This will later be used to limit the size of the compiled model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import MarianMTModel, MarianTokenizer\\n\",\n \"\\n\",\n \"model_cpu = MarianMTModel.from_pretrained(model_name)\\n\",\n \"model_cpu.config.max_length = max_decoder_length\\n\",\n \"model_cpu.eval()\\n\",\n \"\\n\",\n \"tokenizer = MarianTokenizer.from_pretrained(model_name)\\n\",\n \"\\n\",\n \"sample_text = \\\"I am a small frog.\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"infer(model_cpu, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Padded Model\\n\",\n \"In order to perform inference on Neuron, the model must be changed in a way that it supports tracing and fixed-sized inputs. One way in which this is possible is to use a pad the model inputs to the maximum possible tensor sizes. The benefit of using a padded model is that it supports variable length text generation up to a specified length `max_decoder_length`. A consequence of padding is that it can negatively impact performance due to large data transfers.\\n\",\n \"\\n\",\n \"### PaddedEncoder & PaddedDecoder Modules\\n\",\n \"Here you will define wrappers around the encoder and decoder portions of the generation model that are compatible with `torch.jit.trace` as well as fixed-sized inputs.\\n\",\n \"\\n\",\n \"The following are important features which are distinct from the default configuration:\\n\",\n \"\\n\",\n \"1. Disabled `return_dict`. When this is enabled, the network uses `dataclass` type outputs which are not compatible with `torch.jit.trace`.\\n\",\n \"2. Disabled `use_cache`. When this option is enabled, the network expects a collection of cache tensors which grow upon each iteration. Since Neuron requires fixed sized inputs, this must be disabled.\\n\",\n \"3. The `GenerationMixin:beam_search` implementation uses only the logits for the current iteration index from the original decoder layer output. Since inputs must be padded, performance can be improved by selecting only a subset of the hidden state prior to the final linear layer. For efficiency on Neuron, this reduction uses an elementwise-multiply to mask out the unused hidden values and then sums along an axis.\\n\",\n \"4. Since a reduction step is insterted between the decoder output and the final logit calculation, the original `model` attribute is not used. Instead the `PaddedDecoder` class combines the decoder, reducer, and linear layers into a combined forward pass. In the original model there is a clear distinction between the decoder layer and the final linear layer. These layers are fused together to get one large fully optimized graph.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"from torch.nn import functional as F\\n\",\n \"\\n\",\n \"\\n\",\n \"class PaddedEncoder(torch.nn.Module):\\n\",\n \"\\n\",\n \" def __init__(self, model):\\n\",\n \" super().__init__()\\n\",\n \" self.encoder = model.model.encoder\\n\",\n \" self.main_input_name = 'input_ids'\\n\",\n \" \\n\",\n \" def forward(self, input_ids, attention_mask):\\n\",\n \" return self.encoder(input_ids, attention_mask=attention_mask, return_dict=False)\\n\",\n \"\\n\",\n \"\\n\",\n \"class PaddedDecoder(torch.nn.Module):\\n\",\n \"\\n\",\n \" def __init__(self, model):\\n\",\n \" super().__init__()\\n\",\n \" self.weight = model.model.shared.weight.clone().detach()\\n\",\n \" self.bias = model.final_logits_bias.clone().detach()\\n\",\n \" self.decoder = model.model.decoder\\n\",\n \"\\n\",\n \" def forward(self, input_ids, attention_mask, encoder_outputs, index):\\n\",\n \"\\n\",\n \" # Invoke the decoder\\n\",\n \" hidden, = self.decoder(\\n\",\n \" input_ids=input_ids,\\n\",\n \" encoder_hidden_states=encoder_outputs,\\n\",\n \" encoder_attention_mask=attention_mask,\\n\",\n \" return_dict=False,\\n\",\n \" use_cache=False,\\n\",\n \" )\\n\",\n \"\\n\",\n \" _, n_length, _ = hidden.shape\\n\",\n \"\\n\",\n \" # Create selection mask\\n\",\n \" mask = torch.arange(n_length, dtype=torch.float32) == index\\n\",\n \" mask = mask.view(1, -1, 1)\\n\",\n \"\\n\",\n \" # Broadcast mask\\n\",\n \" masked = torch.multiply(hidden, mask)\\n\",\n \"\\n\",\n \" # Reduce along 1st dimension\\n\",\n \" hidden = torch.sum(masked, 1, keepdims=True)\\n\",\n \"\\n\",\n \" # Compute final linear layer for token probabilities\\n\",\n \" logits = F.linear(\\n\",\n \" hidden,\\n\",\n \" self.weight,\\n\",\n \" bias=self.bias\\n\",\n \" )\\n\",\n \" return logits\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### PaddedGenerator - GenerationMixin Class\\n\",\n \"\\n\",\n \"\\n\",\n \"On text generation tasks, HuggingFace Transformers defines a [GenerationMixin](https://huggingface.co/transformers/v4.0.1/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin) base class which provides standard methods and algorithms to generate text. For this tutorial, you will be using the beam search algorithm on encoder/decoder architectures.\\n\",\n \"\\n\",\n \"To be able to use these methods, you will be defining your own class derived from the GenerationMixin class to run a beam search. This will invoke the encoder and decoder layers in a way that is compatible with fixed sized inputs and traced modules. This means you must import the base class and the output objects ([Seq2SeqLMOutput](https://huggingface.co/transformers/v4.0.1/main_classes/output.html#transformers.modeling_outputs.Seq2SeqLMOutput), [BaseModelOutput](https://huggingface.co/transformers/v4.0.1/main_classes/output.html#transformers.modeling_outputs.BaseModelOutput)) used by the [beam_search](https://huggingface.co/transformers/v4.0.1/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.beam_search) algorithm.\\n\",\n \"\\n\",\n \"The `GenerationMixin:generate` method will use `GenerationMixin:beam_search` which requires that you to define your own class implementation that invokes the `PaddedEncoder` and `PaddedDecoder` modules using padded inputs. The standard generator model implementation will not work by default because it is intended to infer with variable-sized (growing) input tensors. \\n\",\n \"\\n\",\n \"The `from_model` method is defined to create the `PaddedGenerator` from an existing pretrained generator class.\\n\",\n \"\\n\",\n \"To invoke the Encoder and Decoder traced modules in a way that is compatible with the `GenerationMixin:beam_search` implementation, the `get_encoder`, `__call__`, and `prepare_inputs_for_generation` methods are overriden.\\n\",\n \"\\n\",\n \"Lastly, the class defines methods for serialization so that the model can be easily saved and loaded.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"\\n\",\n \"from transformers import GenerationMixin, AutoConfig\\n\",\n \"from transformers.modeling_outputs import Seq2SeqLMOutput, BaseModelOutput\\n\",\n \"from transformers.modeling_utils import PreTrainedModel\\n\",\n \"\\n\",\n \"\\n\",\n \"class PaddedGenerator(PreTrainedModel, GenerationMixin):\\n\",\n \"\\n\",\n \" @classmethod\\n\",\n \" def from_model(cls, model):\\n\",\n \" generator = cls(model.config)\\n\",\n \" generator.encoder = PaddedEncoder(model)\\n\",\n \" generator.decoder = PaddedDecoder(model)\\n\",\n \" return generator\\n\",\n \" \\n\",\n \" def prepare_inputs_for_generation(\\n\",\n \" self,\\n\",\n \" input_ids,\\n\",\n \" encoder_outputs=None,\\n\",\n \" attention_mask=None,\\n\",\n \" **kwargs,\\n\",\n \" ):\\n\",\n \" # Pad the inputs for Neuron\\n\",\n \" current_length = input_ids.shape[1]\\n\",\n \" pad_size = self.config.max_length - current_length\\n\",\n \" return dict(\\n\",\n \" input_ids=F.pad(input_ids, (0, pad_size)),\\n\",\n \" attention_mask=attention_mask,\\n\",\n \" encoder_outputs=encoder_outputs.last_hidden_state,\\n\",\n \" current_length=torch.tensor(current_length - 1),\\n\",\n \" )\\n\",\n \"\\n\",\n \" def get_encoder(self):\\n\",\n \" def encode(input_ids, attention_mask, **kwargs): \\n\",\n \" output, = self.encoder(input_ids, attention_mask)\\n\",\n \" return BaseModelOutput(\\n\",\n \" last_hidden_state=output,\\n\",\n \" )\\n\",\n \" return encode\\n\",\n \"\\n\",\n \" def forward(self, input_ids, attention_mask, encoder_outputs, current_length, **kwargs):\\n\",\n \" logits = self.decoder(input_ids, attention_mask, encoder_outputs, current_length)\\n\",\n \" return Seq2SeqLMOutput(logits=logits)\\n\",\n \"\\n\",\n \" @property\\n\",\n \" def device(self): # Attribute required by beam search\\n\",\n \" return torch.device('cpu')\\n\",\n \" \\n\",\n \" def save_pretrained(self, directory):\\n\",\n \" if os.path.isfile(directory):\\n\",\n \" print(f\\\"Provided path ({directory}) should be a directory, not a file\\\")\\n\",\n \" return\\n\",\n \" os.makedirs(directory, exist_ok=True)\\n\",\n \" torch.jit.save(self.encoder, os.path.join(directory, 'encoder.pt'))\\n\",\n \" torch.jit.save(self.decoder, os.path.join(directory, 'decoder.pt'))\\n\",\n \" self.config.save_pretrained(directory)\\n\",\n \"\\n\",\n \" @classmethod\\n\",\n \" def from_pretrained(cls, directory):\\n\",\n \" config = AutoConfig.from_pretrained(directory)\\n\",\n \" obj = cls(config)\\n\",\n \" obj.encoder = torch.jit.load(os.path.join(directory, 'encoder.pt'))\\n\",\n \" obj.decoder = torch.jit.load(os.path.join(directory, 'decoder.pt'))\\n\",\n \" setattr(obj.encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\\n\",\n \" return obj\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Padded CPU Inference\\n\",\n \"To start, it is important to ensure that the transformations we have made to the model were successful. Using the classes defined above we can test that the padded model execution on CPU is identical to the original output also running on CPU.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"padded_model_cpu = PaddedGenerator.from_model(model_cpu)\\n\",\n \"infer(padded_model_cpu, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Padded Neuron Tracing & Inference\\n\",\n \"\\n\",\n \"Now that the padded version of model is confirmed to produce the same outputs as the non-padded version, the model can be compiled for Neuron.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch_neuron\\n\",\n \"\\n\",\n \"\\n\",\n \"def trace(model, num_texts, num_beams, max_decoder_length, max_encoder_length):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Traces the encoder and decoder modules for use on Neuron.\\n\",\n \"\\n\",\n \" This function fixes the network to the given sizes. Once the model has been\\n\",\n \" compiled to a given size, the inputs to these networks must always be of\\n\",\n \" fixed size.\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" model (PaddedGenerator): The padded generator to compile for Neuron\\n\",\n \" num_texts (int): The number of input texts to translate at once\\n\",\n \" num_beams (int): The number of beams to compute per text\\n\",\n \" max_decoder_length (int): The maximum number of tokens to be generated\\n\",\n \" max_encoder_length (int): The maximum number of input tokens that will be encoded\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" # Trace the encoder\\n\",\n \" inputs = (\\n\",\n \" torch.ones((num_texts, max_encoder_length), dtype=torch.long),\\n\",\n \" torch.ones((num_texts, max_encoder_length), dtype=torch.long),\\n\",\n \" )\\n\",\n \" encoder = torch_neuron.trace(model.encoder, inputs)\\n\",\n \"\\n\",\n \" # Trace the decoder (with expanded inputs)\\n\",\n \" batch_size = num_texts * num_beams\\n\",\n \" inputs = (\\n\",\n \" torch.ones((batch_size, max_decoder_length), dtype=torch.long),\\n\",\n \" torch.ones((batch_size, max_encoder_length), dtype=torch.long),\\n\",\n \" torch.ones((batch_size, max_encoder_length, model.config.d_model), dtype=torch.float),\\n\",\n \" torch.tensor(0),\\n\",\n \" )\\n\",\n \" decoder = torch_neuron.trace(model.decoder, inputs)\\n\",\n \" \\n\",\n \" traced = PaddedGenerator(model.config)\\n\",\n \" traced.encoder = encoder\\n\",\n \" traced.decoder = decoder\\n\",\n \" setattr(encoder, 'main_input_name', 'input_ids') # Attribute required by beam search\\n\",\n \" return traced\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"padded_model_neuron = trace(padded_model_cpu, num_texts, num_beams, max_decoder_length, max_encoder_length)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Comparing the Neuron execution to the original CPU implementation, you will see the exact same generated text.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# CPU execution for comparison\\n\",\n \"infer(padded_model_neuron, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Padded Neuron Serialization\\n\",\n \"Finally, we can test that we can serialize and reload the model so that it can be used later in its precompiled format.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"padded_model_neuron.save_pretrained('NeuronPaddedMarianMT')\\n\",\n \"padded_model_loaded = PaddedGenerator.from_pretrained('NeuronPaddedMarianMT')\\n\",\n \"infer(padded_model_loaded, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Greedy Unrolled Model\\n\",\n \"An unrolled version of the model can achieve better performance in some cases since all operations will be executed on the Neuron hardware without returning to CPU. The consequence of this type of model is that since the generation loop execution never returns to CPU, the entire sequence up to `max_decoder_length` is performed in a single forward pass.\\n\",\n \"\\n\",\n \"The following module performs greedy text generation. Unlike the original beam search text generation, this implementation always selects the most probable token and does not generate multiple result texts.\\n\",\n \"\\n\",\n \"### GreedyUnrolledGenerator Module\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"class GreedyUnrolledGenerator(torch.nn.Module):\\n\",\n \" \\n\",\n \" def __init__(self, model):\\n\",\n \" super().__init__()\\n\",\n \" self.config = model.config\\n\",\n \" self.model = model\\n\",\n \" \\n\",\n \" def forward(self, input_ids, attention_mask):\\n\",\n \" \\n\",\n \" # Generate the encoder state for the input tokens. This is only done once and the state is reused.\\n\",\n \" encoder_outputs, = self.model.model.encoder(input_ids, attention_mask=attention_mask, return_dict=False)\\n\",\n \" \\n\",\n \" # Set the intial state for the decode loop. This will grow per decoder iteration\\n\",\n \" tokens = torch.full((input_ids.size(0), 2), self.config.decoder_start_token_id)\\n\",\n \" \\n\",\n \" # Iteratively invoke the decoder on incrementally generated `tokens` to generate a `next_token`.\\n\",\n \" # Note that unlike the GeneratorMixin.generate function, there is no early-exit if the stop token \\n\",\n \" # has been reached. This will always run a fixed number of iterations.\\n\",\n \" for i in range(self.config.max_length):\\n\",\n \" \\n\",\n \" hidden, = self.model.model.decoder(\\n\",\n \" input_ids=tokens,\\n\",\n \" encoder_hidden_states=encoder_outputs,\\n\",\n \" encoder_attention_mask=attention_mask,\\n\",\n \" return_dict=False,\\n\",\n \" use_cache=False,\\n\",\n \" ) # size: [batch, current_length, vocab_size]\\n\",\n \" \\n\",\n \" logits = F.linear(\\n\",\n \" hidden[:, -1, :],\\n\",\n \" self.model.model.shared.weight,\\n\",\n \" bias=self.model.final_logits_bias\\n\",\n \" )\\n\",\n \" next_tokens = torch.argmax(logits, dim=1, keepdims=True)\\n\",\n \" tokens = torch.cat([tokens, next_tokens], dim=1)\\n\",\n \" \\n\",\n \" return tokens\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Greedy CPU Inference\\n\",\n \"The inference code must be updated since the `generate` method is no longer used. This is because the entire generative inference loop occurs within the `GreedyUnrolledGenerator.forward` method.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def infer_greedy(model, tokenizer, text):\\n\",\n \" batch = tokenizer(text, max_length=max_decoder_length, truncation=True, padding='max_length', return_tensors=\\\"pt\\\")\\n\",\n \" inputs = batch['input_ids'], batch['attention_mask']\\n\",\n \" tokens = greedy_cpu(*inputs)\\n\",\n \" print('Texts:')\\n\",\n \" for i, t in enumerate(tokens):\\n\",\n \" result = tokenizer.decode(t, skip_special_tokens=True)\\n\",\n \" print(i + 1, result)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Like in previous section of this tutorial, first the greedy model is executed on CPU to validate that the correct results were produced. In this example, the generated text matches the first result of the original beam search.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model_cpu.config.max_length = 8 # This controls the number of decoder loops. Reduced to improve compilation speed.\\n\",\n \"greedy_cpu = GreedyUnrolledGenerator(model_cpu)\\n\",\n \"infer_greedy(greedy_cpu, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Greedy Neuron Tracing & Inference\\n\",\n \"Similarly the tracing is simplified since the now the `GreedyUnrolledGenerator.forward` can be compiled as a single unit. \\n\",\n \"\\n\",\n \"For compilation efficiency, two changes will be made compared to normal compilaition:\\n\",\n \"- `torch.jit.freeze` is used because it can *sometimes* speed up compilation by in the case where a module is re-used multiple times. In this case, it is more efficient because the `self.model.model.decoder` is used in a loop. \\n\",\n \"- The `torch_neuron.trace` option `fallback` is set to `False`. This forces all operations to execute on Neuron. Most of the time this is not recommended or efficient. In this case, it is more efficient because it means a single subgraph is produced rather than many. Usually one subgraph would be produced per decoder iteration since `aten::embedding` is executed in a loop. The `aten::embedding` operation is otherwise exected on CPU by default since this is usually more efficient than executing on Neuron.\\n\",\n \"\\n\",\n \"You may notice that compilation will take significantly longer with the unrolled model since the model inserts new operations into the compute graph for every single decoder iteration. This creates a much larger model graph even though the weights are re-used.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"example = (\\n\",\n \" torch.ones((num_texts, max_encoder_length), dtype=torch.long),\\n\",\n \" torch.ones((num_texts, max_encoder_length), dtype=torch.long),\\n\",\n \")\\n\",\n \"greedy_cpu.eval()\\n\",\n \"greedy_trace = torch.jit.trace(greedy_cpu, example)\\n\",\n \"greedy_frozen = torch.jit.freeze(greedy_trace)\\n\",\n \"greedy_neuron = torch_neuron.trace(greedy_frozen, example, fallback=False)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"infer_greedy(greedy_neuron, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Greedy Neuron Serialization\\n\",\n \"Unlike the previous version of the model that used the `GenerationMixin` base class. This greedy version of the model can be serialized using the regular `torch.jit.save` and `torch.jit.load` utilities since it is a pure torchscript module.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"torch.jit.save(greedy_neuron, 'greedy_neuron.pt')\\n\",\n \"loaded_greedy_neuron = torch.jit.load('greedy_neuron.pt')\\n\",\n \"infer_greedy(loaded_greedy_neuron, tokenizer, sample_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Appendix\\n\",\n \"### BART (Mask Filling Task)\\n\",\n \"\\n\",\n \"These `PaddedGenerator` class can be applied to the BART model for the task of filling in mask tokens.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import BartForConditionalGeneration, BartTokenizer\\n\",\n \"bart_name = \\\"facebook/bart-large\\\"\\n\",\n \"bart_model = BartForConditionalGeneration.from_pretrained(bart_name)\\n\",\n \"bart_model.config.max_length = max_decoder_length\\n\",\n \"bart_tokenizer = BartTokenizer.from_pretrained(bart_name)\\n\",\n \"bart_text = \\\"UN Chief Says There Is No in Syria\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# CPU Execution\\n\",\n \"infer(bart_model, bart_tokenizer, bart_text)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Neuron Execution\\n\",\n \"paddded_bart = PaddedGenerator.from_model(bart_model)\\n\",\n \"bart_neuron = trace(paddded_bart, num_texts, num_beams, max_decoder_length, max_encoder_length)\\n\",\n \"infer(bart_neuron, bart_tokenizer, bart_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Pegasus (Summarization Task)\\n\",\n \"\\n\",\n \"These `PaddedGenerator` class can be applied to the Pegasus model for summarization.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import PegasusForConditionalGeneration, PegasusTokenizer\\n\",\n \"pegasus_name = 'google/pegasus-xsum'\\n\",\n \"pegasus_model = PegasusForConditionalGeneration.from_pretrained(pegasus_name)\\n\",\n \"pegasus_model.config.max_length = max_decoder_length\\n\",\n \"pegasus_tokenizer = PegasusTokenizer.from_pretrained(pegasus_name)\\n\",\n \"pegasus_text = \\\"PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires.\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# CPU Execution\\n\",\n \"infer(pegasus_model, pegasus_tokenizer, pegasus_text)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"#%%\\n\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Neuron Execution\\n\",\n \"paddded_pegasus = PaddedGenerator.from_model(pegasus_model)\\n\",\n \"pegasus_neuron = trace(paddded_pegasus, num_texts, num_beams, max_decoder_length, max_encoder_length)\\n\",\n \"infer(pegasus_neuron, pegasus_tokenizer, pegasus_text)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.10\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/pytorch/yolo_v4.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Evaluate YOLO v4 on Inferentia\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction\\n\",\n \"This tutorial walks through compiling and evaluating YOLO v4 model implemented in PyTorch on Inferentia. \\n\",\n \"\\n\",\n \"The tutorial has five main sections:\\n\",\n \"\\n\",\n \"1. Define YOLO v4 model in PyTorch\\n\",\n \"2. Download the COCO 2017 evaluation dataset and define the data loader function\\n\",\n \"3. Build, Compile, and Save Neuron-Optimized YOLO v4 TorchScript\\n\",\n \"4. Evaluate Accuracy on the COCO 2017 Dataset\\n\",\n \"5. Benchmark COCO Dataset Performance of the Neuron-Optimized TorchScript\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [PyTorch Installation Guide](../../../frameworks/torch/torch-neuron/setup/pytorch-install.html). You can select the kernel from the \\\"Kernel -> Change Kernel\\\" option on the top of this Jupyter notebook page.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies:\\n\",\n \"This tutorial requires the following pip packages:\\n\",\n \"\\n\",\n \"- `torch-neuron`\\n\",\n \"- `torchvision`\\n\",\n \"- `pillow`\\n\",\n \"- `pycocotools`\\n\",\n \"- `neuron-cc[tensorflow]`\\n\",\n \"\\n\",\n \"Many of these packages will be installed by default when configuring your environment using the Neuron PyTorch setup guide. The additional dependencies must be installed here.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install --upgrade pillow pycocotools \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 1: Define YOLO v4 model in PyTorch \\n\",\n \"The following PyTorch model definition is from https://github.com/Tianxiaomo/pytorch-YOLOv4/.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import numpy as np\\n\",\n \"import torch\\n\",\n \"import torch.neuron\\n\",\n \"from torch import nn\\n\",\n \"import torch.nn.functional as F\\n\",\n \"import os\\n\",\n \"import warnings\\n\",\n \"\\n\",\n \"# Setting up NeuronCore groups for inf1.6xlarge with 16 cores\\n\",\n \"n_cores = 16 # This value should be 4 on inf1.xlarge and inf1.2xlarge\\n\",\n \"os.environ['NEURON_RT_NUM_CORES'] = str(n_cores)\\n\",\n \"\\n\",\n \"\\n\",\n \"class Mish(torch.nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \"\\n\",\n \" def forward(self, x):\\n\",\n \" x = x * (torch.tanh(torch.nn.functional.softplus(x)))\\n\",\n \" return x\\n\",\n \"\\n\",\n \"\\n\",\n \"class Upsample(nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super(Upsample, self).__init__()\\n\",\n \"\\n\",\n \" def forward(self, x, target_size, inference=False):\\n\",\n \" assert (x.data.dim() == 4)\\n\",\n \"\\n\",\n \" if inference:\\n\",\n \"\\n\",\n \" return x.view(x.size(0), x.size(1), x.size(2), 1, x.size(3), 1).\\\\\\n\",\n \" expand(x.size(0), x.size(1), x.size(2), target_size[2] // x.size(2), x.size(3), target_size[3] // x.size(3)).\\\\\\n\",\n \" contiguous().view(x.size(0), x.size(1), target_size[2], target_size[3])\\n\",\n \" else:\\n\",\n \" return F.interpolate(x, size=(target_size[2], target_size[3]), mode='nearest')\\n\",\n \"\\n\",\n \"\\n\",\n \"class Conv_Bn_Activation(nn.Module):\\n\",\n \" def __init__(self, in_channels, out_channels, kernel_size, stride, activation, bn=True, bias=False):\\n\",\n \" super().__init__()\\n\",\n \" pad = (kernel_size - 1) // 2\\n\",\n \"\\n\",\n \" self.conv = nn.ModuleList()\\n\",\n \" if bias:\\n\",\n \" self.conv.append(nn.Conv2d(in_channels, out_channels, kernel_size, stride, pad))\\n\",\n \" else:\\n\",\n \" self.conv.append(nn.Conv2d(in_channels, out_channels, kernel_size, stride, pad, bias=False))\\n\",\n \" if bn:\\n\",\n \" self.conv.append(nn.BatchNorm2d(out_channels))\\n\",\n \" if activation == \\\"mish\\\":\\n\",\n \" self.conv.append(Mish())\\n\",\n \" elif activation == \\\"relu\\\":\\n\",\n \" self.conv.append(nn.ReLU(inplace=True))\\n\",\n \" elif activation == \\\"leaky\\\":\\n\",\n \" self.conv.append(nn.LeakyReLU(0.1, inplace=True))\\n\",\n \" elif activation == \\\"linear\\\":\\n\",\n \" pass\\n\",\n \" else:\\n\",\n \" print(\\\"activate error !!! {} {} {}\\\".format(sys._getframe().f_code.co_filename,\\n\",\n \" sys._getframe().f_code.co_name, sys._getframe().f_lineno))\\n\",\n \"\\n\",\n \" def forward(self, x):\\n\",\n \" for l in self.conv:\\n\",\n \" x = l(x)\\n\",\n \" return x\\n\",\n \"\\n\",\n \"\\n\",\n \"class ResBlock(nn.Module):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Sequential residual blocks each of which consists of \\\\\\n\",\n \" two convolution layers.\\n\",\n \" Args:\\n\",\n \" ch (int): number of input and output channels.\\n\",\n \" nblocks (int): number of residual blocks.\\n\",\n \" shortcut (bool): if True, residual tensor addition is enabled.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" def __init__(self, ch, nblocks=1, shortcut=True):\\n\",\n \" super().__init__()\\n\",\n \" self.shortcut = shortcut\\n\",\n \" self.module_list = nn.ModuleList()\\n\",\n \" for i in range(nblocks):\\n\",\n \" resblock_one = nn.ModuleList()\\n\",\n \" resblock_one.append(Conv_Bn_Activation(ch, ch, 1, 1, 'mish'))\\n\",\n \" resblock_one.append(Conv_Bn_Activation(ch, ch, 3, 1, 'mish'))\\n\",\n \" self.module_list.append(resblock_one)\\n\",\n \"\\n\",\n \" def forward(self, x):\\n\",\n \" for module in self.module_list:\\n\",\n \" h = x\\n\",\n \" for res in module:\\n\",\n \" h = res(h)\\n\",\n \" x = x + h if self.shortcut else h\\n\",\n \" return x\\n\",\n \"\\n\",\n \"\\n\",\n \"class DownSample1(nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \" self.conv1 = Conv_Bn_Activation(3, 32, 3, 1, 'mish')\\n\",\n \"\\n\",\n \" self.conv2 = Conv_Bn_Activation(32, 64, 3, 2, 'mish')\\n\",\n \" self.conv3 = Conv_Bn_Activation(64, 64, 1, 1, 'mish')\\n\",\n \" # [route]\\n\",\n \" # layers = -2\\n\",\n \" self.conv4 = Conv_Bn_Activation(64, 64, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" self.conv5 = Conv_Bn_Activation(64, 32, 1, 1, 'mish')\\n\",\n \" self.conv6 = Conv_Bn_Activation(32, 64, 3, 1, 'mish')\\n\",\n \" # [shortcut]\\n\",\n \" # from=-3\\n\",\n \" # activation = linear\\n\",\n \"\\n\",\n \" self.conv7 = Conv_Bn_Activation(64, 64, 1, 1, 'mish')\\n\",\n \" # [route]\\n\",\n \" # layers = -1, -7\\n\",\n \" self.conv8 = Conv_Bn_Activation(128, 64, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" def forward(self, input):\\n\",\n \" x1 = self.conv1(input)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \" x3 = self.conv3(x2)\\n\",\n \" # route -2\\n\",\n \" x4 = self.conv4(x2)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" x6 = self.conv6(x5)\\n\",\n \" # shortcut -3\\n\",\n \" x6 = x6 + x4\\n\",\n \"\\n\",\n \" x7 = self.conv7(x6)\\n\",\n \" # [route]\\n\",\n \" # layers = -1, -7\\n\",\n \" x7 = torch.cat([x7, x3], dim=1)\\n\",\n \" x8 = self.conv8(x7)\\n\",\n \" return x8\\n\",\n \"\\n\",\n \"\\n\",\n \"class DownSample2(nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \" self.conv1 = Conv_Bn_Activation(64, 128, 3, 2, 'mish')\\n\",\n \" self.conv2 = Conv_Bn_Activation(128, 64, 1, 1, 'mish')\\n\",\n \" # r -2\\n\",\n \" self.conv3 = Conv_Bn_Activation(128, 64, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" self.resblock = ResBlock(ch=64, nblocks=2)\\n\",\n \"\\n\",\n \" # s -3\\n\",\n \" self.conv4 = Conv_Bn_Activation(64, 64, 1, 1, 'mish')\\n\",\n \" # r -1 -10\\n\",\n \" self.conv5 = Conv_Bn_Activation(128, 128, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" def forward(self, input):\\n\",\n \" x1 = self.conv1(input)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \" x3 = self.conv3(x1)\\n\",\n \"\\n\",\n \" r = self.resblock(x3)\\n\",\n \" x4 = self.conv4(r)\\n\",\n \"\\n\",\n \" x4 = torch.cat([x4, x2], dim=1)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" return x5\\n\",\n \"\\n\",\n \"\\n\",\n \"class DownSample3(nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \" self.conv1 = Conv_Bn_Activation(128, 256, 3, 2, 'mish')\\n\",\n \" self.conv2 = Conv_Bn_Activation(256, 128, 1, 1, 'mish')\\n\",\n \" self.conv3 = Conv_Bn_Activation(256, 128, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" self.resblock = ResBlock(ch=128, nblocks=8)\\n\",\n \" self.conv4 = Conv_Bn_Activation(128, 128, 1, 1, 'mish')\\n\",\n \" self.conv5 = Conv_Bn_Activation(256, 256, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" def forward(self, input):\\n\",\n \" x1 = self.conv1(input)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \" x3 = self.conv3(x1)\\n\",\n \"\\n\",\n \" r = self.resblock(x3)\\n\",\n \" x4 = self.conv4(r)\\n\",\n \"\\n\",\n \" x4 = torch.cat([x4, x2], dim=1)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" return x5\\n\",\n \"\\n\",\n \"\\n\",\n \"class DownSample4(nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \" self.conv1 = Conv_Bn_Activation(256, 512, 3, 2, 'mish')\\n\",\n \" self.conv2 = Conv_Bn_Activation(512, 256, 1, 1, 'mish')\\n\",\n \" self.conv3 = Conv_Bn_Activation(512, 256, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" self.resblock = ResBlock(ch=256, nblocks=8)\\n\",\n \" self.conv4 = Conv_Bn_Activation(256, 256, 1, 1, 'mish')\\n\",\n \" self.conv5 = Conv_Bn_Activation(512, 512, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" def forward(self, input):\\n\",\n \" x1 = self.conv1(input)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \" x3 = self.conv3(x1)\\n\",\n \"\\n\",\n \" r = self.resblock(x3)\\n\",\n \" x4 = self.conv4(r)\\n\",\n \"\\n\",\n \" x4 = torch.cat([x4, x2], dim=1)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" return x5\\n\",\n \"\\n\",\n \"\\n\",\n \"class DownSample5(nn.Module):\\n\",\n \" def __init__(self):\\n\",\n \" super().__init__()\\n\",\n \" self.conv1 = Conv_Bn_Activation(512, 1024, 3, 2, 'mish')\\n\",\n \" self.conv2 = Conv_Bn_Activation(1024, 512, 1, 1, 'mish')\\n\",\n \" self.conv3 = Conv_Bn_Activation(1024, 512, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" self.resblock = ResBlock(ch=512, nblocks=4)\\n\",\n \" self.conv4 = Conv_Bn_Activation(512, 512, 1, 1, 'mish')\\n\",\n \" self.conv5 = Conv_Bn_Activation(1024, 1024, 1, 1, 'mish')\\n\",\n \"\\n\",\n \" def forward(self, input):\\n\",\n \" x1 = self.conv1(input)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \" x3 = self.conv3(x1)\\n\",\n \"\\n\",\n \" r = self.resblock(x3)\\n\",\n \" x4 = self.conv4(r)\\n\",\n \"\\n\",\n \" x4 = torch.cat([x4, x2], dim=1)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" return x5\\n\",\n \"\\n\",\n \"\\n\",\n \"class Neck(nn.Module):\\n\",\n \" def __init__(self, inference=False):\\n\",\n \" super().__init__()\\n\",\n \" self.inference = inference\\n\",\n \"\\n\",\n \" self.conv1 = Conv_Bn_Activation(1024, 512, 1, 1, 'leaky')\\n\",\n \" self.conv2 = Conv_Bn_Activation(512, 1024, 3, 1, 'leaky')\\n\",\n \" self.conv3 = Conv_Bn_Activation(1024, 512, 1, 1, 'leaky')\\n\",\n \" # SPP\\n\",\n \" self.maxpool1 = nn.MaxPool2d(kernel_size=5, stride=1, padding=5 // 2)\\n\",\n \" self.maxpool2 = nn.MaxPool2d(kernel_size=9, stride=1, padding=9 // 2)\\n\",\n \" self.maxpool3 = nn.MaxPool2d(kernel_size=13, stride=1, padding=13 // 2)\\n\",\n \"\\n\",\n \" # R -1 -3 -5 -6\\n\",\n \" # SPP\\n\",\n \" self.conv4 = Conv_Bn_Activation(2048, 512, 1, 1, 'leaky')\\n\",\n \" self.conv5 = Conv_Bn_Activation(512, 1024, 3, 1, 'leaky')\\n\",\n \" self.conv6 = Conv_Bn_Activation(1024, 512, 1, 1, 'leaky')\\n\",\n \" self.conv7 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" # UP\\n\",\n \" self.upsample1 = Upsample()\\n\",\n \" # R 85\\n\",\n \" self.conv8 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" # R -1 -3\\n\",\n \" self.conv9 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" self.conv10 = Conv_Bn_Activation(256, 512, 3, 1, 'leaky')\\n\",\n \" self.conv11 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" self.conv12 = Conv_Bn_Activation(256, 512, 3, 1, 'leaky')\\n\",\n \" self.conv13 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" self.conv14 = Conv_Bn_Activation(256, 128, 1, 1, 'leaky')\\n\",\n \" # UP\\n\",\n \" self.upsample2 = Upsample()\\n\",\n \" # R 54\\n\",\n \" self.conv15 = Conv_Bn_Activation(256, 128, 1, 1, 'leaky')\\n\",\n \" # R -1 -3\\n\",\n \" self.conv16 = Conv_Bn_Activation(256, 128, 1, 1, 'leaky')\\n\",\n \" self.conv17 = Conv_Bn_Activation(128, 256, 3, 1, 'leaky')\\n\",\n \" self.conv18 = Conv_Bn_Activation(256, 128, 1, 1, 'leaky')\\n\",\n \" self.conv19 = Conv_Bn_Activation(128, 256, 3, 1, 'leaky')\\n\",\n \" self.conv20 = Conv_Bn_Activation(256, 128, 1, 1, 'leaky')\\n\",\n \"\\n\",\n \" def forward(self, input, downsample4, downsample3, inference=False):\\n\",\n \" x1 = self.conv1(input)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \" x3 = self.conv3(x2)\\n\",\n \" # SPP\\n\",\n \" m1 = self.maxpool1(x3)\\n\",\n \" m2 = self.maxpool2(x3)\\n\",\n \" m3 = self.maxpool3(x3)\\n\",\n \" spp = torch.cat([m3, m2, m1, x3], dim=1)\\n\",\n \" # SPP end\\n\",\n \" x4 = self.conv4(spp)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" x6 = self.conv6(x5)\\n\",\n \" x7 = self.conv7(x6)\\n\",\n \" # UP\\n\",\n \" up = self.upsample1(x7, downsample4.size(), self.inference)\\n\",\n \" # R 85\\n\",\n \" x8 = self.conv8(downsample4)\\n\",\n \" # R -1 -3\\n\",\n \" x8 = torch.cat([x8, up], dim=1)\\n\",\n \"\\n\",\n \" x9 = self.conv9(x8)\\n\",\n \" x10 = self.conv10(x9)\\n\",\n \" x11 = self.conv11(x10)\\n\",\n \" x12 = self.conv12(x11)\\n\",\n \" x13 = self.conv13(x12)\\n\",\n \" x14 = self.conv14(x13)\\n\",\n \"\\n\",\n \" # UP\\n\",\n \" up = self.upsample2(x14, downsample3.size(), self.inference)\\n\",\n \" # R 54\\n\",\n \" x15 = self.conv15(downsample3)\\n\",\n \" # R -1 -3\\n\",\n \" x15 = torch.cat([x15, up], dim=1)\\n\",\n \"\\n\",\n \" x16 = self.conv16(x15)\\n\",\n \" x17 = self.conv17(x16)\\n\",\n \" x18 = self.conv18(x17)\\n\",\n \" x19 = self.conv19(x18)\\n\",\n \" x20 = self.conv20(x19)\\n\",\n \" return x20, x13, x6\\n\",\n \"\\n\",\n \"\\n\",\n \"class Yolov4Head(nn.Module):\\n\",\n \" def __init__(self, output_ch, n_classes, inference=False):\\n\",\n \" super().__init__()\\n\",\n \" self.inference = inference\\n\",\n \"\\n\",\n \" self.conv1 = Conv_Bn_Activation(128, 256, 3, 1, 'leaky')\\n\",\n \" self.conv2 = Conv_Bn_Activation(256, output_ch, 1, 1, 'linear', bn=False, bias=True)\\n\",\n \"\\n\",\n \" self.yolo1 = YoloLayer(\\n\",\n \" anchor_mask=[0, 1, 2], num_classes=n_classes,\\n\",\n \" anchors=[12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401],\\n\",\n \" num_anchors=9, stride=8)\\n\",\n \"\\n\",\n \" # R -4\\n\",\n \" self.conv3 = Conv_Bn_Activation(128, 256, 3, 2, 'leaky')\\n\",\n \"\\n\",\n \" # R -1 -16\\n\",\n \" self.conv4 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" self.conv5 = Conv_Bn_Activation(256, 512, 3, 1, 'leaky')\\n\",\n \" self.conv6 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" self.conv7 = Conv_Bn_Activation(256, 512, 3, 1, 'leaky')\\n\",\n \" self.conv8 = Conv_Bn_Activation(512, 256, 1, 1, 'leaky')\\n\",\n \" self.conv9 = Conv_Bn_Activation(256, 512, 3, 1, 'leaky')\\n\",\n \" self.conv10 = Conv_Bn_Activation(512, output_ch, 1, 1, 'linear', bn=False, bias=True)\\n\",\n \" \\n\",\n \" self.yolo2 = YoloLayer(\\n\",\n \" anchor_mask=[3, 4, 5], num_classes=n_classes,\\n\",\n \" anchors=[12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401],\\n\",\n \" num_anchors=9, stride=16)\\n\",\n \"\\n\",\n \" # R -4\\n\",\n \" self.conv11 = Conv_Bn_Activation(256, 512, 3, 2, 'leaky')\\n\",\n \"\\n\",\n \" # R -1 -37\\n\",\n \" self.conv12 = Conv_Bn_Activation(1024, 512, 1, 1, 'leaky')\\n\",\n \" self.conv13 = Conv_Bn_Activation(512, 1024, 3, 1, 'leaky')\\n\",\n \" self.conv14 = Conv_Bn_Activation(1024, 512, 1, 1, 'leaky')\\n\",\n \" self.conv15 = Conv_Bn_Activation(512, 1024, 3, 1, 'leaky')\\n\",\n \" self.conv16 = Conv_Bn_Activation(1024, 512, 1, 1, 'leaky')\\n\",\n \" self.conv17 = Conv_Bn_Activation(512, 1024, 3, 1, 'leaky')\\n\",\n \" self.conv18 = Conv_Bn_Activation(1024, output_ch, 1, 1, 'linear', bn=False, bias=True)\\n\",\n \" \\n\",\n \" self.yolo3 = YoloLayer(\\n\",\n \" anchor_mask=[6, 7, 8], num_classes=n_classes,\\n\",\n \" anchors=[12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401],\\n\",\n \" num_anchors=9, stride=32)\\n\",\n \"\\n\",\n \" def forward(self, input1, input2, input3):\\n\",\n \" x1 = self.conv1(input1)\\n\",\n \" x2 = self.conv2(x1)\\n\",\n \"\\n\",\n \" x3 = self.conv3(input1)\\n\",\n \" # R -1 -16\\n\",\n \" x3 = torch.cat([x3, input2], dim=1)\\n\",\n \" x4 = self.conv4(x3)\\n\",\n \" x5 = self.conv5(x4)\\n\",\n \" x6 = self.conv6(x5)\\n\",\n \" x7 = self.conv7(x6)\\n\",\n \" x8 = self.conv8(x7)\\n\",\n \" x9 = self.conv9(x8)\\n\",\n \" x10 = self.conv10(x9)\\n\",\n \"\\n\",\n \" # R -4\\n\",\n \" x11 = self.conv11(x8)\\n\",\n \" # R -1 -37\\n\",\n \" x11 = torch.cat([x11, input3], dim=1)\\n\",\n \"\\n\",\n \" x12 = self.conv12(x11)\\n\",\n \" x13 = self.conv13(x12)\\n\",\n \" x14 = self.conv14(x13)\\n\",\n \" x15 = self.conv15(x14)\\n\",\n \" x16 = self.conv16(x15)\\n\",\n \" x17 = self.conv17(x16)\\n\",\n \" x18 = self.conv18(x17)\\n\",\n \" \\n\",\n \" if self.inference:\\n\",\n \" y1 = self.yolo1(x2)\\n\",\n \" y2 = self.yolo2(x10)\\n\",\n \" y3 = self.yolo3(x18)\\n\",\n \"\\n\",\n \" return get_region_boxes([y1, y2, y3])\\n\",\n \" \\n\",\n \" else:\\n\",\n \" return [x2, x10, x18]\\n\",\n \"\\n\",\n \"\\n\",\n \"class Yolov4(nn.Module):\\n\",\n \" def __init__(self, yolov4conv137weight=None, n_classes=80, inference=False):\\n\",\n \" super().__init__()\\n\",\n \"\\n\",\n \" output_ch = (4 + 1 + n_classes) * 3\\n\",\n \"\\n\",\n \" # backbone\\n\",\n \" self.down1 = DownSample1()\\n\",\n \" self.down2 = DownSample2()\\n\",\n \" self.down3 = DownSample3()\\n\",\n \" self.down4 = DownSample4()\\n\",\n \" self.down5 = DownSample5()\\n\",\n \" # neck\\n\",\n \" self.neek = Neck(inference)\\n\",\n \" # yolov4conv137\\n\",\n \" if yolov4conv137weight:\\n\",\n \" _model = nn.Sequential(self.down1, self.down2, self.down3, self.down4, self.down5, self.neek)\\n\",\n \" pretrained_dict = torch.load(yolov4conv137weight)\\n\",\n \"\\n\",\n \" model_dict = _model.state_dict()\\n\",\n \" # 1. filter out unnecessary keys\\n\",\n \" pretrained_dict = {k1: v for (k, v), k1 in zip(pretrained_dict.items(), model_dict)}\\n\",\n \" # 2. overwrite entries in the existing state dict\\n\",\n \" model_dict.update(pretrained_dict)\\n\",\n \" _model.load_state_dict(model_dict)\\n\",\n \" \\n\",\n \" # head\\n\",\n \" self.head = Yolov4Head(output_ch, n_classes, inference)\\n\",\n \"\\n\",\n \"\\n\",\n \" def forward(self, input):\\n\",\n \" d1 = self.down1(input)\\n\",\n \" d2 = self.down2(d1)\\n\",\n \" d3 = self.down3(d2)\\n\",\n \" d4 = self.down4(d3)\\n\",\n \" d5 = self.down5(d4)\\n\",\n \"\\n\",\n \" x20, x13, x6 = self.neek(d5, d4, d3)\\n\",\n \"\\n\",\n \" output = self.head(x20, x13, x6)\\n\",\n \" return output\\n\",\n \"\\n\",\n \"\\n\",\n \"def yolo_forward_dynamic(output, conf_thresh, num_classes, anchors, num_anchors, scale_x_y, only_objectness=1,\\n\",\n \" validation=False):\\n\",\n \" # Output would be invalid if it does not satisfy this assert\\n\",\n \" # assert (output.size(1) == (5 + num_classes) * num_anchors)\\n\",\n \"\\n\",\n \" # print(output.size())\\n\",\n \"\\n\",\n \" # Slice the second dimension (channel) of output into:\\n\",\n \" # [ 2, 2, 1, num_classes, 2, 2, 1, num_classes, 2, 2, 1, num_classes ]\\n\",\n \" # And then into\\n\",\n \" # bxy = [ 6 ] bwh = [ 6 ] det_conf = [ 3 ] cls_conf = [ num_classes * 3 ]\\n\",\n \" # batch = output.size(0)\\n\",\n \" # H = output.size(2)\\n\",\n \" # W = output.size(3)\\n\",\n \"\\n\",\n \" bxy_list = []\\n\",\n \" bwh_list = []\\n\",\n \" det_confs_list = []\\n\",\n \" cls_confs_list = []\\n\",\n \"\\n\",\n \" for i in range(num_anchors):\\n\",\n \" begin = i * (5 + num_classes)\\n\",\n \" end = (i + 1) * (5 + num_classes)\\n\",\n \" \\n\",\n \" bxy_list.append(output[:, begin : begin + 2])\\n\",\n \" bwh_list.append(output[:, begin + 2 : begin + 4])\\n\",\n \" det_confs_list.append(output[:, begin + 4 : begin + 5])\\n\",\n \" cls_confs_list.append(output[:, begin + 5 : end])\\n\",\n \"\\n\",\n \" # Shape: [batch, num_anchors * 2, H, W]\\n\",\n \" bxy = torch.cat(bxy_list, dim=1)\\n\",\n \" # Shape: [batch, num_anchors * 2, H, W]\\n\",\n \" bwh = torch.cat(bwh_list, dim=1)\\n\",\n \"\\n\",\n \" # Shape: [batch, num_anchors, H, W]\\n\",\n \" det_confs = torch.cat(det_confs_list, dim=1)\\n\",\n \" # Shape: [batch, num_anchors * H * W]\\n\",\n \" det_confs = det_confs.view(output.size(0), num_anchors * output.size(2) * output.size(3))\\n\",\n \"\\n\",\n \" # Shape: [batch, num_anchors * num_classes, H, W]\\n\",\n \" cls_confs = torch.cat(cls_confs_list, dim=1)\\n\",\n \" # Shape: [batch, num_anchors, num_classes, H * W]\\n\",\n \" cls_confs = cls_confs.view(output.size(0), num_anchors, num_classes, output.size(2) * output.size(3))\\n\",\n \" # Shape: [batch, num_anchors, num_classes, H * W] --> [batch, num_anchors * H * W, num_classes] \\n\",\n \" cls_confs = cls_confs.permute(0, 1, 3, 2).reshape(output.size(0), num_anchors * output.size(2) * output.size(3), num_classes)\\n\",\n \"\\n\",\n \" # Apply sigmoid(), exp() and softmax() to slices\\n\",\n \" #\\n\",\n \" bxy = torch.sigmoid(bxy) * scale_x_y - 0.5 * (scale_x_y - 1)\\n\",\n \" bwh = torch.exp(bwh)\\n\",\n \" det_confs = torch.sigmoid(det_confs)\\n\",\n \" cls_confs = torch.sigmoid(cls_confs)\\n\",\n \"\\n\",\n \" # Prepare C-x, C-y, P-w, P-h (None of them are torch related)\\n\",\n \" grid_x = np.expand_dims(np.expand_dims(np.expand_dims(np.linspace(0, output.size(3) - 1, output.size(3)), axis=0).repeat(output.size(2), 0), axis=0), axis=0)\\n\",\n \" grid_y = np.expand_dims(np.expand_dims(np.expand_dims(np.linspace(0, output.size(2) - 1, output.size(2)), axis=1).repeat(output.size(3), 1), axis=0), axis=0)\\n\",\n \" # grid_x = torch.linspace(0, W - 1, W).reshape(1, 1, 1, W).repeat(1, 1, H, 1)\\n\",\n \" # grid_y = torch.linspace(0, H - 1, H).reshape(1, 1, H, 1).repeat(1, 1, 1, W)\\n\",\n \"\\n\",\n \" anchor_w = []\\n\",\n \" anchor_h = []\\n\",\n \" for i in range(num_anchors):\\n\",\n \" anchor_w.append(anchors[i * 2])\\n\",\n \" anchor_h.append(anchors[i * 2 + 1])\\n\",\n \"\\n\",\n \" device = None\\n\",\n \" cuda_check = output.is_cuda\\n\",\n \" if cuda_check:\\n\",\n \" device = output.get_device()\\n\",\n \"\\n\",\n \" bx_list = []\\n\",\n \" by_list = []\\n\",\n \" bw_list = []\\n\",\n \" bh_list = []\\n\",\n \"\\n\",\n \" # Apply C-x, C-y, P-w, P-h\\n\",\n \" for i in range(num_anchors):\\n\",\n \" ii = i * 2\\n\",\n \" # Shape: [batch, 1, H, W]\\n\",\n \" bx = bxy[:, ii : ii + 1] + torch.tensor(grid_x, device=device, dtype=torch.float32) # grid_x.to(device=device, dtype=torch.float32)\\n\",\n \" # Shape: [batch, 1, H, W]\\n\",\n \" by = bxy[:, ii + 1 : ii + 2] + torch.tensor(grid_y, device=device, dtype=torch.float32) # grid_y.to(device=device, dtype=torch.float32)\\n\",\n \" # Shape: [batch, 1, H, W]\\n\",\n \" bw = bwh[:, ii : ii + 1] * anchor_w[i]\\n\",\n \" # Shape: [batch, 1, H, W]\\n\",\n \" bh = bwh[:, ii + 1 : ii + 2] * anchor_h[i]\\n\",\n \"\\n\",\n \" bx_list.append(bx)\\n\",\n \" by_list.append(by)\\n\",\n \" bw_list.append(bw)\\n\",\n \" bh_list.append(bh)\\n\",\n \"\\n\",\n \"\\n\",\n \" ########################################\\n\",\n \" # Figure out bboxes from slices #\\n\",\n \" ########################################\\n\",\n \" \\n\",\n \" # Shape: [batch, num_anchors, H, W]\\n\",\n \" bx = torch.cat(bx_list, dim=1)\\n\",\n \" # Shape: [batch, num_anchors, H, W]\\n\",\n \" by = torch.cat(by_list, dim=1)\\n\",\n \" # Shape: [batch, num_anchors, H, W]\\n\",\n \" bw = torch.cat(bw_list, dim=1)\\n\",\n \" # Shape: [batch, num_anchors, H, W]\\n\",\n \" bh = torch.cat(bh_list, dim=1)\\n\",\n \"\\n\",\n \" # Shape: [batch, 2 * num_anchors, H, W]\\n\",\n \" bx_bw = torch.cat((bx, bw), dim=1)\\n\",\n \" # Shape: [batch, 2 * num_anchors, H, W]\\n\",\n \" by_bh = torch.cat((by, bh), dim=1)\\n\",\n \"\\n\",\n \" # normalize coordinates to [0, 1]\\n\",\n \" bx_bw /= output.size(3)\\n\",\n \" by_bh /= output.size(2)\\n\",\n \"\\n\",\n \" # Shape: [batch, num_anchors * H * W, 1]\\n\",\n \" bx = bx_bw[:, :num_anchors].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)\\n\",\n \" by = by_bh[:, :num_anchors].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)\\n\",\n \" bw = bx_bw[:, num_anchors:].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)\\n\",\n \" bh = by_bh[:, num_anchors:].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)\\n\",\n \"\\n\",\n \" bx1 = bx - bw * 0.5\\n\",\n \" by1 = by - bh * 0.5\\n\",\n \" bx2 = bx1 + bw\\n\",\n \" by2 = by1 + bh\\n\",\n \"\\n\",\n \" # Shape: [batch, num_anchors * h * w, 4] -> [batch, num_anchors * h * w, 1, 4]\\n\",\n \" boxes = torch.cat((bx1, by1, bx2, by2), dim=2).view(output.size(0), num_anchors * output.size(2) * output.size(3), 1, 4)\\n\",\n \" # boxes = boxes.repeat(1, 1, num_classes, 1)\\n\",\n \"\\n\",\n \" # boxes: [batch, num_anchors * H * W, 1, 4]\\n\",\n \" # cls_confs: [batch, num_anchors * H * W, num_classes]\\n\",\n \" # det_confs: [batch, num_anchors * H * W]\\n\",\n \"\\n\",\n \" det_confs = det_confs.view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)\\n\",\n \" confs = cls_confs * det_confs\\n\",\n \"\\n\",\n \" # boxes: [batch, num_anchors * H * W, 1, 4]\\n\",\n \" # confs: [batch, num_anchors * H * W, num_classes]\\n\",\n \"\\n\",\n \" return boxes, confs\\n\",\n \"\\n\",\n \"class YoloLayer(nn.Module):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Yolo layer\\n\",\n \" model_out: while inference,is post-processing inside or outside the model\\n\",\n \" true:outside\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" def __init__(self, anchor_mask=[], num_classes=0, anchors=[], num_anchors=1, stride=32, model_out=False):\\n\",\n \" super(YoloLayer, self).__init__()\\n\",\n \" self.anchor_mask = anchor_mask\\n\",\n \" self.num_classes = num_classes\\n\",\n \" self.anchors = anchors\\n\",\n \" self.num_anchors = num_anchors\\n\",\n \" self.anchor_step = len(anchors) // num_anchors\\n\",\n \" self.coord_scale = 1\\n\",\n \" self.noobject_scale = 1\\n\",\n \" self.object_scale = 5\\n\",\n \" self.class_scale = 1\\n\",\n \" self.thresh = 0.6\\n\",\n \" self.stride = stride\\n\",\n \" self.seen = 0\\n\",\n \" self.scale_x_y = 1\\n\",\n \"\\n\",\n \" self.model_out = model_out\\n\",\n \"\\n\",\n \" def forward(self, output, target=None):\\n\",\n \" if self.training:\\n\",\n \" return output\\n\",\n \" masked_anchors = []\\n\",\n \" for m in self.anchor_mask:\\n\",\n \" masked_anchors += self.anchors[m * self.anchor_step:(m + 1) * self.anchor_step]\\n\",\n \" masked_anchors = [anchor / self.stride for anchor in masked_anchors]\\n\",\n \"\\n\",\n \" return yolo_forward_dynamic(output, self.thresh, self.num_classes, masked_anchors, len(self.anchor_mask),scale_x_y=self.scale_x_y)\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_region_boxes(boxes_and_confs):\\n\",\n \"\\n\",\n \" # print('Getting boxes from boxes and confs ...')\\n\",\n \"\\n\",\n \" boxes_list = []\\n\",\n \" confs_list = []\\n\",\n \"\\n\",\n \" for item in boxes_and_confs:\\n\",\n \" boxes_list.append(item[0])\\n\",\n \" confs_list.append(item[1])\\n\",\n \"\\n\",\n \" # boxes: [batch, num1 + num2 + num3, 1, 4]\\n\",\n \" # confs: [batch, num1 + num2 + num3, num_classes]\\n\",\n \" boxes = torch.cat(boxes_list, dim=1)\\n\",\n \" confs = torch.cat(confs_list, dim=1)\\n\",\n \" \\n\",\n \" return boxes, confs\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 2: Download the COCO 2017 evaluation dataset and define the data loader function\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Download dataset\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"!curl -LO http://images.cocodataset.org/zips/val2017.zip\\n\",\n \"!curl -LO http://images.cocodataset.org/annotations/annotations_trainval2017.zip\\n\",\n \"!unzip -q val2017.zip\\n\",\n \"!unzip annotations_trainval2017.zip\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!ls\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Define data loader\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import json\\n\",\n \"import time\\n\",\n \"import torchvision\\n\",\n \"import torchvision.transforms as transforms\\n\",\n \"import torchvision.datasets as dset\\n\",\n \"from pycocotools.coco import COCO\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_image_filenames(root=os.getcwd()):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Generate paths to the coco dataset image files.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" root (str): The root folder contains.\\n\",\n \" \\n\",\n \" Yields:\\n\",\n \" filename (str): The path to an image file.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" image_path = os.path.join(root, 'val2017')\\n\",\n \" for root, dirs, files in os.walk(image_path):\\n\",\n \" for filename in files:\\n\",\n \" yield os.path.join(image_path, filename)\\n\",\n \"\\n\",\n \" \\n\",\n \"def get_coco_dataloader(coco2017_root, transform, subset_indices=None):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Create the dataset loader and ground truth coco dataset.\\n\",\n \" \\n\",\n \" Arguments:\\n\",\n \" coco2017_root (str): The root directory to load the data/labels from.\\n\",\n \" transform (torchvision.Transform): A transform to apply to the images.\\n\",\n \" subset_indices (list): Indices used to create a subset of the dataset.\\n\",\n \"\\n\",\n \" Returns: \\n\",\n \" loader (iterable): Produces transformed images and labels.\\n\",\n \" cocoGt (pycocotools.coco.COCO): Contains the ground truth in coco \\n\",\n \" format.\\n\",\n \" label_info (dict): A mapping from label id to the human-readable name.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" # Create the dataset\\n\",\n \" coco2017_img_path = os.path.join(coco2017_root, 'val2017')\\n\",\n \" coco2017_ann_path = os.path.join(\\n\",\n \" coco2017_root, 'annotations/instances_val2017.json')\\n\",\n \"\\n\",\n \" # check the number of images in val2017 - Should be 5000\\n\",\n \" num_files = len(list(get_image_filenames(coco2017_root)))\\n\",\n \" print('\\\\nNumber of images in val2017 = {}\\\\n'.format(num_files))\\n\",\n \"\\n\",\n \" # load annotations to decode classification results\\n\",\n \" with open(coco2017_ann_path) as f:\\n\",\n \" annotate_json = json.load(f)\\n\",\n \" label_info = {label[\\\"id\\\"]: label[\\\"name\\\"]\\n\",\n \" for label in annotate_json['categories']}\\n\",\n \"\\n\",\n \" # initialize COCO ground truth dataset\\n\",\n \" cocoGt = COCO(coco2017_ann_path)\\n\",\n \"\\n\",\n \" # create the dataset using torchvision's coco detection dataset\\n\",\n \" coco_val_data = dset.CocoDetection(\\n\",\n \" root=coco2017_img_path, \\n\",\n \" annFile=coco2017_ann_path, \\n\",\n \" transform=transform\\n\",\n \" )\\n\",\n \"\\n\",\n \" if subset_indices is not None:\\n\",\n \" # Create a smaller subset of the data for testing - e.g. to pinpoint error at image 516\\n\",\n \" coco_val_data = torch.utils.data.Subset(coco_val_data, subset_indices)\\n\",\n \"\\n\",\n \" # create the dataloader using torch dataloader\\n\",\n \" loader = torch.utils.data.DataLoader(coco_val_data, batch_size=1, shuffle=False)\\n\",\n \"\\n\",\n \" return loader, cocoGt, label_info\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Load dataset\\n\",\n \"Here 2 dataset loaders are created and the resulting data is displayed\\n\",\n \"- `orig_coco_val_data_loader`: Contains the original unmodified image\\n\",\n \"- `coco_val_data_loader`: Contains images of a standardized size of 608x608 pixels \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"coco2017_root = './'\\n\",\n \"orig_coco_val_data_loader, *_ = get_coco_dataloader(coco2017_root, transforms.ToTensor())\\n\",\n \"transform = transforms.Compose([transforms.Resize([608, 608]), transforms.ToTensor()])\\n\",\n \"coco_val_data_loader, cocoGt, label_info = get_coco_dataloader(coco2017_root, transform)\\n\",\n \"image_orig, _ = next(iter(orig_coco_val_data_loader))\\n\",\n \"print(image_orig.shape)\\n\",\n \"image, image_info = next(iter(coco_val_data_loader))\\n\",\n \"image_id = image_info[0][\\\"image_id\\\"].item()\\n\",\n \"print(image.shape)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Define some helper functions for deployment (inference)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def postprocess(boxes, scores, score_threshold=0.05, iou_threshold=0.5):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Classifies and filters bounding boxes from Yolo V4 output.\\n\",\n \" \\n\",\n \" Performs classification, filtering, and non-maximum suppression to remove\\n\",\n \" boxes that are irrelevant. The result is the filtered set of boxes, the \\n\",\n \" associated label confidence score, and the predicted label.\\n\",\n \" \\n\",\n \" See: https://pytorch.org/docs/stable/torchvision/ops.html#torchvision.ops.nms\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" boxes (torch.Tensor): The Yolo V4 bounding boxes.\\n\",\n \" scores (torch.Tensor): The categories scores for each box.\\n\",\n \" score_threshold (float): Ignore boxes with scores below threshold.\\n\",\n \" iou_threshold (float): Discards boxes with intersection above threshold. \\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" boxes (torch.Tensor): The filtered Yolo V4 bounding boxes.\\n\",\n \" scores (torch.Tensor): The label score for each box.\\n\",\n \" labels (torch.Tensor): The label for each box.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" \\n\",\n \" # shape: [n_batch, n_boxes, 1, 4] => [n_boxes, 4] # Assumes n_batch size is 1\\n\",\n \" boxes = boxes.squeeze()\\n\",\n \"\\n\",\n \" # shape: [n_batch, n_boxes, 80] => [n_boxes, 80] # Assumes n_batch size is 1\\n\",\n \" scores = scores.squeeze()\\n\",\n \"\\n\",\n \" # Classify each box according to the maximum category score\\n\",\n \" score, column = torch.max(scores, dim=1)\\n\",\n \"\\n\",\n \" # Filter out rows for scores which are below threshold\\n\",\n \" mask = score > score_threshold\\n\",\n \"\\n\",\n \" # Filter model output data\\n\",\n \" boxes = boxes[mask]\\n\",\n \" score = score[mask]\\n\",\n \" idxs = column[mask]\\n\",\n \"\\n\",\n \" # Perform non-max suppression on all categories at once. shape: [n_keep,]\\n\",\n \" keep = torchvision.ops.batched_nms(\\n\",\n \" boxes=boxes, \\n\",\n \" scores=score, \\n\",\n \" idxs=idxs,\\n\",\n \" iou_threshold=iou_threshold,\\n\",\n \" )\\n\",\n \"\\n\",\n \" # The image category id associated with each column\\n\",\n \" categories = torch.tensor([\\n\",\n \" 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16,\\n\",\n \" 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 31,\\n\",\n \" 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,\\n\",\n \" 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56,\\n\",\n \" 57, 58, 59, 60, 61, 62, 63, 64, 65, 67, 70, 72,\\n\",\n \" 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 84, 85,\\n\",\n \" 86, 87, 88, 89, 90\\n\",\n \" ])\\n\",\n \" \\n\",\n \" boxes = boxes[keep] # shape: [n_keep, 4]\\n\",\n \" score = score[keep] # shape: [n_keep,]\\n\",\n \" idxs = idxs[keep]\\n\",\n \" label = categories[idxs] # shape: [n_keep,]\\n\",\n \" \\n\",\n \" return boxes, score, label\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_results_as_dict(boxes, scores, labels, image_orig):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Transforms post-processed output into dictionary output.\\n\",\n \" \\n\",\n \" This translates the model coordinate bounding boxes (x1, y1, x2, y2) \\n\",\n \" into a rectangular description (x, y, width, height) scaled to the \\n\",\n \" original image size.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" boxes (torch.Tensor): The Yolo V4 bounding boxes.\\n\",\n \" scores (torch.Tensor): The label score for each box.\\n\",\n \" labels (torch.Tensor): The label for each box.\\n\",\n \" image_orig (torch.Tensor): The image to scale the bounding boxes to.\\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" output (dict): The dictionary of rectangle bounding boxes.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" h_size, w_size = image_orig.shape[-2:]\\n\",\n \"\\n\",\n \" x1 = boxes[:, 0] * w_size\\n\",\n \" y1 = boxes[:, 1] * h_size\\n\",\n \" x2 = boxes[:, 2] * w_size\\n\",\n \" y2 = boxes[:, 3] * h_size\\n\",\n \"\\n\",\n \" width = x2 - x1\\n\",\n \" height = y2 - y1\\n\",\n \"\\n\",\n \" boxes = torch.stack([x1, y1, width, height]).T\\n\",\n \" return {\\n\",\n \" 'boxes': boxes.detach().numpy(),\\n\",\n \" 'labels': labels.detach().numpy(),\\n\",\n \" 'scores': scores.detach().numpy(),\\n\",\n \" }\\n\",\n \"\\n\",\n \"\\n\",\n \"def prepare_for_coco_detection(predictions):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Convert dictionary model predictions into an expected COCO dataset format.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" predictions (dict): The list of box coordinates, scores, and labels.\\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" output (list[dict]): The list of bounding boxes.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" coco_results = []\\n\",\n \" for original_id, prediction in predictions.items():\\n\",\n \" if len(prediction) == 0:\\n\",\n \" continue\\n\",\n \"\\n\",\n \" boxes = prediction[\\\"boxes\\\"].tolist()\\n\",\n \" scores = prediction[\\\"scores\\\"].tolist()\\n\",\n \" labels = prediction[\\\"labels\\\"].tolist()\\n\",\n \"\\n\",\n \" coco_results.extend(\\n\",\n \" [\\n\",\n \" {\\n\",\n \" \\\"image_id\\\": original_id,\\n\",\n \" \\\"category_id\\\": labels[k],\\n\",\n \" \\\"bbox\\\": box,\\n\",\n \" \\\"score\\\": scores[k],\\n\",\n \" }\\n\",\n \" for k, box in enumerate(boxes)\\n\",\n \" ]\\n\",\n \" )\\n\",\n \" return coco_results\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Download pretrained checkpoint\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import requests\\n\",\n \"\\n\",\n \"def download_file_from_google_drive(id, destination):\\n\",\n \" response = requests.post('https://drive.google.com/uc?id='+id+'&confirm=t')\\n\",\n \" save_response_content(response, destination)\\n\",\n \"\\n\",\n \"def save_response_content(response, destination):\\n\",\n \" CHUNK_SIZE = 32768\\n\",\n \" with open(destination, \\\"wb\\\") as f:\\n\",\n \" for chunk in response.iter_content(CHUNK_SIZE):\\n\",\n \" if chunk: # filter out keep-alive new chunks\\n\",\n \" f.write(chunk)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"download_file_from_google_drive('1wv_LiFeCRYwtpkqREPeI13-gPELBDwuJ', './yolo_v4.pth')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 3: Build, Compile, and Save Neuron-Optimized YOLO v4 TorchScript\\n\",\n \"### Construct model and load pretrained checkpoint\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"model = Yolov4(yolov4conv137weight=None, n_classes=80, inference=True)\\n\",\n \"weightfile = \\\"./yolo_v4.pth\\\"\\n\",\n \"pretrained_dict = torch.load(weightfile, map_location=torch.device('cpu'))\\n\",\n \"model.load_state_dict(pretrained_dict)\\n\",\n \"model.eval()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Execute inference for a single image and display output\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"import matplotlib.pyplot as plt\\n\",\n \"import matplotlib.patches as patches\\n\",\n \"\\n\",\n \"image_orig, _ = next(iter(orig_coco_val_data_loader))\\n\",\n \"image, _ = next(iter(coco_val_data_loader))\\n\",\n \"boxes, scores = model(image)\\n\",\n \"boxes, scores, labels = postprocess(boxes, scores)\\n\",\n \"result_dict = get_results_as_dict(boxes, scores, labels, image_orig)\\n\",\n \"\\n\",\n \"fig, ax = plt.subplots(figsize=(10, 10))\\n\",\n \"ax.imshow(image_orig.numpy().squeeze(0).transpose(1, 2, 0))\\n\",\n \"for xywh, _ in zip(result_dict['boxes'], result_dict['labels']):\\n\",\n \" x, y, w, h = xywh\\n\",\n \" rect = patches.Rectangle((x, y), w, h, linewidth=1, edgecolor='g', facecolor='none')\\n\",\n \" ax.add_patch(rect)\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"tags\": []\n },\n \"source\": [\n \"### Run compilation with manually specified device placement\\n\",\n \"\\n\",\n \"First, inspect the model without running compilation by adding the `skip_compiler=True` argument to the `torch.neuron.trace` call.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"model_neuron_for_inspection = torch.neuron.trace(model, image, skip_compiler=True)\\n\",\n \"print(model_neuron_for_inspection)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Inspecting the model, we discover that there are many `aten::slice` operations in some submodules called `YoloLayer`. Although these operations are supported by the neuron-cc compiler, they are not going to run efficiently on the Inferentia hardware. To work it around, we recommend to manually place these operators on CPU.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"To manually place `YoloLayer` on CPU, we may make use of the `subgraph_builder_function` argument in `torch.neuron.trace`. It is a callback function that returns `True` or `False` based on information available in `node`. The typical use is a condition based on either `node.name` or `node.type_string`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"tags\": []\n },\n \"outputs\": [],\n \"source\": [\n \"def subgraph_builder_function(node):\\n\",\n \" return 'YoloLayer' not in node.name\\n\",\n \"\\n\",\n \"model_neuron = torch.neuron.trace(model, image, subgraph_builder_function=subgraph_builder_function)\\n\",\n \"model_neuron.save('yolo_v4_neuron.pt')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Compilation is now finished and the compiled model has been saved to a local file called 'yolo_v4_neuron.pt'. Saving is important due to the slow compilation process.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 4: Evaluate Accuracy on the COCO 2017 Dataset\\n\",\n \"### Load compiled model and run inference\\n\",\n \"To validate accuracy of the compiled model, lets run inference on the COCO 2017 validation dataset. We start by defining a helper function `run_inference`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def run_inference(dataloader, dataloader_orig, model, convert=True, modelName=''):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Run Yolo V4 inference on the COCO dataset.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" dataloader (iterable): Data loader of input processed images and labels.\\n\",\n \" dataloader_orig (iterable): Data loader with original images.\\n\",\n \" model (torch.nn.Module): The torch model to run inference against.\\n\",\n \" convert (bool): Set to False when using a vanilla torchvision model that \\n\",\n \" does not need to be transformed into coco format.\\n\",\n \" \\n\",\n \" Returns: \\n\",\n \" imgIds (list): The list of images with predictions.\\n\",\n \" cocoDt (pycocotools.coco.COCO): Contains the predictions from the model \\n\",\n \" in coco format.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" print('\\\\n================ Starting Inference on {} Images using {} model ================\\\\n'.format(\\n\",\n \" len(dataloader), modelName))\\n\",\n \"\\n\",\n \" modelName = str(modelName).replace(\\\" \\\", \\\"_\\\")\\n\",\n \"\\n\",\n \" # convert predicition to cocoDt\\n\",\n \" # code from def evaluate in https://github.com/pytorch/vision/blob/master/references/detection/engine.py\\n\",\n \" imgIds = []\\n\",\n \" results = []\\n\",\n \" skippedImages = []\\n\",\n \"\\n\",\n \" # time inference\\n\",\n \" inference_time = 0.0\\n\",\n \" for idx, ((image, targets), (image_orig, _)) in enumerate(zip(dataloader, dataloader_orig)):\\n\",\n \" # if target is empty, skip the image because it breaks the scripted model\\n\",\n \" if not targets:\\n\",\n \" skippedImages.append(idx)\\n\",\n \" continue\\n\",\n \"\\n\",\n \" # get the predictions\\n\",\n \" start_time = time.time()\\n\",\n \" boxes, scores = model(image)\\n\",\n \" delta = time.time() - start_time\\n\",\n \" inference_time += delta\\n\",\n \" boxes, scores, labels = postprocess(boxes, scores)\\n\",\n \" outputs = get_results_as_dict(boxes, scores, labels, image_orig)\\n\",\n \"\\n\",\n \" res = {target[\\\"image_id\\\"].item(): output for target,\\n\",\n \" output in zip(targets, [outputs])}\\n\",\n \"\\n\",\n \" # add the image id to imgIds\\n\",\n \" image_id = targets[0][\\\"image_id\\\"].item()\\n\",\n \" imgIds.append(image_id)\\n\",\n \"\\n\",\n \" # convert the predicition into cocoDt results\\n\",\n \" pred = prepare_for_coco_detection(res)\\n\",\n \" results.extend(pred)\\n\",\n \"\\n\",\n \" print('\\\\n==================== Performance Measurement ====================')\\n\",\n \" print('Finished inference on {} images in {:.2f} seconds'.format(\\n\",\n \" len(dataloader), inference_time))\\n\",\n \" print('=================================================================\\\\n')\\n\",\n \"\\n\",\n \" # create bbox detections file\\n\",\n \" # following code in https://github.com/aws/aws-neuron-sdk/blob/master/src/examples/tensorflow/yolo_v4_demo/evaluate.ipynb\\n\",\n \" resultsfile = modelName + '_bbox_detections.json'\\n\",\n \" print('Generating json file...')\\n\",\n \" with open(resultsfile, 'w') as f:\\n\",\n \" json.dump(results, f)\\n\",\n \"\\n\",\n \" # return COCO api object with loadRes\\n\",\n \" cocoDt = cocoGt.loadRes(resultsfile)\\n\",\n \"\\n\",\n \" return imgIds, cocoDt\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The next step is to simply load the compiled model from disk and then run inference.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model_neuron = torch.jit.load('yolo_v4_neuron.pt')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"imgIds, cocoDt = run_inference(coco_val_data_loader, orig_coco_val_data_loader, model_neuron)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"We then use the standard `pycocotools` routines to generate a report of bounding box precision/recall.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from pycocotools.cocoeval import COCOeval\\n\",\n \"\\n\",\n \"cocoEval = COCOeval(cocoGt, cocoDt, 'bbox')\\n\",\n \"cocoEval.params.imgIds = imgIds\\n\",\n \"cocoEval.evaluate()\\n\",\n \"cocoEval.accumulate()\\n\",\n \"cocoEval.summarize()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"For reference, we may perform the same evaluation on the CPU model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"imgIdsRef, cocoDtRef = run_inference(coco_val_data_loader, orig_coco_val_data_loader, model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"cocoEval = COCOeval(cocoGt, cocoDtRef, 'bbox')\\n\",\n \"cocoEval.params.imgIds = imgIdsRef\\n\",\n \"cocoEval.evaluate()\\n\",\n \"cocoEval.accumulate()\\n\",\n \"cocoEval.summarize()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 5: Benchmark COCO Dataset Performance of the Neuron-Optimized TorchScript\\n\",\n \"The following code snippet sets up data parallel on 16 NeuronCores and runs saturated multi-threaded inference on the Inferentia accelerator. Note that the number of cores (`n_cores`) should be set to the number of available NeuronCores on the current instance.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"import torch.neuron\\n\",\n \"import torchvision\\n\",\n \"import torchvision.transforms as transforms\\n\",\n \"import torchvision.datasets as dset\\n\",\n \"import multiprocessing as mp\\n\",\n \"from concurrent.futures import ThreadPoolExecutor\\n\",\n \"import PIL\\n\",\n \"import os\\n\",\n \"import time\\n\",\n \"\\n\",\n \"n_threads = 16\\n\",\n \"\\n\",\n \"def get_image_filenames(root=os.getcwd()):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Generate paths to the coco dataset image files.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" root (str): The root folder contains.\\n\",\n \" \\n\",\n \" Yields:\\n\",\n \" filename (str): The path to an image file.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" image_path = os.path.join(root, 'val2017')\\n\",\n \" for root, dirs, files in os.walk(image_path):\\n\",\n \" for filename in files:\\n\",\n \" yield os.path.join(image_path, filename)\\n\",\n \"\\n\",\n \"def preprocess(path):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Load an image and convert to the expected Yolo V4 tensor format.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" path (str): The image file to load from disk. \\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" result (torch.Tensor): The image for prediction. Shape: [1, 3, 608, 608]\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" image = PIL.Image.open(path).convert('RGB')\\n\",\n \" resized = torchvision.transforms.functional.resize(image, [608, 608])\\n\",\n \" tensor = torchvision.transforms.functional.to_tensor(resized)\\n\",\n \" return tensor.unsqueeze(0).to(torch.float32)\\n\",\n \"\\n\",\n \"\\n\",\n \"def load_model(filename='yolo_v4_neuron.pt'):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Load and pre-warm the Yolo V4 model.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" filename (str): The location to load the model from.\\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" model (torch.nn.Module): The torch model.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" \\n\",\n \" # Load model from disk\\n\",\n \" model = torch.jit.load(filename)\\n\",\n \"\\n\",\n \" # Warm up model on neuron by running a single example image\\n\",\n \" filename = next(iter(get_image_filenames()))\\n\",\n \" image = preprocess(filename)\\n\",\n \" model(image)\\n\",\n \"\\n\",\n \" return model\\n\",\n \"\\n\",\n \"\\n\",\n \"def task(model, filename):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" The thread task to perform prediction.\\n\",\n \" \\n\",\n \" This does the full end-to-end processing of an image from loading from disk\\n\",\n \" all the way to classifying and filtering bounding boxes.\\n\",\n \" \\n\",\n \" Args:\\n\",\n \" model (torch.nn.Module): The model to run processing with\\n\",\n \" filename (str): The image file to load from disk. \\n\",\n \" \\n\",\n \" Returns:\\n\",\n \" boxes (torch.Tensor): The Yolo V4 bounding boxes.\\n\",\n \" scores (torch.Tensor): The label score for each box.\\n\",\n \" labels (torch.Tensor): The label for each box. \\n\",\n \" \\\"\\\"\\\"\\n\",\n \" image = preprocess(filename)\\n\",\n \" begin = time.time()\\n\",\n \" boxes, scores = model(image)\\n\",\n \" delta = time.time() - begin\\n\",\n \" return postprocess(boxes, scores), delta\\n\",\n \"\\n\",\n \"\\n\",\n \"def benchmark():\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Run a benchmark on the entire COCO dataset against the neuron model.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" \\n\",\n \" # Load a model into each NeuronCore\\n\",\n \" models = [load_model() for _ in range(n_cores)]\\n\",\n \" \\n\",\n \" # Create input/output lists\\n\",\n \" filenames = list(get_image_filenames())\\n\",\n \" results = list()\\n\",\n \" latency = list()\\n\",\n \" \\n\",\n \" # We want to keep track of average completion time per thread\\n\",\n \" sum_time = 0.0\\n\",\n \" \\n\",\n \" # Submit all tasks and wait for them to finish\\n\",\n \" with ThreadPoolExecutor(n_threads) as pool:\\n\",\n \" for i, filename in enumerate(filenames):\\n\",\n \" result = pool.submit(task, models[i % len(models)], filename)\\n\",\n \" results.append(result)\\n\",\n \" for result in results:\\n\",\n \" results, times = result.result() # Note: Outputs unused for benchmark\\n\",\n \" latency.append(times)\\n\",\n \" sum_time += times\\n\",\n \" \\n\",\n \" print('Duration: ', sum_time / n_threads)\\n\",\n \" print('Images Per Second:', len(filenames) / (sum_time / n_threads))\\n\",\n \" print(\\\"Latency P50: {:.1f}\\\".format(np.percentile(latency[1000:], 50)*1000.0))\\n\",\n \" print(\\\"Latency P90: {:.1f}\\\".format(np.percentile(latency[1000:], 90)*1000.0))\\n\",\n \" print(\\\"Latency P95: {:.1f}\\\".format(np.percentile(latency[1000:], 95)*1000.0))\\n\",\n \" print(\\\"Latency P99: {:.1f}\\\".format(np.percentile(latency[1000:], 99)*1000.0))\\n\",\n \"\\n\",\n \"benchmark()\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/tensorflow/bert_demo/LICENSE", "content": "Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n \nPermission is hereby granted, free of charge, to any person obtaining a copy of this\nsoftware and associated documentation files (the \"Software\"), to deal in the Software\nwithout restriction, including without limitation the rights to use, copy, modify,\nmerge, publish, distribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,\nINCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A\nPARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT\nHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\nSOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n" }, { "path": "src/examples/tensorflow/bert_demo/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/tensorflow/bert_demo/bert_client.py", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\nimport sys\nimport os\nimport argparse\nimport random\nimport time\nimport grpc\nimport mrpc_pb2\nsys.path.append(os.path.dirname(__file__))\nimport mrpc_pb2_grpc\nimport mrpc_feature\n\nlatencies = []\n\ndef client():\n parser = argparse.ArgumentParser()\n parser.add_argument('--port', default=60061, help='gRPC port')\n parser.add_argument('--pair', default=None, help='Text pair')\n parser.add_argument('--cycle', type=int, default=1, help='Number of inference cycles')\n parser.add_argument('--save-accuracy', default=None, help='Save accuracy to file')\n args = parser.parse_args()\n text_pair = mrpc_pb2.TextPair()\n if args.pair is not None:\n text_a, text_b = args.pair\n text_pair.text_a = text_a.encode()\n text_pair.text_b = text_b.encode()\n else:\n eval_data_path = os.path.join(os.path.dirname(__file__), 'glue_mrpc_dev.tsv')\n tsv = mrpc_feature.read_tsv(eval_data_path)\n with grpc.insecure_channel('127.0.0.1:{}'.format(args.port)) as channel:\n stub = mrpc_pb2_grpc.mrpcStub(channel)\n num_correct = 0\n very_start = time.time()\n for _ in range(args.cycle):\n if args.pair is None:\n data = random.choice(tsv[1:])\n text_pair.text_a = data[3].encode()\n text_pair.text_b = data[4].encode()\n start = time.time()\n yes_no = stub.paraphrase(text_pair)\n elapsed = time.time() - start\n if data is None:\n evaluation = ''\n else:\n if yes_no.prediction.decode() == data[0]:\n num_correct += 1\n evaluation = 'correct, ' if yes_no.prediction.decode() == data[0] else 'incorrect, '\n print('{} ({}latency {} s)'.format(yes_no.message.decode(), evaluation, elapsed))\n latencies.append(elapsed)\n if args.cycle > 1:\n accuracy = num_correct / args.cycle\n print('took {} s for {} cycles, accuracy {}'.format(time.time() - very_start, args.cycle, accuracy))\n if args.save_accuracy is not None:\n with open(args.save_accuracy, 'w') as f:\n f.write(str(accuracy))\n\ndef write_latencies():\n with open('latencies.txt', 'a') as f:\n for l in latencies:\n f.write(str(l) + '\\n')\n\nif __name__ == '__main__':\n client()\n write_latencies()\n" }, { "path": "src/examples/tensorflow/bert_demo/bert_model.py", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\nimport os\nimport argparse\nimport shlex\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.neuron import fuse\nfrom tensorflow.core.framework import attr_value_pb2\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--input_saved_model', required=True, help='Original SaveModel')\n parser.add_argument('--output_saved_model', required=True, help='Output SavedModel that runs on Inferentia')\n parser.add_argument('--dtype', default='float16', help='Data type for weights')\n parser.add_argument('--batch_size', type=int, default=4)\n parser.add_argument('--sequence_length', type=int, default=128)\n parser.add_argument('--crude_gelu', action='store_true')\n parser.add_argument('--aggressive_optimizations', action='store_true')\n args = parser.parse_args()\n if os.path.exists(args.output_saved_model):\n raise OSError('output_saved_model {} already exists'.format(args.output_saved_model))\n dtype = tf.float16 if args.dtype == 'float16' else tf.float32\n if args.aggressive_optimizations:\n args.crude_gelu = True\n bert = NeuronBERTMRPC(\n args.input_saved_model,\n dtype=dtype,\n batch_size=args.batch_size,\n seq_len=args.sequence_length,\n crude_gelu=args.crude_gelu,\n aggressive_fp16_cast=args.aggressive_optimizations,\n )\n\n fuser = fuse(compiler_args=['--fp32-cast', 'matmult'], timeout=360000)\n bert.encoder = fuser(bert.encoder)\n\n input_ids = bert.input_ids\n input_mask = bert.input_mask\n segment_ids = bert.segment_ids\n with tf.Session(graph=tf.Graph()) as sess:\n input_ids_ph_shape = input_ids.shape.as_list()\n input_ids_ph_shape[0] = None\n input_ids_ph = tf.placeholder(input_ids.dtype, input_ids_ph_shape, name='input_ids')\n\n input_mask_ph_shape = input_mask.shape.as_list()\n input_mask_ph_shape[0] = None\n input_mask_ph = tf.placeholder(input_mask.dtype, input_mask_ph_shape, name='input_mask')\n\n segment_ids_ph_shape = segment_ids.shape.as_list()\n segment_ids_ph_shape[0] = None\n segment_ids_ph = tf.placeholder(segment_ids.dtype, segment_ids_ph_shape, name='segment_ids')\n\n dummy_reshapes = []\n discard_op_names = set()\n\n with tf.name_scope('bert/embeddings'):\n expand_dims = tf.expand_dims(input_ids_ph, axis=-1)\n batch_size = tf.shape(input_ids_ph)[0]\n reshape = tf.reshape(expand_dims, [batch_size * bert.seq_len])\n gatherv2 = tf.gather(bert.weights_dict['bert/embeddings/word_embeddings:0'], reshape, axis=0)\n reshape_1 = tf.reshape(gatherv2, [batch_size, bert.seq_len, bert.hid_size])\n reshape_2 = tf.reshape(segment_ids_ph, [batch_size * bert.seq_len])\n one_hot = tf.one_hot(reshape_2, depth=2)\n matmul = tf.matmul(one_hot, bert.weights_dict['bert/embeddings/token_type_embeddings:0'])\n reshape_3 = tf.reshape(matmul, [batch_size, bert.seq_len, bert.hid_size])\n slice0 = tf.slice(bert.weights_dict['bert/embeddings/position_embeddings:0'], begin=[0, 0], size=[bert.seq_len, -1])\n add_1 = reshape_1 + reshape_3 + slice0\n input_tensor = tf.reshape(add_1, [batch_size, bert.seq_len, bert.hid_size])\n with tf.name_scope('bert/encoder'):\n reshape = tf.reshape(input_mask_ph, [batch_size, 1, 1, bert.seq_len])\n bias_tensor = tf.cast(reshape, tf.float32)\n bias_tensor = 1.0 - bias_tensor\n bias_tensor = bias_tensor * -10000.0\n bias_tensor = tf.cast(bias_tensor, bert.dtype)\n tensor = bert.layer_norm(input_tensor, 'embeddings', force_float32=True)\n\n tensor = tf.reshape(tensor, [bert.batch_size, bert.seq_len, bert.hid_size])\n dummy_reshapes.append(tensor)\n discard_op_names.add(tensor.op.name)\n bias_tensor = tf.reshape(bias_tensor, [bert.batch_size, 1, 1, bert.seq_len])\n dummy_reshapes.append(bias_tensor)\n discard_op_names.add(bias_tensor.op.name)\n\n logits = bert.encoder(tensor, bias_tensor)\n with tf.name_scope('loss'):\n if bert.dtype is not tf.float32:\n logits = tf.cast(logits, tf.float32)\n probabilities = tf.nn.softmax(logits)\n for rts in dummy_reshapes:\n neuron_op = rts.consumers()[0]\n neuron_op._update_input(list(neuron_op.inputs).index(rts), rts.op.inputs[0])\n try:\n sess.run(probabilities)\n except:\n pass\n graph_def = sess.graph.as_graph_def()\n new_graph_def = tf.GraphDef()\n new_graph_def.node.MergeFrom(node for node in graph_def.node if node.name not in discard_op_names)\n neuron_op_node = [node for node in new_graph_def.node if node.op == 'NeuronOp'][0]\n neuron_op_node.attr['input_batch_axis'].list.i[:] = [0, 0]\n neuron_op_node.attr['output_batch_axis'].list.i[:] = [0]\n\n with tf.Session(graph=tf.Graph()) as sess:\n tf.import_graph_def(new_graph_def, name='')\n inputs = {\n 'input_ids': sess.graph.get_tensor_by_name(input_ids_ph.name),\n 'input_mask': sess.graph.get_tensor_by_name(input_mask_ph.name),\n 'segment_ids': sess.graph.get_tensor_by_name(segment_ids_ph.name),\n }\n outputs = {\n 'probabilities': sess.graph.get_tensor_by_name(probabilities.name)\n }\n try:\n sess.run(probabilities)\n except:\n pass\n neuron_op = [op for op in sess.graph.get_operations() if op.type == 'NeuronOp'][0]\n if not neuron_op.get_attr('executable'):\n raise AttributeError('Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly (`pip install neuron-cc` to install neuron-cc).')\n tf.saved_model.simple_save(sess, args.output_saved_model, inputs, outputs)\n\n\nclass NeuronBERTMRPC:\n\n def __init__(self, bert_saved_model, dtype=tf.float16, batch_size=4, seq_len=128, crude_gelu=False, aggressive_fp16_cast=False):\n predictor = tf.contrib.predictor.from_saved_model(bert_saved_model)\n sess = predictor.session\n self.input_ids = predictor.feed_tensors['input_ids']\n self.input_mask = predictor.feed_tensors['input_mask']\n self.segment_ids = predictor.feed_tensors['segment_ids']\n weights_dict = {}\n for op in sess.graph.get_operations():\n if op.type == 'Const':\n tensor = op.outputs[0]\n weights_dict[tensor.name] = tensor\n if op.type == 'Identity' and op.name.endswith('read'):\n tensor = op.outputs[0]\n weights_dict[tensor.op.inputs[0].name] = tensor\n self.weights_dict = sess.run(weights_dict)\n self.dtype = dtype\n self.batch_size = batch_size\n self.seq_len = seq_len\n self.hid_size, self.inter_size = self.weights_dict['bert/encoder/layer_0/intermediate/dense/kernel:0'].shape\n self.num_heads = sess.graph.get_tensor_by_name('bert/encoder/layer_0/attention/self/Reshape:0').shape.as_list()[2]\n self.head_size = self.hid_size // self.num_heads\n self.eps = self.weights_dict['bert/encoder/layer_0/attention/output/LayerNorm/batchnorm/add/y:0']\n self.crude_gelu = crude_gelu\n self.layer_norm_dtype = tf.float16 if aggressive_fp16_cast else tf.float32\n sess.close()\n\n def encoder(self, tensor, bias_tensor):\n tensor = tf.reshape(tensor, [self.batch_size * self.seq_len, self.hid_size])\n for layer_id in range(24):\n mid_layer_name = 'layer_{}'.format(layer_id)\n tensor = self.self_attention(tensor, bias_tensor, mid_layer_name)\n tensor = self.layer_norm(tensor, 'encoder/' + mid_layer_name + '/attention/output')\n tensor = self.fully_connected(tensor, mid_layer_name)\n tensor = self.layer_norm(tensor, 'encoder/' + mid_layer_name + '/output')\n logits = self.pooler_loss(tensor)\n return logits\n\n def fully_connected(self, input_tensor, layer_name):\n inter_kernel = self.weights_dict['bert/encoder/{}/intermediate/dense/kernel:0'.format(layer_name)]\n inter_bias = self.weights_dict['bert/encoder/{}/intermediate/dense/bias:0'.format(layer_name)]\n out_kernel = self.weights_dict['bert/encoder/{}/output/dense/kernel:0'.format(layer_name)]\n out_bias = self.weights_dict['bert/encoder/{}/output/dense/bias:0'.format(layer_name)]\n with tf.name_scope('bert/encoder/{}/fully_connected/intermediate/dense'.format(layer_name)):\n matmul = tf.matmul(input_tensor, inter_kernel.astype(self.dtype.as_numpy_dtype))\n bias_add = tf.nn.bias_add(matmul, inter_bias.astype(self.dtype.as_numpy_dtype))\n gelu = self.gelu_sigmoid(bias_add) if self.crude_gelu else self.gelu_tanh(bias_add)\n with tf.name_scope('bert/encoder/{}/fully_connected/output/dense'.format(layer_name)):\n matmul = tf.matmul(gelu, out_kernel.astype(self.dtype.as_numpy_dtype))\n bias_add = tf.nn.bias_add(matmul, out_bias.astype(self.dtype.as_numpy_dtype))\n output_tensor = bias_add + input_tensor\n return output_tensor\n\n def self_attention(self, input_tensor, bias_tensor, layer_name):\n query_kernel = self.weights_dict['bert/encoder/{}/attention/self/query/kernel:0'.format(layer_name)] * 0.125\n query_bias = self.weights_dict['bert/encoder/{}/attention/self/query/bias:0'.format(layer_name)] * 0.125\n key_kernel = self.weights_dict['bert/encoder/{}/attention/self/key/kernel:0'.format(layer_name)]\n key_bias = self.weights_dict['bert/encoder/{}/attention/self/key/bias:0'.format(layer_name)]\n value_kernel = self.weights_dict['bert/encoder/{}/attention/self/value/kernel:0'.format(layer_name)]\n value_bias = self.weights_dict['bert/encoder/{}/attention/self/value/bias:0'.format(layer_name)]\n output_kernel = self.weights_dict['bert/encoder/{}/attention/output/dense/kernel:0'.format(layer_name)]\n output_bias = self.weights_dict['bert/encoder/{}/attention/output/dense/bias:0'.format(layer_name)]\n with tf.name_scope('bert/encoder/{}/attention/self'.format(layer_name)):\n matmul = tf.matmul(input_tensor, query_kernel.astype(self.dtype.as_numpy_dtype))\n query = tf.nn.bias_add(matmul, query_bias.astype(self.dtype.as_numpy_dtype))\n query_r = tf.reshape(query, [self.batch_size, self.seq_len, self.num_heads, self.head_size])\n query_rt = tf.transpose(query_r, [0, 2, 1, 3])\n matmul = tf.matmul(input_tensor, key_kernel.astype(self.dtype.as_numpy_dtype))\n key = tf.nn.bias_add(matmul, key_bias.astype(self.dtype.as_numpy_dtype))\n key_r = tf.reshape(key, [self.batch_size, self.seq_len, self.num_heads, self.head_size])\n key_rt = tf.transpose(key_r, [0, 2, 1, 3]) # [b, n, l, h]\n query_key = tf.matmul(query_rt, key_rt, transpose_b=True) # [b, n, lq, h] @ [b, n, lk, h] -> [b, n, lq, lk]\n bias_query_key = tf.add(query_key, bias_tensor)\n softmax_weights = tf.nn.softmax(bias_query_key)\n matmul = tf.matmul(input_tensor, value_kernel.astype(self.dtype.as_numpy_dtype))\n value = tf.nn.bias_add(matmul, value_bias.astype(self.dtype.as_numpy_dtype))\n value_r = tf.reshape(value, [self.batch_size, self.seq_len, self.num_heads, self.head_size])\n value_rt = tf.transpose(value_r, [0, 2, 3, 1])\n weighted_value_rt = tf.matmul(softmax_weights, value_rt, transpose_b=True) # [b, n, lq, lk] @ [b, n, h, lv] -> [b, n, lq, h]\n weighted_value_r = tf.transpose(weighted_value_rt, [0, 2, 1, 3]) # [b, lq, n, h]\n weighted_value = tf.reshape(weighted_value_r, [self.batch_size * self.seq_len, self.hid_size])\n with tf.name_scope('bert/encoder/{}/attention/output'.format(layer_name)):\n matmul = tf.matmul(weighted_value, output_kernel.astype(self.dtype.as_numpy_dtype))\n unnorm_output = tf.nn.bias_add(matmul, output_bias.astype(self.dtype.as_numpy_dtype))\n output_tensor = tf.add(input_tensor, unnorm_output)\n return output_tensor\n\n def layer_norm(self, input_tensor, layer_name, force_float32=False):\n dtype = tf.float32 if force_float32 else self.layer_norm_dtype\n gamma = dtype.as_numpy_dtype(self.weights_dict['bert/{}/LayerNorm/gamma:0'.format(layer_name)])\n beta = dtype.as_numpy_dtype(self.weights_dict['bert/{}/LayerNorm/beta:0'.format(layer_name)])\n with tf.name_scope('bert/{}/LayerNorm'.format(layer_name)):\n input_tensor = tf.cast(input_tensor, dtype)\n mean = tf.reduce_mean(input_tensor, axis=[-1], keepdims=True, name='mean')\n residuals = tf.subtract(input_tensor, mean, name='residuals')\n var = tf.reduce_mean(residuals * residuals, axis=[-1], keepdims=True, name='var')\n rsqrt = tf.rsqrt(var + dtype.as_numpy_dtype(self.eps))\n norm_output = tf.multiply(residuals, rsqrt, name='normalized')\n output_tensor = norm_output * gamma + beta\n output_tensor = tf.cast(output_tensor, self.dtype)\n return output_tensor\n\n def pooler_loss(self, input_tensor):\n pooler_kernel = self.weights_dict['bert/pooler/dense/kernel:0']\n pooler_bias = self.weights_dict['bert/pooler/dense/bias:0']\n loss_kernel = self.weights_dict['output_weights:0'].T\n loss_bias = self.weights_dict['output_bias:0']\n with tf.name_scope('bert/pooler_loss'):\n reshape = tf.reshape(input_tensor, [self.batch_size, self.seq_len, self.hid_size])\n reshape_1 = tf.reshape(reshape[:, 0:1, :], [self.batch_size, self.hid_size])\n matmul = tf.matmul(reshape_1, pooler_kernel.astype(self.dtype.as_numpy_dtype))\n bias_add = tf.nn.bias_add(matmul, pooler_bias.astype(self.dtype.as_numpy_dtype))\n tanh = tf.tanh(bias_add)\n matmul = tf.matmul(tanh, loss_kernel.astype(self.dtype.as_numpy_dtype))\n output_tensor = tf.nn.bias_add(matmul, loss_bias.astype(self.dtype.as_numpy_dtype))\n return output_tensor\n\n def gelu_tanh(self, tensor):\n pow3 = 0.044714998453855515 * tensor * tensor * tensor + tensor\n shifted = (tf.tanh(0.7978845834732056 * pow3) + 1.0) * tensor\n return tf.multiply(shifted, 0.5)\n\n def gelu_sigmoid(self, tensor):\n return tf.sigmoid(1.702 * tensor) * tensor\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/bert_demo/bert_model_server.py", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\nimport os\nimport argparse\nimport subprocess\nimport time\n\n\n_ONE_DAY_IN_SECONDS = 60 * 60 * 24\n\n\ndef serve():\n parser = argparse.ArgumentParser()\n parser.add_argument('--serving', required=True, help='Path to tf-serving binary')\n parser.add_argument('--dir', required=True, help='TensorFlow SavedModel dir')\n parser.add_argument('--port', default=8500, help='gRPC port')\n parser.add_argument('--parallel', type=int, default=8, help='Number of predictors')\n args = parser.parse_args()\n model = os.path.abspath(args.dir)\n model_with_version = os.path.join(model, '1')\n if not os.path.exists(model_with_version):\n os.makedirs(model_with_version)\n os.symlink(os.path.join(model, 'variables'), os.path.join(model_with_version, 'variables'))\n os.symlink(os.path.join(model, 'saved_model.pb'), os.path.join(model_with_version, 'saved_model.pb'))\n process_list = []\n for _ in range(args.parallel):\n proc = subprocess.Popen([\n args.serving, '--model_base_path={}'.format(model), '--port={}'.format(args.port),\n '--tensorflow_intra_op_parallelism=1', '--tensorflow_inter_op_parallelism=1'\n ])\n process_list.append(proc)\n try:\n time.sleep(_ONE_DAY_IN_SECONDS)\n except KeyboardInterrupt:\n for proc in process_list:\n proc.terminate()\n proc.wait()\n\n\nif __name__ == '__main__':\n serve()\n" }, { "path": "src/examples/tensorflow/bert_demo/bert_no_model.py", "content": "# bert_no_model.py\nimport argparse\nimport tensorflow as tf\nimport tensorflow.neuron as tfn\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--input_saved_model', required=True, help='Original SaveModel')\n parser.add_argument('--output_saved_model', required=True, help='Output SavedModel that runs on Inferentia')\n parser.add_argument('--batch_size', type=int, default=1)\n args = parser.parse_args()\n pred = tf.contrib.predictor.from_saved_model(args.input_saved_model)\n no_fuse_ops = [op.name for op in pred.graph.get_operations()]\n\n def force_fuse_condition(op_name):\n exclude_scopes = [\n 'bert/encoder/strided_slice',\n 'bert/encoder/ones',\n 'bert/encoder/Reshape',\n 'bert/encoder/Shape',\n 'bert/encoder/Cast',\n ]\n for scope in exclude_scopes:\n if op_name == scope or op_name.startswith('{}/'.format(scope)):\n return False\n return op_name.startswith('bert/encoder') or op_name.startswith('bert/pooler')\n\n force_fuse_ops = [op.name for op in pred.graph.get_operations() if force_fuse_condition(op.name)]\n compilation_result = tfn.saved_model.compile(\n args.input_saved_model, args.output_saved_model,\n batch_size=args.batch_size,\n no_fuse_ops=no_fuse_ops,\n force_fuse_ops=force_fuse_ops,\n )\n print(compilation_result)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/bert_demo/bert_server.py", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\nimport sys\nimport os\nimport collections\nimport argparse\nimport time\nimport csv\nimport random\nfrom concurrent import futures\nimport multiprocessing\nfrom multiprocessing.dummy import Pool\nfrom threading import Lock\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport grpc\nimport numpy as np\nimport tensorflow as tf\nimport mrpc_feature\nimport tokenization\nimport mrpc_pb2\nsys.path.append(os.path.dirname(__file__))\nimport mrpc_pb2_grpc\n\n\n_ONE_DAY_IN_SECONDS = 60 * 60 * 24\n\ntotal_tpt = 0\nnum_tpt = 0\n\nclass BERTService(mrpc_pb2_grpc.mrpcServicer):\n\n def __init__(self, model_path, parallel, batch_size, bootstrap, vocab_txt, num_thread_per_predictor=2):\n num_queues = parallel * num_thread_per_predictor\n config = tf.ConfigProto(inter_op_parallelism_threads=num_queues, intra_op_parallelism_threads=1)\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version >= LooseVersion('1.15.0.1.0.1333.0'):\n neuroncore_group_sizes = '{}x1'.format(parallel)\n predictor = tf.contrib.predictor.from_saved_model(model_path, config=config)\n self.predictor_list = [predictor for _ in range(num_queues)]\n else:\n neuroncore_group_sizes = ','.join('1' for _ in range(parallel))\n predictor_list = [tf.contrib.predictor.from_saved_model(model_path, config=config) for _ in range(parallel)]\n self.predictor_list = []\n for pred in predictor_list:\n self.predictor_list.extend(pred for _ in range(num_thread_per_predictor))\n os.environ['NEURONCORE_GROUP_SIZES'] = neuroncore_group_sizes\n if self.predictor_list[0].feed_tensors['input_ids'].shape.is_fully_defined():\n self.batch_size = self.predictor_list[0].feed_tensors['input_ids'].shape.as_list()[0]\n else:\n self.batch_size = batch_size\n self.bootstrap = bootstrap\n self.tokenizer = tokenization.FullTokenizer(vocab_file=vocab_txt, do_lower_case=True)\n self.num_infer = 0\n self.num_correct = 0\n self.output_name = list(self.predictor_list[0].fetch_tensors.keys())[0]\n self.iid = 0\n self.throughput_list = []\n self.latency_list = []\n self.max_len_latency_list = 1000\n self.iid_lock = Lock()\n if bootstrap:\n self.request_queue_list = [collections.deque() for _ in self.predictor_list]\n eval_data_path = os.path.join(os.path.dirname(__file__), 'glue_mrpc_dev.tsv')\n tsv = mrpc_feature.read_tsv(eval_data_path)\n for request_queue in self.request_queue_list:\n for _ in range(1024):\n data_list = random.choices(tsv[1:], k=self.batch_size)\n model_feed_dict_list = [mrpc_feature.text_pair_to_model_feed_dict(data[3], data[4], self.tokenizer) for data in data_list]\n label_list = [int(data[0]) for data in data_list]\n batch_labels = np.array(label_list)\n batch_feeds = {\n key: np.concatenate([feed[key] for feed in model_feed_dict_list], axis=0)\n for key in model_feed_dict_list[0].keys()\n }\n request_queue.append((batch_feeds, batch_labels))\n else:\n self.request_queue_list = [[] for _ in self.predictor_list]\n self.result_map = {}\n self.alive = True\n dummy_feed = {\n 'input_ids': np.zeros([1, 128], dtype=np.int32),\n 'input_mask': np.zeros([1, 128], dtype=np.int32),\n 'segment_ids': np.zeros([1, 128], dtype=np.int32),\n }\n self.dummy_feeds = [(None, dummy_feed) for _ in range(self.batch_size)]\n model_feed_dict_list = [dummy_feed for _ in range(self.batch_size)]\n batch_feeds = {\n key: np.concatenate([feed[key] for feed in model_feed_dict_list], axis=0)\n for key in model_feed_dict_list[0].keys()\n }\n pool = Pool(len(self.predictor_list))\n for pred in self.predictor_list:\n pool.apply_async(pred, (batch_feeds,))\n time.sleep(1)\n pool.close()\n pool.join()\n\n def cleanup(self):\n for pred in self.predictor_list:\n print(pred)\n pred.session.close()\n\n def current_throughput(self):\n last_num_infer = self.num_infer\n global total_tpt\n global num_tpt\n while self.alive:\n current_num_infer = self.num_infer\n throughput = current_num_infer - last_num_infer\n self.throughput_list.append(throughput)\n print('current throughput {}'.format(throughput))\n last_num_infer = current_num_infer\n if throughput != 0:\n total_tpt += throughput\n num_tpt += 1\n time.sleep(1)\n\n def current_throughput_accuracy(self):\n last_num_infer = self.num_infer\n while self.alive:\n current_num_infer = self.num_infer\n accuracy = 0.0 if self.num_infer == 0 else self.num_correct / self.num_infer\n print('current throughput {}, accuracy {}'.format(current_num_infer - last_num_infer, accuracy))\n last_num_infer = current_num_infer\n if throughput != 0:\n total_tpt += throughput\n num_tpt += 1\n time.sleep(1)\n\n def paraphrase(self, text_pair, context):\n iid = self.put_input(text_pair.text_a, text_pair.text_b)\n yes_no = mrpc_pb2.YesNo()\n if self.get_output(iid) == 1:\n yes_no.message = b'paraphrase!'\n yes_no.prediction = b'1'\n else:\n yes_no.message = b'not paraphrase!'\n yes_no.prediction = b'0'\n return yes_no\n\n def put_input(self, text_a, text_b):\n model_feed_dict = mrpc_feature.text_pair_to_model_feed_dict(text_a, text_b, self.tokenizer)\n with self.iid_lock:\n self.iid += 1\n iid = self.iid\n self.request_queue_list[iid % len(self.request_queue_list)].append((iid, model_feed_dict))\n return iid\n\n def process_input(self, idx):\n print('input processor is waiting')\n request_queue = self.request_queue_list[idx]\n predictor = self.predictor_list[idx]\n while self.alive:\n if len(request_queue) > 0:\n sublist = request_queue[:self.batch_size]\n request_queue[:self.batch_size] = []\n if len(sublist) != self.batch_size:\n print('batch with {} garbage entries!'.format(self.batch_size - len(sublist)))\n if len(sublist) < self.batch_size:\n pad_batch_size = self.batch_size - len(sublist)\n sublist.extend(self.dummy_feeds[:pad_batch_size])\n iid_list = [iid for iid, _ in sublist]\n model_feed_dict_list = [feed for _, feed in sublist]\n batch_feeds = {\n key: np.concatenate([feed[key] for feed in model_feed_dict_list], axis=0)\n for key in model_feed_dict_list[0].keys()\n }\n start = time.time()\n batch_predictions = predictor(batch_feeds)[self.output_name].argmax(-1)\n latency = time.time() - start\n if len(self.latency_list) < self.max_len_latency_list:\n self.latency_list.append(latency)\n self.result_map.update({iid: pred for iid, pred in zip(iid_list, batch_predictions)})\n time.sleep(0.001)\n\n def process_input_bootstrap(self, idx):\n print('input processor is waiting')\n request_queue = self.request_queue_list[idx]\n predictor = self.predictor_list[idx]\n while self.alive:\n if len(request_queue) > 0:\n batch_feeds, batch_labels = request_queue.popleft()\n batch_predictions = predictor(batch_feeds)[self.output_name].argmax(-1)\n self.num_infer += self.batch_size\n self.num_correct += (batch_predictions == batch_labels).sum()\n continue\n time.sleep(0.0001)\n\n def get_output(self, iid):\n while iid not in self.result_map:\n time.sleep(0.001)\n self.num_infer += 1\n return self.result_map.pop(iid)\n\n\ndef serve():\n parser = argparse.ArgumentParser()\n parser.add_argument('--port', default=60061, help='gRPC port')\n parser.add_argument('--dir', required=True, help='TensorFlow SavedModel dir')\n parser.add_argument('--parallel', type=int, default=4, help='Number of predictors')\n parser.add_argument('--thread', type=int, default=2, help='Number of threads used by each predictor')\n parser.add_argument('--batch', type=int, default=4, help='Batch size')\n parser.add_argument('--bootstrap', action='store_true',\n help='Server loads a dataset and run inference itself')\n args = parser.parse_args()\n vocab_txt = os.path.join(os.path.dirname(__file__), 'uncased_L-24_H-1024_A-16.vocab.txt')\n bert_service = BERTService(args.dir, args.parallel, args.batch, args.bootstrap, vocab_txt, args.thread)\n server = grpc.server(\n futures.ThreadPoolExecutor(max_workers=128),\n options=[('grpc.max_send_message_length', -1),\n ('grpc.max_receive_message_length', -1)])\n mrpc_pb2_grpc.add_mrpcServicer_to_server(bert_service, server)\n server.add_insecure_port('[::]:{}'.format(args.port))\n server.start()\n try:\n pool = Pool(len(bert_service.predictor_list) + 1) # +1 for bert_service.current_throughput\n if args.bootstrap:\n monitor_func = bert_service.current_throughput_accuracy\n process_func = bert_service.process_input_bootstrap\n else:\n monitor_func = bert_service.current_throughput\n process_func = bert_service.process_input\n pool.apply_async(monitor_func)\n if args.parallel == 1:\n process_func(0)\n else:\n for idx in range(len(bert_service.predictor_list)):\n pool.apply_async(process_func, (idx,))\n pool.close()\n time.sleep(_ONE_DAY_IN_SECONDS)\n except KeyboardInterrupt:\n pass\n bert_service.cleanup()\n bert_service.alive = False\n server.stop(0)\n\n\nif __name__ == '__main__':\n serve()\n print(f'Average Throughput: {total_tpt/num_tpt}')\n" }, { "path": "src/examples/tensorflow/bert_demo/download_mrpc_data.py", "content": "import os\nimport sys\nimport argparse\nimport urllib.request\n\nMRPC_TRAIN = 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt'\nMRPC_TEST = 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_test.txt'\n\n# This function is taken from https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e.\ndef format_mrpc(data_dir, path_to_data, path_to_dev_tsv):\n print(\"Processing MRPC...\")\n mrpc_dir = os.path.join(data_dir, \"MRPC\")\n if not os.path.isdir(mrpc_dir):\n os.mkdir(mrpc_dir)\n if path_to_data:\n mrpc_train_file = os.path.join(path_to_data, \"msr_paraphrase_train.txt\")\n mrpc_test_file = os.path.join(path_to_data, \"msr_paraphrase_test.txt\")\n else:\n try:\n mrpc_train_file = os.path.join(mrpc_dir, \"msr_paraphrase_train.txt\")\n mrpc_test_file = os.path.join(mrpc_dir, \"msr_paraphrase_test.txt\")\n urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file)\n urllib.request.urlretrieve(MRPC_TEST, mrpc_test_file)\n except urllib.error.HTTPError:\n print(\"Error downloading MRPC\")\n return\n assert os.path.isfile(mrpc_train_file), \"Train data not found at %s\" % mrpc_train_file\n assert os.path.isfile(mrpc_test_file), \"Test data not found at %s\" % mrpc_test_file\n\n with open(mrpc_test_file, encoding='utf-8') as data_fh, \\\n open(os.path.join(mrpc_dir, \"test.tsv\"), 'w', encoding='utf-8') as test_fh:\n header = data_fh.readline()\n test_fh.write(\"index\\t#1 ID\\t#2 ID\\t#1 String\\t#2 String\\n\")\n for idx, row in enumerate(data_fh):\n label, id1, id2, s1, s2 = row.strip().split('\\t')\n test_fh.write(\"%d\\t%s\\t%s\\t%s\\t%s\\n\" % (idx, id1, id2, s1, s2))\n\n dev_ids = []\n with open(path_to_dev_tsv, encoding='utf-8') as dev_fh:\n header = dev_fh.readline()\n for row in dev_fh:\n _, id1, id2, _, _ = row.strip().split('\\t')\n dev_ids.append([id1, id2])\n\n with open(mrpc_train_file, encoding='utf-8') as data_fh, \\\n open(os.path.join(mrpc_dir, \"train.tsv\"), 'w', encoding='utf-8') as train_fh, \\\n open(os.path.join(mrpc_dir, \"dev.tsv\"), 'w', encoding='utf-8') as dev_fh:\n header = data_fh.readline()\n train_fh.write(header)\n dev_fh.write(header)\n for row in data_fh:\n label, id1, id2, s1, s2 = row.strip().split('\\t')\n if [id1, id2] in dev_ids:\n dev_fh.write(\"%s\\t%s\\t%s\\t%s\\t%s\\n\" % (label, id1, id2, s1, s2))\n else:\n train_fh.write(\"%s\\t%s\\t%s\\t%s\\t%s\\n\" % (label, id1, id2, s1, s2))\n \n print(\"\\tCompleted!\")\n\ndef main(arguments):\n parser = argparse.ArgumentParser()\n parser.add_argument('--data_dir', help='directory to save data to', type=str, default='glue_data')\n parser.add_argument('--path_to_mrpc', help='path to directory containing extracted MRPC data, msr_paraphrase_train.txt and msr_paraphrase_text.txt',\n type=str, default='')\n parser.add_argument('--path_to_dev_tsv', help='path to directory containing the glue_mrpc_dev.tsv', type=str, default='glue_mrpc_dev.tsv')\n args = parser.parse_args(arguments)\n\n if not os.path.isdir(args.data_dir):\n os.mkdir(args.data_dir)\n format_mrpc(args.data_dir, args.path_to_mrpc, args.path_to_dev_tsv)\n\nif __name__ == '__main__':\n sys.exit(main(sys.argv[1:]))" }, { "path": "src/examples/tensorflow/bert_demo/glue_mrpc_dev.tsv", "content": "Quality\t#1 ID\t#2 ID\t#1 String\t#2 String\n1\t1355540\t1355592\tHe said the foodservice pie business doesn 't fit the company 's long-term growth strategy .\t\" The foodservice pie business does not fit our long-term growth strategy .\n0\t2029631\t2029565\tMagnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war .\tHis wife said he was \" 100 percent behind George Bush \" and looked forward to using his years of training in the war .\n0\t487993\t487952\tThe dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat .\tThe dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .\n1\t1989515\t1989458\tThe AFL-CIO is waiting until October to decide if it will endorse a candidate .\tThe AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .\n0\t1783137\t1782659\tNo dates have been set for the civil or the criminal trial .\tNo dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .\n1\t3039165\t3039036\tWal-Mart said it would check all of its million-plus domestic workers to ensure they were legally employed .\tIt has also said it would review all of its domestic employees more than 1 million to ensure they have legal status .\n0\t1490811\t1490840\tWhile dioxin levels in the environment were up last year , they have dropped by 75 percent since the 1970s , said Caswell .\tThe Institute said dioxin levels in the environment have fallen by as much as 76 percent since the 1970s .\n1\t426112\t426210\tThis integrates with Rational PurifyPlus and allows developers to work in supported versions of Java , Visual C # and Visual Basic .NET.\tIBM said the Rational products were also integrated with Rational PurifyPlus , which allows developers to work in Java , Visual C # and VisualBasic .Net.\n1\t1439663\t1439808\tThe top rate will go to 4.45 percent for all residents with taxable incomes above $ 500,000 .\tFor residents with incomes above $ 500,000 , the income-tax rate will increase to 4.45 percent .\n1\t3147370\t3147525\tThe results appear in the January issue of Cancer , an American Cancer Society journal , being published online today .\tThe results appear in the January issue of Cancer , an American Cancer Society ( news - web sites ) journal , being published online Monday .\n1\t3300040\t3299992\tThe delegates said raising and distributing funds has been complicated by the U.S. crackdown on jihadi charitable foundations , bank accounts of terror-related organizations and money transfers .\tBin Laden ’ s men pointed out that raising and distributing funds has been complicated by the U.S. crackdown on jihadi charitable foundations , bank accounts of terror-related organizations and money transfers .\n0\t524136\t524119\t\" Sanitation is poor ... there could be typhoid and cholera , \" he said .\t\" Sanitation is poor , drinking water is generally left behind . . . there could be typhoid and cholera . \"\n0\t969512\t969295\tThe broader Standard & Poor 's 500 Index .SPX gave up 11.91 points , or 1.19 percent , at 986.60 .\tThe technology-laced Nasdaq Composite Index was down 25.36 points , or 1.53 percent , at 1,628.26 .\n1\t1685339\t1685429\tThe only announced Republican to replace Davis is Rep. Darrell Issa of Vista , who has spent $ 1.71 million of his own money to force a recall .\tSo far the only declared major party candidate is Rep. Darrell Issa , a Republican who has spent $ 1.5 million of his own money to fund the recall .\n1\t1967578\t1967664\tThe decision to issue new guidance has been prompted by intelligence passed to Britain by the FBI in a secret briefing in late July .\tScotland Yard 's decision to issue new guidance has been prompted by new intelligence passed to Britain by the FBI in late July .\n1\t2047034\t2046820\tUnable to find a home for him , a judge told mental health authorities they needed to find supervised housing and treatment for DeVries somewhere in California .\tThe judge had told the state Department of Mental Health to find supervised housing and treatment for DeVries somewhere in California .\n1\t2046630\t2046644\tThe decision came a year after Whipple ended federal oversight of the district 's racial balance , facilities , budget , and busing .\tThe decision came a year after Whipple ended federal oversight of school busing as well as the district 's racial balance , facilities and budget .\n0\t2221603\t2221633\tIn midafternoon trading , the Nasdaq composite index was up 8.34 , or 0.5 percent , to 1,790.47 .\tThe Nasdaq Composite Index .IXIC dipped 8.59 points , or 0.48 percent , to 1,773.54 .\n1\t129995\t129864\tMorgan Stanley raised its rating on the beverage maker to \" overweight \" from \" equal-weight \" saying in part that pricing power with its bottlers should improve in 2004 .\tMorgan Stanley raised its rating on the company to \" overweight \" from \" equal-weight , \" saying the beverage maker 's pricing power with bottlers should improve in 2004 .\n0\t919683\t919782\tThe pound also made progress against the dollar , reached fresh three-year highs at $ 1.6789 .\tThe British pound flexed its muscle against the dollar , last up 1 percent at $ 1.6672 .\n0\t970740\t971209\tFriday , Stanford ( 47-15 ) blanked the Gamecocks 8-0 .\tStanford ( 46-15 ) has a team full of such players this season .\n1\t2745055\t2745022\tLast month Intel raised its revenue guidance for the quarter to between $ 7.6 billion and $ 7.8 billion .\tAt the end of the second quarter , Intel initially predicted sales of between $ 6.9 billion and $ 7.5 billion .\n0\t2199097\t2199072\tThe driver , Eugene Rogers , helped to remove children from the bus , Wood said .\tAt the accident scene , the driver was \" covered in blood \" but helped to remove children , Wood said .\n1\t1609290\t1609098\tONG KONG , July 9 Tens of thousands of demonstrators gathered tonight before the legislature building here to call for free elections and the resignation of Hong Kong 's leader .\tTens of thousands of demonstrators gathered yesterday evening to stand before this city 's legislature building and call for free elections and the resignation of Hong Kong 's leader .\n1\t1597193\t1597119\tSaddam loyalists have been blamed for sabotaging the nation 's infrastructure , as well as frequent attacks on U.S. soldiers .\tHussein loyalists have been blamed for sabotaging the nation 's infrastructure and attacking US soldiers .\n1\t2758944\t2758975\tIts closest living relatives are a family frogs called sooglossidae that are found only in the Seychelles in the Indian Ocean .\tIts closest relative is found in the Seychelles Archipelago , near Madagascar in the Indian Ocean .\n0\t2584416\t2584653\tCooley said he expects Muhammad will similarly be called as a witness at a pretrial hearing for Malvo .\tLee Boyd Malvo will be called as a witness Wednesday in a pretrial hearing for fellow sniper suspect John Allen Muhammad .\n1\t86007\t86373\t\" Instead of pursuing the most imminent and real threats - international terrorists , \" Graham said , \" this Bush administration chose to settle old scores . \"\t\" Instead of pursuing the most imminent and real threats - international terrorists - this Bush administration has chosen to settle old scores , \" Graham said .\n1\t1602860\t1602844\tHe said they lied on a sworn affidavit that requires them to list prior marriages .\tMorgenthau said the women , all U.S. citizens , lied on a sworn affidavit that requires them to list prior marriages .\n1\t1201306\t1201329\tThe association said 28.2 million DVDs were rented in the week that ended June 15 , compared with 27.3 million VHS cassettes .\tThe Video Software Dealers Association said 28.2 million DVDs were rented out last week , compared to 27.3 million VHS cassettes .\n0\t461779\t461815\tWith these assets , Funny Cide has a solid chance to become the first Triple Crown winner since Affirmed in 1978 .\tFunny Cide is looking to become horse racing 's first Triple Crown winner in a generation .\n1\t1438666\t1438643\tIntel was disappointed and assessing its \" options in the event Mr. Hamidi resumes his spamming activity against Intel , \" spokesman Chuck Mulloy said .\tIntel spokesman Chuck Mulloy said the company was disappointed and assessing its \" options in the event Mr. Hamidi resumes his spamming activity against Intel . \"\n1\t3261484\t3261306\tMr Annan also warned the US should not use the war on terror as an excuse to suppress \" long-cherished freedoms \" .\tAnnan warned that the dangers of extremism after September 11 should not be used as an excuse to suppress \" long-cherished \" freedoms .\n1\t1277539\t1277527\tAt community colleges , tuition will jump to $ 2,800 from $ 2,500 .\tCommunity college students will see their tuition rise by $ 300 to $ 2,800 or 12 percent .\n1\t3035788\t3035918\tHe made a point of saying during Tuesdays debate that the Confederate flag was a racist symbol .\tThough Dean made a point of saying during the debate that the Confederate flag is a racist symbol .\n0\t132553\t132725\tBush wanted \" to see an aircraft landing the same way that the pilots saw an aircraft landing , \" White House press secretary Ari Fleischer said yesterday .\tOn Tuesday , before Byrd 's speech , Fleischer said Bush wanted ' ' to see an aircraft landing the same way that the pilots saw an aircraft landing .\n0\t2259788\t2259747\tOn Monday the Palestinian Prime Minister , Mahmoud Abbas , will report to the Palestinian parliament on his Government 's achievements in its first 100 days in office .\tPalestinian Prime Minister Mahmoud Abbas must defend the record of his first 100 days in office before Parliament today as the death toll in the occupied territories continues to rise .\n0\t2307064\t2307235\tThe civilian unemployment rate improved marginally last month -- slipping to 6.1 percent -- even as companies slashed payrolls by 93,000 .\tThe civilian unemployment rate improved marginally last month _ sliding down to 6.1 percent _ as companies slashed payrolls by 93,000 amid continuing mixed signals about the nation 's economic health .\n1\t3046488\t3046824\tPer-user pricing is $ 29 for Workplace Messaging , $ 89 for Team Collaboration and $ 35 for Collaborative Learning .\tWorkplace Messaging is $ 29 , Workplace Team Collaboration is $ 89 , and Collaborative Learning is $ 35 .\n1\t86020\t86007\t\" Instead of pursuing the most imminent and real threats – international terrorism – this Bush administration chose to settle old scores , \" Mr. Graham said .\t\" Instead of pursuing the most imminent and real threats - international terrorists , \" Graham said , \" this Bush administration chose to settle old scores . \"\n0\t1100998\t1100441\tSARS has killed about 800 people and affected more than 8400 since being detected in China in November .\tSARS has killed about 800 people and sickened more than 8,400 worldwide , mostly in Asia .\n1\t2268396\t2268480\tAuthorities had no evidence to suggest the two incidents were connected .\tThere was no immediate evidence that the two incidents were connected , police said .\n0\t1984039\t1983986\t\" Jeremy 's a good guy , \" Barber said , adding : \" Jeremy is living the dream life of the New York athlete .\tHe also said Shockey is \" living the dream life of a New York athlete .\n0\t2697659\t2697747\tRatliff 's daughters , Margaret and Martha Ratliff , were adopted by Peterson after their mother 's death .\tPeterson helped raise Ratliff 's two daughters , Margaret and Martha Ratliff , who supported him throughout the trial .\n0\t2175939\t2176090\tAfter losing as much as 84.56 earlier , the Dow Jones industrial average closed up 22.81 , or 0.2 percent , at 9,340.45 .\tIn midday trading , the Dow Jones industrial average lost 68.84 , or 0.7 percent , to 9,248.80 .\n1\t886618\t886456\tRumsfeld , who has been feuding for two years with Army leadership , passed over nine active-duty four-star generals .\tRumsfeld has been feuding for a long time with Army leadership , and he passed over nine active-duty four-star generals .\n1\t588637\t588864\tConsumers who said jobs are difficult to find jumped from 29.4 to 32.6 , while those claiming work was plentiful slipped from 13 to 12.6 .\tConsumers who said jobs are difficult to find jumped to 32.6 from 29.4 , while those saying work was plentiful slipped to 12.6 from 13 in April .\n0\t2252795\t2252970\tHe has no immediate plans for television advertising , believing it is unnecessary this early .\tA Lieberman aide said there were no immediate plans for television advertising .\n1\t1756329\t1756394\t\" I think it happened very quickly , \" Houston Police Department homicide investigator Phil Yochum said of the crime .\t\" I think it happened very quickly , \" said Investigator Phil Yochum of the Houston Police Department 's homicide division .\n1\t1673112\t1673068\tUnited issued a statement saying it will \" work professionally and cooperatively with all its unions . \"\tSenior vice president Sara Fields said the airline \" will work professionally and cooperatively with all our unions . \"\n1\t2357324\t2357271\t\" But they never climb out of the pot of beer again . \"\tIt 's just that they never climb out of the beer again . \"\n1\t780408\t780363\tChief financial officer Andy Bryant has said that hike had a greater affect volume than officials expected .\tBryant has said that hike had a greater effect on demand than officials expected .\n1\t821523\t821385\tRobert Liscouski , the Assistant Secretary of Homeland Security for Infrastructure Protection , will oversee NCSD .\tNCSD 's chief will be Robert Liscouski , the assistant secretary of Homeland Security for Infrastructure Protection .\n1\t2304696\t2304863\tHP 's shipments increased 48 percent year-over-year , compared to an increase of 31 percent for Dell .\tHPs shipments increased 48 per cent year-on-year , compared to an increase of 31 per cent for Dell .\n1\t2531749\t2531607\tChirac , who can pardon a law-breaker , refused Humbert 's request last year but kept in close touch with the family .\tChirac , who has the authority to pardon law-breakers , refused Humbert 's request to be allowed to die last year but kept in close touch with the family .\n1\t3180014\t3179967\tThe charges allege that he was part of the conspiracy to kill and kidnap persons in a foreign country .\tThe government now charges that Sattar conspired with Rahman to kill and kidnap individuals in foreign countries .\n1\t726966\t726945\tIn the 2002 study , the margin of error ranged from 1.8 to 4.4 percentage points .\tIt has a margin of error of plus or minus three to four percentage points .\n1\t2638861\t2638982\tMr. Clinton 's national security adviser , Sandy Berger , said that the White House wasn 't informed of the FBI activities .\tClinton ’ s national security adviser , Sandy Berger , said in an interview that the White House was not informed of the FBI activities .\n1\t2495223\t2495307\t\" This decision is clearly incorrect , \" FTC Chairman Timothy Muris said in a written statement .\tThe decision is \" clearly incorrect , \" FTC Chairman Tim Muris said .\n1\t55187\t54831\tProsecutors allege that Nichols and co-conspirator Timothy McVeigh worked together to prepare a bomb that destroyed the Alfred P. Murrah Federal Building .\tProsecutors allege that Nichols and coconspirator Timothy McVeigh worked together to prepare a 4,000-pound fuel-and-fertilizer bomb that destroyed the Murrah building .\n0\t2763381\t2763517\tTerri Schiavo , 39 , is expected to die sometime in the next two weeks in the Tampa-area hospice where she has spent the past several years .\tTerri Schiavo , 39 , underwent the procedure at the Tampa Bay area hospice where she has been living for several years , said her father , Bob Schindler .\n1\t1990975\t1991132\tSecretary of State Colin Powell designated the Chechen leader believed responsible for last year 's hostage standoff in a Moscow theater as a threat to U.S. security Friday .\tU.S. Secretary of State Colin Powell on Friday designated Chechen rebel leader Shamil Basayev a threat to the security of the United States and to U.S. citizens .\n1\t2204353\t2204418\t\" Today , we are trying to convey this problem to Russian President Vladimir Putin and US President George W Bush . \"\t\" Today , we are trying to convey this problem to Russian President Vladimir Putin ( news - web sites ) and President Bush ( news - web sites ) . \"\n1\t60122\t60445\tThat would be a potential setback to Chief Executive Phil Condit 's strategy of bolstering defense-related sales during a slump in jetliner deliveries .\tThe inquiry may hinder Chief Executive Phil Condit 's strategy of bolstering defense-related sales during a slump in jetliner deliveries .\n1\t961836\t962243\tPeopleSoft also said its board had officially rejected Oracle 's offer .\tThursday morning , PeopleSoft 's board rejected the Oracle takeover offer .\n0\t3140260\t3140288\tThe Dow Jones industrial average ended the day down 10.89 at 9,837.94 , after advancing 111.04 Wednesday .\tThe Dow Jones industrial average fell 10.89 points , or 0.11 percent , to 9,837.94 .\n1\t1720166\t1720115\tCortisol levels in the saliva of day care children were highest and rose most steeply in those judged by day care center personnel to be the shyest .\tCortisol levels in the saliva of day-care children were highest and rose most steeply in those whom day-care centre staffed judged to be the shyest .\n1\t2573262\t2573319\t\" The idea that Tony Abbott is in some way a one-dimensional political head-kicker couldn 't be more wrong , \" Mr Howard said .\t\" The idea that Tony Abbott is in some way a one-dimensional political head kicker couldn 't be more wrong . \"\n0\t1353356\t1353174\t\" Biotech products , if anything , may be safer than conventional products because of all the testing , \" Fraley said , adding that 18 countries have adopted biotechnology .\t\" Biotech products , if anything , may be safer than conventional products because of all the testing , \" said Robert Fraley , Monsanto 's executive vice president .\n1\t2738677\t2738741\tThe rate of skin cancer has tripled since the 1950s in Norway and Sweden , according to the study .\tThe study also found that skin cancer nearly tripled in Norway and Sweden since the 1950s .\n1\t1638813\t1639087\tWe acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , \" Rumsfeld said .\tRather , the US acted because the administration saw \" existing evidence in a new light , through the prism of our experience on September 11 \" .\n1\t1605350\t1605425\tTrans fat makes up only 1 percent to 3 percent of the total fat Americans consume , compared with 14 percent for saturated fat .\tTrans fat accounts for 2.5 percent of Americans ' daily calories , compared to 11 percent to 12 percent for saturated fat .\n1\t2494149\t2494073\tHowever , a recent slide in prices and OPEC 's expectations of a surge in oil inventories have compounded its fears about a further softening of the market .\tA 14 percent slide in crude prices this month and expectations of a build up in oil inventories compounded OPEC 's fears of a further softening of the market .\n1\t3023029\t3023229\tPeterson , 31 , is now charged with murder in the deaths of his 27-year-old wife and their unborn son .\tPeterson , 31 , is charged with two counts of first-degree murder in the slayings of his wife , Laci , and their unborn son , Conner .\n1\t1351550\t1351155\tCarlson on Tuesday said he would not recuse himself from the case .\tService officials said Carlson refused to recuse himself from the case .\n1\t981185\t981234\tThe program will grow to include ports in Dubai , Turkey and Malaysia , among others .\tThe program will be expanded to include areas of the Middle East such as Dubai , Turkey and Malaysia , Mr. Ridge said .\n0\t2111629\t2111786\tMcCabe said he was considered a witness , not a suspect .\t\" He is not considered a suspect , \" McCabe said .\n1\t655498\t655391\tThe woman was exposed to the SARS virus while in the hospital but was not a health care worker , said Dr. Colin D ’ Cunha , Ontario ’ s commissioner of public health .\tThe woman was exposed to the SARS virus while in the hospital but was not a health-care worker , said Dr Colin D 'Cunha , Ontario 's commissioner of public health .\n1\t533823\t533909\tHe added that those \" are not solely American principles , nor are they exclusively Western . \"\t\" These are not solely American principles nor are they exclusively Western , \" Rumsfeld said .\n1\t581592\t581570\t\" If we don 't march into Tehran , I think we will be in pretty good shape , \" he said .\t\" As long as we don 't march on Tehran , I think we are going to be in pretty good shape , \" he said .\n0\t1010655\t1010430\tOn Saturday , a 149mph serve against Agassi equalled Rusedski 's world record .\tOn Saturday , Roddick equalled the world record with a 149 m.p.h. serve in beating Andre Agassi .\n1\t2241925\t2242066\tChad Kolton , emergency management spokesman with the Department of Homeland Security , said the government is open to new technologies and methods to communicate more quickly and efficiently .\tChad Kolton , emergency management spokesman with the Department of Homeland Security , said the government is open to new ways to communicate .\n1\t2796978\t2797024\t\" APEC leaders are painfully aware that security and prosperity are inseparable , \" Thai Prime Minister Thaksin Shinawatra told business leaders .\t\" APEC leaders are painfully aware that security and prosperity are inseparable , \" Thaksin said .\n0\t101746\t101775\tDanbury prosecutor Warren Murray could not be reached for comment Monday .\tProsecutors could not be reached for comment after the legal papers were obtained late Monday afternoon .\n1\t327839\t327748\tWittig resigned last year after being indicted on federal bank fraud charges involving a real estate loan unrelated to Westar business .\tWittig resigned in late November about two weeks after being indicted on bank fraud charges in a real estate case unrelated to the company .\n0\t2988297\t2988555\tShattered Glass , \" starring Hayden Christensen as Stephen Glass , debuted well with $ 80,000 in eight theaters .\t\" Shattered Glass \" _ starring Hayden Christensen as Stephen Glass , The New Republic journalist fired for fabricating stories _ debuted well with $ 80,000 in eight theaters .\n1\t2217613\t2217659\tHe was arrested Friday night at an Alpharetta seafood restaurant while dining with his wife , singer Whitney Houston .\tHe was arrested again Friday night at an Alpharetta restaurant where he was having dinner with his wife .\n0\t2128530\t2128455\tHowever , EPA officials would not confirm the 20 percent figure .\tOnly in the past few weeks have officials settled on the 20 percent figure .\n1\t2208376\t2208198\tUniversity of Michigan President Mary Sue Coleman said in a statement on the university 's Web site , \" Our fundamental values haven 't changed .\t\" Our fundamental values haven 't changed , \" Mary Sue Coleman , president of the university , said in a statement in Ann Arbor .\n1\t1980654\t1980641\tThe first products are likely to be dongles costing between US $ 100 and US $ 150 that will establish connections between consumer electronics devices and PCs .\tThe first products will likely be dongles costing $ 100 to $ 150 that will establish connections between consumer electronics devices and PCs .\n0\t589579\t589557\tHowever , Lapidus expects foreign brands ' sales to be up 4 percent , driven by strong truck sales at Honda Motor Co .\tLapidus expects Ford to be down 5 percent , Chrysler down 10 percent and foreign brands up 4 percent driven by strong truck sales at Honda .\n1\t1636060\t1635946\tMichel , who remains in the government , denied that US pressure had provoked the government 's move .\tMichel , who has stayed in the new government , denied that it was U.S. pressure which had provoked the government 's move .\n1\t1630585\t1630657\tSome of the computers also are used to send spam e-mail messages to drum up traffic to the sites .\tSome are also used to send spam e-mail messages to boost traffic to the sites .\n0\t447728\t447699\tIndonesia 's army has often been accused of human rights abuses during GAM 's battle for independence , charges it has generally denied while accusing the separatists of committing rights violations .\tIndonesia 's army has been accused of human rights abuses during its earlier battles with GAM , charges it has generally denied .\n1\t1606495\t1606619\tBush also hoped to polish his anti-AIDS credentials in Uganda , which has been hailed as an African pioneer in fighting the killer disease .\tPresident Bush flies to Uganda Friday hoping to polish his anti- AIDS credentials in a country hailed as an African pioneer in fighting the epidemic .\n1\t1550897\t1550977\tLater this year , the command will send trainers with soldiers from four North African nations on patrolling and intelligence gathering missions .\tThis fall the command will send trainers to work with soldiers from four North African nations on patrolling and gathering intelligence .\n0\t490376\t490490\tThe reports helped overcome investor jitters after the euro briefly hit an all-time high against the dollar Tuesday .\tStocks slipped at the open after the euro hit record highs against the dollar .\n1\t3084554\t3084612\tSales for the quarter beat expectations , rising 37 percent year-on-year to 1.76 billion euros .\tSales rose 37 per cent year-on-year to 1.76bn , beating expectations .\n1\t315647\t315778\tIf the MTA 's appeal to a higher court is successful , the $ 2 bus and subway base fare won 't be rolled back .\tIf the MTA 's appeal is successful , the $ 2 bus and subway base fare won 't change .\n1\t3428298\t3428362\tRobert Walsh , 40 , remained in critical but stable condition Friday at Staten Island University Hospital 's north campus .\tWalsh , also 40 , was in critical but stable condition at Staten Island University Hospital last night .\n1\t2523564\t2523358\tThe Guru microcontroller serves four functions : hardware monitoring , overclocking management , BIOS ( Basic Input Output System ) update and a troubleshooting-assistance feature called Black Box .\tThe µGuru microcontroller serves four functions : hardware monitoring , overclocking management , BIOS update and a troubleshooting-assistance feature called Black Box .\n1\t2079200\t2079131\tU.S. corporate bond yield spreads tightened in spotty trading on Friday as Wall Street labored to get back on its feet after the largest power outage ever in North America .\tU.S. stocks rose slightly on feather-light volume on Friday , as Wall Street regrouped after the biggest-ever power outage in North America .\n1\t818091\t817811\tThe company said it would issue revised guidance for the full fiscal year next month when it releases its Q2 results .\tThe company said it would renew its guidance for 2003 when it announces its second quarter results in mid-July .\n1\t1580638\t1580663\t\" I stand 100 percent by it , and I think our intelligence services gave us the correct information at the time . \"\tI stand 100 percent by it , and I think that our intelligence services gave us the correct intelligence and information at the time , \" Blair said .\n0\t1919740\t1919926\t\" I don 't know if the person I 'm talking to now may end up being someone else at another time that may not follow the rules , \" Parrish said .\t\" I don 't know whether the person I 'm talking to now may end up being someone else , \" Parrish said .\n1\t2748287\t2748550\t\" I think it 's going to be a close vote , but I think the grant proposal is going to win , \" McConnell said .\t\" I think it 's going to be a close vote , but I think the grant proposal 's going to win , \" said Sen. Mitch McConnell , assistant majority leader .\n1\t3394891\t3394775\tTwenty-eight people were believed to have been spending Christmas Day with the caretaker of the St Sophia 's camp , when the mudslide smashed into two cabins .\tTwenty-seven people were believed to have been spending Christmas Day with the caretaker of Saint Sophia Camp , a Greek Orthodox facility , when the mudslide roared through .\n0\t2963943\t2963880\tOne , Capt. Doug McDonald , remained hospitalized in critical condition on Thursday .\tHer 20-year-old sister , Allyson , was severely burned and remained hospitalized in critical condition .\n0\t1865364\t1865251\tThe United States finally relented during President Bush 's visit to Africa earlier this month .\tDuring President Bush 's trip to Africa earlier this month , however , Washington said it would support the increase .\n1\t263690\t263819\t\" There is no conscious policy of the United States , I can assure you of this , to move the dollar at all , \" he said .\tHe also said there is no conscious policy by the United States to move the value of the dollar .\n1\t283751\t283290\tIt 's the first such drill since the September 11 terrorist attacks on New York and Washington .\tIt is the nation 's first large-scale counterterrorism exercise since the Sept . 11 terrorist attacks .\n1\t2517014\t2516995\tMyanmar 's pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital , her personal physician said .\tMyanmar 's pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery , her personal physician said Friday .\n1\t1330643\t1330622\tAccording to the Merchant Marine Ministry , the 37-year-old ship is registered to Alpha Shipping Inc. based in the Pacific Ocean nation of Marshall Islands .\tThe Baltic Sky is a 37-year-old ship registered to Alpha Shipping Inc. based in the Pacific Ocean nation of Marshall Islands .\n1\t3111452\t3111428\tIn an unusual move , the U.S. Patent and Trademark Office is reconsidering a patent affecting Internet pages that critics contend could disrupt millions of Web sites .\tIn an unusual move that critics contend could disrupt millions of Web sites , the U.S. Patent and Trademark Office is reconsidering a patent affecting Internet pages .\n0\t1167835\t1167651\tKansas Department of Health and Environment records show there were 88 abortions performed on girls age 14 and younger last year .\tStatistics from the Kansas Department of Health and Environment show that 11,844 abortions were performed in the state last year .\n0\t1423836\t1423708\tA European Union spokesman said the Commission was consulting EU member states \" with a view to taking appropriate action if necessary \" on the matter .\tLaos 's second most important export destination - said it was consulting EU member states ' ' with a view to taking appropriate action if necessary ' ' on the matter .\n1\t2090911\t2091154\tWaiting crowds filling the streets on both sides overwhelmed the peacekeepers soon after daylight , sweeping past the barbed wire barricades .\tBut waiting crowds filling the streets rushed the bridges soon after daylight , overrunning razor-wire barricades .\n1\t2265271\t2265152\tBarry Callebaut will be able to use Brach 's retail network to sell products made from its German subsidiary Stollwerck , which makes chocolate products not sold in the United States .\tBarry Callebaut will be able to use Brach 's retail network to sell products made from its German subsidiary Stollwerck , which makes chocolate products unknown to the American market .\n1\t3062202\t3062308\tBy skirting the FDA 's oversight , Eagan said , the quality of the imported drugs is \" less predictable \" than for those obtained in the United States .\tBy skirting the FDA 's oversight , Eagan said the quality of the imported drugs is \" less predictable \" than U.S. drugs .\n1\t2155514\t2155377\tHe said : \" For the first time there is an easy and affordable way of making this treasure trove of BBC content available to all . \"\t\" For the first time , there is an easy and affordable way of making this treasure trove of BBC content available to all , \" Dyke said .\n1\t1552068\t1551928\tThree such vigilante-style attacks forced the hacker organizer , who identified himself only as \" Eleonora [ 67 ] , \" to extend the contest until 7 p.m. EST Sunday .\tThree such vigilante-style attacks forced the hacker organiser , who identified himself only as \" Eleonora67 ] , \" to extend the contest until 8am ( AEST ) today .\n1\t936978\t937500\tEric Gagne pitched a perfect ninth for his 23rd save in as many opportunities .\tGagne struck out two in a perfect ninth inning for his 23rd save .\n0\t985015\t984975\tOne way or another , Harry Potter And The Order Of The Phoenix will be in your hands by Saturday .\tJust about everything about \" Harry Potter and the Order of the Phoenix \" will set records .\n1\t1430357\t1430425\t\" Allison just proves you don 't need to wait until August or September to have a disaster , \" said Josh Lichter , a meteorologist with the Houston-Galveston weather office .\t\" Allison just proves you don 't need to wait until August or September to have a disaster , \" Lichter said .\n1\t3039310\t3039413\tToday , analysts say , UN members can no longer ignore the shifts since the September 11 2001 attacks .\tOn Wednesday , analysts say , UN members can no longer ignore the shifts since the attacks in the US of September 11 2001 .\n1\t34513\t34742\tPolice say CIBA was involved in the importation of qat , a narcotic substance legal in Britain but banned in the United States .\tMr McKinlay said that CIBA was involved in the importation of qat , a narcotic substance legal in Britain but banned in the US .\n1\t368067\t368018\tChiron already has nearly 20 percent acceptances from PowderJect 's shareholders .\tChiron has acceptances from holders of nearly 20 percent of PowderJect shares .\n0\t611663\t611716\tErnst & Young has denied any wrongdoing and plans to fight the allegations .\tErnst & Young has denied the SEC 's claims , and called its recommendations \" irresponsible \" .\n1\t98432\t98657\tThe attack followed several days of disturbances in the city where American soldiers exchanged fire with an unknown number of attackers as civilians carried out demonstrations against the American presence .\tThe attack came after several days of disturbance in the city in which U.S. soldiers exchanged fire with an unknown number of attackers as civilians protested the American presence .\n1\t3039007\t3038845\tNo company employee has received an individual target letter at this time .\tShe said no company official had received \" an individual target letter at this time . \"\n1\t1708040\t1708062\tSecond-quarter results reflected a gain of 10 cents per diluted share , while the 2002 results included a loss of 19 cents per diluted share .\tThe second-quarter results had a non-operating gain of 10 cents a share while the 2002 second-quarter performance had a net non-operating loss of 19 cents a share .\n0\t1757264\t1757375\tHe allegedly told his ex-wife in an angry phone call that he had no intention of following their new custody agreement .\tThe two had battled over custody and he allegedly told her in an angry phone call that he had no intention of following their new custody agreement .\n1\t383417\t383558\tWorldwide , more than 50 million people have seen \" Les Miz , \" with gross receipts of $ 1.8 billion .\tWorldwide , Les Misérables has been seen by over 50 million people , with a total gross of over $ 2 billion .\n0\t2766112\t2766084\tIn fiction : Edward P. Jones ( \" The Known World \" ) and Scott Spencer ( \" A Ship Made of Paper \" ) .\tThe fifth nominee for fiction is Scott Spencer , for A Ship Made of Paper .\n1\t1261116\t1261234\t\" Overwhelmingly the Windows brand really resonated with them . \"\t\" Windows was the part of the experience that really resonated with people . \"\n1\t3028143\t3028234\tThe Centers for Medicare and Medicaid Services , the federal agency that runs Medicare , last year began a similar effort for nursing homes .\tThe Centers for Medicare and Medicaid launched a similar consumer tool for nursing homes last year .\n0\t249699\t249623\tVivace was founded in 1999 and has raised over $ 118 million in three rounds of venture financing .\tDuring difficult times for technology venture capital , Vivace raised over $ 118 million in three rounds of venture financing .\n0\t3448488\t3448449\tThe Dow Jones industrial average < .DJI > added 28 points , or 0.27 percent , at 10,557 , hitting its highest level in 21 months .\tThe Dow Jones industrial average < .DJI > rose 49 points , or 0.47 percent , to 10,578 .\n1\t2749322\t2749663\tThe Democratic candidates also began announcing their fund-raising totals before Wednesday 's deadline to file quarterly reports with the Federal Election Commission .\tThe Democratic candidates also began announcing their fund-raising totals in advance of the deadline today to file quarterly reports with the Federal Election Commission .\n0\t2204592\t2204588\tSun Microsystems Inc. on Thursday said it had added 100 new third-party systems and 100 new components to its Hardware Compatibility List for the Solaris x86 operating system Platform Edition .\tThe vendor has added 100 new third-party systems and 100 new components to the operating system 's Hardware Compatibility List ( HCL ) .\n1\t2889005\t2888954\tProsecutors said PW Marketing violated the state 's 1998 anti-spam law by sending unsolicited e-mail without a toll-free number for recipients to call to stop additional mailings .\tProsecutors said PW Marketing violated the 1998 anti-spam law because these unsolicited e-mails were sent without a free call number for recipients to phone to stop additional mailings .\n0\t1657632\t1657619\tThe Neighbours star and singer spent yesterday resting at her family home in Sydney and will have more tests today .\tGoodrem spent yesterday resting in her family home in Sydney and will have more tests today to determine her exact treatment .\n0\t555617\t555528\tThe 3 rd Armored Cavalry Regiment is 5,200 strong and the largest combat unit at Fort Carson .\tBroomhead , 34 , was assigned to the 2nd Squadron , 3rd Armored Cavalry Regiment .\n1\t2396937\t2396818\t\" The risk of inflation becoming undesirably low remains the predominant concern for the foreseeable future , \" the Fed said in a statement accompanying the unanimous decision .\t\" The risk of inflation becoming undesirably low remains the predominant concern for the foreseeable future , \" the policy-setting Federal Open Market Committee said .\n0\t2339738\t2339771\t\" It is bad for Symbian , \" said Per Lindberg , analyst at Dresdner Kleinwort Wasserstein .\t\" Motorola has displayed clear disloyalty \" to Symbian , said Per Lindberg , an analyst at Dresdner Kleinwort Wasserstein in London .\n0\t1616174\t1616206\tBob Richter , a spokesman for House Speaker Tom Craddick , had no comment about the ruling .\tBob Richter , spokesman for Craddick , R-Midland , said the speaker had not seen the ruling and could not comment .\n1\t635783\t635802\tBut Ms Ward said the headroom under its financial covenants was \" tight \" and that there could be another downgrade if Southcorp breached any of its banking covenants .\tBut Ms Ward said the headroom under its financial covenants was \" tight \" and that there could be a rating downgrade if Southcorp did breach any banking covenants .\n1\t3444633\t3444733\tHe added : ``I 've never heard of more reprehensiblebehaviour by a doctor .\tThe Harrisons ’ lawyer Paul LiCalsi said : “ I ’ ve never heard of more reprehensible behaviour by a doctor .\n1\t555553\t555528\tBroomhead was assigned to 2nd Squadron , 3rd Armor Cavalry Regiment , based at Fort Carson .\tBroomhead , 34 , was assigned to the 2nd Squadron , 3rd Armored Cavalry Regiment .\n1\t1112021\t1111925\tOther staff members , however , defended the document , saying it would still help policy-makers and the agency improve efforts to address the climate issue .\tSome E.P.A. staff members defended the document , saying that although pared down it would still help policy makers and the agency address the climate issue .\n0\t2749410\t2749625\tPresident Bush raised a record-breaking $ 49.5 million for his re-election campaign over the last three months , with contributions from 262,000 Americans , the president 's campaign chairman said Tuesday .\tPresident Bush has raised $ 83.9 million since beginning his re-election campaign in May , and has $ 70 million of that left to spend , his campaign said Tuesday .\n1\t1629064\t1629043\tAn episode is declared when the ozone reaches .20 parts per million parts of air for one hour .\tA Stage 1 episode is declared when ozone levels reach 0.20 parts per million .\n1\t789691\t789665\t\" He may not have been there , \" the defence official said on Thursday .\t\" He may not have been there , \" said a defence official speaking on condition of anonymity .\n1\t844421\t844679\tThe U.N. troops are in Congo to protect U.N. installations and personnel , and they can only fire in self defense and have been unable to stem the violence .\tThe troops - whose mandate is to protect U.N. installations and personnel - can only fire in self-defense and have been unable to stem the violence .\n1\t58540\t58567\tNorth American markets grabbed early gains Monday morning , as earnings season begins to slow and economic indicators take the spotlight .\tNorth American futures pointed to a strong start to the first trading session of the week Monday , as earnings season slows and economic indicators take the spotlight .\n1\t781439\t781461\tXerox itself paid a $ 10 million fine last year to settle similar SEC charges .\tXerox itself previously paid a $ 10-million penalty to settle the SEC accusations .\n1\t1909579\t1909408\t\" This deal makes sense for both companies , \" said National Chief Executive Brian Halla .\t\" This deal makes sense for both companies , \" Halla said in a prepared statement .\n0\t787432\t787464\tThe blasts killed two people and injured more than 150 others .\tThe Atlanta Olympic Games attack killed one woman and injured more than 100 other people .\n0\t52758\t52343\tMorrill 's wife , Ellie , sobbed and hugged Bondeson 's sister-in-law during the service .\tAt the service Morrill 's widow , Ellie , sobbed and hugged Bondeson 's sister-in-law as people consoled her .\n1\t1675025\t1675047\tSpansion products are to be available from both AMD and Fujitsu , AMD said .\tSpansion Flash memory solutions are available worldwide from AMD and Fujitsu .\n1\t2131318\t2131372\tAbout 1,500 police will be deployed for the visit .\tAround 1,500 police are to be deployed at Niigata for the ferry 's visit .\n1\t325763\t325928\tGamarekian told The News she remembers only the woman 's first name - and refused to reveal it .\tShe told the New York Daily News she remembers only the intern 's first name , which she refused to reveal .\n1\t2638975\t2638855\tOne of the FBI ’ s key operatives , who had a falling out with the bureau , provided an account of the operation at a friend ’ s closed immigration court proceeding .\tOne of the FBI 's key operatives , who has had a falling-out with the bureau , provided an account of the operation at a friend 's closed immigration court proceeding .\n1\t2198694\t2198937\tA nationally board certified teacher with a master 's degree , Kelley makes a salary of $ 65,000 in his 30th year .\tA nationally board certified teacher with a master 's degree , Kelley , in his 30th year teaching , makes $ 65,000 .\n1\t1825432\t1825301\tA man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $ 100,000 bail during an early morning court appearance Saturday .\tThe Queens man arrested for allegedly threatening to shoot City Councilman Hiram Monserrate was held on $ 100,000 bail Saturday , a spokesman for the Queens district attorney said .\n1\t2906104\t2906322\tThey were being held Sunday in the Camden County Jail on $ 100,000 bail .\tThey remained in Camden County Jail on Sunday on $ 100,000 bail .\n1\t722278\t722383\tMs Stewart , the chief executive , was not expected to attend .\tMs Stewart , 61 , its chief executive officer and chairwoman , did not attend .\n0\t101747\t101777\tChristina 's aunt , Shelley Riling , said the defense 's claims were preposterous .\tChristina 's aunt , Shelley Riling , said she will address the court .\n1\t2224884\t2224819\tThe Justice Department Aug. 19 gave pre-clearance for the Oct. 7 date for the election to recall Gov. Gray Davis , saying it would not affect minority voting rights .\tThe Justice Department on Aug. 19 sanctioned the Oct. 7 date for recall election , saying it would not affect voting rights .\n0\t977938\t978162\tLord Falconer hailed the changes as \" a new beginning as far as the courts , Crown Prosecution Service and police are concerned \" .\t\" It 's a new beginning as far as the courts , Crown Prosecution Service and police are concerned , making the criminal justice system work better . \"\n0\t1015010\t1014963\tGE stock closed at $ 30.65 a share , down about 42 cents , on the New York Stock Exchange .\tGE 's shares closed at $ 30.65 on Friday on the New York Stock Exchange .\n1\t1513190\t1513246\tAt least 27 US troops have been killed in hostile fire since Bush 's statement .\tAt least 26 American troops have been killed in hostile fire since major combat was officially declared over on May 1 .\n1\t2385348\t2385394\tA recent poll showed Edwards with a narrow lead in South Carolina , and he plans a rally there later on Tuesday .\tA recent poll showed Edwards in a virtual four-way tie at the top in South Carolina , and he plans a rally there later on Tuesday .\n1\t2317018\t2317252\tNovember 17 's last victim was British defence attache Stephen Saunders , who was shot on an Athens road in June 2000 .\tNovember 17 's last victim was British defense attache Stephen Saunders , who was shot and killed at point-blank range on a busy Athens road in June 2000 .\n0\t1831696\t1831660\tThe agency charged that one WD Energy worker discussed false reporting with traders at two other energy companies .\tThe agency found further that a WD Energy employee discussed false reporting with traders at two other energy companies , which the CFTC didn 't identify .\n1\t1528383\t1528083\tZulifquar Ali , a worshipper slightly wounded by shrapnel , said the assailants first targeted the mosque 's security guards .\tWitness Zulfiqar Ali , who was slightly wounded by shrapnel , said the attackers had focused on the mosque 's guards .\n1\t917965\t918315\tFor the second year in a row , rises in hospital costs accounted for much of the inflation , accounting for 51 percent of the overall cost increase .\tFor the second year in a row , rises in hospital costs dominated the increase , accounting for 51 percent of the overall cost spiral .\n0\t3218713\t3218830\tQ : Can I buy coverage for prescription drugs right away ?\tCongress has added a new benefit - an option to buy insurance coverage for prescription drugs .\n1\t221079\t221003\tThe airline also said it has the option to buy 380 more airplanes , orders that would be split evenly between the two manufacturers .\tThe airline has the option to buy 380 more , split evenly between the two manufacturers .\n1\t2546175\t2546198\tDr Mark McClean , Jonathan 's family doctor , said if the drug had been administered earlier Jonathan would have retained more of his brain functions .\tDr Mark McClean , the family 's GP , said had the drug been administered to Jonathan earlier , he would have retained more of his brain function .\n0\t799346\t799268\tThe chain operates more than 3,400 stores , and has annual revenue of about $ 15.8 billion .\tThe chain , which has been under new management since late 1999 , has more than 3,400 stores and $ 15.8 billion in annual revenue .\n0\t2673104\t2673130\tAll patients developed some or all of the symptoms of E. coli food poisoning : bloody diarrhea , vomiting , abdominal cramping and nausea .\tSymptoms of the E. coli infection include bloody diarrhea , nausea , vomiting and abdominal cramping .\n1\t1354501\t1354476\tFederal regulators have turned from sour to sweet on a proposed $ 2.8 billion merger of ice cream giants Nestle Holdings Inc. and Dreyer 's Grand Ice Cream Inc .\tFederal regulators have changed their minds on a proposed $ 2.8 billion merger of ice cream giants Nestle Holdings and Dreyer 's Grand Ice Cream .\n1\t3070979\t3070949\tEnvironmental campaigners are using this weekend ’ s lunar eclipse to highlight the huge increase in light pollution across the UK .\tEnvironmental campaigners used the eclipse to highlight the surge in light pollution across Britain .\n0\t1264509\t1264471\tAvailable July 7 , the software supports the Solaris , IBM AIX , Red Hat Linux and Windows operating systems .\tThe OpForce product currently works with Solaris , AIX , Red Hat Linux and Windows servers .\n1\t103280\t103431\tJustice Minister Martin Cauchon and Prime Minister Jean Chrétien have both said the Liberal government will introduce legislation soon to decriminalize possession of small amounts of pot for personal use .\tJustice Minister Martin Cauchon and Prime Minister Jean Chretien both have said the government will introduce legislation to decriminalize possession of small amounts of pot .\n0\t110731\t110648\tBut Chauncey Billups demonstrated he 's also capable of big games , scoring 77 points over the final two games against the Magic .\tBillups scored 77 points in the final two games of the first-round series against the Magic .\n1\t2274844\t2274714\tKelly killed himself after being exposed as the source for a BBC report which claimed the government had embellished evidence of Iraq 's banned weapons to justify the war .\tHe killed himself after being exposed as the source for a BBC report which claimed the government exaggerated the case for war against Iraq .\n0\t1050307\t1050144\tAnd it 's going to be a wild ride , \" said Allan Hoffenblum , a Republican consultant .\tNow the rest is just mechanical , \" said Allan Hoffenblum , a Republican consultant .\n1\t2810634\t2810670\tWhile the Ibrahims had one separation operation , Goodrich and Dr. David Staffenberg plan about three for the Aguirres , with several weeks between each .\tInstead of one long operation to separate the twins , Goodrich and Dr. David Staffenberg plan about three , with several weeks between each .\n1\t3073773\t3073779\tLay had contended that turning over the documents would violate his Fifth Amendment right against self-incrimination .\tLay had refused to turn over the papers , asserting his Fifth Amendment right against self-incrimination .\n0\t261202\t260995\tThe WHO experts didn 't say how many cases in Hebei were in rural areas .\tHebei has reported 191 cases and eight deaths , though the WHO experts did not say how many were in rural areas .\n1\t1824224\t1824209\tNearly 300 mutinous troops who seized a Manila shopping and apartment complex demanding the government resign gave up and retreated peacefully after some 19 hours .\tMutinous troops who seized a Manila shopping and apartment complex demanding the government resign ended a 19-hour standoff late Sunday and returned to barracks without a shot fired .\n1\t548867\t548785\tIn three years , Lend Lease has slipped from a top-five stock , when its share price was around $ 24 , to 37th .\tIn the space of three years , Lend Lease has slipped from a top-five 5 stock when its share price hovered around $ 24 to 37th on the list .\n0\t2796658\t2796682\tAbout two hours later , his body , wrapped in a blanket , was found dumped a few blocks away .\tThen his body was dumped a few blocks away , found in a driveway on Argyle Road .\n1\t1808166\t1808434\tColumbia broke up over Texas upon re-entry on Feb. 1 .\tColumbia broke apart in the skies above Texas on Feb. 1 .\n1\t853475\t853342\tA year or two later , 259 , or 10 per cent , of the youths reported that they had started to smoke , or had taken just a few puffs .\tWithin two years , 259 , or 10 percent , of the youths reported they had started to smoke or had at least taken a few puffs .\n0\t977772\t977804\tThe Lord Chancellor was guardian of the Great Seal , used to stamp all official documents from the sovereign .\tFalconer will hold on , for now , to the Lord Chancellor 's Great Seal , used to sign off instructions from the sovereign .\n1\t577854\t578500\tCindy Yeast , a 50-year-old Washington-area publicist , says she began taking supplements two years ago in part to avoid mild dementia that affects her elderly parents .\tShe started taking supplements two years ago - partly to stave off mild dementia that affects her elderly parents .\n1\t2829194\t2829229\tThe two are not related , but have referred to each other as father and son .\tHe 's not related to Malvo , but the two have referred to each other as father and son .\n1\t2074182\t2074668\tGibson said last month in a press statement that \" neither I nor my film are anti-Semitic .\tGibson said in a June statement that he and his film are not anti-Semitic .\n0\t2758265\t2758282\tThe world 's largest software company said it recognized the difficulty the multiple patches posed for companies , and set out to make it easier for them to apply the updates .\tThe world 's largest software company said it recognized the difficulty the multiple patches posed for companies trying to apply them .\n1\t1958079\t1958143\tThe Dow Jones industrial average .DJI ended up 64.64 points , or 0.71 percent , at 9,191.09 , according to the latest available data .\tThe blue-chip Dow Jones industrial average .DJI added 38 points , or 0.42 percent , to 9,165 .\n1\t544217\t544325\tThe vote came just two days after Kurds swept City Council elections , taking the largest single block of votes on the 30-seat council .\tThe vote for mayor followed City Council elections that gave Kurds the largest block of votes on the 30-seat council .\n1\t2385288\t2385256\tLarge swells and dangerous surf already were being felt along sections of the coast .\tAlready large swells and dangerous surf have arrived along the mid-Atlantic .\n0\t2324708\t2325028\tBased on a separate survey of households , the unemployment rate fell in August to 6.1 percent from 6.2 percent .\tLabor Department analysts discounted a slight improvement in the national unemployment rate , which fell in August to 6.1 percent from 6.2 percent .\n1\t2139506\t2139427\t\" We will work with the board to ensure a smooth transition . \"\tHe said federal regulators would work with the corporation to ensure a \" smooth transition . \"\n1\t2965576\t2965701\tGasps could be heard in the courtroom when the photo was displayed .\tGasps could be heard as the photo was projected onto the screen .\n1\t2931098\t2931144\tGilead had earnings of $ 73.1 million , or 33 cents a share , compared with $ 20.8 million , or 10 cents , in the year-ago quarter .\tQuarterly profit climbed to $ 73.1 million , or 33 cents a share , from $ 20.8 million , or 10 cents , a year earlier , the company said .\n0\t644788\t644816\t\" I had one bad stretch of holes that put me out of contention to win , \" Woods said .\t\" I had one bad stretch of holes that put me out of contention , \" Woods said , referring to his 42 on the front nine Saturday .\n0\t2551891\t2551563\tThe poll had a margin of error of plus or minus 2 percentage points .\tIt had a margin of sampling error of plus or minus four percentage points and was conducted Thursday through Saturday .\n1\t1089053\t1089297\tSen. Patrick Leahy of Vermont , the committee 's senior Democrat , later said the problem is serious but called Hatch 's suggestion too drastic .\tSen. Patrick Leahy , the committee 's senior Democrat , later said the problem is serious but called Hatch 's idea too drastic a remedy to be considered .\n1\t3435735\t3435717\tThe broad Standard & Poor 's 500 < .SPX > eased 0.37 of a point , or 0.03 percent , at 1,121 .\tThe Standard & Poor 's 500 Index < .SPX > slipped 0.26 point , or 0.02 percent , to 1,121.96 .\n0\t1954\t2142\tWatertown , Saugus and Framingham also are going smoke-free Monday , joining a growing number of cities around the country .\tAlong with Boston , Watertown , Saugus and Framingham also are going smoke-free Monday .\n1\t3400796\t3400822\tThat is evident from their failure , three times in a row , to get a big enough turnout to elect a president .\tThree times in a row , they failed to get a big _ enough turnout to elect a president .\n1\t1220668\t1220801\tWe firmly believe we have an absolute right to use the common word ' spike ' as the name of our network . \"\tWe firmly believe that we have an absolute right to use the common word ' spike ' to name our network .\n1\t1889954\t1889847\tSources who knew of the bidding said last week that cable TV company Comcast Corp. was also looking at VUE .\tLate last week , sources told Reuters cable TV company Comcast Corp. CMCSA.O also was looking at buying VUE assets .\n1\t315785\t315653\tBut MTA officials appropriated the money to the 2003 and 2004 budgets without notifying riders or even the MTA board members considering the 50-cent hike , Hevesi found .\tMTA officials appropriated the surplus money to later years ' budgets without notifying riders or the MTA board members when the 50-cent hike was being considered , he said .\n0\t1521034\t1520582\tWhite , who had suffered kidney failure from years of high blood pressure , died at Cedars-Sinai Medical Center around 9 : 30 a.m. , said manager Ned Shankman .\tWhite , who had kidney failure from years of high blood pressure , had been undergoing dialysis and had been hospitalized since a September stroke .\n1\t2083598\t2083810\tAbout 10 percent of high school and 16 percent of elementary students must be proficient at math .\tIn math , 16 percent of elementary and middle school students and 9.6 percent of high school students must be proficient .\n1\t1910610\t1910455\tThe legal ruling follows three days of intense speculation Hewlett-Packard Co. may be bidding for the company .\tThe legal ruling follows three days of wild volatility in RIM 's stock over speculation that PC giant Hewlett-Packard Co. may be bidding for the company .\n1\t3113791\t3113782\tThe European Commission , the EU 's antitrust enforcer , is expected to issue its decision next spring — unless a settlement is reached .\tThe European Commission is expected to issue its decision in the case next spring — unless a settlement is reached .\n1\t3214517\t3214483\t\" So Sebastian did his best to convincingly confess to a crime that he didn 't commit in order to survive , \" she told jurors .\t\" Sebastian did his best to confess convincingly to a crime he didn 't do in order to survive , \" Ms. Richardson declared .\n0\t2083612\t2083810\tTwenty percent of Latino students and 23 percent of black students performed at proficient or higher .\tIn math , 16 percent of elementary and middle school students and 9.6 percent of high school students must be proficient .\n1\t661390\t661218\tHe is charged in three bombings in Atlanta including a blast at the 1996 Olympics and one in Alabama .\tHe is charged in three bombings in Atlanta - including a blast at the 1996 Olympics - along with the bombing in Alabama .\n1\t1269572\t1269682\tThe men were remanded in custody and are due to appear again before court on July 8 .\tThey were remanded in custody and will appear in court again on July 8 .\n1\t1095780\t1095652\t\" No matter who becomes the sponsor for stock-car racing 's top series , NASCAR will need an all-star event , \" Wheeler said in a statement .\tNo matter who becomes the sponsor for stock-car racings top series , NASCAR will need an all-star event , Wheeler said Tuesday .\n1\t116294\t116332\tThe Phillies were upset that Counsell had stolen second in the sixth inning with Arizona leading 7-1 .\tThe Phillies were apparently upset when Counsell stole during the sixth with the Diamondbacks up 7-1 .\n1\t941617\t941673\tHe said his hatred for such people grew from these discussions and had helped convince him violence was the answer .\tHis hatred for these people had germinated from these discussions and helped cement his belief that violence was the panacea .\n1\t2640607\t2640576\t\" There is no need for one deadline for all to create the ASEAN Economic Community , \" Thaksin said .\tThus , he said , there did not have to one deadline to create the economic community .\n1\t3310210\t3310286\tThe announcement was made during the recording of a Christmas concert attended by top Vatican cardinals , bishops , and many elite from Italian society , witnesses said .\tThe broadside came during the recording on Saturday night of a Christmas concert attended by top Vatican cardinals , bishops and many elite of Italian society , witnesses said .\n1\t3376093\t3376101\tThe additional contribution brings total U.S. food aid to North Korea this year to 100,000 tonnes .\tThe donation of 60,000 tons brings the total of U.S. contributions for the year to 100,000 .\n1\t1549586\t1549609\tLeon Williams ' body was found inside his third-floor apartment at 196 Bay St. , in Tompkinsville .\tThe dead man , Leon Williams , was found in his third-floor apartment .\n1\t460211\t460445\tThe player 's eyes were bloodshot and a blood-alcohol test produced a reading of 0.18 - well above Tennessee 's level of presumed intoxication of 0.10 , the report said .\tHe failed a field sobriety test and a blood-alcohol test produced a reading of 0.18 – well above Tennessee 's level of presumed intoxication of 0.10 , the report said .\n1\t1196962\t1197061\tBut Virgin wants to operate Concorde on routes to New York , Barbados and Dubai .\tBranson said that his preference would be to operate a fully commercial service on routes to New York , Barbados and Dubai .\n0\t862804\t862715\tHe tried to fight off officers and was taken to a hospital after a police dog bit him but was later released .\tCruz tried to fight off officers and was hospitalized after a police dog bit him , Sgt. Steve Dixon said .\n1\t1726935\t1726879\tThe announcement , which economists said was not a surprise , may be bittersweet for the millions of Americans without jobs .\tEconomists said the announcement was not a surprise , and politicians said it offered little comfort to the millions of Americans without jobs .\n0\t331980\t332110\tAsked if the delegates could leave on Friday , police intelligence chief in Aceh , Surya Dharma , told reporters they could not because they did not have proper permission .\tAsked if the delegates could leave on Friday , police intelligence chief Surya Dharma told reporters : \" Of course they may not go .\n1\t173879\t173832\tDealers said the dollar also drew some downside support as Japanese investors are expected to keep snapping up foreign bonds amid the yen 's rise against the dollar .\tDealers said the dollar also drew some downside support as Japanese investors are expected to keep snapping up foreign bonds amid ever-falling domestic interest rates .\n0\t2834988\t2835026\tIran has until the end of the month to satisfy the agency it has no plans for nuclear weapons .\tThe Iranians have until the end of the month to answer all the agency 's questions about their past nuclear activities .\n1\t2587300\t2587243\tHer father , Florin Cioaba , the king of Transylvania 's Gypsies , had her brought back and she was married against her will .\tHer father , Roma King Florin Cioaba , had her brought back and she was promptly married against her will .\n0\t554905\t554627\tClaire had advanced to the third round of the 76th annual Scripps Howard National Spelling Bee .\tOne by one they strolled to the microphone , all 251 youngsters in the 76th Scripps Howard National Spelling Bee .\n1\t1912524\t1912648\tCitigroup Inc . C.N , the world 's largest financial services company , on Wednesday promoted Marjorie Magner to chairman and chief executive of its global consumer group .\tCitigroup ( C ) on Wednesday named Marjorie Magner chairman and chief executive of its colossal global consumer business .\n1\t3255597\t3255668\t\" They 've been in the stores for over six weeks , \" says Carney .\tThe quarterlies usually stay in stores for between six to eight weeks , \" Carney added .\n1\t629316\t629289\tLet me just say this : the evidence that we have of weapons of mass destruction was evidence drawn up and accepted by the joint intelligence community .\t\" The evidence that we had of weapons of mass destruction was drawn up and accepted by the Joint Intelligence Committee , \" he said .\n1\t54181\t53570\tRidge said no actual explosives or other harmful substances will be used .\tRidge said no real explosives or harmful devices will be used in the exercise .\n1\t723557\t724115\tThus far , Stewart 's company appears ready to stand behind her .\tFor now , the company 's management appears to be standing behind Stewart .\n0\t2607718\t2607708\tBut late Thursday night , the campaign issued a statement saying there would be no news conference and no big announcement .\tBut late yesterday , the campaign and the state Democratic Party said there would be no news conference .\n1\t753858\t753890\tThere 's also a flaw that results because IE does not implement an appropriate block on a file download dialog box .\tThe second vulnerability is a result of IE not implementing a block on a file download dialog box .\n1\t587009\t586969\tAnother $ 100-million in savings will come from management layoffs and pay cuts .\tThe airline expects to save another $ 100-million a year through management layoffs and pay cuts .\n1\t308567\t308525\tHe called on Prime Minister John Howard to establish a royal commission on child sex abuse .\tThe Senate motion also called on Prime Minister John Howard to hold a royal commission into child sex abuse .\n0\t665419\t665612\t\" We think that the United States of America should support the free speech of all groups , \" Mr. White said , objecting to Mr. Olson 's recommendation .\tWe think that the United States of America should support the free speech of all groups , he said .\n1\t2763517\t2763576\tTerri Schiavo , 39 , underwent the procedure at the Tampa Bay area hospice where she has been living for several years , said her father , Bob Schindler .\tThe tube was removed Wednesday from Terri Schiavo , 39 , at the Tampa Bay-area hospice where she has lived for several years .\n0\t3107118\t3107136\tAfter 18 months , Nissen found that Lipitor stopped plaque buildup in the patients ' arteries .\tAfter 18 months , the atorvastatin patients had no change in the plaque in their arteries .\n1\t780604\t780466\tToll , Australia 's second-largest transport company , last week offered NZ75 a share for Tranz Rail .\tToll last week offered to buy the company for NZ75c a share , or $ NZ158 million .\n0\t1989213\t1989116\t\" This child was literally neglected to death , \" Armstrong County District Attorney Scott Andreassi said .\tArmstrong County District Attorney Scott Andreassi said the many family photos in the home did not include Kristen .\n1\t1462409\t1462504\tWal-Mart , the nation 's largest private employer , has expanded its antidiscrimination policy to protect gay and lesbian employees , company officials said Tuesday .\tWal-Mart Stores Inc . , the nation 's largest private employer , will now include gays and lesbians in its anti-discrimination policy , company officials said Wednesday .\n1\t260952\t260924\tMetro , bus and local rail services in France 's four largest towns -- Paris , Lyon , Lille and Marseille -- were severely disrupted , Europe 1 radio reported .\tSubway , bus and suburban rail services in France 's four largest cities -- Paris , Lyon , Lille and Marseille -- were severely disrupted , transport authorities said .\n1\t1224743\t1225510\tIn the undergraduate case , Rehnquist said the use of race was not \" narrowly tailored \" to achieve the university 's asserted interest in diversity .\tRehnquist wrote that the system was not narrowly tailored to achieve the interest in educational diversity .\n0\t3329379\t3329416\tSP2 is basically about security enhancements to Windows , such as the improved Internet Connection Firewall ( ICF ) .\tThe firewall in the current Windows XP was known as the Internet Connection Firewall ( ICF ) .\n1\t2362761\t2362698\tA landslide in central Chungchong province derailed a Seoul-bound train and 28 passengers were injured , television said .\tIn central Chungchong province , a landslide caused a Seoul-bound Saemaeul Express train to derail , injuring 28 people , local television said .\n0\t1465073\t1464854\tThey will help draft a plan to attack obesity that Kraft will implement over three to four years .\tThe team will help draft a plan by the end of the year to attack obesity .\n1\t195728\t196099\tBut that amount would probably be impossible to pass in the Senate , where Republican moderates have refused to go above $ 350 billion .\tSuch an amount would probably be unable to summon a majority of the Senate , where Republican moderates have refused to go above $ 350 billion .\n1\t2587767\t2587673\tIn the clash with police , Lt. Mothana Ali said about 1,000 demonstrators had gone to the station demanding jobs .\tIn Baghdad , police Lieut . Mothana Ali said about 1,000 demonstrators arrived at the station demanding jobs .\n0\t1490044\t1489975\tCorixa shares rose 54 cents to $ 7.74 yesterday on the Nasdaq Stock Market .\tShares of Corixa rose 54 cents , or about 8 percent , to close at $ 7.74 .\n1\t958161\t957782\tCommittee approval , expected today , would set the stage for debate on the Senate floor beginning Monday .\tThat would clear the way for debate in the full Senate beginning on Monday .\n1\t1033204\t1033365\tO 'Brien was charged with leaving the scene of a fatal accident , a felony .\tBishop Thomas O 'Brien , 67 , was booked on a charge of leaving the scene of a fatal accident .\n0\t2996241\t2996734\tTom Hamilton said his daughter was conscious and alert and in stable condition after the attack Friday morning .\tBethany , who remained in stable condition after the attack Friday morning , talked of the attack Saturday .\n0\t2015389\t2015410\tThe Calgary woman , who is in her twenties , donated blood on Aug. 7 .\tThe woman -- who has no symptoms of illness -- donated blood Aug. 7 .\n1\t221515\t221509\tQuattrone lawyer John W. Keker said his client is innocent .\tIn a statement Monday , his lawyer John Keker said ``Frank Quattrone is innocent .\n0\t2283737\t2283794\tIn the weeks leading up to the execution , several Florida officials received anonymous threatening letters .\tSeveral Florida officials connected to the case have received threatening letters , accompanied by rifle bullets .\n1\t2826681\t2826474\tThe disagreement over online music sales was disclosed in documents filed last week with the judge and made available by the court yesterday .\tThe fight over online music sales was disclosed in documents made available Monday by the court .\n1\t2249237\t2249305\tParson was charged with intentionally causing and attempting to cause damage to protected computers .\tParson is charged with one count of intentionally causing damage to a protected computer .\n1\t389239\t389299\t\" The court and the public need to know much more of the details of the defendant 's seemingly massive fraud , \" the judge said .\t\" The court and the public need to know more of the defendants ' seemingly massive fraud , \" he said .\n1\t2652187\t2652218\tThe U.S. Supreme Court will hear arguments on Wednesday on whether companies can be sued under the Americans with Disabilities Act for refusing to rehire rehabilitated drug users .\tThe high court will hear arguments today on whether companies can be sued under the ADA for refusing to rehire rehabilitated drug users .\n1\t2945693\t2945847\tThe IRS said taxpayers can avoid undelivered checks by having refunds deposited directly into their checking or savings accounts .\tThe IRS said taxpayers can avoid problems with lost or stolen refunds by having refunds deposited directly into personal checking or savings accounts .\n1\t2065523\t2065836\t\" More than 70,000 men and women from bases in Southern California were deployed in Iraq .\tIn all , more than 70,000 troops based in Southern California were deployed to Iraq .\n1\t2222998\t2223097\tBP shares slipped 0.8 percent to 433.50 pence ( $ 6.85 ) each in afternoon trading on the London Stock Exchange .\tBP shares slipped 48 cents to $ 41.72 Friday in trading on the New York Stock Exchange .\n1\t2561999\t2561941\tBecause of the accounting charge , the company now says it lost $ 1.04 billion , or 32 cents a share , in the quarter ended June 30 .\tIncluding the charge , the Santa Clara , Calif.-based company said Monday it lost $ 1.04 billion , or 32 cents per share , in the period ending June 30 .\n0\t2324704\t2325023\tFriday 's report raised new worries that a weak job market could shackle the budding economic recovery despite a slight improvement in the overall unemployment rate .\tU.S. companies slashed payrolls for a seventh straight month in August , raising new worries that a weak jobs market could shackle the budding economic recovery .\n1\t2336453\t2336545\tFederal Emergency Management Administration designated $ 20 million to establish the registry .\tThe registry was launched with $ 20 million from the Federal Emergency Management Agency .\n1\t720572\t720486\tBREAST cancer cases in the UK have hit an all-time high with more than 40,000 women diagnosed with the disease each year , Cancer Re-search UK revealed yesterday .\tCases of breast cancer in Britain have reached a record high , with the number of women diagnosed with the disease passing the 40,000 mark for the first time .\n1\t1605818\t1605806\t\" It was never our intention to sell the product , \" said Health Minister Anne McClellan , a skeptic of medical marijuana use .\t\" It was never the intention of us to sell product , \" federal Health Minister Anne McLellan said yesterday in Edmonton .\n0\t2440680\t2440474\tGM , the world 's largest automaker , has 115,000 active UAW workers and another 340,000 retirees and spouses .\tThey cover more than 300,000 UAW workers and 500,000 retirees and spouses .\n0\t726399\t726078\tRosenthal is hereby sentenced to custody of the Federal Bureau of prisons for one day with credit for time served , \" Breyer said to tumultuous cheers in the courtroom .\t\" Rosenthal is hereby sentenced to custody of the Federal Bureau of Prisons for one day with credit for time served . \"\n1\t533903\t533818\t\" We are committed to helping the Iraqi people get on the path to a free society , \" Rumsfeld said in a speech to the Council on Foreign Relations .\t\" We are committed to helping the Iraqi people get on the path to a free society , \" he said .\n1\t1166473\t1166857\tMr. Young said he was disappointed that the government didn 't see the severe acute respiratory syndrome crisis as worthy of federal disaster-relief money .\tYoung said he was disappointed the government didn 't see the SARS crisis as worthy of federal disaster relief money .\n1\t144089\t143697\tThe 12-nation currency has risen by 33 percent against the dollar over the past 15 months .\tThe euro is up 9 percent against the dollar in the past six weeks .\n1\t3439854\t3439874\tIn February 2000 , the officers — Kenneth Boss , Sean Carroll , Edward McMellon and Richard Murphy — were acquitted of all charges in the killing .\tThe officers -- Kenneth Boss , Sean Carroll , Edward McMellon and Richard Murphy -- were acquitted in 2000 of state murder charges .\n1\t3464314\t3464302\tI was surprised it turned out me talking and the president just listening .\t\" I was surprised it turned out me talking and the president just listening . . . It was mostly a monologue . \"\n1\t2008984\t2009175\tThe state 's House delegation currently consists of 17 Democrats and 15 Republicans .\tDemocrats hold a 17-15 edge in the state 's U.S. House delegation .\n0\t816867\t816831\tFreddie also said Leland C. Brendsel will retire as chairman and chief executive and resign from the board .\tHe replaces Leland Brendsel , 61 , who retired as chairman and chief executive .\n1\t192285\t192327\tWe 'll be listening carefully to the [ IAEA ] director general 's report at the next board meeting .\t\" We 'll be listening carefully to the ( IAEA ) director-general 's report at the next board meeting . \"\n1\t2688145\t2688162\tIn that position , Elias will report to Joe Tucci , president and CEO of EMC .\tAs executive vice president of new ventures , Elias will report to Joe Tucci , EMC 's president and chief executive .\n1\t3294207\t3294290\tBut with the PM due to leave tomorrow afternoon for personal reasons there was a risk he might not be present when the final decision was made .\tBut with the Prime Minister due to leave tomorrow , a day early , he may not be present when the final decision is made .\n0\t205100\t205145\tA pro-independence radical , Miodrag Zivkovic , of the Liberal Alliance , came in second with 31 percent of the vote .\tMiodrag Zivkovic , of the Liberal Alliance of Montenegro , won 31 percent of the vote while the independent Dragan Hajdukovic got four percent .\n0\t3242051\t3241897\tMr. Kerkorian tried unsuccessfully to take over Chrysler in 1995 , but did win representation on its board .\tKerkorian and Tracinda had also tried to take over Chrysler in 1995 .\n0\t1076861\t1077018\tGlover spoke at a news conference that included about 20 relatives of the victims .\tAbout 20 family members of the victims were invited to the news conference .\n1\t2095803\t2095786\tDrax faced a financial crisis late last year after it lost its most lucrative sales contract , held with insolvent utility TXU Europe .\tDrax ’ s troubles began late last year when it lost its most lucrative sales contract , with the insolvent utility TXU Europe .\n1\t2112330\t2112376\tBut I would rather be talking about high standards than low standards . \"\t\" I would rather be talking about positive numbers rather than negative .\n1\t3389318\t3389271\tIt was not immediately known how many people were on flight UTA 141 , which could carry 141 passengers and crew .\tIt was still not known exactly how many people were on the plane , which could carry 141 passengers and crew .\n1\t698948\t698933\tThe market remains pinned in a narrow range after a powerful rally drove the broad Standard & Poor 's 500 index .SPX up more than 20 percent since mid-March .\tThe market remains pinned in a narrow range after a powerful rally pushed the broad S & P 500 index up more than 20 percent since mid-March .\n1\t539585\t539355\tWitnesses said they believed the man planned to crash the Launceston-bound Qantas flight 1737 , which was carrying 47 passengers and six crew .\tWitnesses believe he wanted to crash Flight 1737 , which had 47 passengers and six crew .\n1\t684848\t684557\tAs Samudra sat down to hear the indictment , he looked over to his nine lawyers and shouted ``God is Great ' ' three times .\tAs he sat down to hear the indictment , Samudra looked over to his nine lawyers and shouted \" Takbir ! \" , or \" Proclaim ! \" , a religious rallying cry .\n1\t347017\t347002\tIn hardest-hit Taipei , traffic has disappeared from once bustling streets , ubiquitous department stores stand mostly empty and restaurants are eerily quiet .\tIn hardest-hit Taipei , traffic has disappeared from once-bustling streets and department stores and restaurants are virtually empty .\n1\t1592037\t1592076\tIn a statement , Lee said he \" no longer believes that Viacom deliberately intended to trade on my name when naming Spike TV . \"\tSpike Lee no longer believes that Viacom deliberately intended to trade on his name by calling its own venture \" Spike TV , \" according to a statement read in court Tuesday .\n0\t3013483\t3013540\tSingapore Prime Minister Goh Chok Tong says China plays an important role in the integration of Asia , including managing the stresses and strains both within and between countries .\tHAINAN PROVINCE , China : Singapore Prime Minister Goh Chok Tong said China plays an important role in the integration of Asia .\n1\t2020252\t2020081\tThe worm attacks Windows computers via a hole in the operating system , an issue Microsoft on July 16 had warned about .\tThe worm attacks Windows computers via a hole in the operating system , which Microsoft warned of 16 July .\n0\t2614947\t2614904\tThe premium edition adds OfficeFront Page 2003 , Acceleration Server 2000 , and SQL Server 2000 .\tThe premium edition adds ISA Server , SQL Server and a specialized edition of BizTalk 2004 .\n0\t1744257\t1744378\tIn the year-ago quarter , the steelmaker recorded a profit of $ 16.2 million , or 15 cents per share , on sales of $ 1.14 billion .\tIn the second quarter last year , AK Steel reported a profit of $ 16.2 million , or 15 cents a share .\n0\t1119721\t1119714\tSony claimed that the reader 's capacitance sensing technology cannot be fooled by paper copies and does not require cleaning .\tIts capacitance sensing technology electronically reads a fingerprint ; Sony says it can 't be fooled by paper copies and doesn 't require cleaning .\n1\t1186754\t1187056\tAmazon.com shipped out more than a million copies of the new book , making Saturday the largest distribution day of a single item in e-commerce history .\tAmazon.com shipped more than a million copies by Saturday afternoon , making Saturday the largest distribution day of a single item in e-commerce history .\n1\t2842562\t2842582\tThe show 's closure affected third-quarter earnings per share by a penny .\tThe company said this impacted earnings by a penny a share .\n0\t431076\t431242\tAfter the two-hour meeting on May 14 , publisher Arthur O. Sulzberger Jr . , executive editor Howell Raines and managing editor Gerald Boyd pledged quick remedies to staff grievances .\tThe committee will make recommendations to Publisher Arthur Sulzberger , Executive Editor Howell Raines and Managing Editor Gerald Boyd .\n1\t1393764\t1393984\tIt 's been a busy couple of days for security gurus assigned to keep their companies safe and sound .\tIt 's been a busy couple of days for enterprise security gurus tasked with the job of keeping their companies safe and sound .\n0\t2916199\t2916164\tLu reclined in a soft chair wearing a woolly coat near the blackened capsule .\t\" It 's great to be back home , \" said Lu , dressed in a woolly coat near the blackened capsule .\n1\t2530671\t2530542\tGov. Bob Riley proposed the budget cuts after Alabama voters rejected his $ 1.2 billion tax plan Sept . 9 .\tAfter Alabama voters rejected his $ 1.2 billion tax plan Sept . 9 , Riley forecast significant cuts in state programs .\n1\t219064\t218969\t\" It is probably not the easiest time to come in and take over the shuttle program , but then again , I look forward to the challenge , \" he said .\t\" It 's probably not the easiest time to come in and take over the shuttle program , but I look forward to the challenge , \" Parsons told reporters at NASA headquarters .\n0\t2377289\t2377259\tEstonia 's place in the European mainstream and safeguard its independence regained in 1991 .\tEstonia was forcibly incorporated in the Soviet Union in 1940 and regained its independence only in 1991 .\n0\t2110220\t2110199\tFranklin County Judge-Executive Teresa Barton said a firefighter was struck by lightning and was taken to the Frankfort Regional Medical Center .\tA county firefighter , was struck by lightning and was in stable condition at Frankfort Regional Medical Center .\n0\t1864253\t1863810\tPolice suspected that Shaichat , 20 , had been abducted either by Palestinians or by Israeli Arabs .\tNobody claimed responsibility for Schaichat 's death , but police suspect that the 20-year-old soldier was abducted either by Palestinians or Israeli Arabs .\n0\t3150803\t3150839\tDuring this year 's August to October quarter , Lowe 's opened 38 new stores , including two relocations .\tDuring the third quarter , Lowe 's opened 38 new stores and now has 932 stores in 45 states .\n0\t969381\t969512\tThe technology-laced Nasdaq Composite Index < .IXIC > declined 25.78 points , or 1.56 percent , to 1,627.84 .\tThe broader Standard & Poor 's 500 Index .SPX gave up 11.91 points , or 1.19 percent , at 986.60 .\n1\t271891\t271839\tSony said the PSP would also feature a 4.5-inch LCD screen , Memory Stick expansion slots .\tIt also features a 4.5 in back-lit LCD screen and memory expansion facilities .\n0\t2829648\t2829613\tClinton did not mention that two Democratic senators , Charles Robb of Virginia and Wendell Ford of Kentucky , voted to shelve the McCain bill .\tTwo Democrats , Sen. Charles Robb of Virginia and Wendell Ford of Kentucky , voted with the 40 Republicans .\n1\t886904\t887158\tSome of the company 's software developers will join Microsoft , but details haven 't been finalized , said Mike Nash , corporate vice president of Microsoft 's security business unit .\tSome of the companys software developers will join Microsoft , but details havent been finalized , said Mike Nash , corporate vice president of Microsofts security business unit .\n0\t2632692\t2632767\tWal-Mart has said it plans to open at least 40 Supercenters in the state in the coming years ; analysts expect four or more to be in San Diego County .\tAt least 40 of the outlets will be in California , and analysts expect four or more to be in San Diego County .\n1\t2240399\t2240149\tCintas is battling efforts to unionize 17,000 of its workers and to let unions organize the workers by signing cards , rather than by a lengthy election process .\tCintas is battling efforts to unionize 17,000 of its workers and labor 's demands to let its workers organize by signing cards , rather than by a lengthy election process .\n1\t805457\t805985\tThe opposition would resort to rolling mass action \" at strategic times of our choice and without warning to the dictatorship , \" he said .\t\" From now onwards we will embark on rolling mass action at strategic times of our choice and without any warning to the dictatorship , \" he said .\n1\t2896308\t2896334\tFederal Agriculture Minister Warren Truss said the Government still did not know the real reason the sheep were rejected at the Saudi port of Jeddah on August 21 .\tHe said the Government still did not know the real reason the original Saudi buyer pulled out on August 21 .\n1\t2110775\t2110924\tTom Kraynak , manager of operations and resources for the Canton , Ohio-based East Central Area Reliability Council , said that scenario is one among many that investigators are considering .\tTom Kraynak , manager of operations and resources for the Canton , Ohio-based East Central Area Reliability Council , said investigators are considering the scenario .\n1\t1762569\t1762526\tHester said Sanmina was the best fit among several purchase offers the company received from electronics manufacturers and computer makers .\tHester said Sanmina 's offer was the best among several Newisys received from electronics manufacturers and computer makers .\n0\t2706154\t2706185\tThe other inmate fell but Selenski shimmed down the makeshift rope to a second-story roof and used the mattress to scale a razor-wire fence , Fischi said .\tAfter the other inmate fell , Selenski used the mattress to scale a 10-foot , razor-wire fence , Fischi said .\n1\t1057995\t1057778\tThe hearing , expected to last a week , will determine whether Akbar faces a court-martial .\tThe purpose of the hearing is to determine whether Akbar should be court-martialled .\n1\t1386884\t1386857\tHe said he has begun a court action to seize Beacon Hill 's assets and has frozen more than $ 13 million Beacon Hill had when it closed .\tHe said he has initiated a forfeiture action in court and frozen more than $ 13 million Beacon Hill had when it closed .\n1\t3093023\t3092996\tSpeaking for the first time yesterday , Brigitte 's maternal aunt said his family was unaware he had was in prison or that he had remarried .\tBrigitte 's maternal aunt said his family was unaware he had been sent to prison , or that he had remarried in Sydney .\n1\t1661381\t1661317\t\" Close co-operation between our law enforcement agencies , close co-operation between our intelligence services lie at the heart of the ongoing fight against terrorism . \"\tClose cooperation between regional law enforcement agencies and intelligence services was at the heart of the fight against terrorism , he said .\n0\t2926039\t2925982\tThe mother of a Briton held by Colombian guerrillasspoke of her relief yesterday after hearing that he might be freed in the next few weeks .\tThe parents of a Briton being held hostage by Colombian rebels spoke yesterday of their optimism that he would be freed in time for his birthday next month .\n0\t637168\t637447\tWe strongly disagree with Novell 's position and view it as a desperate measure to curry favor with the Linux community .\tMcBride characterized Novell 's move as \" a desperate measure to curry favor with the Linux community . \"\n1\t696677\t696932\tAfter more than two years ' detention under the State Security Bureau , the four were found guilty of subversion in Beijing 's No. 1 Intermediate Court last Wednesday .\tAfter more than two years in detention by the State Security Bureau , the four were found guilty last Wednesday of subversion .\n1\t3122429\t3122305\tMr Russell , 46 , a coal miner from Brisbane , said : \" They are obviously hurting , so we are basically going over there to help them . \"\t\" They are obviously hurting so we are basically going over there to help them , \" Russell , 46 , said .\n1\t1348909\t1348954\tThe New York Democrat and former first lady has said she will not run for the White House in 2004 , but has not ruled out a race in later years .\tThe former first lady has said she will not run for the White House in 2004 but has not ruled out a race later on .\n0\t162203\t162101\tIt does not affect the current Windows Media Player 9.0 Series .\tWindows Media Player has had security problems before .\n0\t71501\t71627\tThe seizure took place at 4 a.m. on March 18 , just hours before the first American air assault .\tThe time was about 4 a.m. on March 18 , just hours before the first pinpoint missiles rained down on the capital .\n1\t2907762\t2907649\tDonations stemming from the Sept . 11 attacks helped push up contributions to human service organizations and large branches of the United Way by 15 percent and 28.6 percent , respectively .\tDonations stemming from the Sept . 11 attacks helped push up contributions to human service organizations by 15 percent and to large branches of the United Way by 28.6 percent .\n1\t2167771\t2167744\tIn May , Mr. Hatfill said he was struck by a vehicle being driven by an FBI employee who was tailing him in Georgetown .\tLast May , Hatfill was struck by a vehicle being driven by an FBI employee who was tailing him in Washington 's Georgetown neighborhood .\n1\t3320577\t3320553\t\" I will support a constitutional amendment which would honor marriage between a man and a woman , codify that , \" he said .\t\" If necessary , I will support a constitutional amendment which would honour marriage between a man and a woman , codify that . \"\n1\t849291\t849442\tIBM of the US and Infineon Technologies of Germany will today announce a technological development that could threaten multi-billion dollar memory chip markets .\tIBMof the US andInfineon Technologies of Germany willon Tuesdayannounce a technological development that could threaten multi-billion dollar memory chip markets .\n0\t763948\t763991\tCosta 's semifinal opponent is Spaniard Juan Carlos Ferrero , whom he beat in last year 's final .\tCosta will play Juan Carlos Ferrero next in a rematch of last year 's final .\n1\t1908763\t1908744\tA former employee of a local power company pleaded guilty Wednesday to setting off a bomb that knocked out a power substation during the Winter Olympics last year .\tA former Utah Power meter reader pleaded guilty Wednesday to bombing a power substation during the 2002 Winter Olympics .\n0\t1876120\t1876059\tThyroid hormones are known to help in weight loss by stimulating metabolism - and cutting cholesterol - but come with the unwanted side effect of speeding up the heartbeat .\tThyroid hormones are known to help in weight loss by stimulating metabolism , and they can help cut cholesterol too .\n1\t518089\t518133\tJudge Craig Doran said it wasn 't his role to determine if Hovan was \" an evil man \" but maintained that \" he has committed an evil act . \"\tJudge Craig Doran said he couldn 't determine if Hovan was \" an evil man \" but said he \" has committed an evil act . \"\n0\t224932\t224868\tThe Hartford shares rose $ 2.88 , or 6.6 percent , to close Monday at $ 46.50 on the New York Stock Exchange .\tShares of Hartford rose $ 2.88 to $ 46.50 in New York Stock Exchange composite trading .\n1\t1771131\t1771091\tIt also offers a built-in NAND flash boot loader so that high-density NAND flash memory can be used without having to install an additional support chip .\tThe S3C2440 has a built-in NAND flash boot loader , for example , so that high-density NAND flash memory can be installed without an additional support chip .\n0\t2728425\t2728251\tIt decided instead to issue them before the stock market opened Monday after the downgrade of its debt late Friday by Moody 's , the credit rating agency .\tIt decided instead to issue them before the stock market opened Monday to counteract the downgrade of its debt late Friday by Moody 's to one step above junk status .\n0\t953733\t953537\tAltria shares fell 2.5 percent or $ 1.11 to $ 42.57 and were the Dow 's biggest percentage loser .\tIts shares fell $ 9.61 to $ 50.26 , ranking as the NYSE 's most-active issue and its biggest percentage loser .\n1\t349215\t349241\tIt will be followed in November by a third movie , \" The Matrix Revolutions . \"\tThe film is the second of a trilogy , which will wrap up in November with \" The Matrix Revolutions . \"\n1\t2919853\t2919804\tMassachusetts regulators and the Securities and Exchange Commission on Tuesday pressed securities fraud charges against Putnam Investments and two of its former portfolio managers for alleged improper mutual fund trading .\tState and federal securities regulators filed civil charges against Putnam Investments and two portfolio managers in the ever-expanding mutual fund trading scandal .\n1\t954526\t954607\tHe is blocking them until the Air Force assigns four additional C-130 cargo planes to Gowen Field , an Idaho Air National Guard base in Boise .\tHe is holding them up until the Air Force agrees to assign four additional C-130 cargo planes to the Idaho Air National Guard .\n1\t69773\t69792\tCisco pared spending to compensate for sluggish sales .\tIn response to sluggish sales , Cisco pared spending .\n0\t2823575\t2823513\tThe study , published Monday in the journal Molecular Brain Research , is likely to also apply to humans , its authors said .\tThe study , conducted on the brains of developing mice , was being published today in the journal Molecular Brain Research .\n1\t2455942\t2455978\tMy decision today is not based on any one event . \"\tGovernor Rowland said his decision was \" not based on any one event . \"\n1\t131979\t131957\tNelson , 27 , is being retried on civil-rights charges stemming from the disturbance which led to Rosenbaum 's death .\tNelson , 27 , is being retried on civil rights charges stemming from the disturbance that led to Rosenbaum 's death .\n0\t2010705\t2010779\t\" The government elements who have been causing trouble are still in place .\tThe government elements who have been causing trouble are still in place , they are attacking us . \"\n1\t54142\t53641\tNext Monday at about 2 p.m. ( CST ) , hospital officials in and near Chicago will notice a sudden increase in people complaining of flu-like symptoms .\tAround the same time , hospital officials in and near Chicago will notice a sudden increase in people complaining of flu-like symptoms .\n1\t1015249\t1015204\tWal-Mart Stores Inc . , Kohl 's Corp. , Family Dollar Stores Inc. and Big Lots Inc. were among the merchants posting May sales that fell below Wall Street 's modest expectations .\tWal- Mart , Kohl 's Corp. , Family Dollar Stores Inc . , and Big Lots Inc. posted May sales that fell below Wall Street 's modest expectations .\n0\t753928\t753890\tThe patch also fixes a vulnerability that results because IE does not implement an appropriate block on a file download dialog box .\tThe second vulnerability is a result of IE not implementing a block on a file download dialog box .\n1\t3022833\t3023029\tPeterson , a former fertilizer salesman , is charged with murder in the deaths of his 27-year-old wife and the baby boy she was carrying .\tPeterson , 31 , is now charged with murder in the deaths of his 27-year-old wife and their unborn son .\n0\t751520\t751373\tSPOT products run a Microsoft operating system and the company 's DirectBand radio technology developed with SCA Data Systems .\tThe DirectBand network was developed with the assistance of SCA Data Systems .\n0\t218848\t218851\tHe replaces Ron Dittemore , who announced his resignation in April .\tDittemore announced his plans to resign on April 23 .\n1\t3181118\t3181443\tDetectives told Deasean 's father , Stelly Chisolm , a college student , and mother , Kimberly Hill , of the arrest shortly after Perry was apprehended .\tShortly after his arrest , detectives told Deasean 's father , Stelly Chisolm , a college student , and mother , Kimberly Hill , a medical assistant , about the development .\n1\t515581\t515752\tThey were among about 40 people attending the traditional Jewish ceremony colored by some non-traditional touches .\tHe said about 40 people attended the traditional Jewish ceremony colored by some nontraditional touches .\n1\t347022\t347003\tTaiwan had been relatively free of the viral infection until a fiasco at a Taipei hospital in late April caused the number of infections to skyrocket .\tTaiwan had been relatively free of the viral infection until a severe outbreak at a Taipei hospital in late April .\n1\t3311600\t3311633\tMr. Rowland attended a party in South Windsor for the families of Connecticut National Guard soldiers called to active duty .\tRowland was making an appearance at a holiday party for families of Connecticut National Guard soldiers assigned to duty in Iraq and Afghanistan .\n0\t3439114\t3439084\tRoss Garber , Rowland 's lawyer , said Tuesday he would attend the meeting and would ask to speak on the issue .\tRoss Garber , Rowland 's legal counsel , said the governor would have no comment on the condo deal .\n0\t487951\t488007\tThe euro was at 1.5281 versus the Swiss franc EURCHF = , up 0.2 percent on the session , after hitting its highest since mid-2001 around 1.5292 earlier in the session .\tThe euro was steady versus the Swiss franc after hitting its highest since mid-2001 of 1.5261 earlier in the session .\n0\t314997\t315030\tOn the stand Wednesday , she said she was referring only to the kissing .\tOn the stand Wednesday , she testified that she was referring to the kissing before the alleged rape .\n0\t4733\t4557\tGarner said the group would probably be expanded to include , for example , a Christian and perhaps another Sunni leader .\tThe group has already met several times and Gen. Garner said it probably will be expanded to include a Christian and perhaps another Sunni Muslim leader .\n1\t2820371\t2820525\tBlair 's Foreign Secretary Jack Straw was to take his place on Monday to give a statement to parliament on the European Union .\tBlair 's office said his Foreign Secretary Jack Straw would take his place on Monday to give a statement to parliament on the EU meeting the prime minister attended last week .\n1\t801552\t801516\t\" There were more people surrounding the clubhouse than the Unabomber 's house up in the hills , \" Baker said .\t\" There are more people surrounding the clubhouse than surrounded the Unabomber 's home in the hills .\n1\t1704987\t1705268\tCharles O. Prince , 53 , was named as Mr. Weill 's successor .\tMr. Weill 's longtime confidant , Charles O. Prince , 53 , was named as his successor .\n1\t396041\t396188\tOfficials are also meeting with the International Organization for Epizootics ( OIE ) , which establishes animal-health standards for the world .\tCanadian officials were also expected to meet yesterday with the International Organization for Epizootics ( OIE ) , which establishes animal-health standards for the world .\n0\t1014983\t1014963\tGE stock closed Friday at $ 30.65 a share , down about 42 cents , on the New York Stock Exchange .\tGE 's shares closed at $ 30.65 on Friday on the New York Stock Exchange .\n1\t2320654\t2320666\tThe Midwestern research center will focus on the development of diagnostic , therapeutic and vaccine products for anthrax , botulism , tularemia , hemorrhagic fever viruses and plague .\tThe Midwestern center will focus on diagnosis , treatment and vaccines for anthrax , botulism , tularemia , hemorrhagic fever viruses and plague .\n1\t1057876\t1057778\tThe hearing is to determine whether there is enough evidence to order Akbar to a general court-martial proceeding .\tThe purpose of the hearing is to determine whether Akbar should be court-martialled .\n0\t2116843\t2116883\tIn the United States , heart attacks kill about 460,000 year , in Canada about 80,000 .\tIn the United States , heart attacks kill about 460,000 yearly , according to the National Institutes of Health .\n1\t1461629\t1461781\tNinety-five percent of international cargo to the United States is carried by ship .\tShips carry 95 percent of international cargo to the United States .\n0\t374015\t374162\t\" It 's a major victory for Maine , and it 's a major victory for other states .\tThe Maine program could be a model for other states .\n1\t2493369\t2493428\tNews that oil producers were lowering their output starting in November exacerbated a sell-off that was already under way on Wall Street .\tNews that the Organization of Petroleum Exporting Countries was lowering output starting in November exacerbated a stock sell-off already under way yesterday .\n1\t490355\t490378\tThey note that after several weeks of rallies on upbeat earnings , investors are looking for stronger evidence of a recovery before sending stocks higher .\tAfter several weeks of market rallies on upbeat earnings , many investors are looking for more concrete signs of an economic recovery .\n1\t2691044\t2691264\tMost economists had expected a more dire report , with many anticipating the fifth month of job losses in six months .\tMost economists had been expecting a far more dire report , with many expecting to see the fifth month of job losses in six months in September .\n1\t1831453\t1831491\tBut software license revenues , a measure financial analysts watch closely , decreased 21 percent to $ 107.6 million .\tLicense sales , a key measure of demand , fell 21 percent to $ 107.6 million .\n1\t2380695\t2380822\tKing , brand-name writer , master of the horror story and e-book pioneer , is receiving this year 's medal for Distinguished Contributions to American Letters .\tStephen King , master of the horror story and e-book pioneer , is receiving this year 's medal for Distinguished Contributions to American Letters from the National Book Foundation .\n1\t2577517\t2577531\tThe Denver-based natural gas producer and marketer said the inaccurate reporting was discovered after it received a subpoena from the U.S. Commodity Futures Trading Commission .\tThe natural gas producer and marketer said the inaccurate reporting was discovered in response to a subpoena from the U.S. Commodity Futures Trading Commission , or CFTC .\n1\t3267026\t3266930\tThe steel tariffs , which the U.S. president imposed in March 2002 , will officially end at midnight , instead of March 2005 as initially planned .\tThe U.S. steel tariffs , which Bush imposed in March 2002 , were to officially end at midnight Thursday ( 0500 GMT ) , instead of March 2005 as initially planned .\n1\t360875\t360943\tBusiness Week 's online edition reported on Friday that WorldCom and the SEC could announce a settlement as early as Monday .\tBusinessWeek Online has learned that the settlement could come as early as Monday , May 19 .\n1\t162632\t162653\tOnly one of the five buildings in the Baghdad compound of the United Nations Development Program escaped being burned , the UN said on its Web site .\tOnly one of the five buildings in the compound in Baghdad run by the UN Development Program , escaped being burned , the UN said on its Web site .\n1\t1128884\t1128865\tShares of Salix have rocketed 64 percent since Axcan made its first offer on April 10 .\tSince the initial takeover offer , Salix shares have risen about 35 percent .\n1\t3264732\t3264648\tThe jury verdict , reached Wednesday after less than four hours of deliberation , followed a 2 week trial , during which Waagner represented himself .\tThe quick conviction followed a 2 1 / 2 week trial , during which the Venango County man represented himself .\n1\t1721433\t1721267\tIt 's happened five times in the last 11 years : A disaster puts this Southwestern town in the headlines during the summer tourist season .\tIt 's happened five times in the last decade : A disaster puts this tourist town in the headlines during summer , its busiest season .\n0\t146112\t146127\tThe broader Standard & Poor 's 500 Index .SPX edged down 9 points , or 0.98 percent , to 921 .\tThe technology-laced Nasdaq Composite Index < .IXIC > shed 15 points , or 0.98 percent , to 1,492 .\n1\t389117\t389052\tThe company emphasized that McDonald 's USA does not import any raw beef or hamburger patties from Canada for McDonald 's use in the United States .\tMcDonald 's said in a statement that it does not import any raw beef or hamburger patties from Canada for use in the United States .\n1\t872784\t872834\tGregory Parseghian , a former investment banker , was appointed chief executive .\tGreg Parseghian was appointed the new chief executive .\n0\t2977500\t2977547\tTheir contract will expire at 12 : 01 a.m. Wednesday instead of 12 : 01 a.m. Sunday , said Rian Wathen , organizing director for United Food and Commercial Workers Local 700 .\t\" It has outraged the membership , \" said Rian Wathen , organizing director of United Food and Commercial Workers Local 700 .\n1\t3107137\t3107119\tBut plaque volume increased by 2.7 percent in pravastatin patients .\tThe volume of plaque in Pravachol patients ' arteries rose by 3 % .\n1\t1619244\t1619274\tToday in the US , the book - kept under wraps by its publishers , G. P. Putnam 's Sons , since its inception - will appear in bookstores .\tTomorrow the book , kept under wraps by G. P. Putnam 's Sons since its inception , will appear in bookstores .\n0\t3061836\t3062031\tThe S & P / TSX composite rose 87.74 points on the week , while the TSX Venture Exchange composite gained 44.49 points .\tOn the week , the Dow Jones industrial average rose 11.56 points , while the Nasdaq Stock Market gained 39.42 points .\n1\t485999\t486011\tEx-KGB agent Putin added that the Beatles were considered ' propaganda of an alien ideology ' .\tIn Soviet times the Beatles ' music \" was considered propaganda of an alien ideology .\n" }, { "path": "src/examples/tensorflow/bert_demo/latency_printer.py", "content": "latency_list = []\nwith open('latencies.txt', 'r') as f:\n for line in f:\n latency_list.append(float(line.rstrip()))\n\nlatency_list = sorted(latency_list)\nl = len(latency_list)\n\nprint(f'p50 latency is {latency_list[int(.5 * l)]} seconds')\nprint(f'p90 latency is {latency_list[int(.9 * l)]} seconds')\nprint(f'p95 latency is {latency_list[int(.95 * l)]} seconds')\nprint(f'p99 latency is {latency_list[int(.99 * l)]} seconds')\n" }, { "path": "src/examples/tensorflow/bert_demo/mrpc.proto", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\nsyntax = \"proto3\";\n\npackage mrpc;\n\nservice mrpc {\n rpc paraphrase (TextPair) returns (YesNo) {}\n}\n\nmessage TextPair {\n bytes text_a = 1;\n bytes text_b = 2;\n}\n\nmessage YesNo {\n bytes message = 1;\n bytes prediction = 2;\n}\n" }, { "path": "src/examples/tensorflow/bert_demo/mrpc_feature.py", "content": "# coding=utf-8\n# Copyright 2018 The Google AI Language Team Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Extract pre-computed feature vectors from BERT.\"\"\"\n\nimport os\nimport csv\nimport time\nimport numpy as np\nimport tokenization\n\n\nclass InputExample(object):\n \"\"\"A single training/test example for simple sequence classification.\"\"\"\n\n def __init__(self, guid, text_a, text_b=None, label=None):\n \"\"\"Constructs a InputExample.\n\n Args:\n guid: Unique id for the example.\n text_a: string. The untokenized text of the first sequence. For single\n sequence tasks, only this sequence must be specified.\n text_b: (Optional) string. The untokenized text of the second sequence.\n Only must be specified for sequence pair tasks.\n label: (Optional) string. The label of the example. This should be\n specified for train and dev examples, but not for test examples.\n \"\"\"\n self.guid = guid\n self.text_a = text_a\n self.text_b = text_b\n self.label = label\n\n\nclass PaddingInputExample(object):\n \"\"\"Fake example so the num input examples is a multiple of the batch size.\n\n When running eval/predict on the TPU, we need to pad the number of examples\n to be a multiple of the batch size, because the TPU requires a fixed batch\n size. The alternative is to drop the last batch, which is bad because it means\n the entire output data won't be generated.\n\n We use this class instead of `None` because treating `None` as padding\n battches could cause silent errors.\n \"\"\"\n\n\nclass InputFeatures(object):\n \"\"\"A single set of features of data.\"\"\"\n\n def __init__(self,\n input_ids,\n input_mask,\n segment_ids,\n label_id,\n is_real_example=True):\n self.input_ids = input_ids\n self.input_mask = input_mask\n self.segment_ids = segment_ids\n self.label_id = label_id\n self.is_real_example = is_real_example\n\n\ndef convert_single_example(ex_index, example, label_list, max_seq_length,\n tokenizer):\n \"\"\"Converts a single `InputExample` into a single `InputFeatures`.\"\"\"\n\n if isinstance(example, PaddingInputExample):\n return InputFeatures(\n input_ids=[0] * max_seq_length,\n input_mask=[0] * max_seq_length,\n segment_ids=[0] * max_seq_length,\n label_id=0,\n is_real_example=False)\n\n label_map = {}\n for (i, label) in enumerate(label_list):\n label_map[label] = i\n\n tokens_a = tokenizer.tokenize(example.text_a)\n tokens_b = None\n if example.text_b:\n tokens_b = tokenizer.tokenize(example.text_b)\n\n if tokens_b:\n # Modifies `tokens_a` and `tokens_b` in place so that the total\n # length is less than the specified length.\n # Account for [CLS], [SEP], [SEP] with \"- 3\"\n _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)\n else:\n # Account for [CLS] and [SEP] with \"- 2\"\n if len(tokens_a) > max_seq_length - 2:\n tokens_a = tokens_a[0:(max_seq_length - 2)]\n\n # The convention in BERT is:\n # (a) For sequence pairs:\n # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1\n # (b) For single sequences:\n # tokens: [CLS] the dog is hairy . [SEP]\n # type_ids: 0 0 0 0 0 0 0\n #\n # Where \"type_ids\" are used to indicate whether this is the first\n # sequence or the second sequence. The embedding vectors for `type=0` and\n # `type=1` were learned during pre-training and are added to the wordpiece\n # embedding vector (and position vector). This is not *strictly* necessary\n # since the [SEP] token unambiguously separates the sequences, but it makes\n # it easier for the model to learn the concept of sequences.\n #\n # For classification tasks, the first vector (corresponding to [CLS]) is\n # used as the \"sentence vector\". Note that this only makes sense because\n # the entire model is fine-tuned.\n tokens = []\n segment_ids = []\n tokens.append(\"[CLS]\")\n segment_ids.append(0)\n for token in tokens_a:\n tokens.append(token)\n segment_ids.append(0)\n tokens.append(\"[SEP]\")\n segment_ids.append(0)\n\n if tokens_b:\n for token in tokens_b:\n tokens.append(token)\n segment_ids.append(1)\n tokens.append(\"[SEP]\")\n segment_ids.append(1)\n\n input_ids = tokenizer.convert_tokens_to_ids(tokens)\n\n # The mask has 1 for real tokens and 0 for padding tokens. Only real\n # tokens are attended to.\n input_mask = [1] * len(input_ids)\n\n # Zero-pad up to the sequence length.\n while len(input_ids) < max_seq_length:\n input_ids.append(0)\n input_mask.append(0)\n segment_ids.append(0)\n\n assert len(input_ids) == max_seq_length\n assert len(input_mask) == max_seq_length\n assert len(segment_ids) == max_seq_length\n\n label_id = label_map[example.label]\n\n feature = InputFeatures(\n input_ids=input_ids,\n input_mask=input_mask,\n segment_ids=segment_ids,\n label_id=label_id,\n is_real_example=True)\n return feature\n\n\ndef read_tsv(input_file, quotechar=None):\n \"\"\"Reads a tab separated value file.\"\"\"\n with open(input_file, \"r\") as f:\n reader = csv.reader(f, delimiter=\"\\t\", quotechar=quotechar)\n lines = []\n for line in reader:\n lines.append(line)\n return lines\n\n\ndef create_examples(lines, set_type):\n \"\"\"Creates examples for the training and dev sets.\"\"\"\n examples = []\n for (i, line) in enumerate(lines):\n if i == 0:\n continue\n guid = \"%s-%s\" % (set_type, i)\n text_a = tokenization.convert_to_unicode(line[3])\n text_b = tokenization.convert_to_unicode(line[4])\n if set_type == \"test\":\n label = \"0\"\n else:\n label = tokenization.convert_to_unicode(line[0])\n examples.append(\n InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))\n return examples\n\n\ndef _truncate_seq_pair(tokens_a, tokens_b, max_length):\n \"\"\"Truncates a sequence pair in place to the maximum length.\"\"\"\n\n # This is a simple heuristic which will always truncate the longer sequence\n # one token at a time. This makes more sense than truncating an equal percent\n # of tokens from each, since if one sequence is very short then each token\n # that's truncated likely contains more information than a longer sequence.\n while True:\n total_length = len(tokens_a) + len(tokens_b)\n if total_length <= max_length:\n break\n if len(tokens_a) > len(tokens_b):\n tokens_a.pop()\n else:\n tokens_b.pop()\n\n\ndef get_eval_model_feed_dict_list(mrpc_tsv, vocab_txt):\n tsv = read_tsv(mrpc_tsv)\n result = create_examples(tsv, \"dev\")\n model_feed_dict_list = []\n for example in result:\n tokenizer = tokenization.FullTokenizer(vocab_file=vocab_txt, do_lower_case=True)\n label_list = ['0', '1']\n feature = convert_single_example(ex_index=0, example=example, label_list=label_list,\n max_seq_length=128, tokenizer=tokenizer)\n pre_model_feed_dict = {\n 'input_ids': feature.input_ids,\n 'input_mask': feature.input_mask,\n 'segment_ids': feature.segment_ids,\n 'label_id': feature.label_id,\n 'is_real_example': feature.is_real_example,\n }\n model_feed_dict = {}\n for key, value in pre_model_feed_dict.items():\n if key in {'label_id', 'is_real_example'}:\n value = np.tile(np.int32(value), reps=[1])\n else:\n value = np.tile(np.int32(value), reps=[1, 1])\n model_feed_dict[key] = value\n model_feed_dict_list.append(model_feed_dict)\n return model_feed_dict_list\n\n\ndef text_pair_to_model_feed_dict(text_a, text_b, tokenizer):\n fake_tsv = [['index', '#1 ID', '#2 ID', '#1 String', '#2 String'],\n ['', '', '', text_a, text_b]]\n result = create_examples(fake_tsv, \"test\")\n example = result[0]\n label_list = ['0', '1']\n feature = convert_single_example(ex_index=0, example=example, label_list=label_list,\n max_seq_length=128, tokenizer=tokenizer)\n return {\n 'input_ids': np.tile(np.int32(feature.input_ids), reps=[1, 1]),\n 'input_mask': np.tile(np.int32(feature.input_mask), reps=[1, 1]),\n 'segment_ids': np.tile(np.int32(feature.segment_ids), reps=[1, 1]),\n }\n" }, { "path": "src/examples/tensorflow/bert_demo/mrpc_pb2.py", "content": "# -*- coding: utf-8 -*-\n# Generated by the protocol buffer compiler. DO NOT EDIT!\n# source: mrpc.proto\n\nimport sys\n_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))\nfrom google.protobuf import descriptor as _descriptor\nfrom google.protobuf import message as _message\nfrom google.protobuf import reflection as _reflection\nfrom google.protobuf import symbol_database as _symbol_database\n# @@protoc_insertion_point(imports)\n\n_sym_db = _symbol_database.Default()\n\n\n\n\nDESCRIPTOR = _descriptor.FileDescriptor(\n name='mrpc.proto',\n package='mrpc',\n syntax='proto3',\n serialized_options=None,\n serialized_pb=_b('\\n\\nmrpc.proto\\x12\\x04mrpc\\\"*\\n\\x08TextPair\\x12\\x0e\\n\\x06text_a\\x18\\x01 \\x01(\\x0c\\x12\\x0e\\n\\x06text_b\\x18\\x02 \\x01(\\x0c\\\",\\n\\x05YesNo\\x12\\x0f\\n\\x07message\\x18\\x01 \\x01(\\x0c\\x12\\x12\\n\\nprediction\\x18\\x02 \\x01(\\x0c\\x32\\x33\\n\\x04mrpc\\x12+\\n\\nparaphrase\\x12\\x0e.mrpc.TextPair\\x1a\\x0b.mrpc.YesNo\\\"\\x00\\x62\\x06proto3')\n)\n\n\n\n\n_TEXTPAIR = _descriptor.Descriptor(\n name='TextPair',\n full_name='mrpc.TextPair',\n filename=None,\n file=DESCRIPTOR,\n containing_type=None,\n fields=[\n _descriptor.FieldDescriptor(\n name='text_a', full_name='mrpc.TextPair.text_a', index=0,\n number=1, type=12, cpp_type=9, label=1,\n has_default_value=False, default_value=_b(\"\"),\n message_type=None, enum_type=None, containing_type=None,\n is_extension=False, extension_scope=None,\n serialized_options=None, file=DESCRIPTOR),\n _descriptor.FieldDescriptor(\n name='text_b', full_name='mrpc.TextPair.text_b', index=1,\n number=2, type=12, cpp_type=9, label=1,\n has_default_value=False, default_value=_b(\"\"),\n message_type=None, enum_type=None, containing_type=None,\n is_extension=False, extension_scope=None,\n serialized_options=None, file=DESCRIPTOR),\n ],\n extensions=[\n ],\n nested_types=[],\n enum_types=[\n ],\n serialized_options=None,\n is_extendable=False,\n syntax='proto3',\n extension_ranges=[],\n oneofs=[\n ],\n serialized_start=20,\n serialized_end=62,\n)\n\n\n_YESNO = _descriptor.Descriptor(\n name='YesNo',\n full_name='mrpc.YesNo',\n filename=None,\n file=DESCRIPTOR,\n containing_type=None,\n fields=[\n _descriptor.FieldDescriptor(\n name='message', full_name='mrpc.YesNo.message', index=0,\n number=1, type=12, cpp_type=9, label=1,\n has_default_value=False, default_value=_b(\"\"),\n message_type=None, enum_type=None, containing_type=None,\n is_extension=False, extension_scope=None,\n serialized_options=None, file=DESCRIPTOR),\n _descriptor.FieldDescriptor(\n name='prediction', full_name='mrpc.YesNo.prediction', index=1,\n number=2, type=12, cpp_type=9, label=1,\n has_default_value=False, default_value=_b(\"\"),\n message_type=None, enum_type=None, containing_type=None,\n is_extension=False, extension_scope=None,\n serialized_options=None, file=DESCRIPTOR),\n ],\n extensions=[\n ],\n nested_types=[],\n enum_types=[\n ],\n serialized_options=None,\n is_extendable=False,\n syntax='proto3',\n extension_ranges=[],\n oneofs=[\n ],\n serialized_start=64,\n serialized_end=108,\n)\n\nDESCRIPTOR.message_types_by_name['TextPair'] = _TEXTPAIR\nDESCRIPTOR.message_types_by_name['YesNo'] = _YESNO\n_sym_db.RegisterFileDescriptor(DESCRIPTOR)\n\nTextPair = _reflection.GeneratedProtocolMessageType('TextPair', (_message.Message,), {\n 'DESCRIPTOR' : _TEXTPAIR,\n '__module__' : 'mrpc_pb2'\n # @@protoc_insertion_point(class_scope:mrpc.TextPair)\n })\n_sym_db.RegisterMessage(TextPair)\n\nYesNo = _reflection.GeneratedProtocolMessageType('YesNo', (_message.Message,), {\n 'DESCRIPTOR' : _YESNO,\n '__module__' : 'mrpc_pb2'\n # @@protoc_insertion_point(class_scope:mrpc.YesNo)\n })\n_sym_db.RegisterMessage(YesNo)\n\n\n\n_MRPC = _descriptor.ServiceDescriptor(\n name='mrpc',\n full_name='mrpc.mrpc',\n file=DESCRIPTOR,\n index=0,\n serialized_options=None,\n serialized_start=110,\n serialized_end=161,\n methods=[\n _descriptor.MethodDescriptor(\n name='paraphrase',\n full_name='mrpc.mrpc.paraphrase',\n index=0,\n containing_service=None,\n input_type=_TEXTPAIR,\n output_type=_YESNO,\n serialized_options=None,\n ),\n])\n_sym_db.RegisterServiceDescriptor(_MRPC)\n\nDESCRIPTOR.services_by_name['mrpc'] = _MRPC\n\n# @@protoc_insertion_point(module_scope)\n" }, { "path": "src/examples/tensorflow/bert_demo/mrpc_pb2_grpc.py", "content": "# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\n# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!\nimport grpc\n\nimport mrpc_pb2 as mrpc__pb2\n\n\nclass mrpcStub(object):\n # missing associated documentation comment in .proto file\n pass\n\n def __init__(self, channel):\n \"\"\"Constructor.\n\n Args:\n channel: A grpc.Channel.\n \"\"\"\n self.paraphrase = channel.unary_unary(\n '/mrpc.mrpc/paraphrase',\n request_serializer=mrpc__pb2.TextPair.SerializeToString,\n response_deserializer=mrpc__pb2.YesNo.FromString,\n )\n\n\nclass mrpcServicer(object):\n # missing associated documentation comment in .proto file\n pass\n\n def paraphrase(self, request, context):\n # missing associated documentation comment in .proto file\n pass\n context.set_code(grpc.StatusCode.UNIMPLEMENTED)\n context.set_details('Method not implemented!')\n raise NotImplementedError('Method not implemented!')\n\n\ndef add_mrpcServicer_to_server(servicer, server):\n rpc_method_handlers = {\n 'paraphrase': grpc.unary_unary_rpc_method_handler(\n servicer.paraphrase,\n request_deserializer=mrpc__pb2.TextPair.FromString,\n response_serializer=mrpc__pb2.YesNo.SerializeToString,\n ),\n }\n generic_handler = grpc.method_handlers_generic_handler(\n 'mrpc.mrpc', rpc_method_handlers)\n server.add_generic_rpc_handlers((generic_handler,))\n" }, { "path": "src/examples/tensorflow/bert_demo/protoc.sh", "content": "#!/bin/bash\n# coding=utf-8\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n Program to gather information from a system\n\"\"\"\n\npython -m grpc_tools.protoc -I . --python_out=. --grpc_python_out=. mrpc.proto\n" }, { "path": "src/examples/tensorflow/bert_demo/setup.py", "content": "\nimport setuptools\n\n\nsetuptools.setup(\n name='bert-demo',\n version='2019.12.13',\n description='BERT Client-Server Demo',\n author='Amazon AWS',\n author_email='aws-neuron-support@amazon.com',\n license='BSD',\n classifiers=[\n 'Development Status :: 1 - Planning',\n 'Intended Audience :: Developers',\n 'Topic :: Scientific/Engineering :: Artificial Intelligence',\n 'License :: OSI Approved :: BSD License',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.5',\n 'Programming Language :: Python :: 3.6',\n 'Programming Language :: Python :: 3.7',\n ],\n keywords='bert',\n include_package_data=True,\n packages=setuptools.PEP420PackageFinder.find(),\n package_data={'': [\n '*',\n ]},\n entry_points={\n 'console_scripts': [\n 'neuron_bert_model=bert_demo.bert_model:main',\n 'bert_server=bert_demo.bert_server:serve',\n 'bert_client=bert_demo.bert_client:client',\n ],\n },\n install_requires=[\n ],\n)\n" }, { "path": "src/examples/tensorflow/bert_demo/tokenization.py", "content": "# coding=utf-8\n# Copyright 2018 The Google AI Language Team Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Tokenization classes.\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport collections\nimport re\nimport unicodedata\nimport six\n\n\ndef validate_case_matches_checkpoint(do_lower_case, init_checkpoint):\n \"\"\"Checks whether the casing config is consistent with the checkpoint name.\"\"\"\n\n # The casing has to be passed in by the user and there is no explicit check\n # as to whether it matches the checkpoint. The casing information probably\n # should have been stored in the bert_config.json file, but it's not, so\n # we have to heuristically detect it to validate.\n\n if not init_checkpoint:\n return\n\n m = re.match(\"^.*?([A-Za-z0-9_-]+)/bert_model.ckpt\", init_checkpoint)\n if m is None:\n return\n\n model_name = m.group(1)\n\n lower_models = [\n \"uncased_L-24_H-1024_A-16\", \"uncased_L-12_H-768_A-12\",\n \"multilingual_L-12_H-768_A-12\", \"chinese_L-12_H-768_A-12\"\n ]\n\n cased_models = [\n \"cased_L-12_H-768_A-12\", \"cased_L-24_H-1024_A-16\",\n \"multi_cased_L-12_H-768_A-12\"\n ]\n\n is_bad_config = False\n if model_name in lower_models and not do_lower_case:\n is_bad_config = True\n actual_flag = \"False\"\n case_name = \"lowercased\"\n opposite_flag = \"True\"\n\n if model_name in cased_models and do_lower_case:\n is_bad_config = True\n actual_flag = \"True\"\n case_name = \"cased\"\n opposite_flag = \"False\"\n\n if is_bad_config:\n raise ValueError(\n \"You passed in `--do_lower_case=%s` with `--init_checkpoint=%s`. \"\n \"However, `%s` seems to be a %s model, so you \"\n \"should pass in `--do_lower_case=%s` so that the fine-tuning matches \"\n \"how the model was pre-training. If this error is wrong, please \"\n \"just comment out this check.\" % (actual_flag, init_checkpoint,\n model_name, case_name, opposite_flag))\n\n\ndef convert_to_unicode(text):\n \"\"\"Converts `text` to Unicode (if it's not already), assuming utf-8 input.\"\"\"\n if six.PY3:\n if isinstance(text, str):\n return text\n elif isinstance(text, bytes):\n return text.decode(\"utf-8\", \"ignore\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n elif six.PY2:\n if isinstance(text, str):\n return text.decode(\"utf-8\", \"ignore\")\n elif isinstance(text, unicode):\n return text\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n else:\n raise ValueError(\"Not running on Python2 or Python 3?\")\n\n\ndef printable_text(text):\n \"\"\"Returns text encoded in a way suitable for print or `tf.logging`.\"\"\"\n\n # These functions want `str` for both Python2 and Python3, but in one case\n # it's a Unicode string and in the other it's a byte string.\n if six.PY3:\n if isinstance(text, str):\n return text\n elif isinstance(text, bytes):\n return text.decode(\"utf-8\", \"ignore\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n elif six.PY2:\n if isinstance(text, str):\n return text\n elif isinstance(text, unicode):\n return text.encode(\"utf-8\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n else:\n raise ValueError(\"Not running on Python2 or Python 3?\")\n\n\ndef load_vocab(vocab_file):\n \"\"\"Loads a vocabulary file into a dictionary.\"\"\"\n vocab = collections.OrderedDict()\n index = 0\n with open(vocab_file, \"r\") as reader:\n while True:\n token = convert_to_unicode(reader.readline())\n if not token:\n break\n token = token.strip()\n vocab[token] = index\n index += 1\n return vocab\n\n\ndef convert_by_vocab(vocab, items):\n \"\"\"Converts a sequence of [tokens|ids] using the vocab.\"\"\"\n output = []\n for item in items:\n output.append(vocab[item])\n return output\n\n\ndef convert_tokens_to_ids(vocab, tokens):\n return convert_by_vocab(vocab, tokens)\n\n\ndef convert_ids_to_tokens(inv_vocab, ids):\n return convert_by_vocab(inv_vocab, ids)\n\n\ndef whitespace_tokenize(text):\n \"\"\"Runs basic whitespace cleaning and splitting on a piece of text.\"\"\"\n text = text.strip()\n if not text:\n return []\n tokens = text.split()\n return tokens\n\n\nclass FullTokenizer(object):\n \"\"\"Runs end-to-end tokenziation.\"\"\"\n\n def __init__(self, vocab_file, do_lower_case=True):\n self.vocab = load_vocab(vocab_file)\n self.inv_vocab = {v: k for k, v in self.vocab.items()}\n self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)\n self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)\n\n def tokenize(self, text):\n split_tokens = []\n for token in self.basic_tokenizer.tokenize(text):\n for sub_token in self.wordpiece_tokenizer.tokenize(token):\n split_tokens.append(sub_token)\n\n return split_tokens\n\n def convert_tokens_to_ids(self, tokens):\n return convert_by_vocab(self.vocab, tokens)\n\n def convert_ids_to_tokens(self, ids):\n return convert_by_vocab(self.inv_vocab, ids)\n\n\nclass BasicTokenizer(object):\n \"\"\"Runs basic tokenization (punctuation splitting, lower casing, etc.).\"\"\"\n\n def __init__(self, do_lower_case=True):\n \"\"\"Constructs a BasicTokenizer.\n\n Args:\n do_lower_case: Whether to lower case the input.\n \"\"\"\n self.do_lower_case = do_lower_case\n\n def tokenize(self, text):\n \"\"\"Tokenizes a piece of text.\"\"\"\n text = convert_to_unicode(text)\n text = self._clean_text(text)\n\n # This was added on November 1st, 2018 for the multilingual and Chinese\n # models. This is also applied to the English models now, but it doesn't\n # matter since the English models were not trained on any Chinese data\n # and generally don't have any Chinese data in them (there are Chinese\n # characters in the vocabulary because Wikipedia does have some Chinese\n # words in the English Wikipedia.).\n text = self._tokenize_chinese_chars(text)\n\n orig_tokens = whitespace_tokenize(text)\n split_tokens = []\n for token in orig_tokens:\n if self.do_lower_case:\n token = token.lower()\n token = self._run_strip_accents(token)\n split_tokens.extend(self._run_split_on_punc(token))\n\n output_tokens = whitespace_tokenize(\" \".join(split_tokens))\n return output_tokens\n\n def _run_strip_accents(self, text):\n \"\"\"Strips accents from a piece of text.\"\"\"\n text = unicodedata.normalize(\"NFD\", text)\n output = []\n for char in text:\n cat = unicodedata.category(char)\n if cat == \"Mn\":\n continue\n output.append(char)\n return \"\".join(output)\n\n def _run_split_on_punc(self, text):\n \"\"\"Splits punctuation on a piece of text.\"\"\"\n chars = list(text)\n i = 0\n start_new_word = True\n output = []\n while i < len(chars):\n char = chars[i]\n if _is_punctuation(char):\n output.append([char])\n start_new_word = True\n else:\n if start_new_word:\n output.append([])\n start_new_word = False\n output[-1].append(char)\n i += 1\n\n return [\"\".join(x) for x in output]\n\n def _tokenize_chinese_chars(self, text):\n \"\"\"Adds whitespace around any CJK character.\"\"\"\n output = []\n for char in text:\n cp = ord(char)\n if self._is_chinese_char(cp):\n output.append(\" \")\n output.append(char)\n output.append(\" \")\n else:\n output.append(char)\n return \"\".join(output)\n\n def _is_chinese_char(self, cp):\n \"\"\"Checks whether CP is the codepoint of a CJK character.\"\"\"\n # This defines a \"chinese character\" as anything in the CJK Unicode block:\n # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)\n #\n # Note that the CJK Unicode block is NOT all Japanese and Korean characters,\n # despite its name. The modern Korean Hangul alphabet is a different block,\n # as is Japanese Hiragana and Katakana. Those alphabets are used to write\n # space-separated words, so they are not treated specially and handled\n # like the all of the other languages.\n if ((cp >= 0x4E00 and cp <= 0x9FFF) or #\n (cp >= 0x3400 and cp <= 0x4DBF) or #\n (cp >= 0x20000 and cp <= 0x2A6DF) or #\n (cp >= 0x2A700 and cp <= 0x2B73F) or #\n (cp >= 0x2B740 and cp <= 0x2B81F) or #\n (cp >= 0x2B820 and cp <= 0x2CEAF) or\n (cp >= 0xF900 and cp <= 0xFAFF) or #\n (cp >= 0x2F800 and cp <= 0x2FA1F)): #\n return True\n\n return False\n\n def _clean_text(self, text):\n \"\"\"Performs invalid character removal and whitespace cleanup on text.\"\"\"\n output = []\n for char in text:\n cp = ord(char)\n if cp == 0 or cp == 0xfffd or _is_control(char):\n continue\n if _is_whitespace(char):\n output.append(\" \")\n else:\n output.append(char)\n return \"\".join(output)\n\n\nclass WordpieceTokenizer(object):\n \"\"\"Runs WordPiece tokenziation.\"\"\"\n\n def __init__(self, vocab, unk_token=\"[UNK]\", max_input_chars_per_word=200):\n self.vocab = vocab\n self.unk_token = unk_token\n self.max_input_chars_per_word = max_input_chars_per_word\n\n def tokenize(self, text):\n \"\"\"Tokenizes a piece of text into its word pieces.\n\n This uses a greedy longest-match-first algorithm to perform tokenization\n using the given vocabulary.\n\n For example:\n input = \"unaffable\"\n output = [\"un\", \"##aff\", \"##able\"]\n\n Args:\n text: A single token or whitespace separated tokens. This should have\n already been passed through `BasicTokenizer.\n\n Returns:\n A list of wordpiece tokens.\n \"\"\"\n\n text = convert_to_unicode(text)\n\n output_tokens = []\n for token in whitespace_tokenize(text):\n chars = list(token)\n if len(chars) > self.max_input_chars_per_word:\n output_tokens.append(self.unk_token)\n continue\n\n is_bad = False\n start = 0\n sub_tokens = []\n while start < len(chars):\n end = len(chars)\n cur_substr = None\n while start < end:\n substr = \"\".join(chars[start:end])\n if start > 0:\n substr = \"##\" + substr\n if substr in self.vocab:\n cur_substr = substr\n break\n end -= 1\n if cur_substr is None:\n is_bad = True\n break\n sub_tokens.append(cur_substr)\n start = end\n\n if is_bad:\n output_tokens.append(self.unk_token)\n else:\n output_tokens.extend(sub_tokens)\n return output_tokens\n\n\ndef _is_whitespace(char):\n \"\"\"Checks whether `chars` is a whitespace character.\"\"\"\n # \\t, \\n, and \\r are technically contorl characters but we treat them\n # as whitespace since they are generally considered as such.\n if char == \" \" or char == \"\\t\" or char == \"\\n\" or char == \"\\r\":\n return True\n cat = unicodedata.category(char)\n if cat == \"Zs\":\n return True\n return False\n\n\ndef _is_control(char):\n \"\"\"Checks whether `chars` is a control character.\"\"\"\n # These are technically control characters but we count them as whitespace\n # characters.\n if char == \"\\t\" or char == \"\\n\" or char == \"\\r\":\n return False\n cat = unicodedata.category(char)\n if cat in (\"Cc\", \"Cf\"):\n return True\n return False\n\n\ndef _is_punctuation(char):\n \"\"\"Checks whether `chars` is a punctuation character.\"\"\"\n cp = ord(char)\n # We treat all non-letter/number ASCII as punctuation.\n # Characters such as \"^\", \"$\", and \"`\" are not in the Unicode\n # Punctuation class but we treat them as punctuation anyways, for\n # consistency.\n if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or\n (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):\n return True\n cat = unicodedata.category(char)\n if cat.startswith(\"P\"):\n return True\n return False\n" }, { "path": "src/examples/tensorflow/bert_demo/tune_save.sh", "content": "#!/bin/bash\n\npushd $BERT_REPO_DIR\n\npython run_classifier.py \\\n --task_name=MRPC \\\n --do_train=true \\\n --do_eval=true \\\n --do_predict=true \\\n --data_dir=$GLUE_DIR/MRPC \\\n --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n --max_seq_length=128 \\\n --train_batch_size=32 \\\n --learning_rate=2e-5 \\\n --num_train_epochs=3.0 \\\n --output_dir=$BERT_REPO_DIR/MRPC_finetune\n\npython run_classifier.py \\\n --task_name=MRPC \\\n --do_predict=true \\\n --data_dir=$GLUE_DIR/MRPC \\\n --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n --max_seq_length=128 \\\n --output_dir=$BERT_REPO_DIR/MRPC_finetune\n\npopd\n\n" }, { "path": "src/examples/tensorflow/bert_demo/uncased_L-24_H-1024_A-16.vocab.txt", "content": "[PAD]\n[unused0]\n[unused1]\n[unused2]\n[unused3]\n[unused4]\n[unused5]\n[unused6]\n[unused7]\n[unused8]\n[unused9]\n[unused10]\n[unused11]\n[unused12]\n[unused13]\n[unused14]\n[unused15]\n[unused16]\n[unused17]\n[unused18]\n[unused19]\n[unused20]\n[unused21]\n[unused22]\n[unused23]\n[unused24]\n[unused25]\n[unused26]\n[unused27]\n[unused28]\n[unused29]\n[unused30]\n[unused31]\n[unused32]\n[unused33]\n[unused34]\n[unused35]\n[unused36]\n[unused37]\n[unused38]\n[unused39]\n[unused40]\n[unused41]\n[unused42]\n[unused43]\n[unused44]\n[unused45]\n[unused46]\n[unused47]\n[unused48]\n[unused49]\n[unused50]\n[unused51]\n[unused52]\n[unused53]\n[unused54]\n[unused55]\n[unused56]\n[unused57]\n[unused58]\n[unused59]\n[unused60]\n[unused61]\n[unused62]\n[unused63]\n[unused64]\n[unused65]\n[unused66]\n[unused67]\n[unused68]\n[unused69]\n[unused70]\n[unused71]\n[unused72]\n[unused73]\n[unused74]\n[unused75]\n[unused76]\n[unused77]\n[unused78]\n[unused79]\n[unused80]\n[unused81]\n[unused82]\n[unused83]\n[unused84]\n[unused85]\n[unused86]\n[unused87]\n[unused88]\n[unused89]\n[unused90]\n[unused91]\n[unused92]\n[unused93]\n[unused94]\n[unused95]\n[unused96]\n[unused97]\n[unused98]\n[UNK]\n[CLS]\n[SEP]\n[MASK]\n[unused99]\n[unused100]\n[unused101]\n[unused102]\n[unused103]\n[unused104]\n[unused105]\n[unused106]\n[unused107]\n[unused108]\n[unused109]\n[unused110]\n[unused111]\n[unused112]\n[unused113]\n[unused114]\n[unused115]\n[unused116]\n[unused117]\n[unused118]\n[unused119]\n[unused120]\n[unused121]\n[unused122]\n[unused123]\n[unused124]\n[unused125]\n[unused126]\n[unused127]\n[unused128]\n[unused129]\n[unused130]\n[unused131]\n[unused132]\n[unused133]\n[unused134]\n[unused135]\n[unused136]\n[unused137]\n[unused138]\n[unused139]\n[unused140]\n[unused141]\n[unused142]\n[unused143]\n[unused144]\n[unused145]\n[unused146]\n[unused147]\n[unused148]\n[unused149]\n[unused150]\n[unused151]\n[unused152]\n[unused153]\n[unused154]\n[unused155]\n[unused156]\n[unused157]\n[unused158]\n[unused159]\n[unused160]\n[unused161]\n[unused162]\n[unused163]\n[unused164]\n[unused165]\n[unused166]\n[unused167]\n[unused168]\n[unused169]\n[unused170]\n[unused171]\n[unused172]\n[unused173]\n[unused174]\n[unused175]\n[unused176]\n[unused177]\n[unused178]\n[unused179]\n[unused180]\n[unused181]\n[unused182]\n[unused183]\n[unused184]\n[unused185]\n[unused186]\n[unused187]\n[unused188]\n[unused189]\n[unused190]\n[unused191]\n[unused192]\n[unused193]\n[unused194]\n[unused195]\n[unused196]\n[unused197]\n[unused198]\n[unused199]\n[unused200]\n[unused201]\n[unused202]\n[unused203]\n[unused204]\n[unused205]\n[unused206]\n[unused207]\n[unused208]\n[unused209]\n[unused210]\n[unused211]\n[unused212]\n[unused213]\n[unused214]\n[unused215]\n[unused216]\n[unused217]\n[unused218]\n[unused219]\n[unused220]\n[unused221]\n[unused222]\n[unused223]\n[unused224]\n[unused225]\n[unused226]\n[unused227]\n[unused228]\n[unused229]\n[unused230]\n[unused231]\n[unused232]\n[unused233]\n[unused234]\n[unused235]\n[unused236]\n[unused237]\n[unused238]\n[unused239]\n[unused240]\n[unused241]\n[unused242]\n[unused243]\n[unused244]\n[unused245]\n[unused246]\n[unused247]\n[unused248]\n[unused249]\n[unused250]\n[unused251]\n[unused252]\n[unused253]\n[unused254]\n[unused255]\n[unused256]\n[unused257]\n[unused258]\n[unused259]\n[unused260]\n[unused261]\n[unused262]\n[unused263]\n[unused264]\n[unused265]\n[unused266]\n[unused267]\n[unused268]\n[unused269]\n[unused270]\n[unused271]\n[unused272]\n[unused273]\n[unused274]\n[unused275]\n[unused276]\n[unused277]\n[unused278]\n[unused279]\n[unused280]\n[unused281]\n[unused282]\n[unused283]\n[unused284]\n[unused285]\n[unused286]\n[unused287]\n[unused288]\n[unused289]\n[unused290]\n[unused291]\n[unused292]\n[unused293]\n[unused294]\n[unused295]\n[unused296]\n[unused297]\n[unused298]\n[unused299]\n[unused300]\n[unused301]\n[unused302]\n[unused303]\n[unused304]\n[unused305]\n[unused306]\n[unused307]\n[unused308]\n[unused309]\n[unused310]\n[unused311]\n[unused312]\n[unused313]\n[unused314]\n[unused315]\n[unused316]\n[unused317]\n[unused318]\n[unused319]\n[unused320]\n[unused321]\n[unused322]\n[unused323]\n[unused324]\n[unused325]\n[unused326]\n[unused327]\n[unused328]\n[unused329]\n[unused330]\n[unused331]\n[unused332]\n[unused333]\n[unused334]\n[unused335]\n[unused336]\n[unused337]\n[unused338]\n[unused339]\n[unused340]\n[unused341]\n[unused342]\n[unused343]\n[unused344]\n[unused345]\n[unused346]\n[unused347]\n[unused348]\n[unused349]\n[unused350]\n[unused351]\n[unused352]\n[unused353]\n[unused354]\n[unused355]\n[unused356]\n[unused357]\n[unused358]\n[unused359]\n[unused360]\n[unused361]\n[unused362]\n[unused363]\n[unused364]\n[unused365]\n[unused366]\n[unused367]\n[unused368]\n[unused369]\n[unused370]\n[unused371]\n[unused372]\n[unused373]\n[unused374]\n[unused375]\n[unused376]\n[unused377]\n[unused378]\n[unused379]\n[unused380]\n[unused381]\n[unused382]\n[unused383]\n[unused384]\n[unused385]\n[unused386]\n[unused387]\n[unused388]\n[unused389]\n[unused390]\n[unused391]\n[unused392]\n[unused393]\n[unused394]\n[unused395]\n[unused396]\n[unused397]\n[unused398]\n[unused399]\n[unused400]\n[unused401]\n[unused402]\n[unused403]\n[unused404]\n[unused405]\n[unused406]\n[unused407]\n[unused408]\n[unused409]\n[unused410]\n[unused411]\n[unused412]\n[unused413]\n[unused414]\n[unused415]\n[unused416]\n[unused417]\n[unused418]\n[unused419]\n[unused420]\n[unused421]\n[unused422]\n[unused423]\n[unused424]\n[unused425]\n[unused426]\n[unused427]\n[unused428]\n[unused429]\n[unused430]\n[unused431]\n[unused432]\n[unused433]\n[unused434]\n[unused435]\n[unused436]\n[unused437]\n[unused438]\n[unused439]\n[unused440]\n[unused441]\n[unused442]\n[unused443]\n[unused444]\n[unused445]\n[unused446]\n[unused447]\n[unused448]\n[unused449]\n[unused450]\n[unused451]\n[unused452]\n[unused453]\n[unused454]\n[unused455]\n[unused456]\n[unused457]\n[unused458]\n[unused459]\n[unused460]\n[unused461]\n[unused462]\n[unused463]\n[unused464]\n[unused465]\n[unused466]\n[unused467]\n[unused468]\n[unused469]\n[unused470]\n[unused471]\n[unused472]\n[unused473]\n[unused474]\n[unused475]\n[unused476]\n[unused477]\n[unused478]\n[unused479]\n[unused480]\n[unused481]\n[unused482]\n[unused483]\n[unused484]\n[unused485]\n[unused486]\n[unused487]\n[unused488]\n[unused489]\n[unused490]\n[unused491]\n[unused492]\n[unused493]\n[unused494]\n[unused495]\n[unused496]\n[unused497]\n[unused498]\n[unused499]\n[unused500]\n[unused501]\n[unused502]\n[unused503]\n[unused504]\n[unused505]\n[unused506]\n[unused507]\n[unused508]\n[unused509]\n[unused510]\n[unused511]\n[unused512]\n[unused513]\n[unused514]\n[unused515]\n[unused516]\n[unused517]\n[unused518]\n[unused519]\n[unused520]\n[unused521]\n[unused522]\n[unused523]\n[unused524]\n[unused525]\n[unused526]\n[unused527]\n[unused528]\n[unused529]\n[unused530]\n[unused531]\n[unused532]\n[unused533]\n[unused534]\n[unused535]\n[unused536]\n[unused537]\n[unused538]\n[unused539]\n[unused540]\n[unused541]\n[unused542]\n[unused543]\n[unused544]\n[unused545]\n[unused546]\n[unused547]\n[unused548]\n[unused549]\n[unused550]\n[unused551]\n[unused552]\n[unused553]\n[unused554]\n[unused555]\n[unused556]\n[unused557]\n[unused558]\n[unused559]\n[unused560]\n[unused561]\n[unused562]\n[unused563]\n[unused564]\n[unused565]\n[unused566]\n[unused567]\n[unused568]\n[unused569]\n[unused570]\n[unused571]\n[unused572]\n[unused573]\n[unused574]\n[unused575]\n[unused576]\n[unused577]\n[unused578]\n[unused579]\n[unused580]\n[unused581]\n[unused582]\n[unused583]\n[unused584]\n[unused585]\n[unused586]\n[unused587]\n[unused588]\n[unused589]\n[unused590]\n[unused591]\n[unused592]\n[unused593]\n[unused594]\n[unused595]\n[unused596]\n[unused597]\n[unused598]\n[unused599]\n[unused600]\n[unused601]\n[unused602]\n[unused603]\n[unused604]\n[unused605]\n[unused606]\n[unused607]\n[unused608]\n[unused609]\n[unused610]\n[unused611]\n[unused612]\n[unused613]\n[unused614]\n[unused615]\n[unused616]\n[unused617]\n[unused618]\n[unused619]\n[unused620]\n[unused621]\n[unused622]\n[unused623]\n[unused624]\n[unused625]\n[unused626]\n[unused627]\n[unused628]\n[unused629]\n[unused630]\n[unused631]\n[unused632]\n[unused633]\n[unused634]\n[unused635]\n[unused636]\n[unused637]\n[unused638]\n[unused639]\n[unused640]\n[unused641]\n[unused642]\n[unused643]\n[unused644]\n[unused645]\n[unused646]\n[unused647]\n[unused648]\n[unused649]\n[unused650]\n[unused651]\n[unused652]\n[unused653]\n[unused654]\n[unused655]\n[unused656]\n[unused657]\n[unused658]\n[unused659]\n[unused660]\n[unused661]\n[unused662]\n[unused663]\n[unused664]\n[unused665]\n[unused666]\n[unused667]\n[unused668]\n[unused669]\n[unused670]\n[unused671]\n[unused672]\n[unused673]\n[unused674]\n[unused675]\n[unused676]\n[unused677]\n[unused678]\n[unused679]\n[unused680]\n[unused681]\n[unused682]\n[unused683]\n[unused684]\n[unused685]\n[unused686]\n[unused687]\n[unused688]\n[unused689]\n[unused690]\n[unused691]\n[unused692]\n[unused693]\n[unused694]\n[unused695]\n[unused696]\n[unused697]\n[unused698]\n[unused699]\n[unused700]\n[unused701]\n[unused702]\n[unused703]\n[unused704]\n[unused705]\n[unused706]\n[unused707]\n[unused708]\n[unused709]\n[unused710]\n[unused711]\n[unused712]\n[unused713]\n[unused714]\n[unused715]\n[unused716]\n[unused717]\n[unused718]\n[unused719]\n[unused720]\n[unused721]\n[unused722]\n[unused723]\n[unused724]\n[unused725]\n[unused726]\n[unused727]\n[unused728]\n[unused729]\n[unused730]\n[unused731]\n[unused732]\n[unused733]\n[unused734]\n[unused735]\n[unused736]\n[unused737]\n[unused738]\n[unused739]\n[unused740]\n[unused741]\n[unused742]\n[unused743]\n[unused744]\n[unused745]\n[unused746]\n[unused747]\n[unused748]\n[unused749]\n[unused750]\n[unused751]\n[unused752]\n[unused753]\n[unused754]\n[unused755]\n[unused756]\n[unused757]\n[unused758]\n[unused759]\n[unused760]\n[unused761]\n[unused762]\n[unused763]\n[unused764]\n[unused765]\n[unused766]\n[unused767]\n[unused768]\n[unused769]\n[unused770]\n[unused771]\n[unused772]\n[unused773]\n[unused774]\n[unused775]\n[unused776]\n[unused777]\n[unused778]\n[unused779]\n[unused780]\n[unused781]\n[unused782]\n[unused783]\n[unused784]\n[unused785]\n[unused786]\n[unused787]\n[unused788]\n[unused789]\n[unused790]\n[unused791]\n[unused792]\n[unused793]\n[unused794]\n[unused795]\n[unused796]\n[unused797]\n[unused798]\n[unused799]\n[unused800]\n[unused801]\n[unused802]\n[unused803]\n[unused804]\n[unused805]\n[unused806]\n[unused807]\n[unused808]\n[unused809]\n[unused810]\n[unused811]\n[unused812]\n[unused813]\n[unused814]\n[unused815]\n[unused816]\n[unused817]\n[unused818]\n[unused819]\n[unused820]\n[unused821]\n[unused822]\n[unused823]\n[unused824]\n[unused825]\n[unused826]\n[unused827]\n[unused828]\n[unused829]\n[unused830]\n[unused831]\n[unused832]\n[unused833]\n[unused834]\n[unused835]\n[unused836]\n[unused837]\n[unused838]\n[unused839]\n[unused840]\n[unused841]\n[unused842]\n[unused843]\n[unused844]\n[unused845]\n[unused846]\n[unused847]\n[unused848]\n[unused849]\n[unused850]\n[unused851]\n[unused852]\n[unused853]\n[unused854]\n[unused855]\n[unused856]\n[unused857]\n[unused858]\n[unused859]\n[unused860]\n[unused861]\n[unused862]\n[unused863]\n[unused864]\n[unused865]\n[unused866]\n[unused867]\n[unused868]\n[unused869]\n[unused870]\n[unused871]\n[unused872]\n[unused873]\n[unused874]\n[unused875]\n[unused876]\n[unused877]\n[unused878]\n[unused879]\n[unused880]\n[unused881]\n[unused882]\n[unused883]\n[unused884]\n[unused885]\n[unused886]\n[unused887]\n[unused888]\n[unused889]\n[unused890]\n[unused891]\n[unused892]\n[unused893]\n[unused894]\n[unused895]\n[unused896]\n[unused897]\n[unused898]\n[unused899]\n[unused900]\n[unused901]\n[unused902]\n[unused903]\n[unused904]\n[unused905]\n[unused906]\n[unused907]\n[unused908]\n[unused909]\n[unused910]\n[unused911]\n[unused912]\n[unused913]\n[unused914]\n[unused915]\n[unused916]\n[unused917]\n[unused918]\n[unused919]\n[unused920]\n[unused921]\n[unused922]\n[unused923]\n[unused924]\n[unused925]\n[unused926]\n[unused927]\n[unused928]\n[unused929]\n[unused930]\n[unused931]\n[unused932]\n[unused933]\n[unused934]\n[unused935]\n[unused936]\n[unused937]\n[unused938]\n[unused939]\n[unused940]\n[unused941]\n[unused942]\n[unused943]\n[unused944]\n[unused945]\n[unused946]\n[unused947]\n[unused948]\n[unused949]\n[unused950]\n[unused951]\n[unused952]\n[unused953]\n[unused954]\n[unused955]\n[unused956]\n[unused957]\n[unused958]\n[unused959]\n[unused960]\n[unused961]\n[unused962]\n[unused963]\n[unused964]\n[unused965]\n[unused966]\n[unused967]\n[unused968]\n[unused969]\n[unused970]\n[unused971]\n[unused972]\n[unused973]\n[unused974]\n[unused975]\n[unused976]\n[unused977]\n[unused978]\n[unused979]\n[unused980]\n[unused981]\n[unused982]\n[unused983]\n[unused984]\n[unused985]\n[unused986]\n[unused987]\n[unused988]\n[unused989]\n[unused990]\n[unused991]\n[unused992]\n[unused993]\n!\n\"\n#\n$\n%\n&\n'\n(\n)\n*\n+\n,\n-\n.\n/\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n:\n;\n<\n=\n>\n?\n@\n[\n\\\n]\n^\n_\n`\na\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nm\nn\no\np\nq\nr\ns\nt\nu\nv\nw\nx\ny\nz\n{\n|\n}\n~\n¡\n¢\n£\n¤\n¥\n¦\n§\n¨\n©\nª\n«\n¬\n®\n°\n±\n²\n³\n´\nµ\n¶\n·\n¹\nº\n»\n¼\n½\n¾\n¿\n×\nß\næ\nð\n÷\nø\nþ\nđ\nħ\nı\nł\nŋ\nœ\nƒ\nɐ\nɑ\nɒ\nɔ\nɕ\nə\nɛ\nɡ\nɣ\nɨ\nɪ\nɫ\nɬ\nɯ\nɲ\nɴ\nɹ\nɾ\nʀ\nʁ\nʂ\nʃ\nʉ\nʊ\nʋ\nʌ\nʎ\nʐ\nʑ\nʒ\nʔ\nʰ\nʲ\nʳ\nʷ\nʸ\nʻ\nʼ\nʾ\nʿ\nˈ\nː\nˡ\nˢ\nˣ\nˤ\nα\nβ\nγ\nδ\nε\nζ\nη\nθ\nι\nκ\nλ\nμ\nν\nξ\nο\nπ\nρ\nς\nσ\nτ\nυ\nφ\nχ\nψ\nω\nа\nб\nв\nг\nд\nе\nж\nз\nи\nк\nл\nм\nн\nо\nп\nр\nс\nт\nу\nф\nх\nц\nч\nш\nщ\nъ\nы\nь\nэ\nю\nя\nђ\nє\nі\nј\nљ\nњ\nћ\nӏ\nա\nբ\nգ\nդ\nե\nթ\nի\nլ\nկ\nհ\nմ\nյ\nն\nո\nպ\nս\nվ\nտ\nր\nւ\nք\n־\nא\nב\nג\nד\nה\nו\nז\nח\nט\nי\nך\nכ\nל\nם\nמ\nן\nנ\nס\nע\nף\nפ\nץ\nצ\nק\nר\nש\nת\n،\nء\nا\nب\nة\nت\nث\nج\nح\nخ\nد\nذ\nر\nز\nس\nش\nص\nض\nط\nظ\nع\nغ\nـ\nف\nق\nك\nل\nم\nن\nه\nو\nى\nي\nٹ\nپ\nچ\nک\nگ\nں\nھ\nہ\nی\nے\nअ\nआ\nउ\nए\nक\nख\nग\nच\nज\nट\nड\nण\nत\nथ\nद\nध\nन\nप\nब\nभ\nम\nय\nर\nल\nव\nश\nष\nस\nह\nा\nि\nी\nो\n।\n॥\nং\nঅ\nআ\nই\nউ\nএ\nও\nক\nখ\nগ\nচ\nছ\nজ\nট\nড\nণ\nত\nথ\nদ\nধ\nন\nপ\nব\nভ\nম\nয\nর\nল\nশ\nষ\nস\nহ\nা\nি\nী\nে\nக\nச\nட\nத\nந\nன\nப\nம\nய\nர\nல\nள\nவ\nா\nி\nு\nே\nை\nನ\nರ\nಾ\nක\nය\nර\nල\nව\nා\nก\nง\nต\nท\nน\nพ\nม\nย\nร\nล\nว\nส\nอ\nา\nเ\n་\n།\nག\nང\nད\nན\nཔ\nབ\nམ\nའ\nར\nལ\nས\nမ\nა\nბ\nგ\nდ\nე\nვ\nთ\nი\nკ\nლ\nმ\nნ\nო\nრ\nს\nტ\nუ\nᄀ\nᄂ\nᄃ\nᄅ\nᄆ\nᄇ\nᄉ\nᄊ\nᄋ\nᄌ\nᄎ\nᄏ\nᄐ\nᄑ\nᄒ\nᅡ\nᅢ\nᅥ\nᅦ\nᅧ\nᅩ\nᅪ\nᅭ\nᅮ\nᅯ\nᅲ\nᅳ\nᅴ\nᅵ\nᆨ\nᆫ\nᆯ\nᆷ\nᆸ\nᆼ\nᴬ\nᴮ\nᴰ\nᴵ\nᴺ\nᵀ\nᵃ\nᵇ\nᵈ\nᵉ\nᵍ\nᵏ\nᵐ\nᵒ\nᵖ\nᵗ\nᵘ\nᵢ\nᵣ\nᵤ\nᵥ\nᶜ\nᶠ\n‐\n‑\n‒\n–\n—\n―\n‖\n‘\n’\n‚\n“\n”\n„\n†\n‡\n•\n…\n‰\n′\n″\n›\n‿\n⁄\n⁰\nⁱ\n⁴\n⁵\n⁶\n⁷\n⁸\n⁹\n⁺\n⁻\nⁿ\n₀\n₁\n₂\n₃\n₄\n₅\n₆\n₇\n₈\n₉\n₊\n₍\n₎\nₐ\nₑ\nₒ\nₓ\nₕ\nₖ\nₗ\nₘ\nₙ\nₚ\nₛ\nₜ\n₤\n₩\n€\n₱\n₹\nℓ\n№\nℝ\n™\n⅓\n⅔\n←\n↑\n→\n↓\n↔\n↦\n⇄\n⇌\n⇒\n∂\n∅\n∆\n∇\n∈\n−\n∗\n∘\n√\n∞\n∧\n∨\n∩\n∪\n≈\n≡\n≤\n≥\n⊂\n⊆\n⊕\n⊗\n⋅\n─\n│\n■\n▪\n●\n★\n☆\n☉\n♠\n♣\n♥\n♦\n♭\n♯\n⟨\n⟩\nⱼ\n⺩\n⺼\n⽥\n、\n。\n〈\n〉\n《\n》\n「\n」\n『\n』\n〜\nあ\nい\nう\nえ\nお\nか\nき\nく\nけ\nこ\nさ\nし\nす\nせ\nそ\nた\nち\nっ\nつ\nて\nと\nな\nに\nぬ\nね\nの\nは\nひ\nふ\nへ\nほ\nま\nみ\nむ\nめ\nも\nや\nゆ\nよ\nら\nり\nる\nれ\nろ\nを\nん\nァ\nア\nィ\nイ\nウ\nェ\nエ\nオ\nカ\nキ\nク\nケ\nコ\nサ\nシ\nス\nセ\nタ\nチ\nッ\nツ\nテ\nト\nナ\nニ\nノ\nハ\nヒ\nフ\nヘ\nホ\nマ\nミ\nム\nメ\nモ\nャ\nュ\nョ\nラ\nリ\nル\nレ\nロ\nワ\nン\n・\nー\n一\n三\n上\n下\n不\n世\n中\n主\n久\n之\n也\n事\n二\n五\n井\n京\n人\n亻\n仁\n介\n代\n仮\n伊\n会\n佐\n侍\n保\n信\n健\n元\n光\n八\n公\n内\n出\n分\n前\n劉\n力\n加\n勝\n北\n区\n十\n千\n南\n博\n原\n口\n古\n史\n司\n合\n吉\n同\n名\n和\n囗\n四\n国\n國\n土\n地\n坂\n城\n堂\n場\n士\n夏\n外\n大\n天\n太\n夫\n奈\n女\n子\n学\n宀\n宇\n安\n宗\n定\n宣\n宮\n家\n宿\n寺\n將\n小\n尚\n山\n岡\n島\n崎\n川\n州\n巿\n帝\n平\n年\n幸\n广\n弘\n張\n彳\n後\n御\n德\n心\n忄\n志\n忠\n愛\n成\n我\n戦\n戸\n手\n扌\n政\n文\n新\n方\n日\n明\n星\n春\n昭\n智\n曲\n書\n月\n有\n朝\n木\n本\n李\n村\n東\n松\n林\n森\n楊\n樹\n橋\n歌\n止\n正\n武\n比\n氏\n民\n水\n氵\n氷\n永\n江\n沢\n河\n治\n法\n海\n清\n漢\n瀬\n火\n版\n犬\n王\n生\n田\n男\n疒\n発\n白\n的\n皇\n目\n相\n省\n真\n石\n示\n社\n神\n福\n禾\n秀\n秋\n空\n立\n章\n竹\n糹\n美\n義\n耳\n良\n艹\n花\n英\n華\n葉\n藤\n行\n街\n西\n見\n訁\n語\n谷\n貝\n貴\n車\n軍\n辶\n道\n郎\n郡\n部\n都\n里\n野\n金\n鈴\n镇\n長\n門\n間\n阝\n阿\n陳\n陽\n雄\n青\n面\n風\n食\n香\n馬\n高\n龍\n龸\nfi\nfl\n！\n（\n）\n，\n－\n．\n／\n：\n？\n～\nthe\nof\nand\nin\nto\nwas\nhe\nis\nas\nfor\non\nwith\nthat\nit\nhis\nby\nat\nfrom\nher\n##s\nshe\nyou\nhad\nan\nwere\nbut\nbe\nthis\nare\nnot\nmy\nthey\none\nwhich\nor\nhave\nhim\nme\nfirst\nall\nalso\ntheir\nhas\nup\nwho\nout\nbeen\nwhen\nafter\nthere\ninto\nnew\ntwo\nits\n##a\ntime\nwould\nno\nwhat\nabout\nsaid\nwe\nover\nthen\nother\nso\nmore\n##e\ncan\nif\nlike\nback\nthem\nonly\nsome\ncould\n##i\nwhere\njust\n##ing\nduring\nbefore\n##n\ndo\n##o\nmade\nschool\nthrough\nthan\nnow\nyears\nmost\nworld\nmay\nbetween\ndown\nwell\nthree\n##d\nyear\nwhile\nwill\n##ed\n##r\n##y\nlater\n##t\ncity\nunder\naround\ndid\nsuch\nbeing\nused\nstate\npeople\npart\nknow\nagainst\nyour\nmany\nsecond\nuniversity\nboth\nnational\n##er\nthese\ndon\nknown\noff\nway\nuntil\nre\nhow\neven\nget\nhead\n...\ndidn\n##ly\nteam\namerican\nbecause\nde\n##l\nborn\nunited\nfilm\nsince\nstill\nlong\nwork\nsouth\nus\nbecame\nany\nhigh\nagain\nday\nfamily\nsee\nright\nman\neyes\nhouse\nseason\nwar\nstates\nincluding\ntook\nlife\nnorth\nsame\neach\ncalled\nname\nmuch\nplace\nhowever\ngo\nfour\ngroup\nanother\nfound\nwon\narea\nhere\ngoing\n10\naway\nseries\nleft\nhome\nmusic\nbest\nmake\nhand\nnumber\ncompany\nseveral\nnever\nlast\njohn\n000\nvery\nalbum\ntake\nend\ngood\ntoo\nfollowing\nreleased\ngame\nplayed\nlittle\nbegan\ndistrict\n##m\nold\nwant\nthose\nside\nheld\nown\nearly\ncounty\nll\nleague\nuse\nwest\n##u\nface\nthink\n##es\n2010\ngovernment\n##h\nmarch\ncame\nsmall\ngeneral\ntown\njune\n##on\nline\nbased\nsomething\n##k\nseptember\nthought\nlooked\nalong\ninternational\n2011\nair\njuly\nclub\nwent\njanuary\noctober\nour\naugust\napril\nyork\n12\nfew\n2012\n2008\neast\nshow\nmember\ncollege\n2009\nfather\npublic\n##us\ncome\nmen\nfive\nset\nstation\nchurch\n##c\nnext\nformer\nnovember\nroom\nparty\nlocated\ndecember\n2013\nage\ngot\n2007\n##g\nsystem\nlet\nlove\n2006\nthough\nevery\n2014\nlook\nsong\nwater\ncentury\nwithout\nbody\nblack\nnight\nwithin\ngreat\nwomen\nsingle\nve\nbuilding\nlarge\npopulation\nriver\nnamed\nband\nwhite\nstarted\n##an\nonce\n15\n20\nshould\n18\n2015\nservice\ntop\nbuilt\nbritish\nopen\ndeath\nking\nmoved\nlocal\ntimes\nchildren\nfebruary\nbook\nwhy\n11\ndoor\nneed\npresident\norder\nfinal\nroad\nwasn\nalthough\ndue\nmajor\ndied\nvillage\nthird\nknew\n2016\nasked\nturned\nst\nwanted\nsay\n##p\ntogether\nreceived\nmain\nson\nserved\ndifferent\n##en\nbehind\nhimself\nfelt\nmembers\npower\nfootball\nlaw\nvoice\nplay\n##in\nnear\npark\nhistory\n30\nhaving\n2005\n16\n##man\nsaw\nmother\n##al\narmy\npoint\nfront\nhelp\nenglish\nstreet\nart\nlate\nhands\ngames\naward\n##ia\nyoung\n14\nput\npublished\ncountry\ndivision\nacross\ntold\n13\noften\never\nfrench\nlondon\ncenter\nsix\nred\n2017\nled\ndays\ninclude\nlight\n25\nfind\ntell\namong\nspecies\nreally\naccording\ncentral\nhalf\n2004\nform\noriginal\ngave\noffice\nmaking\nenough\nlost\nfull\nopened\nmust\nincluded\nlive\ngiven\ngerman\nplayer\nrun\nbusiness\nwoman\ncommunity\ncup\nmight\nmillion\nland\n2000\ncourt\ndevelopment\n17\nshort\nround\nii\nkm\nseen\nclass\nstory\nalways\nbecome\nsure\nresearch\nalmost\ndirector\ncouncil\nla\n##2\ncareer\nthings\nusing\nisland\n##z\ncouldn\ncar\n##is\n24\nclose\nforce\n##1\nbetter\nfree\nsupport\ncontrol\nfield\nstudents\n2003\neducation\nmarried\n##b\nnothing\nworked\nothers\nrecord\nbig\ninside\nlevel\nanything\ncontinued\ngive\njames\n##3\nmilitary\nestablished\nnon\nreturned\nfeel\ndoes\ntitle\nwritten\nthing\nfeet\nwilliam\nfar\nco\nassociation\nhard\nalready\n2002\n##ra\nchampionship\nhuman\nwestern\n100\n##na\ndepartment\nhall\nrole\nvarious\nproduction\n21\n19\nheart\n2001\nliving\nfire\nversion\n##ers\n##f\ntelevision\nroyal\n##4\nproduced\nworking\nact\ncase\nsociety\nregion\npresent\nradio\nperiod\nlooking\nleast\ntotal\nkeep\nengland\nwife\nprogram\nper\nbrother\nmind\nspecial\n22\n##le\nam\nworks\nsoon\n##6\npolitical\ngeorge\nservices\ntaken\ncreated\n##7\nfurther\nable\nreached\ndavid\nunion\njoined\nupon\ndone\nimportant\nsocial\ninformation\neither\n##ic\n##x\nappeared\nposition\nground\nlead\nrock\ndark\nelection\n23\nboard\nfrance\nhair\ncourse\narms\nsite\npolice\ngirl\ninstead\nreal\nsound\n##v\nwords\nmoment\n##te\nsomeone\n##8\nsummer\nproject\nannounced\nsan\nless\nwrote\npast\nfollowed\n##5\nblue\nfounded\nal\nfinally\nindia\ntaking\nrecords\namerica\n##ne\n1999\ndesign\nconsidered\nnorthern\ngod\nstop\nbattle\ntoward\neuropean\noutside\ndescribed\ntrack\ntoday\nplaying\nlanguage\n28\ncall\n26\nheard\nprofessional\nlow\naustralia\nmiles\ncalifornia\nwin\nyet\ngreen\n##ie\ntrying\nblood\n##ton\nsouthern\nscience\nmaybe\neverything\nmatch\nsquare\n27\nmouth\nvideo\nrace\nrecorded\nleave\nabove\n##9\ndaughter\npoints\nspace\n1998\nmuseum\nchange\nmiddle\ncommon\n##0\nmove\ntv\npost\n##ta\nlake\nseven\ntried\nelected\nclosed\nten\npaul\nminister\n##th\nmonths\nstart\nchief\nreturn\ncanada\nperson\nsea\nrelease\nsimilar\nmodern\nbrought\nrest\nhit\nformed\nmr\n##la\n1997\nfloor\nevent\ndoing\nthomas\n1996\nrobert\ncare\nkilled\ntraining\nstar\nweek\nneeded\nturn\nfinished\nrailway\nrather\nnews\nhealth\nsent\nexample\nran\nterm\nmichael\ncoming\ncurrently\nyes\nforces\ndespite\ngold\nareas\n50\nstage\nfact\n29\ndead\nsays\npopular\n2018\noriginally\ngermany\nprobably\ndeveloped\nresult\npulled\nfriend\nstood\nmoney\nrunning\nmi\nsigned\nword\nsongs\nchild\neventually\nmet\ntour\naverage\nteams\nminutes\nfestival\ncurrent\ndeep\nkind\n1995\ndecided\nusually\neastern\nseemed\n##ness\nepisode\nbed\nadded\ntable\nindian\nprivate\ncharles\nroute\navailable\nidea\nthroughout\ncentre\naddition\nappointed\nstyle\n1994\nbooks\neight\nconstruction\npress\nmean\nwall\nfriends\nremained\nschools\nstudy\n##ch\n##um\ninstitute\noh\nchinese\nsometimes\nevents\npossible\n1992\naustralian\ntype\nbrown\nforward\ntalk\nprocess\nfood\ndebut\nseat\nperformance\ncommittee\nfeatures\ncharacter\narts\nherself\nelse\nlot\nstrong\nrussian\nrange\nhours\npeter\narm\n##da\nmorning\ndr\nsold\n##ry\nquickly\ndirected\n1993\nguitar\nchina\n##w\n31\nlist\n##ma\nperformed\nmedia\nuk\nplayers\nsmile\n##rs\nmyself\n40\nplaced\ncoach\nprovince\ntowards\nwouldn\nleading\nwhole\nboy\nofficial\ndesigned\ngrand\ncensus\n##el\neurope\nattack\njapanese\nhenry\n1991\n##re\n##os\ncross\ngetting\nalone\naction\nlower\nnetwork\nwide\nwashington\njapan\n1990\nhospital\nbelieve\nchanged\nsister\n##ar\nhold\ngone\nsir\nhadn\nship\n##ka\nstudies\nacademy\nshot\nrights\nbelow\nbase\nbad\ninvolved\nkept\nlargest\n##ist\nbank\nfuture\nespecially\nbeginning\nmark\nmovement\nsection\nfemale\nmagazine\nplan\nprofessor\nlord\nlonger\n##ian\nsat\nwalked\nhill\nactually\ncivil\nenergy\nmodel\nfamilies\nsize\nthus\naircraft\ncompleted\nincludes\ndata\ncaptain\n##or\nfight\nvocals\nfeatured\nrichard\nbridge\nfourth\n1989\nofficer\nstone\nhear\n##ism\nmeans\nmedical\ngroups\nmanagement\nself\nlips\ncompetition\nentire\nlived\ntechnology\nleaving\nfederal\ntournament\nbit\npassed\nhot\nindependent\nawards\nkingdom\nmary\nspent\nfine\ndoesn\nreported\n##ling\njack\nfall\nraised\nitself\nstay\ntrue\nstudio\n1988\nsports\nreplaced\nparis\nsystems\nsaint\nleader\ntheatre\nwhose\nmarket\ncapital\nparents\nspanish\ncanadian\nearth\n##ity\ncut\ndegree\nwriting\nbay\nchristian\nawarded\nnatural\nhigher\nbill\n##as\ncoast\nprovided\nprevious\nsenior\nft\nvalley\norganization\nstopped\nonto\ncountries\nparts\nconference\nqueen\nsecurity\ninterest\nsaying\nallowed\nmaster\nearlier\nphone\nmatter\nsmith\nwinning\ntry\nhappened\nmoving\ncampaign\nlos\n##ley\nbreath\nnearly\nmid\n1987\ncertain\ngirls\ndate\nitalian\nafrican\nstanding\nfell\nartist\n##ted\nshows\ndeal\nmine\nindustry\n1986\n##ng\neveryone\nrepublic\nprovide\ncollection\nlibrary\nstudent\n##ville\nprimary\nowned\nolder\nvia\nheavy\n1st\nmakes\n##able\nattention\nanyone\nafrica\n##ri\nstated\nlength\nended\nfingers\ncommand\nstaff\nskin\nforeign\nopening\ngovernor\nokay\nmedal\nkill\nsun\ncover\njob\n1985\nintroduced\nchest\nhell\nfeeling\n##ies\nsuccess\nmeet\nreason\nstandard\nmeeting\nnovel\n1984\ntrade\nsource\nbuildings\n##land\nrose\nguy\ngoal\n##ur\nchapter\nnative\nhusband\npreviously\nunit\nlimited\nentered\nweeks\nproducer\noperations\nmountain\ntakes\ncovered\nforced\nrelated\nroman\ncomplete\nsuccessful\nkey\ntexas\ncold\n##ya\nchannel\n1980\ntraditional\nfilms\ndance\nclear\napproximately\n500\nnine\nvan\nprince\nquestion\nactive\ntracks\nireland\nregional\nsilver\nauthor\npersonal\nsense\noperation\n##ine\neconomic\n1983\nholding\ntwenty\nisbn\nadditional\nspeed\nhour\nedition\nregular\nhistoric\nplaces\nwhom\nshook\nmovie\nkm²\nsecretary\nprior\nreport\nchicago\nread\nfoundation\nview\nengine\nscored\n1982\nunits\nask\nairport\nproperty\nready\nimmediately\nlady\nmonth\nlisted\ncontract\n##de\nmanager\nthemselves\nlines\n##ki\nnavy\nwriter\nmeant\n##ts\nruns\n##ro\npractice\nchampionships\nsinger\nglass\ncommission\nrequired\nforest\nstarting\nculture\ngenerally\ngiving\naccess\nattended\ntest\ncouple\nstand\ncatholic\nmartin\ncaught\nexecutive\n##less\neye\n##ey\nthinking\nchair\nquite\nshoulder\n1979\nhope\ndecision\nplays\ndefeated\nmunicipality\nwhether\nstructure\noffered\nslowly\npain\nice\ndirection\n##ion\npaper\nmission\n1981\nmostly\n200\nnoted\nindividual\nmanaged\nnature\nlives\nplant\n##ha\nhelped\nexcept\nstudied\ncomputer\nfigure\nrelationship\nissue\nsignificant\nloss\ndie\nsmiled\ngun\nago\nhighest\n1972\n##am\nmale\nbring\ngoals\nmexico\nproblem\ndistance\ncommercial\ncompletely\nlocation\nannual\nfamous\ndrive\n1976\nneck\n1978\nsurface\ncaused\nitaly\nunderstand\ngreek\nhighway\nwrong\nhotel\ncomes\nappearance\njoseph\ndouble\nissues\nmusical\ncompanies\ncastle\nincome\nreview\nassembly\nbass\ninitially\nparliament\nartists\nexperience\n1974\nparticular\nwalk\nfoot\nengineering\ntalking\nwindow\ndropped\n##ter\nmiss\nbaby\nboys\nbreak\n1975\nstars\nedge\nremember\npolicy\ncarried\ntrain\nstadium\nbar\nsex\nangeles\nevidence\n##ge\nbecoming\nassistant\nsoviet\n1977\nupper\nstep\nwing\n1970\nyouth\nfinancial\nreach\n##ll\nactor\nnumerous\n##se\n##st\nnodded\narrived\n##ation\nminute\n##nt\nbelieved\nsorry\ncomplex\nbeautiful\nvictory\nassociated\ntemple\n1968\n1973\nchance\nperhaps\nmetal\n##son\n1945\nbishop\n##et\nlee\nlaunched\nparticularly\ntree\nle\nretired\nsubject\nprize\ncontains\nyeah\ntheory\nempire\n##ce\nsuddenly\nwaiting\ntrust\nrecording\n##to\nhappy\nterms\ncamp\nchampion\n1971\nreligious\npass\nzealand\nnames\n2nd\nport\nancient\ntom\ncorner\nrepresented\nwatch\nlegal\nanti\njustice\ncause\nwatched\nbrothers\n45\nmaterial\nchanges\nsimply\nresponse\nlouis\nfast\n##ting\nanswer\n60\nhistorical\n1969\nstories\nstraight\ncreate\nfeature\nincreased\nrate\nadministration\nvirginia\nel\nactivities\ncultural\noverall\nwinner\nprograms\nbasketball\nlegs\nguard\nbeyond\ncast\ndoctor\nmm\nflight\nresults\nremains\ncost\neffect\nwinter\n##ble\nlarger\nislands\nproblems\nchairman\ngrew\ncommander\nisn\n1967\npay\nfailed\nselected\nhurt\nfort\nbox\nregiment\nmajority\njournal\n35\nedward\nplans\n##ke\n##ni\nshown\npretty\nirish\ncharacters\ndirectly\nscene\nlikely\noperated\nallow\nspring\n##j\njunior\nmatches\nlooks\nmike\nhouses\nfellow\n##tion\nbeach\nmarriage\n##ham\n##ive\nrules\noil\n65\nflorida\nexpected\nnearby\ncongress\nsam\npeace\nrecent\niii\nwait\nsubsequently\ncell\n##do\nvariety\nserving\nagreed\nplease\npoor\njoe\npacific\nattempt\nwood\ndemocratic\npiece\nprime\n##ca\nrural\nmile\ntouch\nappears\ntownship\n1964\n1966\nsoldiers\n##men\n##ized\n1965\npennsylvania\ncloser\nfighting\nclaimed\nscore\njones\nphysical\neditor\n##ous\nfilled\ngenus\nspecific\nsitting\nsuper\nmom\n##va\ntherefore\nsupported\nstatus\nfear\ncases\nstore\nmeaning\nwales\nminor\nspain\ntower\nfocus\nvice\nfrank\nfollow\nparish\nseparate\ngolden\nhorse\nfifth\nremaining\nbranch\n32\npresented\nstared\n##id\nuses\nsecret\nforms\n##co\nbaseball\nexactly\n##ck\nchoice\nnote\ndiscovered\ntravel\ncomposed\ntruth\nrussia\nball\ncolor\nkiss\ndad\nwind\ncontinue\nring\nreferred\nnumbers\ndigital\ngreater\n##ns\nmetres\nslightly\ndirect\nincrease\n1960\nresponsible\ncrew\nrule\ntrees\ntroops\n##no\nbroke\ngoes\nindividuals\nhundred\nweight\ncreek\nsleep\nmemory\ndefense\nprovides\nordered\ncode\nvalue\njewish\nwindows\n1944\nsafe\njudge\nwhatever\ncorps\nrealized\ngrowing\npre\n##ga\ncities\nalexander\ngaze\nlies\nspread\nscott\nletter\nshowed\nsituation\nmayor\ntransport\nwatching\nworkers\nextended\n##li\nexpression\nnormal\n##ment\nchart\nmultiple\nborder\n##ba\nhost\n##ner\ndaily\nmrs\nwalls\npiano\n##ko\nheat\ncannot\n##ate\nearned\nproducts\ndrama\nera\nauthority\nseasons\njoin\ngrade\n##io\nsign\ndifficult\nmachine\n1963\nterritory\nmainly\n##wood\nstations\nsquadron\n1962\nstepped\niron\n19th\n##led\nserve\nappear\nsky\nspeak\nbroken\ncharge\nknowledge\nkilometres\nremoved\nships\narticle\ncampus\nsimple\n##ty\npushed\nbritain\n##ve\nleaves\nrecently\ncd\nsoft\nboston\nlatter\neasy\nacquired\npoland\n##sa\nquality\nofficers\npresence\nplanned\nnations\nmass\nbroadcast\njean\nshare\nimage\ninfluence\nwild\noffer\nemperor\nelectric\nreading\nheaded\nability\npromoted\nyellow\nministry\n1942\nthroat\nsmaller\npolitician\n##by\nlatin\nspoke\ncars\nwilliams\nmales\nlack\npop\n80\n##ier\nacting\nseeing\nconsists\n##ti\nestate\n1961\npressure\njohnson\nnewspaper\njr\nchris\nolympics\nonline\nconditions\nbeat\nelements\nwalking\nvote\n##field\nneeds\ncarolina\ntext\nfeaturing\nglobal\nblock\nshirt\nlevels\nfrancisco\npurpose\nfemales\net\ndutch\nduke\nahead\ngas\ntwice\nsafety\nserious\nturning\nhighly\nlieutenant\nfirm\nmaria\namount\nmixed\ndaniel\nproposed\nperfect\nagreement\naffairs\n3rd\nseconds\ncontemporary\npaid\n1943\nprison\nsave\nkitchen\nlabel\nadministrative\nintended\nconstructed\nacademic\nnice\nteacher\nraces\n1956\nformerly\ncorporation\nben\nnation\nissued\nshut\n1958\ndrums\nhousing\nvictoria\nseems\nopera\n1959\ngraduated\nfunction\nvon\nmentioned\npicked\nbuild\nrecognized\nshortly\nprotection\npicture\nnotable\nexchange\nelections\n1980s\nloved\npercent\nracing\nfish\nelizabeth\ngarden\nvolume\nhockey\n1941\nbeside\nsettled\n##ford\n1940\ncompeted\nreplied\ndrew\n1948\nactress\nmarine\nscotland\nsteel\nglanced\nfarm\nsteve\n1957\nrisk\ntonight\npositive\nmagic\nsingles\neffects\ngray\nscreen\ndog\n##ja\nresidents\nbus\nsides\nnone\nsecondary\nliterature\npolish\ndestroyed\nflying\nfounder\nhouseholds\n1939\nlay\nreserve\nusa\ngallery\n##ler\n1946\nindustrial\nyounger\napproach\nappearances\nurban\nones\n1950\nfinish\navenue\npowerful\nfully\ngrowth\npage\nhonor\njersey\nprojects\nadvanced\nrevealed\nbasic\n90\ninfantry\npair\nequipment\nvisit\n33\nevening\nsearch\ngrant\neffort\nsolo\ntreatment\nburied\nrepublican\nprimarily\nbottom\nowner\n1970s\nisrael\ngives\njim\ndream\nbob\nremain\nspot\n70\nnotes\nproduce\nchampions\ncontact\ned\nsoul\naccepted\nways\ndel\n##ally\nlosing\nsplit\nprice\ncapacity\nbasis\ntrial\nquestions\n##ina\n1955\n20th\nguess\nofficially\nmemorial\nnaval\ninitial\n##ization\nwhispered\nmedian\nengineer\n##ful\nsydney\n##go\ncolumbia\nstrength\n300\n1952\ntears\nsenate\n00\ncard\nasian\nagent\n1947\nsoftware\n44\ndraw\nwarm\nsupposed\ncom\npro\n##il\ntransferred\nleaned\n##at\ncandidate\nescape\nmountains\nasia\npotential\nactivity\nentertainment\nseem\ntraffic\njackson\nmurder\n36\nslow\nproduct\norchestra\nhaven\nagency\nbbc\ntaught\nwebsite\ncomedy\nunable\nstorm\nplanning\nalbums\nrugby\nenvironment\nscientific\ngrabbed\nprotect\n##hi\nboat\ntypically\n1954\n1953\ndamage\nprincipal\ndivided\ndedicated\nmount\nohio\n##berg\npick\nfought\ndriver\n##der\nempty\nshoulders\nsort\nthank\nberlin\nprominent\naccount\nfreedom\nnecessary\nefforts\nalex\nheadquarters\nfollows\nalongside\ndes\nsimon\nandrew\nsuggested\noperating\nlearning\nsteps\n1949\nsweet\ntechnical\nbegin\neasily\n34\nteeth\nspeaking\nsettlement\nscale\n##sh\nrenamed\nray\nmax\nenemy\nsemi\njoint\ncompared\n##rd\nscottish\nleadership\nanalysis\noffers\ngeorgia\npieces\ncaptured\nanimal\ndeputy\nguest\norganized\n##lin\ntony\ncombined\nmethod\nchallenge\n1960s\nhuge\nwants\nbattalion\nsons\nrise\ncrime\ntypes\nfacilities\ntelling\npath\n1951\nplatform\nsit\n1990s\n##lo\ntells\nassigned\nrich\npull\n##ot\ncommonly\nalive\n##za\nletters\nconcept\nconducted\nwearing\nhappen\nbought\nbecomes\nholy\ngets\nocean\ndefeat\nlanguages\npurchased\ncoffee\noccurred\ntitled\n##q\ndeclared\napplied\nsciences\nconcert\nsounds\njazz\nbrain\n##me\npainting\nfleet\ntax\nnick\n##ius\nmichigan\ncount\nanimals\nleaders\nepisodes\n##line\ncontent\n##den\nbirth\n##it\nclubs\n64\npalace\ncritical\nrefused\nfair\nleg\nlaughed\nreturning\nsurrounding\nparticipated\nformation\nlifted\npointed\nconnected\nrome\nmedicine\nlaid\ntaylor\nsanta\npowers\nadam\ntall\nshared\nfocused\nknowing\nyards\nentrance\nfalls\n##wa\ncalling\n##ad\nsources\nchosen\nbeneath\nresources\nyard\n##ite\nnominated\nsilence\nzone\ndefined\n##que\ngained\nthirty\n38\nbodies\nmoon\n##ard\nadopted\nchristmas\nwidely\nregister\napart\niran\npremier\nserves\ndu\nunknown\nparties\n##les\ngeneration\n##ff\ncontinues\nquick\nfields\nbrigade\nquiet\nteaching\nclothes\nimpact\nweapons\npartner\nflat\ntheater\nsupreme\n1938\n37\nrelations\n##tor\nplants\nsuffered\n1936\nwilson\nkids\nbegins\n##age\n1918\nseats\narmed\ninternet\nmodels\nworth\nlaws\n400\ncommunities\nclasses\nbackground\nknows\nthanks\nquarter\nreaching\nhumans\ncarry\nkilling\nformat\nkong\nhong\nsetting\n75\narchitecture\ndisease\nrailroad\ninc\npossibly\nwish\narthur\nthoughts\nharry\ndoors\ndensity\n##di\ncrowd\nillinois\nstomach\ntone\nunique\nreports\nanyway\n##ir\nliberal\nder\nvehicle\nthick\ndry\ndrug\nfaced\nlargely\nfacility\ntheme\nholds\ncreation\nstrange\ncolonel\n##mi\nrevolution\nbell\npolitics\nturns\nsilent\nrail\nrelief\nindependence\ncombat\nshape\nwrite\ndetermined\nsales\nlearned\n4th\nfinger\noxford\nproviding\n1937\nheritage\nfiction\nsituated\ndesignated\nallowing\ndistribution\nhosted\n##est\nsight\ninterview\nestimated\nreduced\n##ria\ntoronto\nfootballer\nkeeping\nguys\ndamn\nclaim\nmotion\nsport\nsixth\nstayed\n##ze\nen\nrear\nreceive\nhanded\ntwelve\ndress\naudience\ngranted\nbrazil\n##well\nspirit\n##ated\nnoticed\netc\nolympic\nrepresentative\neric\ntight\ntrouble\nreviews\ndrink\nvampire\nmissing\nroles\nranked\nnewly\nhousehold\nfinals\nwave\ncritics\n##ee\nphase\nmassachusetts\npilot\nunlike\nphiladelphia\nbright\nguns\ncrown\norganizations\nroof\n42\nrespectively\nclearly\ntongue\nmarked\ncircle\nfox\nkorea\nbronze\nbrian\nexpanded\nsexual\nsupply\nyourself\ninspired\nlabour\nfc\n##ah\nreference\nvision\ndraft\nconnection\nbrand\nreasons\n1935\nclassic\ndriving\ntrip\njesus\ncells\nentry\n1920\nneither\ntrail\nclaims\natlantic\norders\nlabor\nnose\nafraid\nidentified\nintelligence\ncalls\ncancer\nattacked\npassing\nstephen\npositions\nimperial\ngrey\njason\n39\nsunday\n48\nswedish\navoid\nextra\nuncle\nmessage\ncovers\nallows\nsurprise\nmaterials\nfame\nhunter\n##ji\n1930\ncitizens\nfigures\ndavis\nenvironmental\nconfirmed\nshit\ntitles\ndi\nperforming\ndifference\nacts\nattacks\n##ov\nexisting\nvotes\nopportunity\nnor\nshop\nentirely\ntrains\nopposite\npakistan\n##pa\ndevelop\nresulted\nrepresentatives\nactions\nreality\npressed\n##ish\nbarely\nwine\nconversation\nfaculty\nnorthwest\nends\ndocumentary\nnuclear\nstock\ngrace\nsets\neat\nalternative\n##ps\nbag\nresulting\ncreating\nsurprised\ncemetery\n1919\ndrop\nfinding\nsarah\ncricket\nstreets\ntradition\nride\n1933\nexhibition\ntarget\near\nexplained\nrain\ncomposer\ninjury\napartment\nmunicipal\neducational\noccupied\nnetherlands\nclean\nbillion\nconstitution\nlearn\n1914\nmaximum\nclassical\nfrancis\nlose\nopposition\njose\nontario\nbear\ncore\nhills\nrolled\nending\ndrawn\npermanent\nfun\n##tes\n##lla\nlewis\nsites\nchamber\nryan\n##way\nscoring\nheight\n1934\n##house\nlyrics\nstaring\n55\nofficials\n1917\nsnow\noldest\n##tic\norange\n##ger\nqualified\ninterior\napparently\nsucceeded\nthousand\ndinner\nlights\nexistence\nfans\nheavily\n41\ngreatest\nconservative\nsend\nbowl\nplus\nenter\ncatch\n##un\neconomy\nduty\n1929\nspeech\nauthorities\nprincess\nperformances\nversions\nshall\ngraduate\npictures\neffective\nremembered\npoetry\ndesk\ncrossed\nstarring\nstarts\npassenger\nsharp\n##ant\nacres\nass\nweather\nfalling\nrank\nfund\nsupporting\ncheck\nadult\npublishing\nheads\ncm\nsoutheast\nlane\n##burg\napplication\nbc\n##ura\nles\ncondition\ntransfer\nprevent\ndisplay\nex\nregions\nearl\nfederation\ncool\nrelatively\nanswered\nbesides\n1928\nobtained\nportion\n##town\nmix\n##ding\nreaction\nliked\ndean\nexpress\npeak\n1932\n##tte\ncounter\nreligion\nchain\nrare\nmiller\nconvention\naid\nlie\nvehicles\nmobile\nperform\nsquad\nwonder\nlying\ncrazy\nsword\n##ping\nattempted\ncenturies\nweren\nphilosophy\ncategory\n##ize\nanna\ninterested\n47\nsweden\nwolf\nfrequently\nabandoned\nkg\nliterary\nalliance\ntask\nentitled\n##ay\nthrew\npromotion\nfactory\ntiny\nsoccer\nvisited\nmatt\nfm\nachieved\n52\ndefence\ninternal\npersian\n43\nmethods\n##ging\narrested\notherwise\ncambridge\nprogramming\nvillages\nelementary\ndistricts\nrooms\ncriminal\nconflict\nworry\ntrained\n1931\nattempts\nwaited\nsignal\nbird\ntruck\nsubsequent\nprogramme\n##ol\nad\n49\ncommunist\ndetails\nfaith\nsector\npatrick\ncarrying\nlaugh\n##ss\ncontrolled\nkorean\nshowing\norigin\nfuel\nevil\n1927\n##ent\nbrief\nidentity\ndarkness\naddress\npool\nmissed\npublication\nweb\nplanet\nian\nanne\nwings\ninvited\n##tt\nbriefly\nstandards\nkissed\n##be\nideas\nclimate\ncausing\nwalter\nworse\nalbert\narticles\nwinners\ndesire\naged\nnortheast\ndangerous\ngate\ndoubt\n1922\nwooden\nmulti\n##ky\npoet\nrising\nfunding\n46\ncommunications\ncommunication\nviolence\ncopies\nprepared\nford\ninvestigation\nskills\n1924\npulling\nelectronic\n##ak\n##ial\n##han\ncontaining\nultimately\noffices\nsinging\nunderstanding\nrestaurant\ntomorrow\nfashion\nchrist\nward\nda\npope\nstands\n5th\nflow\nstudios\naired\ncommissioned\ncontained\nexist\nfresh\namericans\n##per\nwrestling\napproved\nkid\nemployed\nrespect\nsuit\n1925\nangel\nasking\nincreasing\nframe\nangry\nselling\n1950s\nthin\nfinds\n##nd\ntemperature\nstatement\nali\nexplain\ninhabitants\ntowns\nextensive\nnarrow\n51\njane\nflowers\nimages\npromise\nsomewhere\nobject\nfly\nclosely\n##ls\n1912\nbureau\ncape\n1926\nweekly\npresidential\nlegislative\n1921\n##ai\n##au\nlaunch\nfounding\n##ny\n978\n##ring\nartillery\nstrike\nun\ninstitutions\nroll\nwriters\nlanding\nchose\nkevin\nanymore\npp\n##ut\nattorney\nfit\ndan\nbillboard\nreceiving\nagricultural\nbreaking\nsought\ndave\nadmitted\nlands\nmexican\n##bury\ncharlie\nspecifically\nhole\niv\nhoward\ncredit\nmoscow\nroads\naccident\n1923\nproved\nwear\nstruck\nhey\nguards\nstuff\nslid\nexpansion\n1915\ncat\nanthony\n##kin\nmelbourne\nopposed\nsub\nsouthwest\narchitect\nfailure\nplane\n1916\n##ron\nmap\ncamera\ntank\nlisten\nregarding\nwet\nintroduction\nmetropolitan\nlink\nep\nfighter\ninch\ngrown\ngene\nanger\nfixed\nbuy\ndvd\nkhan\ndomestic\nworldwide\nchapel\nmill\nfunctions\nexamples\n##head\ndeveloping\n1910\nturkey\nhits\npocket\nantonio\npapers\ngrow\nunless\ncircuit\n18th\nconcerned\nattached\njournalist\nselection\njourney\nconverted\nprovincial\npainted\nhearing\naren\nbands\nnegative\naside\nwondered\nknight\nlap\nsurvey\nma\n##ow\nnoise\nbilly\n##ium\nshooting\nguide\nbedroom\npriest\nresistance\nmotor\nhomes\nsounded\ngiant\n##mer\n150\nscenes\nequal\ncomic\npatients\nhidden\nsolid\nactual\nbringing\nafternoon\ntouched\nfunds\nwedding\nconsisted\nmarie\ncanal\nsr\nkim\ntreaty\nturkish\nrecognition\nresidence\ncathedral\nbroad\nknees\nincident\nshaped\nfired\nnorwegian\nhandle\ncheek\ncontest\nrepresent\n##pe\nrepresenting\nbeauty\n##sen\nbirds\nadvantage\nemergency\nwrapped\ndrawing\nnotice\npink\nbroadcasting\n##ong\nsomehow\nbachelor\nseventh\ncollected\nregistered\nestablishment\nalan\nassumed\nchemical\npersonnel\nroger\nretirement\njeff\nportuguese\nwore\ntied\ndevice\nthreat\nprogress\nadvance\n##ised\nbanks\nhired\nmanchester\nnfl\nteachers\nstructures\nforever\n##bo\ntennis\nhelping\nsaturday\nsale\napplications\njunction\nhip\nincorporated\nneighborhood\ndressed\nceremony\n##ds\ninfluenced\nhers\nvisual\nstairs\ndecades\ninner\nkansas\nhung\nhoped\ngain\nscheduled\ndowntown\nengaged\naustria\nclock\nnorway\ncertainly\npale\nprotected\n1913\nvictor\nemployees\nplate\nputting\nsurrounded\n##ists\nfinishing\nblues\ntropical\n##ries\nminnesota\nconsider\nphilippines\naccept\n54\nretrieved\n1900\nconcern\nanderson\nproperties\ninstitution\ngordon\nsuccessfully\nvietnam\n##dy\nbacking\noutstanding\nmuslim\ncrossing\nfolk\nproducing\nusual\ndemand\noccurs\nobserved\nlawyer\neducated\n##ana\nkelly\nstring\npleasure\nbudget\nitems\nquietly\ncolorado\nphilip\ntypical\n##worth\nderived\n600\nsurvived\nasks\nmental\n##ide\n56\njake\njews\ndistinguished\nltd\n1911\nsri\nextremely\n53\nathletic\nloud\nthousands\nworried\nshadow\ntransportation\nhorses\nweapon\narena\nimportance\nusers\ntim\nobjects\ncontributed\ndragon\ndouglas\naware\nsenator\njohnny\njordan\nsisters\nengines\nflag\ninvestment\nsamuel\nshock\ncapable\nclark\nrow\nwheel\nrefers\nsession\nfamiliar\nbiggest\nwins\nhate\nmaintained\ndrove\nhamilton\nrequest\nexpressed\ninjured\nunderground\nchurches\nwalker\nwars\ntunnel\npasses\nstupid\nagriculture\nsoftly\ncabinet\nregarded\njoining\nindiana\n##ea\n##ms\npush\ndates\nspend\nbehavior\nwoods\nprotein\ngently\nchase\nmorgan\nmention\nburning\nwake\ncombination\noccur\nmirror\nleads\njimmy\nindeed\nimpossible\nsingapore\npaintings\ncovering\n##nes\nsoldier\nlocations\nattendance\nsell\nhistorian\nwisconsin\ninvasion\nargued\npainter\ndiego\nchanging\negypt\n##don\nexperienced\ninches\n##ku\nmissouri\nvol\ngrounds\nspoken\nswitzerland\n##gan\nreform\nrolling\nha\nforget\nmassive\nresigned\nburned\nallen\ntennessee\nlocked\nvalues\nimproved\n##mo\nwounded\nuniverse\nsick\ndating\nfacing\npack\npurchase\nuser\n##pur\nmoments\n##ul\nmerged\nanniversary\n1908\ncoal\nbrick\nunderstood\ncauses\ndynasty\nqueensland\nestablish\nstores\ncrisis\npromote\nhoping\nviews\ncards\nreferee\nextension\n##si\nraise\narizona\nimprove\ncolonial\nformal\ncharged\n##rt\npalm\nlucky\nhide\nrescue\nfaces\n95\nfeelings\ncandidates\njuan\n##ell\ngoods\n6th\ncourses\nweekend\n59\nluke\ncash\nfallen\n##om\ndelivered\naffected\ninstalled\ncarefully\ntries\nswiss\nhollywood\ncosts\nlincoln\nresponsibility\n##he\nshore\nfile\nproper\nnormally\nmaryland\nassistance\njump\nconstant\noffering\nfriendly\nwaters\npersons\nrealize\ncontain\ntrophy\n800\npartnership\nfactor\n58\nmusicians\ncry\nbound\noregon\nindicated\nhero\nhouston\nmedium\n##ure\nconsisting\nsomewhat\n##ara\n57\ncycle\n##che\nbeer\nmoore\nfrederick\ngotten\neleven\nworst\nweak\napproached\narranged\nchin\nloan\nuniversal\nbond\nfifteen\npattern\ndisappeared\n##ney\ntranslated\n##zed\nlip\narab\ncapture\ninterests\ninsurance\n##chi\nshifted\ncave\nprix\nwarning\nsections\ncourts\ncoat\nplot\nsmell\nfeed\ngolf\nfavorite\nmaintain\nknife\nvs\nvoted\ndegrees\nfinance\nquebec\nopinion\ntranslation\nmanner\nruled\noperate\nproductions\nchoose\nmusician\ndiscovery\nconfused\ntired\nseparated\nstream\ntechniques\ncommitted\nattend\nranking\nkings\nthrow\npassengers\nmeasure\nhorror\nfan\nmining\nsand\ndanger\nsalt\ncalm\ndecade\ndam\nrequire\nrunner\n##ik\nrush\nassociate\ngreece\n##ker\nrivers\nconsecutive\nmatthew\n##ski\nsighed\nsq\ndocuments\nsteam\nedited\nclosing\ntie\naccused\n1905\n##ini\nislamic\ndistributed\ndirectors\norganisation\nbruce\n7th\nbreathing\nmad\nlit\narrival\nconcrete\ntaste\n08\ncomposition\nshaking\nfaster\namateur\nadjacent\nstating\n1906\ntwin\nflew\n##ran\ntokyo\npublications\n##tone\nobviously\nridge\nstorage\n1907\ncarl\npages\nconcluded\ndesert\ndriven\nuniversities\nages\nterminal\nsequence\nborough\n250\nconstituency\ncreative\ncousin\neconomics\ndreams\nmargaret\nnotably\nreduce\nmontreal\nmode\n17th\nears\nsaved\njan\nvocal\n##ica\n1909\nandy\n##jo\nriding\nroughly\nthreatened\n##ise\nmeters\nmeanwhile\nlanded\ncompete\nrepeated\ngrass\nczech\nregularly\ncharges\ntea\nsudden\nappeal\n##ung\nsolution\ndescribes\npierre\nclassification\nglad\nparking\n##ning\nbelt\nphysics\n99\nrachel\nadd\nhungarian\nparticipate\nexpedition\ndamaged\ngift\nchildhood\n85\nfifty\n##red\nmathematics\njumped\nletting\ndefensive\nmph\n##ux\n##gh\ntesting\n##hip\nhundreds\nshoot\nowners\nmatters\nsmoke\nisraeli\nkentucky\ndancing\nmounted\ngrandfather\nemma\ndesigns\nprofit\nargentina\n##gs\ntruly\nli\nlawrence\ncole\nbegun\ndetroit\nwilling\nbranches\nsmiling\ndecide\nmiami\nenjoyed\nrecordings\n##dale\npoverty\nethnic\ngay\n##bi\ngary\narabic\n09\naccompanied\n##one\n##ons\nfishing\ndetermine\nresidential\nacid\n##ary\nalice\nreturns\nstarred\nmail\n##ang\njonathan\nstrategy\n##ue\nnet\nforty\ncook\nbusinesses\nequivalent\ncommonwealth\ndistinct\nill\n##cy\nseriously\n##ors\n##ped\nshift\nharris\nreplace\nrio\nimagine\nformula\nensure\n##ber\nadditionally\nscheme\nconservation\noccasionally\npurposes\nfeels\nfavor\n##and\n##ore\n1930s\ncontrast\nhanging\nhunt\nmovies\n1904\ninstruments\nvictims\ndanish\nchristopher\nbusy\ndemon\nsugar\nearliest\ncolony\nstudying\nbalance\nduties\n##ks\nbelgium\nslipped\ncarter\n05\nvisible\nstages\niraq\nfifa\n##im\ncommune\nforming\nzero\n07\ncontinuing\ntalked\ncounties\nlegend\nbathroom\noption\ntail\nclay\ndaughters\nafterwards\nsevere\njaw\nvisitors\n##ded\ndevices\naviation\nrussell\nkate\n##vi\nentering\nsubjects\n##ino\ntemporary\nswimming\nforth\nsmooth\nghost\naudio\nbush\noperates\nrocks\nmovements\nsigns\neddie\n##tz\nann\nvoices\nhonorary\n06\nmemories\ndallas\npure\nmeasures\nracial\npromised\n66\nharvard\nceo\n16th\nparliamentary\nindicate\nbenefit\nflesh\ndublin\nlouisiana\n1902\n1901\npatient\nsleeping\n1903\nmembership\ncoastal\nmedieval\nwanting\nelement\nscholars\nrice\n62\nlimit\nsurvive\nmakeup\nrating\ndefinitely\ncollaboration\nobvious\n##tan\nboss\nms\nbaron\nbirthday\nlinked\nsoil\ndiocese\n##lan\nncaa\n##mann\noffensive\nshell\nshouldn\nwaist\n##tus\nplain\nross\norgan\nresolution\nmanufacturing\nadding\nrelative\nkennedy\n98\nwhilst\nmoth\nmarketing\ngardens\ncrash\n72\nheading\npartners\ncredited\ncarlos\nmoves\ncable\n##zi\nmarshall\n##out\ndepending\nbottle\nrepresents\nrejected\nresponded\nexisted\n04\njobs\ndenmark\nlock\n##ating\ntreated\ngraham\nroutes\ntalent\ncommissioner\ndrugs\nsecure\ntests\nreign\nrestored\nphotography\n##gi\ncontributions\noklahoma\ndesigner\ndisc\ngrin\nseattle\nrobin\npaused\natlanta\nunusual\n##gate\npraised\nlas\nlaughing\nsatellite\nhungary\nvisiting\n##sky\ninteresting\nfactors\ndeck\npoems\nnorman\n##water\nstuck\nspeaker\nrifle\ndomain\npremiered\n##her\ndc\ncomics\nactors\n01\nreputation\neliminated\n8th\nceiling\nprisoners\nscript\n##nce\nleather\naustin\nmississippi\nrapidly\nadmiral\nparallel\ncharlotte\nguilty\ntools\ngender\ndivisions\nfruit\n##bs\nlaboratory\nnelson\nfantasy\nmarry\nrapid\naunt\ntribe\nrequirements\naspects\nsuicide\namongst\nadams\nbone\nukraine\nabc\nkick\nsees\nedinburgh\nclothing\ncolumn\nrough\ngods\nhunting\nbroadway\ngathered\nconcerns\n##ek\nspending\nty\n12th\nsnapped\nrequires\nsolar\nbones\ncavalry\n##tta\niowa\ndrinking\nwaste\nindex\nfranklin\ncharity\nthompson\nstewart\ntip\nflash\nlandscape\nfriday\nenjoy\nsingh\npoem\nlistening\n##back\neighth\nfred\ndifferences\nadapted\nbomb\nukrainian\nsurgery\ncorporate\nmasters\nanywhere\n##more\nwaves\nodd\nsean\nportugal\norleans\ndick\ndebate\nkent\neating\npuerto\ncleared\n96\nexpect\ncinema\n97\nguitarist\nblocks\nelectrical\nagree\ninvolving\ndepth\ndying\npanel\nstruggle\n##ged\npeninsula\nadults\nnovels\nemerged\nvienna\nmetro\ndebuted\nshoes\ntamil\nsongwriter\nmeets\nprove\nbeating\ninstance\nheaven\nscared\nsending\nmarks\nartistic\npassage\nsuperior\n03\nsignificantly\nshopping\n##tive\nretained\n##izing\nmalaysia\ntechnique\ncheeks\n##ola\nwarren\nmaintenance\ndestroy\nextreme\nallied\n120\nappearing\n##yn\nfill\nadvice\nalabama\nqualifying\npolicies\ncleveland\nhat\nbattery\nsmart\nauthors\n10th\nsoundtrack\nacted\ndated\nlb\nglance\nequipped\ncoalition\nfunny\nouter\nambassador\nroy\npossibility\ncouples\ncampbell\ndna\nloose\nethan\nsupplies\n1898\ngonna\n88\nmonster\n##res\nshake\nagents\nfrequency\nsprings\ndogs\npractices\n61\ngang\nplastic\neasier\nsuggests\ngulf\nblade\nexposed\ncolors\nindustries\nmarkets\npan\nnervous\nelectoral\ncharts\nlegislation\nownership\n##idae\nmac\nappointment\nshield\ncopy\nassault\nsocialist\nabbey\nmonument\nlicense\nthrone\nemployment\njay\n93\nreplacement\ncharter\ncloud\npowered\nsuffering\naccounts\noak\nconnecticut\nstrongly\nwright\ncolour\ncrystal\n13th\ncontext\nwelsh\nnetworks\nvoiced\ngabriel\njerry\n##cing\nforehead\nmp\n##ens\nmanage\nschedule\ntotally\nremix\n##ii\nforests\noccupation\nprint\nnicholas\nbrazilian\nstrategic\nvampires\nengineers\n76\nroots\nseek\ncorrect\ninstrumental\nund\nalfred\nbacked\nhop\n##des\nstanley\nrobinson\ntraveled\nwayne\nwelcome\naustrian\nachieve\n67\nexit\nrates\n1899\nstrip\nwhereas\n##cs\nsing\ndeeply\nadventure\nbobby\nrick\njamie\ncareful\ncomponents\ncap\nuseful\npersonality\nknee\n##shi\npushing\nhosts\n02\nprotest\nca\nottoman\nsymphony\n##sis\n63\nboundary\n1890\nprocesses\nconsidering\nconsiderable\ntons\n##work\n##ft\n##nia\ncooper\ntrading\ndear\nconduct\n91\nillegal\napple\nrevolutionary\nholiday\ndefinition\nharder\n##van\njacob\ncircumstances\ndestruction\n##lle\npopularity\ngrip\nclassified\nliverpool\ndonald\nbaltimore\nflows\nseeking\nhonour\napproval\n92\nmechanical\ntill\nhappening\nstatue\ncritic\nincreasingly\nimmediate\ndescribe\ncommerce\nstare\n##ster\nindonesia\nmeat\nrounds\nboats\nbaker\northodox\ndepression\nformally\nworn\nnaked\nclaire\nmuttered\nsentence\n11th\nemily\ndocument\n77\ncriticism\nwished\nvessel\nspiritual\nbent\nvirgin\nparker\nminimum\nmurray\nlunch\ndanny\nprinted\ncompilation\nkeyboards\nfalse\nblow\nbelonged\n68\nraising\n78\ncutting\n##board\npittsburgh\n##up\n9th\nshadows\n81\nhated\nindigenous\njon\n15th\nbarry\nscholar\nah\n##zer\noliver\n##gy\nstick\nsusan\nmeetings\nattracted\nspell\nromantic\n##ver\nye\n1895\nphoto\ndemanded\ncustomers\n##ac\n1896\nlogan\nrevival\nkeys\nmodified\ncommanded\njeans\n##ious\nupset\nraw\nphil\ndetective\nhiding\nresident\nvincent\n##bly\nexperiences\ndiamond\ndefeating\ncoverage\nlucas\nexternal\nparks\nfranchise\nhelen\nbible\nsuccessor\npercussion\ncelebrated\nil\nlift\nprofile\nclan\nromania\n##ied\nmills\n##su\nnobody\nachievement\nshrugged\nfault\n1897\nrhythm\ninitiative\nbreakfast\ncarbon\n700\n69\nlasted\nviolent\n74\nwound\nken\nkiller\ngradually\nfilmed\n°c\ndollars\nprocessing\n94\nremove\ncriticized\nguests\nsang\nchemistry\n##vin\nlegislature\ndisney\n##bridge\nuniform\nescaped\nintegrated\nproposal\npurple\ndenied\nliquid\nkarl\ninfluential\nmorris\nnights\nstones\nintense\nexperimental\ntwisted\n71\n84\n##ld\npace\nnazi\nmitchell\nny\nblind\nreporter\nnewspapers\n14th\ncenters\nburn\nbasin\nforgotten\nsurviving\nfiled\ncollections\nmonastery\nlosses\nmanual\ncouch\ndescription\nappropriate\nmerely\ntag\nmissions\nsebastian\nrestoration\nreplacing\ntriple\n73\nelder\njulia\nwarriors\nbenjamin\njulian\nconvinced\nstronger\namazing\ndeclined\nversus\nmerchant\nhappens\noutput\nfinland\nbare\nbarbara\nabsence\nignored\ndawn\ninjuries\n##port\nproducers\n##ram\n82\nluis\n##ities\nkw\nadmit\nexpensive\nelectricity\nnba\nexception\nsymbol\n##ving\nladies\nshower\nsheriff\ncharacteristics\n##je\naimed\nbutton\nratio\neffectively\nsummit\nangle\njury\nbears\nfoster\nvessels\npants\nexecuted\nevans\ndozen\nadvertising\nkicked\npatrol\n1889\ncompetitions\nlifetime\nprinciples\nathletics\n##logy\nbirmingham\nsponsored\n89\nrob\nnomination\n1893\nacoustic\n##sm\ncreature\nlongest\n##tra\ncredits\nharbor\ndust\njosh\n##so\nterritories\nmilk\ninfrastructure\ncompletion\nthailand\nindians\nleon\narchbishop\n##sy\nassist\npitch\nblake\narrangement\ngirlfriend\nserbian\noperational\nhence\nsad\nscent\nfur\ndj\nsessions\nhp\nrefer\nrarely\n##ora\nexists\n1892\n##ten\nscientists\ndirty\npenalty\nburst\nportrait\nseed\n79\npole\nlimits\nrival\n1894\nstable\nalpha\ngrave\nconstitutional\nalcohol\narrest\nflower\nmystery\ndevil\narchitectural\nrelationships\ngreatly\nhabitat\n##istic\nlarry\nprogressive\nremote\ncotton\n##ics\n##ok\npreserved\nreaches\n##ming\ncited\n86\nvast\nscholarship\ndecisions\ncbs\njoy\nteach\n1885\neditions\nknocked\neve\nsearching\npartly\nparticipation\ngap\nanimated\nfate\nexcellent\n##ett\nna\n87\nalternate\nsaints\nyoungest\n##ily\nclimbed\n##ita\n##tors\nsuggest\n##ct\ndiscussion\nstaying\nchoir\nlakes\njacket\nrevenue\nnevertheless\npeaked\ninstrument\nwondering\nannually\nmanaging\nneil\n1891\nsigning\nterry\n##ice\napply\nclinical\nbrooklyn\naim\ncatherine\nfuck\nfarmers\nfigured\nninth\npride\nhugh\nevolution\nordinary\ninvolvement\ncomfortable\nshouted\ntech\nencouraged\ntaiwan\nrepresentation\nsharing\n##lia\n##em\npanic\nexact\ncargo\ncompeting\nfat\ncried\n83\n1920s\noccasions\npa\ncabin\nborders\nutah\nmarcus\n##isation\nbadly\nmuscles\n##ance\nvictorian\ntransition\nwarner\nbet\npermission\n##rin\nslave\nterrible\nsimilarly\nshares\nseth\nuefa\npossession\nmedals\nbenefits\ncolleges\nlowered\nperfectly\nmall\ntransit\n##ye\n##kar\npublisher\n##ened\nharrison\ndeaths\nelevation\n##ae\nasleep\nmachines\nsigh\nash\nhardly\nargument\noccasion\nparent\nleo\ndecline\n1888\ncontribution\n##ua\nconcentration\n1000\nopportunities\nhispanic\nguardian\nextent\nemotions\nhips\nmason\nvolumes\nbloody\ncontroversy\ndiameter\nsteady\nmistake\nphoenix\nidentify\nviolin\n##sk\ndeparture\nrichmond\nspin\nfuneral\nenemies\n1864\ngear\nliterally\nconnor\nrandom\nsergeant\ngrab\nconfusion\n1865\ntransmission\ninformed\nop\nleaning\nsacred\nsuspended\nthinks\ngates\nportland\nluck\nagencies\nyours\nhull\nexpert\nmuscle\nlayer\npractical\nsculpture\njerusalem\nlatest\nlloyd\nstatistics\ndeeper\nrecommended\nwarrior\narkansas\nmess\nsupports\ngreg\neagle\n1880\nrecovered\nrated\nconcerts\nrushed\n##ano\nstops\neggs\nfiles\npremiere\nkeith\n##vo\ndelhi\nturner\npit\naffair\nbelief\npaint\n##zing\nmate\n##ach\n##ev\nvictim\n##ology\nwithdrew\nbonus\nstyles\nfled\n##ud\nglasgow\ntechnologies\nfunded\nnbc\nadaptation\n##ata\nportrayed\ncooperation\nsupporters\njudges\nbernard\njustin\nhallway\nralph\n##ick\ngraduating\ncontroversial\ndistant\ncontinental\nspider\nbite\n##ho\nrecognize\nintention\nmixing\n##ese\negyptian\nbow\ntourism\nsuppose\nclaiming\ntiger\ndominated\nparticipants\nvi\n##ru\nnurse\npartially\ntape\n##rum\npsychology\n##rn\nessential\ntouring\nduo\nvoting\ncivilian\nemotional\nchannels\n##king\napparent\nhebrew\n1887\ntommy\ncarrier\nintersection\nbeast\nhudson\n##gar\n##zo\nlab\nnova\nbench\ndiscuss\ncosta\n##ered\ndetailed\nbehalf\ndrivers\nunfortunately\nobtain\n##lis\nrocky\n##dae\nsiege\nfriendship\nhoney\n##rian\n1861\namy\nhang\nposted\ngovernments\ncollins\nrespond\nwildlife\npreferred\noperator\n##po\nlaura\npregnant\nvideos\ndennis\nsuspected\nboots\ninstantly\nweird\nautomatic\nbusinessman\nalleged\nplacing\nthrowing\nph\nmood\n1862\nperry\nvenue\njet\nremainder\n##lli\n##ci\npassion\nbiological\nboyfriend\n1863\ndirt\nbuffalo\nron\nsegment\nfa\nabuse\n##era\ngenre\nthrown\nstroke\ncolored\nstress\nexercise\ndisplayed\n##gen\nstruggled\n##tti\nabroad\ndramatic\nwonderful\nthereafter\nmadrid\ncomponent\nwidespread\n##sed\ntale\ncitizen\ntodd\nmonday\n1886\nvancouver\noverseas\nforcing\ncrying\ndescent\n##ris\ndiscussed\nsubstantial\nranks\nregime\n1870\nprovinces\nswitch\ndrum\nzane\nted\ntribes\nproof\nlp\ncream\nresearchers\nvolunteer\nmanor\nsilk\nmilan\ndonated\nallies\nventure\nprinciple\ndelivery\nenterprise\n##ves\n##ans\nbars\ntraditionally\nwitch\nreminded\ncopper\n##uk\npete\ninter\nlinks\ncolin\ngrinned\nelsewhere\ncompetitive\nfrequent\n##oy\nscream\n##hu\ntension\ntexts\nsubmarine\nfinnish\ndefending\ndefend\npat\ndetail\n1884\naffiliated\nstuart\nthemes\nvilla\nperiods\ntool\nbelgian\nruling\ncrimes\nanswers\nfolded\nlicensed\nresort\ndemolished\nhans\nlucy\n1881\nlion\ntraded\nphotographs\nwrites\ncraig\n##fa\ntrials\ngenerated\nbeth\nnoble\ndebt\npercentage\nyorkshire\nerected\nss\nviewed\ngrades\nconfidence\nceased\nislam\ntelephone\nretail\n##ible\nchile\nm²\nroberts\nsixteen\n##ich\ncommented\nhampshire\ninnocent\ndual\npounds\nchecked\nregulations\nafghanistan\nsung\nrico\nliberty\nassets\nbigger\noptions\nangels\nrelegated\ntribute\nwells\nattending\nleaf\n##yan\nbutler\nromanian\nforum\nmonthly\nlisa\npatterns\ngmina\n##tory\nmadison\nhurricane\nrev\n##ians\nbristol\n##ula\nelite\nvaluable\ndisaster\ndemocracy\nawareness\ngermans\nfreyja\n##ins\nloop\nabsolutely\npaying\npopulations\nmaine\nsole\nprayer\nspencer\nreleases\ndoorway\nbull\n##ani\nlover\nmidnight\nconclusion\n##sson\nthirteen\nlily\nmediterranean\n##lt\nnhl\nproud\nsample\n##hill\ndrummer\nguinea\n##ova\nmurphy\nclimb\n##ston\ninstant\nattributed\nhorn\nain\nrailways\nsteven\n##ao\nautumn\nferry\nopponent\nroot\ntraveling\nsecured\ncorridor\nstretched\ntales\nsheet\ntrinity\ncattle\nhelps\nindicates\nmanhattan\nmurdered\nfitted\n1882\ngentle\ngrandmother\nmines\nshocked\nvegas\nproduces\n##light\ncaribbean\n##ou\nbelong\ncontinuous\ndesperate\ndrunk\nhistorically\ntrio\nwaved\nraf\ndealing\nnathan\nbat\nmurmured\ninterrupted\nresiding\nscientist\npioneer\nharold\naaron\n##net\ndelta\nattempting\nminority\nmini\nbelieves\nchorus\ntend\nlots\neyed\nindoor\nload\nshots\nupdated\njail\n##llo\nconcerning\nconnecting\nwealth\n##ved\nslaves\narrive\nrangers\nsufficient\nrebuilt\n##wick\ncardinal\nflood\nmuhammad\nwhenever\nrelation\nrunners\nmoral\nrepair\nviewers\narriving\nrevenge\npunk\nassisted\nbath\nfairly\nbreathe\nlists\ninnings\nillustrated\nwhisper\nnearest\nvoters\nclinton\nties\nultimate\nscreamed\nbeijing\nlions\nandre\nfictional\ngathering\ncomfort\nradar\nsuitable\ndismissed\nhms\nban\npine\nwrist\natmosphere\nvoivodeship\nbid\ntimber\n##ned\n##nan\ngiants\n##ane\ncameron\nrecovery\nuss\nidentical\ncategories\nswitched\nserbia\nlaughter\nnoah\nensemble\ntherapy\npeoples\ntouching\n##off\nlocally\npearl\nplatforms\neverywhere\nballet\ntables\nlanka\nherbert\noutdoor\ntoured\nderek\n1883\nspaces\ncontested\nswept\n1878\nexclusive\nslight\nconnections\n##dra\nwinds\nprisoner\ncollective\nbangladesh\ntube\npublicly\nwealthy\nthai\n##ys\nisolated\nselect\n##ric\ninsisted\npen\nfortune\nticket\nspotted\nreportedly\nanimation\nenforcement\ntanks\n110\ndecides\nwider\nlowest\nowen\n##time\nnod\nhitting\n##hn\ngregory\nfurthermore\nmagazines\nfighters\nsolutions\n##ery\npointing\nrequested\nperu\nreed\nchancellor\nknights\nmask\nworker\neldest\nflames\nreduction\n1860\nvolunteers\n##tis\nreporting\n##hl\nwire\nadvisory\nendemic\norigins\nsettlers\npursue\nknock\nconsumer\n1876\neu\ncompound\ncreatures\nmansion\nsentenced\nivan\ndeployed\nguitars\nfrowned\ninvolves\nmechanism\nkilometers\nperspective\nshops\nmaps\nterminus\nduncan\nalien\nfist\nbridges\n##pers\nheroes\nfed\nderby\nswallowed\n##ros\npatent\nsara\nillness\ncharacterized\nadventures\nslide\nhawaii\njurisdiction\n##op\norganised\n##side\nadelaide\nwalks\nbiology\nse\n##ties\nrogers\nswing\ntightly\nboundaries\n##rie\nprepare\nimplementation\nstolen\n##sha\ncertified\ncolombia\nedwards\ngarage\n##mm\nrecalled\n##ball\nrage\nharm\nnigeria\nbreast\n##ren\nfurniture\npupils\nsettle\n##lus\ncuba\nballs\nclient\nalaska\n21st\nlinear\nthrust\ncelebration\nlatino\ngenetic\nterror\n##cia\n##ening\nlightning\nfee\nwitness\nlodge\nestablishing\nskull\n##ique\nearning\nhood\n##ei\nrebellion\nwang\nsporting\nwarned\nmissile\ndevoted\nactivist\nporch\nworship\nfourteen\npackage\n1871\ndecorated\n##shire\nhoused\n##ock\nchess\nsailed\ndoctors\noscar\njoan\ntreat\ngarcia\nharbour\njeremy\n##ire\ntraditions\ndominant\njacques\n##gon\n##wan\nrelocated\n1879\namendment\nsized\ncompanion\nsimultaneously\nvolleyball\nspun\nacre\nincreases\nstopping\nloves\nbelongs\naffect\ndrafted\ntossed\nscout\nbattles\n1875\nfilming\nshoved\nmunich\ntenure\nvertical\nromance\npc\n##cher\nargue\n##ical\ncraft\nranging\nwww\nopens\nhonest\ntyler\nyesterday\nvirtual\n##let\nmuslims\nreveal\nsnake\nimmigrants\nradical\nscreaming\nspeakers\nfiring\nsaving\nbelonging\nease\nlighting\nprefecture\nblame\nfarmer\nhungry\ngrows\nrubbed\nbeam\nsur\nsubsidiary\n##cha\narmenian\nsao\ndropping\nconventional\n##fer\nmicrosoft\nreply\nqualify\nspots\n1867\nsweat\nfestivals\n##ken\nimmigration\nphysician\ndiscover\nexposure\nsandy\nexplanation\nisaac\nimplemented\n##fish\nhart\ninitiated\nconnect\nstakes\npresents\nheights\nhouseholder\npleased\ntourist\nregardless\nslip\nclosest\n##ction\nsurely\nsultan\nbrings\nriley\npreparation\naboard\nslammed\nbaptist\nexperiment\nongoing\ninterstate\norganic\nplayoffs\n##ika\n1877\n130\n##tar\nhindu\nerror\ntours\ntier\nplenty\narrangements\ntalks\ntrapped\nexcited\nsank\nho\nathens\n1872\ndenver\nwelfare\nsuburb\nathletes\ntrick\ndiverse\nbelly\nexclusively\nyelled\n1868\n##med\nconversion\n##ette\n1874\ninternationally\ncomputers\nconductor\nabilities\nsensitive\nhello\ndispute\nmeasured\nglobe\nrocket\nprices\namsterdam\nflights\ntigers\ninn\nmunicipalities\nemotion\nreferences\n3d\n##mus\nexplains\nairlines\nmanufactured\npm\narchaeological\n1873\ninterpretation\ndevon\ncomment\n##ites\nsettlements\nkissing\nabsolute\nimprovement\nsuite\nimpressed\nbarcelona\nsullivan\njefferson\ntowers\njesse\njulie\n##tin\n##lu\ngrandson\nhi\ngauge\nregard\nrings\ninterviews\ntrace\nraymond\nthumb\ndepartments\nburns\nserial\nbulgarian\nscores\ndemonstrated\n##ix\n1866\nkyle\nalberta\nunderneath\nromanized\n##ward\nrelieved\nacquisition\nphrase\ncliff\nreveals\nhan\ncuts\nmerger\ncustom\n##dar\nnee\ngilbert\ngraduation\n##nts\nassessment\ncafe\ndifficulty\ndemands\nswung\ndemocrat\njennifer\ncommons\n1940s\ngrove\n##yo\ncompleting\nfocuses\nsum\nsubstitute\nbearing\nstretch\nreception\n##py\nreflected\nessentially\ndestination\npairs\n##ched\nsurvival\nresource\n##bach\npromoting\ndoubles\nmessages\ntear\n##down\n##fully\nparade\nflorence\nharvey\nincumbent\npartial\nframework\n900\npedro\nfrozen\nprocedure\nolivia\ncontrols\n##mic\nshelter\npersonally\ntemperatures\n##od\nbrisbane\ntested\nsits\nmarble\ncomprehensive\noxygen\nleonard\n##kov\ninaugural\niranian\nreferring\nquarters\nattitude\n##ivity\nmainstream\nlined\nmars\ndakota\nnorfolk\nunsuccessful\n##°\nexplosion\nhelicopter\ncongressional\n##sing\ninspector\nbitch\nseal\ndeparted\ndivine\n##ters\ncoaching\nexamination\npunishment\nmanufacturer\nsink\ncolumns\nunincorporated\nsignals\nnevada\nsqueezed\ndylan\ndining\nphotos\nmartial\nmanuel\neighteen\nelevator\nbrushed\nplates\nministers\nivy\ncongregation\n##len\nslept\nspecialized\ntaxes\ncurve\nrestricted\nnegotiations\nlikes\nstatistical\narnold\ninspiration\nexecution\nbold\nintermediate\nsignificance\nmargin\nruler\nwheels\ngothic\nintellectual\ndependent\nlistened\neligible\nbuses\nwidow\nsyria\nearn\ncincinnati\ncollapsed\nrecipient\nsecrets\naccessible\nphilippine\nmaritime\ngoddess\nclerk\nsurrender\nbreaks\nplayoff\ndatabase\n##ified\n##lon\nideal\nbeetle\naspect\nsoap\nregulation\nstrings\nexpand\nanglo\nshorter\ncrosses\nretreat\ntough\ncoins\nwallace\ndirections\npressing\n##oon\nshipping\nlocomotives\ncomparison\ntopics\nnephew\n##mes\ndistinction\nhonors\ntravelled\nsierra\nibn\n##over\nfortress\nsa\nrecognised\ncarved\n1869\nclients\n##dan\nintent\n##mar\ncoaches\ndescribing\nbread\n##ington\nbeaten\nnorthwestern\n##ona\nmerit\nyoutube\ncollapse\nchallenges\nem\nhistorians\nobjective\nsubmitted\nvirus\nattacking\ndrake\nassume\n##ere\ndiseases\nmarc\nstem\nleeds\n##cus\n##ab\nfarming\nglasses\n##lock\nvisits\nnowhere\nfellowship\nrelevant\ncarries\nrestaurants\nexperiments\n101\nconstantly\nbases\ntargets\nshah\ntenth\nopponents\nverse\nterritorial\n##ira\nwritings\ncorruption\n##hs\ninstruction\ninherited\nreverse\nemphasis\n##vic\nemployee\narch\nkeeps\nrabbi\nwatson\npayment\nuh\n##ala\nnancy\n##tre\nvenice\nfastest\nsexy\nbanned\nadrian\nproperly\nruth\ntouchdown\ndollar\nboards\nmetre\ncircles\nedges\nfavour\ncomments\nok\ntravels\nliberation\nscattered\nfirmly\n##ular\nholland\npermitted\ndiesel\nkenya\nden\noriginated\n##ral\ndemons\nresumed\ndragged\nrider\n##rus\nservant\nblinked\nextend\ntorn\n##ias\n##sey\ninput\nmeal\neverybody\ncylinder\nkinds\ncamps\n##fe\nbullet\nlogic\n##wn\ncroatian\nevolved\nhealthy\nfool\nchocolate\nwise\npreserve\npradesh\n##ess\nrespective\n1850\n##ew\nchicken\nartificial\ngross\ncorresponding\nconvicted\ncage\ncaroline\ndialogue\n##dor\nnarrative\nstranger\nmario\nbr\nchristianity\nfailing\ntrent\ncommanding\nbuddhist\n1848\nmaurice\nfocusing\nyale\nbike\naltitude\n##ering\nmouse\nrevised\n##sley\nveteran\n##ig\npulls\ntheology\ncrashed\ncampaigns\nlegion\n##ability\ndrag\nexcellence\ncustomer\ncancelled\nintensity\nexcuse\n##lar\nliga\nparticipating\ncontributing\nprinting\n##burn\nvariable\n##rk\ncurious\nbin\nlegacy\nrenaissance\n##my\nsymptoms\nbinding\nvocalist\ndancer\n##nie\ngrammar\ngospel\ndemocrats\nya\nenters\nsc\ndiplomatic\nhitler\n##ser\nclouds\nmathematical\nquit\ndefended\noriented\n##heim\nfundamental\nhardware\nimpressive\nequally\nconvince\nconfederate\nguilt\nchuck\nsliding\n##ware\nmagnetic\nnarrowed\npetersburg\nbulgaria\notto\nphd\nskill\n##ama\nreader\nhopes\npitcher\nreservoir\nhearts\nautomatically\nexpecting\nmysterious\nbennett\nextensively\nimagined\nseeds\nmonitor\nfix\n##ative\njournalism\nstruggling\nsignature\nranch\nencounter\nphotographer\nobservation\nprotests\n##pin\ninfluences\n##hr\ncalendar\n##all\ncruz\ncroatia\nlocomotive\nhughes\nnaturally\nshakespeare\nbasement\nhook\nuncredited\nfaded\ntheories\napproaches\ndare\nphillips\nfilling\nfury\nobama\n##ain\nefficient\narc\ndeliver\nmin\nraid\nbreeding\ninducted\nleagues\nefficiency\naxis\nmontana\neagles\n##ked\nsupplied\ninstructions\nkaren\npicking\nindicating\ntrap\nanchor\npractically\nchristians\ntomb\nvary\noccasional\nelectronics\nlords\nreaders\nnewcastle\nfaint\ninnovation\ncollect\nsituations\nengagement\n160\nclaude\nmixture\n##feld\npeer\ntissue\nlogo\nlean\n##ration\n°f\nfloors\n##ven\narchitects\nreducing\n##our\n##ments\nrope\n1859\nottawa\n##har\nsamples\nbanking\ndeclaration\nproteins\nresignation\nfrancois\nsaudi\nadvocate\nexhibited\narmor\ntwins\ndivorce\n##ras\nabraham\nreviewed\njo\ntemporarily\nmatrix\nphysically\npulse\ncurled\n##ena\ndifficulties\nbengal\nusage\n##ban\nannie\nriders\ncertificate\n##pi\nholes\nwarsaw\ndistinctive\njessica\n##mon\nmutual\n1857\ncustoms\ncircular\neugene\nremoval\nloaded\nmere\nvulnerable\ndepicted\ngenerations\ndame\nheir\nenormous\nlightly\nclimbing\npitched\nlessons\npilots\nnepal\nram\ngoogle\npreparing\nbrad\nlouise\nrenowned\n##₂\nliam\n##ably\nplaza\nshaw\nsophie\nbrilliant\nbills\n##bar\n##nik\nfucking\nmainland\nserver\npleasant\nseized\nveterans\njerked\nfail\nbeta\nbrush\nradiation\nstored\nwarmth\nsoutheastern\nnate\nsin\nraced\nberkeley\njoke\nathlete\ndesignation\ntrunk\n##low\nroland\nqualification\narchives\nheels\nartwork\nreceives\njudicial\nreserves\n##bed\nwoke\ninstallation\nabu\nfloating\nfake\nlesser\nexcitement\ninterface\nconcentrated\naddressed\ncharacteristic\namanda\nsaxophone\nmonk\nauto\n##bus\nreleasing\negg\ndies\ninteraction\ndefender\nce\noutbreak\nglory\nloving\n##bert\nsequel\nconsciousness\nhttp\nawake\nski\nenrolled\n##ress\nhandling\nrookie\nbrow\nsomebody\nbiography\nwarfare\namounts\ncontracts\npresentation\nfabric\ndissolved\nchallenged\nmeter\npsychological\nlt\nelevated\nrally\naccurate\n##tha\nhospitals\nundergraduate\nspecialist\nvenezuela\nexhibit\nshed\nnursing\nprotestant\nfluid\nstructural\nfootage\njared\nconsistent\nprey\n##ska\nsuccession\nreflect\nexile\nlebanon\nwiped\nsuspect\nshanghai\nresting\nintegration\npreservation\nmarvel\nvariant\npirates\nsheep\nrounded\ncapita\nsailing\ncolonies\nmanuscript\ndeemed\nvariations\nclarke\nfunctional\nemerging\nboxing\nrelaxed\ncurse\nazerbaijan\nheavyweight\nnickname\neditorial\nrang\ngrid\ntightened\nearthquake\nflashed\nmiguel\nrushing\n##ches\nimprovements\nboxes\nbrooks\n180\nconsumption\nmolecular\nfelix\nsocieties\nrepeatedly\nvariation\naids\ncivic\ngraphics\nprofessionals\nrealm\nautonomous\nreceiver\ndelayed\nworkshop\nmilitia\nchairs\ntrump\ncanyon\n##point\nharsh\nextending\nlovely\nhappiness\n##jan\nstake\neyebrows\nembassy\nwellington\nhannah\n##ella\nsony\ncorners\nbishops\nswear\ncloth\ncontents\nxi\nnamely\ncommenced\n1854\nstanford\nnashville\ncourage\ngraphic\ncommitment\ngarrison\n##bin\nhamlet\nclearing\nrebels\nattraction\nliteracy\ncooking\nruins\ntemples\njenny\nhumanity\ncelebrate\nhasn\nfreight\nsixty\nrebel\nbastard\n##art\nnewton\n##ada\ndeer\n##ges\n##ching\nsmiles\ndelaware\nsingers\n##ets\napproaching\nassists\nflame\n##ph\nboulevard\nbarrel\nplanted\n##ome\npursuit\n##sia\nconsequences\nposts\nshallow\ninvitation\nrode\ndepot\nernest\nkane\nrod\nconcepts\npreston\ntopic\nchambers\nstriking\nblast\narrives\ndescendants\nmontgomery\nranges\nworlds\n##lay\n##ari\nspan\nchaos\npraise\n##ag\nfewer\n1855\nsanctuary\nmud\nfbi\n##ions\nprogrammes\nmaintaining\nunity\nharper\nbore\nhandsome\nclosure\ntournaments\nthunder\nnebraska\nlinda\nfacade\nputs\nsatisfied\nargentine\ndale\ncork\ndome\npanama\n##yl\n1858\ntasks\nexperts\n##ates\nfeeding\nequation\n##las\n##ida\n##tu\nengage\nbryan\n##ax\num\nquartet\nmelody\ndisbanded\nsheffield\nblocked\ngasped\ndelay\nkisses\nmaggie\nconnects\n##non\nsts\npoured\ncreator\npublishers\n##we\nguided\nellis\nextinct\nhug\ngaining\n##ord\ncomplicated\n##bility\npoll\nclenched\ninvestigate\n##use\nthereby\nquantum\nspine\ncdp\nhumor\nkills\nadministered\nsemifinals\n##du\nencountered\nignore\n##bu\ncommentary\n##maker\nbother\nroosevelt\n140\nplains\nhalfway\nflowing\ncultures\ncrack\nimprisoned\nneighboring\nairline\n##ses\n##view\n##mate\n##ec\ngather\nwolves\nmarathon\ntransformed\n##ill\ncruise\norganisations\ncarol\npunch\nexhibitions\nnumbered\nalarm\nratings\ndaddy\nsilently\n##stein\nqueens\ncolours\nimpression\nguidance\nliu\ntactical\n##rat\nmarshal\ndella\narrow\n##ings\nrested\nfeared\ntender\nowns\nbitter\nadvisor\nescort\n##ides\nspare\nfarms\ngrants\n##ene\ndragons\nencourage\ncolleagues\ncameras\n##und\nsucked\npile\nspirits\nprague\nstatements\nsuspension\nlandmark\nfence\ntorture\nrecreation\nbags\npermanently\nsurvivors\npond\nspy\npredecessor\nbombing\ncoup\n##og\nprotecting\ntransformation\nglow\n##lands\n##book\ndug\npriests\nandrea\nfeat\nbarn\njumping\n##chen\n##ologist\n##con\ncasualties\nstern\nauckland\npipe\nserie\nrevealing\nba\n##bel\ntrevor\nmercy\nspectrum\nyang\nconsist\ngoverning\ncollaborated\npossessed\nepic\ncomprises\nblew\nshane\n##ack\nlopez\nhonored\nmagical\nsacrifice\njudgment\nperceived\nhammer\nmtv\nbaronet\ntune\ndas\nmissionary\nsheets\n350\nneutral\noral\nthreatening\nattractive\nshade\naims\nseminary\n##master\nestates\n1856\nmichel\nwounds\nrefugees\nmanufacturers\n##nic\nmercury\nsyndrome\nporter\n##iya\n##din\nhamburg\nidentification\nupstairs\npurse\nwidened\npause\ncared\nbreathed\naffiliate\nsantiago\nprevented\nceltic\nfisher\n125\nrecruited\nbyzantine\nreconstruction\nfarther\n##mp\ndiet\nsake\nau\nspite\nsensation\n##ert\nblank\nseparation\n105\n##hon\nvladimir\narmies\nanime\n##lie\naccommodate\norbit\ncult\nsofia\narchive\n##ify\n##box\nfounders\nsustained\ndisorder\nhonours\nnortheastern\nmia\ncrops\nviolet\nthreats\nblanket\nfires\ncanton\nfollowers\nsouthwestern\nprototype\nvoyage\nassignment\naltered\nmoderate\nprotocol\npistol\n##eo\nquestioned\nbrass\nlifting\n1852\nmath\nauthored\n##ual\ndoug\ndimensional\ndynamic\n##san\n1851\npronounced\ngrateful\nquest\nuncomfortable\nboom\npresidency\nstevens\nrelating\npoliticians\nchen\nbarrier\nquinn\ndiana\nmosque\ntribal\ncheese\npalmer\nportions\nsometime\nchester\ntreasure\nwu\nbend\ndownload\nmillions\nreforms\nregistration\n##osa\nconsequently\nmonitoring\nate\npreliminary\nbrandon\ninvented\nps\neaten\nexterior\nintervention\nports\ndocumented\nlog\ndisplays\nlecture\nsally\nfavourite\n##itz\nvermont\nlo\ninvisible\nisle\nbreed\n##ator\njournalists\nrelay\nspeaks\nbackward\nexplore\nmidfielder\nactively\nstefan\nprocedures\ncannon\nblond\nkenneth\ncentered\nservants\nchains\nlibraries\nmalcolm\nessex\nhenri\nslavery\n##hal\nfacts\nfairy\ncoached\ncassie\ncats\nwashed\ncop\n##fi\nannouncement\nitem\n2000s\nvinyl\nactivated\nmarco\nfrontier\ngrowled\ncurriculum\n##das\nloyal\naccomplished\nleslie\nritual\nkenny\n##00\nvii\nnapoleon\nhollow\nhybrid\njungle\nstationed\nfriedrich\ncounted\n##ulated\nplatinum\ntheatrical\nseated\ncol\nrubber\nglen\n1840\ndiversity\nhealing\nextends\nid\nprovisions\nadministrator\ncolumbus\n##oe\ntributary\nte\nassured\norg\n##uous\nprestigious\nexamined\nlectures\ngrammy\nronald\nassociations\nbailey\nallan\nessays\nflute\nbelieving\nconsultant\nproceedings\ntravelling\n1853\nkit\nkerala\nyugoslavia\nbuddy\nmethodist\n##ith\nburial\ncentres\nbatman\n##nda\ndiscontinued\nbo\ndock\nstockholm\nlungs\nseverely\n##nk\nciting\nmanga\n##ugh\nsteal\nmumbai\niraqi\nrobot\ncelebrity\nbride\nbroadcasts\nabolished\npot\njoel\noverhead\nfranz\npacked\nreconnaissance\njohann\nacknowledged\nintroduce\nhandled\ndoctorate\ndevelopments\ndrinks\nalley\npalestine\n##nis\n##aki\nproceeded\nrecover\nbradley\ngrain\npatch\nafford\ninfection\nnationalist\nlegendary\n##ath\ninterchange\nvirtually\ngen\ngravity\nexploration\namber\nvital\nwishes\npowell\ndoctrine\nelbow\nscreenplay\n##bird\ncontribute\nindonesian\npet\ncreates\n##com\nenzyme\nkylie\ndiscipline\ndrops\nmanila\nhunger\n##ien\nlayers\nsuffer\nfever\nbits\nmonica\nkeyboard\nmanages\n##hood\nsearched\nappeals\n##bad\ntestament\ngrande\nreid\n##war\nbeliefs\ncongo\n##ification\n##dia\nsi\nrequiring\n##via\ncasey\n1849\nregret\nstreak\nrape\ndepends\nsyrian\nsprint\npound\ntourists\nupcoming\npub\n##xi\ntense\n##els\npracticed\necho\nnationwide\nguild\nmotorcycle\nliz\n##zar\nchiefs\ndesired\nelena\nbye\nprecious\nabsorbed\nrelatives\nbooth\npianist\n##mal\ncitizenship\nexhausted\nwilhelm\n##ceae\n##hed\nnoting\nquarterback\nurge\nhectares\n##gue\nace\nholly\n##tal\nblonde\ndavies\nparked\nsustainable\nstepping\ntwentieth\nairfield\ngalaxy\nnest\nchip\n##nell\ntan\nshaft\npaulo\nrequirement\n##zy\nparadise\ntobacco\ntrans\nrenewed\nvietnamese\n##cker\n##ju\nsuggesting\ncatching\nholmes\nenjoying\nmd\ntrips\ncolt\nholder\nbutterfly\nnerve\nreformed\ncherry\nbowling\ntrailer\ncarriage\ngoodbye\nappreciate\ntoy\njoshua\ninteractive\nenabled\ninvolve\n##kan\ncollar\ndetermination\nbunch\nfacebook\nrecall\nshorts\nsuperintendent\nepiscopal\nfrustration\ngiovanni\nnineteenth\nlaser\nprivately\narray\ncirculation\n##ovic\narmstrong\ndeals\npainful\npermit\ndiscrimination\n##wi\naires\nretiring\ncottage\nni\n##sta\nhorizon\nellen\njamaica\nripped\nfernando\nchapters\nplaystation\npatron\nlecturer\nnavigation\nbehaviour\ngenes\ngeorgian\nexport\nsolomon\nrivals\nswift\nseventeen\nrodriguez\nprinceton\nindependently\nsox\n1847\narguing\nentity\ncasting\nhank\ncriteria\noakland\ngeographic\nmilwaukee\nreflection\nexpanding\nconquest\ndubbed\n##tv\nhalt\nbrave\nbrunswick\ndoi\narched\ncurtis\ndivorced\npredominantly\nsomerset\nstreams\nugly\nzoo\nhorrible\ncurved\nbuenos\nfierce\ndictionary\nvector\ntheological\nunions\nhandful\nstability\nchan\npunjab\nsegments\n##lly\naltar\nignoring\ngesture\nmonsters\npastor\n##stone\nthighs\nunexpected\noperators\nabruptly\ncoin\ncompiled\nassociates\nimproving\nmigration\npin\n##ose\ncompact\ncollegiate\nreserved\n##urs\nquarterfinals\nroster\nrestore\nassembled\nhurry\noval\n##cies\n1846\nflags\nmartha\n##del\nvictories\nsharply\n##rated\nargues\ndeadly\nneo\ndrawings\nsymbols\nperformer\n##iel\ngriffin\nrestrictions\nediting\nandrews\njava\njournals\narabia\ncompositions\ndee\npierce\nremoving\nhindi\ncasino\nrunway\ncivilians\nminds\nnasa\nhotels\n##zation\nrefuge\nrent\nretain\npotentially\nconferences\nsuburban\nconducting\n##tto\n##tions\n##tle\ndescended\nmassacre\n##cal\nammunition\nterrain\nfork\nsouls\ncounts\nchelsea\ndurham\ndrives\ncab\n##bank\nperth\nrealizing\npalestinian\nfinn\nsimpson\n##dal\nbetty\n##ule\nmoreover\nparticles\ncardinals\ntent\nevaluation\nextraordinary\n##oid\ninscription\n##works\nwednesday\nchloe\nmaintains\npanels\nashley\ntrucks\n##nation\ncluster\nsunlight\nstrikes\nzhang\n##wing\ndialect\ncanon\n##ap\ntucked\n##ws\ncollecting\n##mas\n##can\n##sville\nmaker\nquoted\nevan\nfranco\naria\nbuying\ncleaning\neva\ncloset\nprovision\napollo\nclinic\nrat\n##ez\nnecessarily\nac\n##gle\n##ising\nvenues\nflipped\ncent\nspreading\ntrustees\nchecking\nauthorized\n##sco\ndisappointed\n##ado\nnotion\nduration\ntrumpet\nhesitated\ntopped\nbrussels\nrolls\ntheoretical\nhint\ndefine\naggressive\nrepeat\nwash\npeaceful\noptical\nwidth\nallegedly\nmcdonald\nstrict\ncopyright\n##illa\ninvestors\nmar\njam\nwitnesses\nsounding\nmiranda\nmichelle\nprivacy\nhugo\nharmony\n##pp\nvalid\nlynn\nglared\nnina\n102\nheadquartered\ndiving\nboarding\ngibson\n##ncy\nalbanian\nmarsh\nroutine\ndealt\nenhanced\ner\nintelligent\nsubstance\ntargeted\nenlisted\ndiscovers\nspinning\nobservations\npissed\nsmoking\nrebecca\ncapitol\nvisa\nvaried\ncostume\nseemingly\nindies\ncompensation\nsurgeon\nthursday\narsenal\nwestminster\nsuburbs\nrid\nanglican\n##ridge\nknots\nfoods\nalumni\nlighter\nfraser\nwhoever\nportal\nscandal\n##ray\ngavin\nadvised\ninstructor\nflooding\nterrorist\n##ale\nteenage\ninterim\nsenses\nduck\nteen\nthesis\nabby\neager\novercome\n##ile\nnewport\nglenn\nrises\nshame\n##cc\nprompted\npriority\nforgot\nbomber\nnicolas\nprotective\n360\ncartoon\nkatherine\nbreeze\nlonely\ntrusted\nhenderson\nrichardson\nrelax\nbanner\ncandy\npalms\nremarkable\n##rio\nlegends\ncricketer\nessay\nordained\nedmund\nrifles\ntrigger\n##uri\n##away\nsail\nalert\n1830\naudiences\npenn\nsussex\nsiblings\npursued\nindianapolis\nresist\nrosa\nconsequence\nsucceed\navoided\n1845\n##ulation\ninland\n##tie\n##nna\ncounsel\nprofession\nchronicle\nhurried\n##una\neyebrow\neventual\nbleeding\ninnovative\ncure\n##dom\ncommittees\naccounting\ncon\nscope\nhardy\nheather\ntenor\ngut\nherald\ncodes\ntore\nscales\nwagon\n##oo\nluxury\ntin\nprefer\nfountain\ntriangle\nbonds\ndarling\nconvoy\ndried\ntraced\nbeings\ntroy\naccidentally\nslam\nfindings\nsmelled\njoey\nlawyers\noutcome\nsteep\nbosnia\nconfiguration\nshifting\ntoll\nbrook\nperformers\nlobby\nphilosophical\nconstruct\nshrine\naggregate\nboot\ncox\nphenomenon\nsavage\ninsane\nsolely\nreynolds\nlifestyle\n##ima\nnationally\nholdings\nconsideration\nenable\nedgar\nmo\nmama\n##tein\nfights\nrelegation\nchances\natomic\nhub\nconjunction\nawkward\nreactions\ncurrency\nfinale\nkumar\nunderwent\nsteering\nelaborate\ngifts\ncomprising\nmelissa\nveins\nreasonable\nsunshine\nchi\nsolve\ntrails\ninhabited\nelimination\nethics\nhuh\nana\nmolly\nconsent\napartments\nlayout\nmarines\n##ces\nhunters\nbulk\n##oma\nhometown\n##wall\n##mont\ncracked\nreads\nneighbouring\nwithdrawn\nadmission\nwingspan\ndamned\nanthology\nlancashire\nbrands\nbatting\nforgive\ncuban\nawful\n##lyn\n104\ndimensions\nimagination\n##ade\ndante\n##ship\ntracking\ndesperately\ngoalkeeper\n##yne\ngroaned\nworkshops\nconfident\nburton\ngerald\nmilton\ncircus\nuncertain\nslope\ncopenhagen\nsophia\nfog\nphilosopher\nportraits\naccent\ncycling\nvarying\ngripped\nlarvae\ngarrett\nspecified\nscotia\nmature\nluther\nkurt\nrap\n##kes\naerial\n750\nferdinand\nheated\nes\ntransported\n##shan\nsafely\nnonetheless\n##orn\n##gal\nmotors\ndemanding\n##sburg\nstartled\n##brook\nally\ngenerate\ncaps\nghana\nstained\ndemo\nmentions\nbeds\nap\nafterward\ndiary\n##bling\nutility\n##iro\nrichards\n1837\nconspiracy\nconscious\nshining\nfootsteps\nobserver\ncyprus\nurged\nloyalty\ndeveloper\nprobability\nolive\nupgraded\ngym\nmiracle\ninsects\ngraves\n1844\nourselves\nhydrogen\namazon\nkatie\ntickets\npoets\n##pm\nplanes\n##pan\nprevention\nwitnessed\ndense\njin\nrandy\ntang\nwarehouse\nmonroe\nbang\narchived\nelderly\ninvestigations\nalec\ngranite\nmineral\nconflicts\ncontrolling\naboriginal\ncarlo\n##zu\nmechanics\nstan\nstark\nrhode\nskirt\nest\n##berry\nbombs\nrespected\n##horn\nimposed\nlimestone\ndeny\nnominee\nmemphis\ngrabbing\ndisabled\n##als\namusement\naa\nfrankfurt\ncorn\nreferendum\nvaries\nslowed\ndisk\nfirms\nunconscious\nincredible\nclue\nsue\n##zhou\ntwist\n##cio\njoins\nidaho\nchad\ndevelopers\ncomputing\ndestroyer\n103\nmortal\ntucker\nkingston\nchoices\nyu\ncarson\n1800\nos\nwhitney\ngeneva\npretend\ndimension\nstaged\nplateau\nmaya\n##une\nfreestyle\n##bc\nrovers\nhiv\n##ids\ntristan\nclassroom\nprospect\n##hus\nhonestly\ndiploma\nlied\nthermal\nauxiliary\nfeast\nunlikely\niata\n##tel\nmorocco\npounding\ntreasury\nlithuania\nconsiderably\n1841\ndish\n1812\ngeological\nmatching\nstumbled\ndestroying\nmarched\nbrien\nadvances\ncake\nnicole\nbelle\nsettling\nmeasuring\ndirecting\n##mie\ntuesday\nbassist\ncapabilities\nstunned\nfraud\ntorpedo\n##list\n##phone\nanton\nwisdom\nsurveillance\nruined\n##ulate\nlawsuit\nhealthcare\ntheorem\nhalls\ntrend\naka\nhorizontal\ndozens\nacquire\nlasting\nswim\nhawk\ngorgeous\nfees\nvicinity\ndecrease\nadoption\ntactics\n##ography\npakistani\n##ole\ndraws\n##hall\nwillie\nburke\nheath\nalgorithm\nintegral\npowder\nelliott\nbrigadier\njackie\ntate\nvarieties\ndarker\n##cho\nlately\ncigarette\nspecimens\nadds\n##ree\n##ensis\n##inger\nexploded\nfinalist\ncia\nmurders\nwilderness\narguments\nnicknamed\nacceptance\nonwards\nmanufacture\nrobertson\njets\ntampa\nenterprises\nblog\nloudly\ncomposers\nnominations\n1838\nai\nmalta\ninquiry\nautomobile\nhosting\nviii\nrays\ntilted\ngrief\nmuseums\nstrategies\nfurious\neuro\nequality\ncohen\npoison\nsurrey\nwireless\ngoverned\nridiculous\nmoses\n##esh\n##room\nvanished\n##ito\nbarnes\nattract\nmorrison\nistanbul\n##iness\nabsent\nrotation\npetition\njanet\n##logical\nsatisfaction\ncustody\ndeliberately\nobservatory\ncomedian\nsurfaces\npinyin\nnovelist\nstrictly\ncanterbury\noslo\nmonks\nembrace\nibm\njealous\nphotograph\ncontinent\ndorothy\nmarina\ndoc\nexcess\nholden\nallegations\nexplaining\nstack\navoiding\nlance\nstoryline\nmajesty\npoorly\nspike\ndos\nbradford\nraven\ntravis\nclassics\nproven\nvoltage\npillow\nfists\nbutt\n1842\ninterpreted\n##car\n1839\ngage\ntelegraph\nlens\npromising\nexpelled\ncasual\ncollector\nzones\n##min\nsilly\nnintendo\n##kh\n##bra\ndownstairs\nchef\nsuspicious\nafl\nflies\nvacant\nuganda\npregnancy\ncondemned\nlutheran\nestimates\ncheap\ndecree\nsaxon\nproximity\nstripped\nidiot\ndeposits\ncontrary\npresenter\nmagnus\nglacier\nim\noffense\nedwin\n##ori\nupright\n##long\nbolt\n##ois\ntoss\ngeographical\n##izes\nenvironments\ndelicate\nmarking\nabstract\nxavier\nnails\nwindsor\nplantation\noccurring\nequity\nsaskatchewan\nfears\ndrifted\nsequences\nvegetation\nrevolt\n##stic\n1843\nsooner\nfusion\nopposing\nnato\nskating\n1836\nsecretly\nruin\nlease\n##oc\nedit\n##nne\nflora\nanxiety\nruby\n##ological\n##mia\ntel\nbout\ntaxi\nemmy\nfrost\nrainbow\ncompounds\nfoundations\nrainfall\nassassination\nnightmare\ndominican\n##win\nachievements\ndeserve\norlando\nintact\narmenia\n##nte\ncalgary\nvalentine\n106\nmarion\nproclaimed\ntheodore\nbells\ncourtyard\nthigh\ngonzalez\nconsole\ntroop\nminimal\nmonte\neveryday\n##ence\n##if\nsupporter\nterrorism\nbuck\nopenly\npresbyterian\nactivists\ncarpet\n##iers\nrubbing\nuprising\n##yi\ncute\nconceived\nlegally\n##cht\nmillennium\ncello\nvelocity\nji\nrescued\ncardiff\n1835\nrex\nconcentrate\nsenators\nbeard\nrendered\nglowing\nbattalions\nscouts\ncompetitors\nsculptor\ncatalogue\narctic\nion\nraja\nbicycle\nwow\nglancing\nlawn\n##woman\ngentleman\nlighthouse\npublish\npredicted\ncalculated\n##val\nvariants\n##gne\nstrain\n##ui\nwinston\ndeceased\n##nus\ntouchdowns\nbrady\ncaleb\nsinking\nechoed\ncrush\nhon\nblessed\nprotagonist\nhayes\nendangered\nmagnitude\neditors\n##tine\nestimate\nresponsibilities\n##mel\nbackup\nlaying\nconsumed\nsealed\nzurich\nlovers\nfrustrated\n##eau\nahmed\nkicking\nmit\ntreasurer\n1832\nbiblical\nrefuse\nterrified\npump\nagrees\ngenuine\nimprisonment\nrefuses\nplymouth\n##hen\nlou\n##nen\ntara\ntrembling\nantarctic\nton\nlearns\n##tas\ncrap\ncrucial\nfaction\natop\n##borough\nwrap\nlancaster\nodds\nhopkins\nerik\nlyon\n##eon\nbros\n##ode\nsnap\nlocality\ntips\nempress\ncrowned\ncal\nacclaimed\nchuckled\n##ory\nclara\nsends\nmild\ntowel\n##fl\n##day\n##а\nwishing\nassuming\ninterviewed\n##bal\n##die\ninteractions\neden\ncups\nhelena\n##lf\nindie\nbeck\n##fire\nbatteries\nfilipino\nwizard\nparted\n##lam\ntraces\n##born\nrows\nidol\nalbany\ndelegates\n##ees\n##sar\ndiscussions\n##ex\nnotre\ninstructed\nbelgrade\nhighways\nsuggestion\nlauren\npossess\norientation\nalexandria\nabdul\nbeats\nsalary\nreunion\nludwig\nalright\nwagner\nintimate\npockets\nslovenia\nhugged\nbrighton\nmerchants\ncruel\nstole\ntrek\nslopes\nrepairs\nenrollment\npolitically\nunderlying\npromotional\ncounting\nboeing\n##bb\nisabella\nnaming\n##и\nkeen\nbacteria\nlisting\nseparately\nbelfast\nussr\n450\nlithuanian\nanybody\nribs\nsphere\nmartinez\ncock\nembarrassed\nproposals\nfragments\nnationals\n##fs\n##wski\npremises\nfin\n1500\nalpine\nmatched\nfreely\nbounded\njace\nsleeve\n##af\ngaming\npier\npopulated\nevident\n##like\nfrances\nflooded\n##dle\nfrightened\npour\ntrainer\nframed\nvisitor\nchallenging\npig\nwickets\n##fold\ninfected\nemail\n##pes\narose\n##aw\nreward\necuador\noblast\nvale\nch\nshuttle\n##usa\nbach\nrankings\nforbidden\ncornwall\naccordance\nsalem\nconsumers\nbruno\nfantastic\ntoes\nmachinery\nresolved\njulius\nremembering\npropaganda\niceland\nbombardment\ntide\ncontacts\nwives\n##rah\nconcerto\nmacdonald\nalbania\nimplement\ndaisy\ntapped\nsudan\nhelmet\nangela\nmistress\n##lic\ncrop\nsunk\nfinest\n##craft\nhostile\n##ute\n##tsu\nboxer\nfr\npaths\nadjusted\nhabit\nballot\nsupervision\nsoprano\n##zen\nbullets\nwicked\nsunset\nregiments\ndisappear\nlamp\nperforms\napp\n##gia\n##oa\nrabbit\ndigging\nincidents\nentries\n##cion\ndishes\n##oi\nintroducing\n##ati\n##fied\nfreshman\nslot\njill\ntackles\nbaroque\nbacks\n##iest\nlone\nsponsor\ndestiny\naltogether\nconvert\n##aro\nconsensus\nshapes\ndemonstration\nbasically\nfeminist\nauction\nartifacts\n##bing\nstrongest\ntwitter\nhalifax\n2019\nallmusic\nmighty\nsmallest\nprecise\nalexandra\nviola\n##los\n##ille\nmanuscripts\n##illo\ndancers\nari\nmanagers\nmonuments\nblades\nbarracks\nspringfield\nmaiden\nconsolidated\nelectron\n##end\nberry\nairing\nwheat\nnobel\ninclusion\nblair\npayments\ngeography\nbee\ncc\neleanor\nreact\n##hurst\nafc\nmanitoba\n##yu\nsu\nlineup\nfitness\nrecreational\ninvestments\nairborne\ndisappointment\n##dis\nedmonton\nviewing\n##row\nrenovation\n##cast\ninfant\nbankruptcy\nroses\naftermath\npavilion\n##yer\ncarpenter\nwithdrawal\nladder\n##hy\ndiscussing\npopped\nreliable\nagreements\nrochester\n##abad\ncurves\nbombers\n220\nrao\nreverend\ndecreased\nchoosing\n107\nstiff\nconsulting\nnaples\ncrawford\ntracy\nka\nribbon\ncops\n##lee\ncrushed\ndeciding\nunified\nteenager\naccepting\nflagship\nexplorer\npoles\nsanchez\ninspection\nrevived\nskilled\ninduced\nexchanged\nflee\nlocals\ntragedy\nswallow\nloading\nhanna\ndemonstrate\n##ela\nsalvador\nflown\ncontestants\ncivilization\n##ines\nwanna\nrhodes\nfletcher\nhector\nknocking\nconsiders\n##ough\nnash\nmechanisms\nsensed\nmentally\nwalt\nunclear\n##eus\nrenovated\nmadame\n##cks\ncrews\ngovernmental\n##hin\nundertaken\nmonkey\n##ben\n##ato\nfatal\narmored\ncopa\ncaves\ngovernance\ngrasp\nperception\ncertification\nfroze\ndamp\ntugged\nwyoming\n##rg\n##ero\nnewman\n##lor\nnerves\ncuriosity\ngraph\n115\n##ami\nwithdraw\ntunnels\ndull\nmeredith\nmoss\nexhibits\nneighbors\ncommunicate\naccuracy\nexplored\nraiders\nrepublicans\nsecular\nkat\nsuperman\npenny\ncriticised\n##tch\nfreed\nupdate\nconviction\nwade\nham\nlikewise\ndelegation\ngotta\ndoll\npromises\ntechnological\nmyth\nnationality\nresolve\nconvent\n##mark\nsharon\ndig\nsip\ncoordinator\nentrepreneur\nfold\n##dine\ncapability\ncouncillor\nsynonym\nblown\nswan\ncursed\n1815\njonas\nhaired\nsofa\ncanvas\nkeeper\nrivalry\n##hart\nrapper\nspeedway\nswords\npostal\nmaxwell\nestonia\npotter\nrecurring\n##nn\n##ave\nerrors\n##oni\ncognitive\n1834\n##²\nclaws\nnadu\nroberto\nbce\nwrestler\nellie\n##ations\ninfinite\nink\n##tia\npresumably\nfinite\nstaircase\n108\nnoel\npatricia\nnacional\n##cation\nchill\neternal\ntu\npreventing\nprussia\nfossil\nlimbs\n##logist\nernst\nfrog\nperez\nrene\n##ace\npizza\nprussian\n##ios\n##vy\nmolecules\nregulatory\nanswering\nopinions\nsworn\nlengths\nsupposedly\nhypothesis\nupward\nhabitats\nseating\nancestors\ndrank\nyield\nhd\nsynthesis\nresearcher\nmodest\n##var\nmothers\npeered\nvoluntary\nhomeland\n##the\nacclaim\n##igan\nstatic\nvalve\nluxembourg\nalto\ncarroll\nfe\nreceptor\nnorton\nambulance\n##tian\njohnston\ncatholics\ndepicting\njointly\nelephant\ngloria\nmentor\nbadge\nahmad\ndistinguish\nremarked\ncouncils\nprecisely\nallison\nadvancing\ndetection\ncrowded\n##10\ncooperative\nankle\nmercedes\ndagger\nsurrendered\npollution\ncommit\nsubway\njeffrey\nlesson\nsculptures\nprovider\n##fication\nmembrane\ntimothy\nrectangular\nfiscal\nheating\nteammate\nbasket\nparticle\nanonymous\ndeployment\n##ple\nmissiles\ncourthouse\nproportion\nshoe\nsec\n##ller\ncomplaints\nforbes\nblacks\nabandon\nremind\nsizes\noverwhelming\nautobiography\nnatalie\n##awa\nrisks\ncontestant\ncountryside\nbabies\nscorer\ninvaded\nenclosed\nproceed\nhurling\ndisorders\n##cu\nreflecting\ncontinuously\ncruiser\ngraduates\nfreeway\ninvestigated\nore\ndeserved\nmaid\nblocking\nphillip\njorge\nshakes\ndove\nmann\nvariables\nlacked\nburden\naccompanying\nque\nconsistently\norganizing\nprovisional\ncomplained\nendless\n##rm\ntubes\njuice\ngeorges\nkrishna\nmick\nlabels\nthriller\n##uch\nlaps\narcade\nsage\nsnail\n##table\nshannon\nfi\nlaurence\nseoul\nvacation\npresenting\nhire\nchurchill\nsurprisingly\nprohibited\nsavannah\ntechnically\n##oli\n170\n##lessly\ntestimony\nsuited\nspeeds\ntoys\nromans\nmlb\nflowering\nmeasurement\ntalented\nkay\nsettings\ncharleston\nexpectations\nshattered\nachieving\ntriumph\nceremonies\nportsmouth\nlanes\nmandatory\nloser\nstretching\ncologne\nrealizes\nseventy\ncornell\ncareers\nwebb\n##ulating\namericas\nbudapest\nava\nsuspicion\n##ison\nyo\nconrad\n##hai\nsterling\njessie\nrector\n##az\n1831\ntransform\norganize\nloans\nchristine\nvolcanic\nwarrant\nslender\nsummers\nsubfamily\nnewer\ndanced\ndynamics\nrhine\nproceeds\nheinrich\ngastropod\ncommands\nsings\nfacilitate\neaster\nra\npositioned\nresponses\nexpense\nfruits\nyanked\nimported\n25th\nvelvet\nvic\nprimitive\ntribune\nbaldwin\nneighbourhood\ndonna\nrip\nhay\npr\n##uro\n1814\nespn\nwelcomed\n##aria\nqualifier\nglare\nhighland\ntiming\n##cted\nshells\neased\ngeometry\nlouder\nexciting\nslovakia\n##sion\n##iz\n##lot\nsavings\nprairie\n##ques\nmarching\nrafael\ntonnes\n##lled\ncurtain\npreceding\nshy\nheal\ngreene\nworthy\n##pot\ndetachment\nbury\nsherman\n##eck\nreinforced\nseeks\nbottles\ncontracted\nduchess\noutfit\nwalsh\n##sc\nmickey\n##ase\ngeoffrey\narcher\nsqueeze\ndawson\neliminate\ninvention\n##enberg\nneal\n##eth\nstance\ndealer\ncoral\nmaple\nretire\npolo\nsimplified\n##ht\n1833\nhid\nwatts\nbackwards\njules\n##oke\ngenesis\nmt\nframes\nrebounds\nburma\nwoodland\nmoist\nsantos\nwhispers\ndrained\nsubspecies\n##aa\nstreaming\nulster\nburnt\ncorrespondence\nmaternal\ngerard\ndenis\nstealing\n##load\ngenius\nduchy\n##oria\ninaugurated\nmomentum\nsuits\nplacement\nsovereign\nclause\nthames\n##hara\nconfederation\nreservation\nsketch\nyankees\nlets\nrotten\ncharm\nhal\nverses\nultra\ncommercially\ndot\nsalon\ncitation\nadopt\nwinnipeg\nmist\nallocated\ncairo\n##boy\njenkins\ninterference\nobjectives\n##wind\n1820\nportfolio\narmoured\nsectors\n##eh\ninitiatives\n##world\nintegrity\nexercises\nrobe\ntap\nab\ngazed\n##tones\ndistracted\nrulers\n111\nfavorable\njerome\ntended\ncart\nfactories\n##eri\ndiplomat\nvalued\ngravel\ncharitable\n##try\ncalvin\nexploring\nchang\nshepherd\nterrace\npdf\npupil\n##ural\nreflects\nups\n##rch\ngovernors\nshelf\ndepths\n##nberg\ntrailed\ncrest\ntackle\n##nian\n##ats\nhatred\n##kai\nclare\nmakers\nethiopia\nlongtime\ndetected\nembedded\nlacking\nslapped\nrely\nthomson\nanticipation\niso\nmorton\nsuccessive\nagnes\nscreenwriter\nstraightened\nphilippe\nplaywright\nhaunted\nlicence\niris\nintentions\nsutton\n112\nlogical\ncorrectly\n##weight\nbranded\nlicked\ntipped\nsilva\nricky\nnarrator\nrequests\n##ents\ngreeted\nsupernatural\ncow\n##wald\nlung\nrefusing\nemployer\nstrait\ngaelic\nliner\n##piece\nzoe\nsabha\n##mba\ndriveway\nharvest\nprints\nbates\nreluctantly\nthreshold\nalgebra\nira\nwherever\ncoupled\n240\nassumption\npicks\n##air\ndesigners\nraids\ngentlemen\n##ean\nroller\nblowing\nleipzig\nlocks\nscrew\ndressing\nstrand\n##lings\nscar\ndwarf\ndepicts\n##nu\nnods\n##mine\ndiffer\nboris\n##eur\nyuan\nflip\n##gie\nmob\ninvested\nquestioning\napplying\n##ture\nshout\n##sel\ngameplay\nblamed\nillustrations\nbothered\nweakness\nrehabilitation\n##of\n##zes\nenvelope\nrumors\nminers\nleicester\nsubtle\nkerry\n##ico\nferguson\n##fu\npremiership\nne\n##cat\nbengali\nprof\ncatches\nremnants\ndana\n##rily\nshouting\npresidents\nbaltic\nought\nghosts\ndances\nsailors\nshirley\nfancy\ndominic\n##bie\nmadonna\n##rick\nbark\nbuttons\ngymnasium\nashes\nliver\ntoby\noath\nprovidence\ndoyle\nevangelical\nnixon\ncement\ncarnegie\nembarked\nhatch\nsurroundings\nguarantee\nneeding\npirate\nessence\n##bee\nfilter\ncrane\nhammond\nprojected\nimmune\npercy\ntwelfth\n##ult\nregent\ndoctoral\ndamon\nmikhail\n##ichi\nlu\ncritically\nelect\nrealised\nabortion\nacute\nscreening\nmythology\nsteadily\n##fc\nfrown\nnottingham\nkirk\nwa\nminneapolis\n##rra\nmodule\nalgeria\nmc\nnautical\nencounters\nsurprising\nstatues\navailability\nshirts\npie\nalma\nbrows\nmunster\nmack\nsoup\ncrater\ntornado\nsanskrit\ncedar\nexplosive\nbordered\ndixon\nplanets\nstamp\nexam\nhappily\n##bble\ncarriers\nkidnapped\n##vis\naccommodation\nemigrated\n##met\nknockout\ncorrespondent\nviolation\nprofits\npeaks\nlang\nspecimen\nagenda\nancestry\npottery\nspelling\nequations\nobtaining\nki\nlinking\n1825\ndebris\nasylum\n##20\nbuddhism\nteddy\n##ants\ngazette\n##nger\n##sse\ndental\neligibility\nutc\nfathers\naveraged\nzimbabwe\nfrancesco\ncoloured\nhissed\ntranslator\nlynch\nmandate\nhumanities\nmackenzie\nuniforms\nlin\n##iana\n##gio\nasset\nmhz\nfitting\nsamantha\ngenera\nwei\nrim\nbeloved\nshark\nriot\nentities\nexpressions\nindo\ncarmen\nslipping\nowing\nabbot\nneighbor\nsidney\n##av\nrats\nrecommendations\nencouraging\nsquadrons\nanticipated\ncommanders\nconquered\n##oto\ndonations\ndiagnosed\n##mond\ndivide\n##iva\nguessed\ndecoration\nvernon\nauditorium\nrevelation\nconversations\n##kers\n##power\nherzegovina\ndash\nalike\nprotested\nlateral\nherman\naccredited\nmg\n##gent\nfreeman\nmel\nfiji\ncrow\ncrimson\n##rine\nlivestock\n##pped\nhumanitarian\nbored\noz\nwhip\n##lene\n##ali\nlegitimate\nalter\ngrinning\nspelled\nanxious\noriental\nwesley\n##nin\n##hole\ncarnival\ncontroller\ndetect\n##ssa\nbowed\neducator\nkosovo\nmacedonia\n##sin\noccupy\nmastering\nstephanie\njaneiro\npara\nunaware\nnurses\nnoon\n135\ncam\nhopefully\nranger\ncombine\nsociology\npolar\nrica\n##eer\nneill\n##sman\nholocaust\n##ip\ndoubled\nlust\n1828\n109\ndecent\ncooling\nunveiled\n##card\n1829\nnsw\nhomer\nchapman\nmeyer\n##gin\ndive\nmae\nreagan\nexpertise\n##gled\ndarwin\nbrooke\nsided\nprosecution\ninvestigating\ncomprised\npetroleum\ngenres\nreluctant\ndifferently\ntrilogy\njohns\nvegetables\ncorpse\nhighlighted\nlounge\npension\nunsuccessfully\nelegant\naided\nivory\nbeatles\namelia\ncain\ndubai\nsunny\nimmigrant\nbabe\nclick\n##nder\nunderwater\npepper\ncombining\nmumbled\natlas\nhorns\naccessed\nballad\nphysicians\nhomeless\ngestured\nrpm\nfreak\nlouisville\ncorporations\npatriots\nprizes\nrational\nwarn\nmodes\ndecorative\novernight\ndin\ntroubled\nphantom\n##ort\nmonarch\nsheer\n##dorf\ngenerals\nguidelines\norgans\naddresses\n##zon\nenhance\ncurling\nparishes\ncord\n##kie\nlinux\ncaesar\ndeutsche\nbavaria\n##bia\ncoleman\ncyclone\n##eria\nbacon\npetty\n##yama\n##old\nhampton\ndiagnosis\n1824\nthrows\ncomplexity\nrita\ndisputed\n##₃\npablo\n##sch\nmarketed\ntrafficking\n##ulus\nexamine\nplague\nformats\n##oh\nvault\nfaithful\n##bourne\nwebster\n##ox\nhighlights\n##ient\n##ann\nphones\nvacuum\nsandwich\nmodeling\n##gated\nbolivia\nclergy\nqualities\nisabel\n##nas\n##ars\nwears\nscreams\nreunited\nannoyed\nbra\n##ancy\n##rate\ndifferential\ntransmitter\ntattoo\ncontainer\npoker\n##och\nexcessive\nresides\ncowboys\n##tum\naugustus\ntrash\nproviders\nstatute\nretreated\nbalcony\nreversed\nvoid\nstorey\npreceded\nmasses\nleap\nlaughs\nneighborhoods\nwards\nschemes\nfalcon\nsanto\nbattlefield\npad\nronnie\nthread\nlesbian\nvenus\n##dian\nbeg\nsandstone\ndaylight\npunched\ngwen\nanalog\nstroked\nwwe\nacceptable\nmeasurements\ndec\ntoxic\n##kel\nadequate\nsurgical\neconomist\nparameters\nvarsity\n##sberg\nquantity\nella\n##chy\n##rton\ncountess\ngenerating\nprecision\ndiamonds\nexpressway\nga\n##ı\n1821\nuruguay\ntalents\ngalleries\nexpenses\nscanned\ncolleague\noutlets\nryder\nlucien\n##ila\nparamount\n##bon\nsyracuse\ndim\nfangs\ngown\nsweep\n##sie\ntoyota\nmissionaries\nwebsites\n##nsis\nsentences\nadviser\nval\ntrademark\nspells\n##plane\npatience\nstarter\nslim\n##borg\ntoe\nincredibly\nshoots\nelliot\nnobility\n##wyn\ncowboy\nendorsed\ngardner\ntendency\npersuaded\norganisms\nemissions\nkazakhstan\namused\nboring\nchips\nthemed\n##hand\nllc\nconstantinople\nchasing\nsystematic\nguatemala\nborrowed\nerin\ncarey\n##hard\nhighlands\nstruggles\n1810\n##ifying\n##ced\nwong\nexceptions\ndevelops\nenlarged\nkindergarten\ncastro\n##ern\n##rina\nleigh\nzombie\njuvenile\n##most\nconsul\n##nar\nsailor\nhyde\nclarence\nintensive\npinned\nnasty\nuseless\njung\nclayton\nstuffed\nexceptional\nix\napostolic\n230\ntransactions\n##dge\nexempt\nswinging\ncove\nreligions\n##ash\nshields\ndairy\nbypass\n190\npursuing\nbug\njoyce\nbombay\nchassis\nsouthampton\nchat\ninteract\nredesignated\n##pen\nnascar\npray\nsalmon\nrigid\nregained\nmalaysian\ngrim\npublicity\nconstituted\ncapturing\ntoilet\ndelegate\npurely\ntray\ndrift\nloosely\nstriker\nweakened\ntrinidad\nmitch\nitv\ndefines\ntransmitted\nming\nscarlet\nnodding\nfitzgerald\nfu\nnarrowly\nsp\ntooth\nstandings\nvirtue\n##₁\n##wara\n##cting\nchateau\ngloves\nlid\n##nel\nhurting\nconservatory\n##pel\nsinclair\nreopened\nsympathy\nnigerian\nstrode\nadvocated\noptional\nchronic\ndischarge\n##rc\nsuck\ncompatible\nlaurel\nstella\nshi\nfails\nwage\ndodge\n128\ninformal\nsorts\nlevi\nbuddha\nvillagers\n##aka\nchronicles\nheavier\nsummoned\ngateway\n3000\neleventh\njewelry\ntranslations\naccordingly\nseas\n##ency\nfiber\npyramid\ncubic\ndragging\n##ista\ncaring\n##ops\nandroid\ncontacted\nlunar\n##dt\nkai\nlisbon\npatted\n1826\nsacramento\ntheft\nmadagascar\nsubtropical\ndisputes\nta\nholidays\npiper\nwillow\nmare\ncane\nitunes\nnewfoundland\nbenny\ncompanions\ndong\nraj\nobserve\nroar\ncharming\nplaque\ntibetan\nfossils\nenacted\nmanning\nbubble\ntina\ntanzania\n##eda\n##hir\nfunk\nswamp\ndeputies\ncloak\nufc\nscenario\npar\nscratch\nmetals\nanthem\nguru\nengaging\nspecially\n##boat\ndialects\nnineteen\ncecil\nduet\ndisability\nmessenger\nunofficial\n##lies\ndefunct\neds\nmoonlight\ndrainage\nsurname\npuzzle\nhonda\nswitching\nconservatives\nmammals\nknox\nbroadcaster\nsidewalk\ncope\n##ried\nbenson\nprinces\npeterson\n##sal\nbedford\nsharks\neli\nwreck\nalberto\ngasp\narchaeology\nlgbt\nteaches\nsecurities\nmadness\ncompromise\nwaving\ncoordination\ndavidson\nvisions\nleased\npossibilities\neighty\njun\nfernandez\nenthusiasm\nassassin\nsponsorship\nreviewer\nkingdoms\nestonian\nlaboratories\n##fy\n##nal\napplies\nverb\ncelebrations\n##zzo\nrowing\nlightweight\nsadness\nsubmit\nmvp\nbalanced\ndude\n##vas\nexplicitly\nmetric\nmagnificent\nmound\nbrett\nmohammad\nmistakes\nirregular\n##hing\n##ass\nsanders\nbetrayed\nshipped\nsurge\n##enburg\nreporters\ntermed\ngeorg\npity\nverbal\nbulls\nabbreviated\nenabling\nappealed\n##are\n##atic\nsicily\nsting\nheel\nsweetheart\nbart\nspacecraft\nbrutal\nmonarchy\n##tter\naberdeen\ncameo\ndiane\n##ub\nsurvivor\nclyde\n##aries\ncomplaint\n##makers\nclarinet\ndelicious\nchilean\nkarnataka\ncoordinates\n1818\npanties\n##rst\npretending\nar\ndramatically\nkiev\nbella\ntends\ndistances\n113\ncatalog\nlaunching\ninstances\ntelecommunications\nportable\nlindsay\nvatican\n##eim\nangles\naliens\nmarker\nstint\nscreens\nbolton\n##rne\njudy\nwool\nbenedict\nplasma\neuropa\nspark\nimaging\nfilmmaker\nswiftly\n##een\ncontributor\n##nor\nopted\nstamps\napologize\nfinancing\nbutter\ngideon\nsophisticated\nalignment\navery\nchemicals\nyearly\nspeculation\nprominence\nprofessionally\n##ils\nimmortal\ninstitutional\ninception\nwrists\nidentifying\ntribunal\nderives\ngains\n##wo\npapal\npreference\nlinguistic\nvince\noperative\nbrewery\n##ont\nunemployment\nboyd\n##ured\n##outs\nalbeit\nprophet\n1813\nbi\n##rr\n##face\n##rad\nquarterly\nasteroid\ncleaned\nradius\ntemper\n##llen\ntelugu\njerk\nviscount\nmenu\n##ote\nglimpse\n##aya\nyacht\nhawaiian\nbaden\n##rl\nlaptop\nreadily\n##gu\nmonetary\noffshore\nscots\nwatches\n##yang\n##arian\nupgrade\nneedle\nxbox\nlea\nencyclopedia\nflank\nfingertips\n##pus\ndelight\nteachings\nconfirm\nroth\nbeaches\nmidway\nwinters\n##iah\nteasing\ndaytime\nbeverly\ngambling\nbonnie\n##backs\nregulated\nclement\nhermann\ntricks\nknot\n##shing\n##uring\n##vre\ndetached\necological\nowed\nspecialty\nbyron\ninventor\nbats\nstays\nscreened\nunesco\nmidland\ntrim\naffection\n##ander\n##rry\njess\nthoroughly\nfeedback\n##uma\nchennai\nstrained\nheartbeat\nwrapping\novertime\npleaded\n##sworth\nmon\nleisure\noclc\n##tate\n##ele\nfeathers\nangelo\nthirds\nnuts\nsurveys\nclever\ngill\ncommentator\n##dos\ndarren\nrides\ngibraltar\n##nc\n##mu\ndissolution\ndedication\nshin\nmeals\nsaddle\nelvis\nreds\nchaired\ntaller\nappreciation\nfunctioning\nniece\nfavored\nadvocacy\nrobbie\ncriminals\nsuffolk\nyugoslav\npassport\nconstable\ncongressman\nhastings\nvera\n##rov\nconsecrated\nsparks\necclesiastical\nconfined\n##ovich\nmuller\nfloyd\nnora\n1822\npaved\n1827\ncumberland\nned\nsaga\nspiral\n##flow\nappreciated\nyi\ncollaborative\ntreating\nsimilarities\nfeminine\nfinishes\n##ib\njade\nimport\n##nse\n##hot\nchampagne\nmice\nsecuring\ncelebrities\nhelsinki\nattributes\n##gos\ncousins\nphases\nache\nlucia\ngandhi\nsubmission\nvicar\nspear\nshine\ntasmania\nbiting\ndetention\nconstitute\ntighter\nseasonal\n##gus\nterrestrial\nmatthews\n##oka\neffectiveness\nparody\nphilharmonic\n##onic\n1816\nstrangers\nencoded\nconsortium\nguaranteed\nregards\nshifts\ntortured\ncollision\nsupervisor\ninform\nbroader\ninsight\ntheaters\narmour\nemeritus\nblink\nincorporates\nmapping\n##50\n##ein\nhandball\nflexible\n##nta\nsubstantially\ngenerous\nthief\n##own\ncarr\nloses\n1793\nprose\nucla\nromeo\ngeneric\nmetallic\nrealization\ndamages\nmk\ncommissioners\nzach\ndefault\n##ther\nhelicopters\nlengthy\nstems\nspa\npartnered\nspectators\nrogue\nindication\npenalties\nteresa\n1801\nsen\n##tric\ndalton\n##wich\nirving\nphotographic\n##vey\ndell\ndeaf\npeters\nexcluded\nunsure\n##vable\npatterson\ncrawled\n##zio\nresided\nwhipped\nlatvia\nslower\necole\npipes\nemployers\nmaharashtra\ncomparable\nva\ntextile\npageant\n##gel\nalphabet\nbinary\nirrigation\nchartered\nchoked\nantoine\noffs\nwaking\nsupplement\n##wen\nquantities\ndemolition\nregain\nlocate\nurdu\nfolks\nalt\n114\n##mc\nscary\nandreas\nwhites\n##ava\nclassrooms\nmw\naesthetic\npublishes\nvalleys\nguides\ncubs\njohannes\nbryant\nconventions\naffecting\n##itt\ndrain\nawesome\nisolation\nprosecutor\nambitious\napology\ncaptive\ndowns\natmospheric\nlorenzo\naisle\nbeef\nfoul\n##onia\nkidding\ncomposite\ndisturbed\nillusion\nnatives\n##ffer\nemi\nrockets\nriverside\nwartime\npainters\nadolf\nmelted\n##ail\nuncertainty\nsimulation\nhawks\nprogressed\nmeantime\nbuilder\nspray\nbreach\nunhappy\nregina\nrussians\n##urg\ndetermining\n##tation\ntram\n1806\n##quin\naging\n##12\n1823\ngarion\nrented\nmister\ndiaz\nterminated\nclip\n1817\ndepend\nnervously\ndisco\nowe\ndefenders\nshiva\nnotorious\ndisbelief\nshiny\nworcester\n##gation\n##yr\ntrailing\nundertook\nislander\nbelarus\nlimitations\nwatershed\nfuller\noverlooking\nutilized\nraphael\n1819\nsynthetic\nbreakdown\nklein\n##nate\nmoaned\nmemoir\nlamb\npracticing\n##erly\ncellular\narrows\nexotic\n##graphy\nwitches\n117\ncharted\nrey\nhut\nhierarchy\nsubdivision\nfreshwater\ngiuseppe\naloud\nreyes\nqatar\nmarty\nsideways\nutterly\nsexually\njude\nprayers\nmccarthy\nsoftball\nblend\ndamien\n##gging\n##metric\nwholly\nerupted\nlebanese\nnegro\nrevenues\ntasted\ncomparative\nteamed\ntransaction\nlabeled\nmaori\nsovereignty\nparkway\ntrauma\ngran\nmalay\n121\nadvancement\ndescendant\n2020\nbuzz\nsalvation\ninventory\nsymbolic\n##making\nantarctica\nmps\n##gas\n##bro\nmohammed\nmyanmar\nholt\nsubmarines\ntones\n##lman\nlocker\npatriarch\nbangkok\nemerson\nremarks\npredators\nkin\nafghan\nconfession\nnorwich\nrental\nemerge\nadvantages\n##zel\nrca\n##hold\nshortened\nstorms\naidan\n##matic\nautonomy\ncompliance\n##quet\ndudley\natp\n##osis\n1803\nmotto\ndocumentation\nsummary\nprofessors\nspectacular\nchristina\narchdiocese\nflashing\ninnocence\nremake\n##dell\npsychic\nreef\nscare\nemploy\nrs\nsticks\nmeg\ngus\nleans\n##ude\naccompany\nbergen\ntomas\n##iko\ndoom\nwages\npools\n##nch\n##bes\nbreasts\nscholarly\nalison\noutline\nbrittany\nbreakthrough\nwillis\nrealistic\n##cut\n##boro\ncompetitor\n##stan\npike\npicnic\nicon\ndesigning\ncommercials\nwashing\nvillain\nskiing\nmicro\ncostumes\nauburn\nhalted\nexecutives\n##hat\nlogistics\ncycles\nvowel\napplicable\nbarrett\nexclaimed\neurovision\neternity\nramon\n##umi\n##lls\nmodifications\nsweeping\ndisgust\n##uck\ntorch\naviv\nensuring\nrude\ndusty\nsonic\ndonovan\noutskirts\ncu\npathway\n##band\n##gun\n##lines\ndisciplines\nacids\ncadet\npaired\n##40\nsketches\n##sive\nmarriages\n##⁺\nfolding\npeers\nslovak\nimplies\nadmired\n##beck\n1880s\nleopold\ninstinct\nattained\nweston\nmegan\nhorace\n##ination\ndorsal\ningredients\nevolutionary\n##its\ncomplications\ndeity\nlethal\nbrushing\nlevy\ndeserted\ninstitutes\nposthumously\ndelivering\ntelescope\ncoronation\nmotivated\nrapids\nluc\nflicked\npays\nvolcano\ntanner\nweighed\n##nica\ncrowds\nfrankie\ngifted\naddressing\ngranddaughter\nwinding\n##rna\nconstantine\ngomez\n##front\nlandscapes\nrudolf\nanthropology\nslate\nwerewolf\n##lio\nastronomy\ncirca\nrouge\ndreaming\nsack\nknelt\ndrowned\nnaomi\nprolific\ntracked\nfreezing\nherb\n##dium\nagony\nrandall\ntwisting\nwendy\ndeposit\ntouches\nvein\nwheeler\n##bbled\n##bor\nbatted\nretaining\ntire\npresently\ncompare\nspecification\ndaemon\nnigel\n##grave\nmerry\nrecommendation\nczechoslovakia\nsandra\nng\nroma\n##sts\nlambert\ninheritance\nsheikh\nwinchester\ncries\nexamining\n##yle\ncomeback\ncuisine\nnave\n##iv\nko\nretrieve\ntomatoes\nbarker\npolished\ndefining\nirene\nlantern\npersonalities\nbegging\ntract\nswore\n1809\n175\n##gic\nomaha\nbrotherhood\n##rley\nhaiti\n##ots\nexeter\n##ete\n##zia\nsteele\ndumb\npearson\n210\nsurveyed\nelisabeth\ntrends\n##ef\nfritz\n##rf\npremium\nbugs\nfraction\ncalmly\nviking\n##birds\ntug\ninserted\nunusually\n##ield\nconfronted\ndistress\ncrashing\nbrent\nturks\nresign\n##olo\ncambodia\ngabe\nsauce\n##kal\nevelyn\n116\nextant\nclusters\nquarry\nteenagers\nluna\n##lers\n##ister\naffiliation\ndrill\n##ashi\npanthers\nscenic\nlibya\nanita\nstrengthen\ninscriptions\n##cated\nlace\nsued\njudith\nriots\n##uted\nmint\n##eta\npreparations\nmidst\ndub\nchallenger\n##vich\nmock\ncf\ndisplaced\nwicket\nbreaths\nenables\nschmidt\nanalyst\n##lum\nag\nhighlight\nautomotive\naxe\njosef\nnewark\nsufficiently\nresembles\n50th\n##pal\nflushed\nmum\ntraits\n##ante\ncommodore\nincomplete\nwarming\ntitular\nceremonial\nethical\n118\ncelebrating\neighteenth\ncao\nlima\nmedalist\nmobility\nstrips\nsnakes\n##city\nminiature\nzagreb\nbarton\nescapes\numbrella\nautomated\ndoubted\ndiffers\ncooled\ngeorgetown\ndresden\ncooked\nfade\nwyatt\nrna\njacobs\ncarlton\nabundant\nstereo\nboost\nmadras\ninning\n##hia\nspur\nip\nmalayalam\nbegged\nosaka\ngroan\nescaping\ncharging\ndose\nvista\n##aj\nbud\npapa\ncommunists\nadvocates\nedged\ntri\n##cent\nresemble\npeaking\nnecklace\nfried\nmontenegro\nsaxony\ngoose\nglances\nstuttgart\ncurator\nrecruit\ngrocery\nsympathetic\n##tting\n##fort\n127\nlotus\nrandolph\nancestor\n##rand\nsucceeding\njupiter\n1798\nmacedonian\n##heads\nhiking\n1808\nhanding\nfischer\n##itive\ngarbage\nnode\n##pies\nprone\nsingular\npapua\ninclined\nattractions\nitalia\npouring\nmotioned\ngrandma\ngarnered\njacksonville\ncorp\nego\nringing\naluminum\n##hausen\nordering\n##foot\ndrawer\ntraders\nsynagogue\n##play\n##kawa\nresistant\nwandering\nfragile\nfiona\nteased\nvar\nhardcore\nsoaked\njubilee\ndecisive\nexposition\nmercer\nposter\nvalencia\nhale\nkuwait\n1811\n##ises\n##wr\n##eed\ntavern\ngamma\n122\njohan\n##uer\nairways\namino\ngil\n##ury\nvocational\ndomains\ntorres\n##sp\ngenerator\nfolklore\noutcomes\n##keeper\ncanberra\nshooter\nfl\nbeams\nconfrontation\n##lling\n##gram\nfeb\naligned\nforestry\npipeline\njax\nmotorway\nconception\ndecay\n##tos\ncoffin\n##cott\nstalin\n1805\nescorted\nminded\n##nam\nsitcom\npurchasing\ntwilight\nveronica\nadditions\npassive\ntensions\nstraw\n123\nfrequencies\n1804\nrefugee\ncultivation\n##iate\nchristie\nclary\nbulletin\ncrept\ndisposal\n##rich\n##zong\nprocessor\ncrescent\n##rol\nbmw\nemphasized\nwhale\nnazis\naurora\n##eng\ndwelling\nhauled\nsponsors\ntoledo\nmega\nideology\ntheatres\ntessa\ncerambycidae\nsaves\nturtle\ncone\nsuspects\nkara\nrusty\nyelling\ngreeks\nmozart\nshades\ncocked\nparticipant\n##tro\nshire\nspit\nfreeze\nnecessity\n##cos\ninmates\nnielsen\ncouncillors\nloaned\nuncommon\nomar\npeasants\nbotanical\noffspring\ndaniels\nformations\njokes\n1794\npioneers\nsigma\nlicensing\n##sus\nwheelchair\npolite\n1807\nliquor\npratt\ntrustee\n##uta\nforewings\nballoon\n##zz\nkilometre\ncamping\nexplicit\ncasually\nshawn\nfoolish\nteammates\nnm\nhassan\ncarrie\njudged\nsatisfy\nvanessa\nknives\nselective\ncnn\nflowed\n##lice\neclipse\nstressed\neliza\nmathematician\ncease\ncultivated\n##roy\ncommissions\nbrowns\n##ania\ndestroyers\nsheridan\nmeadow\n##rius\nminerals\n##cial\ndownstream\nclash\ngram\nmemoirs\nventures\nbaha\nseymour\narchie\nmidlands\nedith\nfare\nflynn\ninvite\ncanceled\ntiles\nstabbed\nboulder\nincorporate\namended\ncamden\nfacial\nmollusk\nunreleased\ndescriptions\nyoga\ngrabs\n550\nraises\nramp\nshiver\n##rose\ncoined\npioneering\ntunes\nqing\nwarwick\ntops\n119\nmelanie\ngiles\n##rous\nwandered\n##inal\nannexed\nnov\n30th\nunnamed\n##ished\norganizational\nairplane\nnormandy\nstoke\nwhistle\nblessing\nviolations\nchased\nholders\nshotgun\n##ctic\noutlet\nreactor\n##vik\ntires\ntearing\nshores\nfortified\nmascot\nconstituencies\nnc\ncolumnist\nproductive\ntibet\n##rta\nlineage\nhooked\noct\ntapes\njudging\ncody\n##gger\nhansen\nkashmir\ntriggered\n##eva\nsolved\ncliffs\n##tree\nresisted\nanatomy\nprotesters\ntransparent\nimplied\n##iga\ninjection\nmattress\nexcluding\n##mbo\ndefenses\nhelpless\ndevotion\n##elli\ngrowl\nliberals\nweber\nphenomena\natoms\nplug\n##iff\nmortality\napprentice\nhowe\nconvincing\naaa\nswimmer\nbarber\nleone\npromptly\nsodium\ndef\nnowadays\narise\n##oning\ngloucester\ncorrected\ndignity\nnorm\nerie\n##ders\nelders\nevacuated\nsylvia\ncompression\n##yar\nhartford\npose\nbackpack\nreasoning\naccepts\n24th\nwipe\nmillimetres\nmarcel\n##oda\ndodgers\nalbion\n1790\noverwhelmed\naerospace\noaks\n1795\nshowcase\nacknowledge\nrecovering\nnolan\nashe\nhurts\ngeology\nfashioned\ndisappearance\nfarewell\nswollen\nshrug\nmarquis\nwimbledon\n124\nrue\n1792\ncommemorate\nreduces\nexperiencing\ninevitable\ncalcutta\nintel\n##court\nmurderer\nsticking\nfisheries\nimagery\nbloom\n280\nbrake\n##inus\ngustav\nhesitation\nmemorable\npo\nviral\nbeans\naccidents\ntunisia\nantenna\nspilled\nconsort\ntreatments\naye\nperimeter\n##gard\ndonation\nhostage\nmigrated\nbanker\naddiction\napex\nlil\ntrout\n##ously\nconscience\n##nova\nrams\nsands\ngenome\npassionate\ntroubles\n##lets\n##set\namid\n##ibility\n##ret\nhiggins\nexceed\nvikings\n##vie\npayne\n##zan\nmuscular\n##ste\ndefendant\nsucking\n##wal\nibrahim\nfuselage\nclaudia\nvfl\neuropeans\nsnails\ninterval\n##garh\npreparatory\nstatewide\ntasked\nlacrosse\nviktor\n##lation\nangola\n##hra\nflint\nimplications\nemploys\nteens\npatrons\nstall\nweekends\nbarriers\nscrambled\nnucleus\ntehran\njenna\nparsons\nlifelong\nrobots\ndisplacement\n5000\n##bles\nprecipitation\n##gt\nknuckles\nclutched\n1802\nmarrying\necology\nmarx\naccusations\ndeclare\nscars\nkolkata\nmat\nmeadows\nbermuda\nskeleton\nfinalists\nvintage\ncrawl\ncoordinate\naffects\nsubjected\norchestral\nmistaken\n##tc\nmirrors\ndipped\nrelied\n260\narches\ncandle\n##nick\nincorporating\nwildly\nfond\nbasilica\nowl\nfringe\nrituals\nwhispering\nstirred\nfeud\ntertiary\nslick\ngoat\nhonorable\nwhereby\nskip\nricardo\nstripes\nparachute\nadjoining\nsubmerged\nsynthesizer\n##gren\nintend\npositively\nninety\nphi\nbeaver\npartition\nfellows\nalexis\nprohibition\ncarlisle\nbizarre\nfraternity\n##bre\ndoubts\nicy\ncbc\naquatic\nsneak\nsonny\ncombines\nairports\ncrude\nsupervised\nspatial\nmerge\nalfonso\n##bic\ncorrupt\nscan\nundergo\n##ams\ndisabilities\ncolombian\ncomparing\ndolphins\nperkins\n##lish\nreprinted\nunanimous\nbounced\nhairs\nunderworld\nmidwest\nsemester\nbucket\npaperback\nminiseries\ncoventry\ndemise\n##leigh\ndemonstrations\nsensor\nrotating\nyan\n##hler\narrange\nsoils\n##idge\nhyderabad\nlabs\n##dr\nbrakes\ngrandchildren\n##nde\nnegotiated\nrover\nferrari\ncontinuation\ndirectorate\naugusta\nstevenson\ncounterpart\ngore\n##rda\nnursery\nrican\nave\ncollectively\nbroadly\npastoral\nrepertoire\nasserted\ndiscovering\nnordic\nstyled\nfiba\ncunningham\nharley\nmiddlesex\nsurvives\ntumor\ntempo\nzack\naiming\nlok\nurgent\n##rade\n##nto\ndevils\n##ement\ncontractor\nturin\n##wl\n##ool\nbliss\nrepaired\nsimmons\nmoan\nastronomical\ncr\nnegotiate\nlyric\n1890s\nlara\nbred\nclad\nangus\npbs\n##ience\nengineered\nposed\n##lk\nhernandez\npossessions\nelbows\npsychiatric\nstrokes\nconfluence\nelectorate\nlifts\ncampuses\nlava\nalps\n##ep\n##ution\n##date\nphysicist\nwoody\n##page\n##ographic\n##itis\njuliet\nreformation\nsparhawk\n320\ncomplement\nsuppressed\njewel\n##½\nfloated\n##kas\ncontinuity\nsadly\n##ische\ninability\nmelting\nscanning\npaula\nflour\njudaism\nsafer\nvague\n##lm\nsolving\ncurb\n##stown\nfinancially\ngable\nbees\nexpired\nmiserable\ncassidy\ndominion\n1789\ncupped\n145\nrobbery\nfacto\namos\nwarden\nresume\ntallest\nmarvin\ning\npounded\nusd\ndeclaring\ngasoline\n##aux\ndarkened\n270\n650\nsophomore\n##mere\nerection\ngossip\ntelevised\nrisen\ndial\n##eu\npillars\n##link\npassages\nprofound\n##tina\narabian\nashton\nsilicon\nnail\n##ead\n##lated\n##wer\n##hardt\nfleming\nfirearms\nducked\ncircuits\nblows\nwaterloo\ntitans\n##lina\natom\nfireplace\ncheshire\nfinanced\nactivation\nalgorithms\n##zzi\nconstituent\ncatcher\ncherokee\npartnerships\nsexuality\nplatoon\ntragic\nvivian\nguarded\nwhiskey\nmeditation\npoetic\n##late\n##nga\n##ake\nporto\nlisteners\ndominance\nkendra\nmona\nchandler\nfactions\n22nd\nsalisbury\nattitudes\nderivative\n##ido\n##haus\nintake\npaced\njavier\nillustrator\nbarrels\nbias\ncockpit\nburnett\ndreamed\nensuing\n##anda\nreceptors\nsomeday\nhawkins\nmattered\n##lal\nslavic\n1799\njesuit\ncameroon\nwasted\ntai\nwax\nlowering\nvictorious\nfreaking\noutright\nhancock\nlibrarian\nsensing\nbald\ncalcium\nmyers\ntablet\nannouncing\nbarack\nshipyard\npharmaceutical\n##uan\ngreenwich\nflush\nmedley\npatches\nwolfgang\npt\nspeeches\nacquiring\nexams\nnikolai\n##gg\nhayden\nkannada\n##type\nreilly\n##pt\nwaitress\nabdomen\ndevastated\ncapped\npseudonym\npharmacy\nfulfill\nparaguay\n1796\nclicked\n##trom\narchipelago\nsyndicated\n##hman\nlumber\norgasm\nrejection\nclifford\nlorraine\nadvent\nmafia\nrodney\nbrock\n##ght\n##used\n##elia\ncassette\nchamberlain\ndespair\nmongolia\nsensors\ndevelopmental\nupstream\n##eg\n##alis\nspanning\n165\ntrombone\nbasque\nseeded\ninterred\nrenewable\nrhys\nleapt\nrevision\nmolecule\n##ages\nchord\nvicious\nnord\nshivered\n23rd\narlington\ndebts\ncorpus\nsunrise\nbays\nblackburn\ncentimetres\n##uded\nshuddered\ngm\nstrangely\ngripping\ncartoons\nisabelle\norbital\n##ppa\nseals\nproving\n##lton\nrefusal\nstrengthened\nbust\nassisting\nbaghdad\nbatsman\nportrayal\nmara\npushes\nspears\nog\n##cock\nreside\nnathaniel\nbrennan\n1776\nconfirmation\ncaucus\n##worthy\nmarkings\nyemen\nnobles\nku\nlazy\nviewer\ncatalan\nencompasses\nsawyer\n##fall\nsparked\nsubstances\npatents\nbraves\narranger\nevacuation\nsergio\npersuade\ndover\ntolerance\npenguin\ncum\njockey\ninsufficient\ntownships\noccupying\ndeclining\nplural\nprocessed\nprojection\npuppet\nflanders\nintroduces\nliability\n##yon\ngymnastics\nantwerp\ntaipei\nhobart\ncandles\njeep\nwes\nobservers\n126\nchaplain\nbundle\nglorious\n##hine\nhazel\nflung\nsol\nexcavations\ndumped\nstares\nsh\nbangalore\ntriangular\nicelandic\nintervals\nexpressing\nturbine\n##vers\nsongwriting\ncrafts\n##igo\njasmine\nditch\nrite\n##ways\nentertaining\ncomply\nsorrow\nwrestlers\nbasel\nemirates\nmarian\nrivera\nhelpful\n##some\ncaution\ndownward\nnetworking\n##atory\n##tered\ndarted\ngenocide\nemergence\nreplies\nspecializing\nspokesman\nconvenient\nunlocked\nfading\naugustine\nconcentrations\nresemblance\nelijah\ninvestigator\nandhra\n##uda\npromotes\nbean\n##rrell\nfleeing\nwan\nsimone\nannouncer\n##ame\n##bby\nlydia\nweaver\n132\nresidency\nmodification\n##fest\nstretches\n##ast\nalternatively\nnat\nlowe\nlacks\n##ented\npam\ntile\nconcealed\ninferior\nabdullah\nresidences\ntissues\nvengeance\n##ided\nmoisture\npeculiar\ngroove\nzip\nbologna\njennings\nninja\noversaw\nzombies\npumping\nbatch\nlivingston\nemerald\ninstallations\n1797\npeel\nnitrogen\nrama\n##fying\n##star\nschooling\nstrands\nresponding\nwerner\n##ost\nlime\ncasa\naccurately\ntargeting\n##rod\nunderway\n##uru\nhemisphere\nlester\n##yard\noccupies\n2d\ngriffith\nangrily\nreorganized\n##owing\ncourtney\ndeposited\n##dd\n##30\nestadio\n##ifies\ndunn\nexiled\n##ying\nchecks\n##combe\n##о\n##fly\nsuccesses\nunexpectedly\nblu\nassessed\n##flower\n##ه\nobserving\nsacked\nspiders\nkn\n##tail\nmu\nnodes\nprosperity\naudrey\ndivisional\n155\nbroncos\ntangled\nadjust\nfeeds\nerosion\npaolo\nsurf\ndirectory\nsnatched\nhumid\nadmiralty\nscrewed\ngt\nreddish\n##nese\nmodules\ntrench\nlamps\nbind\nleah\nbucks\ncompetes\n##nz\n##form\ntranscription\n##uc\nisles\nviolently\nclutching\npga\ncyclist\ninflation\nflats\nragged\nunnecessary\n##hian\nstubborn\ncoordinated\nharriet\nbaba\ndisqualified\n330\ninsect\nwolfe\n##fies\nreinforcements\nrocked\nduel\nwinked\nembraced\nbricks\n##raj\nhiatus\ndefeats\npending\nbrightly\njealousy\n##xton\n##hm\n##uki\nlena\ngdp\ncolorful\n##dley\nstein\nkidney\n##shu\nunderwear\nwanderers\n##haw\n##icus\nguardians\nm³\nroared\nhabits\n##wise\npermits\ngp\nuranium\npunished\ndisguise\nbundesliga\nelise\ndundee\nerotic\npartisan\npi\ncollectors\nfloat\nindividually\nrendering\nbehavioral\nbucharest\nser\nhare\nvalerie\ncorporal\nnutrition\nproportional\n##isa\nimmense\n##kis\npavement\n##zie\n##eld\nsutherland\ncrouched\n1775\n##lp\nsuzuki\ntrades\nendurance\noperas\ncrosby\nprayed\npriory\nrory\nsocially\n##urn\ngujarat\n##pu\nwalton\ncube\npasha\nprivilege\nlennon\nfloods\nthorne\nwaterfall\nnipple\nscouting\napprove\n##lov\nminorities\nvoter\ndwight\nextensions\nassure\nballroom\nslap\ndripping\nprivileges\nrejoined\nconfessed\ndemonstrating\npatriotic\nyell\ninvestor\n##uth\npagan\nslumped\nsquares\n##cle\n##kins\nconfront\nbert\nembarrassment\n##aid\naston\nurging\nsweater\nstarr\nyuri\nbrains\nwilliamson\ncommuter\nmortar\nstructured\nselfish\nexports\n##jon\ncds\n##him\nunfinished\n##rre\nmortgage\ndestinations\n##nagar\ncanoe\nsolitary\nbuchanan\ndelays\nmagistrate\nfk\n##pling\nmotivation\n##lier\n##vier\nrecruiting\nassess\n##mouth\nmalik\nantique\n1791\npius\nrahman\nreich\ntub\nzhou\nsmashed\nairs\ngalway\nxii\nconditioning\nhonduras\ndischarged\ndexter\n##pf\nlionel\n129\ndebates\nlemon\ntiffany\nvolunteered\ndom\ndioxide\nprocession\ndevi\nsic\ntremendous\nadvertisements\ncolts\ntransferring\nverdict\nhanover\ndecommissioned\nutter\nrelate\npac\nracism\n##top\nbeacon\nlimp\nsimilarity\nterra\noccurrence\nant\n##how\nbecky\ncapt\nupdates\narmament\nrichie\npal\n##graph\nhalloween\nmayo\n##ssen\n##bone\ncara\nserena\nfcc\ndolls\nobligations\n##dling\nviolated\nlafayette\njakarta\nexploitation\n##ime\ninfamous\niconic\n##lah\n##park\nkitty\nmoody\nreginald\ndread\nspill\ncrystals\nolivier\nmodeled\nbluff\nequilibrium\nseparating\nnotices\nordnance\nextinction\nonset\ncosmic\nattachment\nsammy\nexpose\nprivy\nanchored\n##bil\nabbott\nadmits\nbending\nbaritone\nemmanuel\npoliceman\nvaughan\nwinged\nclimax\ndresses\ndenny\npolytechnic\nmohamed\nburmese\nauthentic\nnikki\ngenetics\ngrandparents\nhomestead\ngaza\npostponed\nmetacritic\nuna\n##sby\n##bat\nunstable\ndissertation\n##rial\n##cian\ncurls\nobscure\nuncovered\nbronx\npraying\ndisappearing\n##hoe\nprehistoric\ncoke\nturret\nmutations\nnonprofit\npits\nmonaco\n##ي\n##usion\nprominently\ndispatched\npodium\n##mir\nuci\n##uation\n133\nfortifications\nbirthplace\nkendall\n##lby\n##oll\npreacher\nrack\ngoodman\n##rman\npersistent\n##ott\ncountless\njaime\nrecorder\nlexington\npersecution\njumps\nrenewal\nwagons\n##11\ncrushing\n##holder\ndecorations\n##lake\nabundance\nwrath\nlaundry\n£1\ngarde\n##rp\njeanne\nbeetles\npeasant\n##sl\nsplitting\ncaste\nsergei\n##rer\n##ema\nscripts\n##ively\nrub\nsatellites\n##vor\ninscribed\nverlag\nscrapped\ngale\npackages\nchick\npotato\nslogan\nkathleen\narabs\n##culture\ncounterparts\nreminiscent\nchoral\n##tead\nrand\nretains\nbushes\ndane\naccomplish\ncourtesy\ncloses\n##oth\nslaughter\nhague\nkrakow\nlawson\ntailed\nelias\nginger\n##ttes\ncanopy\nbetrayal\nrebuilding\nturf\n##hof\nfrowning\nallegiance\nbrigades\nkicks\nrebuild\npolls\nalias\nnationalism\ntd\nrowan\naudition\nbowie\nfortunately\nrecognizes\nharp\ndillon\nhorrified\n##oro\nrenault\n##tics\nropes\n##α\npresumed\nrewarded\ninfrared\nwiping\naccelerated\nillustration\n##rid\npresses\npractitioners\nbadminton\n##iard\ndetained\n##tera\nrecognizing\nrelates\nmisery\n##sies\n##tly\nreproduction\npiercing\npotatoes\nthornton\nesther\nmanners\nhbo\n##aan\nours\nbullshit\nernie\nperennial\nsensitivity\nilluminated\nrupert\n##jin\n##iss\n##ear\nrfc\nnassau\n##dock\nstaggered\nsocialism\n##haven\nappointments\nnonsense\nprestige\nsharma\nhaul\n##tical\nsolidarity\ngps\n##ook\n##rata\nigor\npedestrian\n##uit\nbaxter\ntenants\nwires\nmedication\nunlimited\nguiding\nimpacts\ndiabetes\n##rama\nsasha\npas\nclive\nextraction\n131\ncontinually\nconstraints\n##bilities\nsonata\nhunted\nsixteenth\nchu\nplanting\nquote\nmayer\npretended\nabs\nspat\n##hua\nceramic\n##cci\ncurtains\npigs\npitching\n##dad\nlatvian\nsore\ndayton\n##sted\n##qi\npatrols\nslice\nplayground\n##nted\nshone\nstool\napparatus\ninadequate\nmates\ntreason\n##ija\ndesires\n##liga\n##croft\nsomalia\nlaurent\nmir\nleonardo\noracle\ngrape\nobliged\nchevrolet\nthirteenth\nstunning\nenthusiastic\n##ede\naccounted\nconcludes\ncurrents\nbasil\n##kovic\ndrought\n##rica\nmai\n##aire\nshove\nposting\n##shed\npilgrimage\nhumorous\npacking\nfry\npencil\nwines\nsmells\n144\nmarilyn\naching\nnewest\nclung\nbon\nneighbours\nsanctioned\n##pie\nmug\n##stock\ndrowning\n##mma\nhydraulic\n##vil\nhiring\nreminder\nlilly\ninvestigators\n##ncies\nsour\n##eous\ncompulsory\npacket\n##rion\n##graphic\n##elle\ncannes\n##inate\ndepressed\n##rit\nheroic\nimportantly\ntheresa\n##tled\nconway\nsaturn\nmarginal\nrae\n##xia\ncorresponds\nroyce\npact\njasper\nexplosives\npackaging\naluminium\n##ttered\ndenotes\nrhythmic\nspans\nassignments\nhereditary\noutlined\noriginating\nsundays\nlad\nreissued\ngreeting\nbeatrice\n##dic\npillar\nmarcos\nplots\nhandbook\nalcoholic\njudiciary\navant\nslides\nextract\nmasculine\nblur\n##eum\n##force\nhomage\ntrembled\nowens\nhymn\ntrey\nomega\nsignaling\nsocks\naccumulated\nreacted\nattic\ntheo\nlining\nangie\ndistraction\nprimera\ntalbot\n##key\n1200\nti\ncreativity\nbilled\n##hey\ndeacon\neduardo\nidentifies\nproposition\ndizzy\ngunner\nhogan\n##yam\n##pping\n##hol\nja\n##chan\njensen\nreconstructed\n##berger\nclearance\ndarius\n##nier\nabe\nharlem\nplea\ndei\ncircled\nemotionally\nnotation\nfascist\nneville\nexceeded\nupwards\nviable\nducks\n##fo\nworkforce\nracer\nlimiting\nshri\n##lson\npossesses\n1600\nkerr\nmoths\ndevastating\nladen\ndisturbing\nlocking\n##cture\ngal\nfearing\naccreditation\nflavor\naide\n1870s\nmountainous\n##baum\nmelt\n##ures\nmotel\ntexture\nservers\nsoda\n##mb\nherd\n##nium\nerect\npuzzled\nhum\npeggy\nexaminations\ngould\ntestified\ngeoff\nren\ndevised\nsacks\n##law\ndenial\nposters\ngrunted\ncesar\ntutor\nec\ngerry\nofferings\nbyrne\nfalcons\ncombinations\nct\nincoming\npardon\nrocking\n26th\navengers\nflared\nmankind\nseller\nuttar\nloch\nnadia\nstroking\nexposing\n##hd\nfertile\nancestral\ninstituted\n##has\nnoises\nprophecy\ntaxation\neminent\nvivid\npol\n##bol\ndart\nindirect\nmultimedia\nnotebook\nupside\ndisplaying\nadrenaline\nreferenced\ngeometric\n##iving\nprogression\n##ddy\nblunt\nannounce\n##far\nimplementing\n##lav\naggression\nliaison\ncooler\ncares\nheadache\nplantations\ngorge\ndots\nimpulse\nthickness\nashamed\naveraging\nkathy\nobligation\nprecursor\n137\nfowler\nsymmetry\nthee\n225\nhears\n##rai\nundergoing\nads\nbutcher\nbowler\n##lip\ncigarettes\nsubscription\ngoodness\n##ically\nbrowne\n##hos\n##tech\nkyoto\ndonor\n##erty\ndamaging\nfriction\ndrifting\nexpeditions\nhardened\nprostitution\n152\nfauna\nblankets\nclaw\ntossing\nsnarled\nbutterflies\nrecruits\ninvestigative\ncoated\nhealed\n138\ncommunal\nhai\nxiii\nacademics\nboone\npsychologist\nrestless\nlahore\nstephens\nmba\nbrendan\nforeigners\nprinter\n##pc\nached\nexplode\n27th\ndeed\nscratched\ndared\n##pole\ncardiac\n1780\nokinawa\nproto\ncommando\ncompelled\noddly\nelectrons\n##base\nreplica\nthanksgiving\n##rist\nsheila\ndeliberate\nstafford\ntidal\nrepresentations\nhercules\nou\n##path\n##iated\nkidnapping\nlenses\n##tling\ndeficit\nsamoa\nmouths\nconsuming\ncomputational\nmaze\ngranting\nsmirk\nrazor\nfixture\nideals\ninviting\naiden\nnominal\n##vs\nissuing\njulio\npitt\nramsey\ndocks\n##oss\nexhaust\n##owed\nbavarian\ndraped\nanterior\nmating\nethiopian\nexplores\nnoticing\n##nton\ndiscarded\nconvenience\nhoffman\nendowment\nbeasts\ncartridge\nmormon\npaternal\nprobe\nsleeves\ninterfere\nlump\ndeadline\n##rail\njenks\nbulldogs\nscrap\nalternating\njustified\nreproductive\nnam\nseize\ndescending\nsecretariat\nkirby\ncoupe\ngrouped\nsmash\npanther\nsedan\ntapping\n##18\nlola\ncheer\ngermanic\nunfortunate\n##eter\nunrelated\n##fan\nsubordinate\n##sdale\nsuzanne\nadvertisement\n##ility\nhorsepower\n##lda\ncautiously\ndiscourse\nluigi\n##mans\n##fields\nnoun\nprevalent\nmao\nschneider\neverett\nsurround\ngovernorate\nkira\n##avia\nwestward\n##take\nmisty\nrails\nsustainability\n134\nunused\n##rating\npacks\ntoast\nunwilling\nregulate\nthy\nsuffrage\nnile\nawe\nassam\ndefinitions\ntravelers\naffordable\n##rb\nconferred\nsells\nundefeated\nbeneficial\ntorso\nbasal\nrepeating\nremixes\n##pass\nbahrain\ncables\nfang\n##itated\nexcavated\nnumbering\nstatutory\n##rey\ndeluxe\n##lian\nforested\nramirez\nderbyshire\nzeus\nslamming\ntransfers\nastronomer\nbanana\nlottery\nberg\nhistories\nbamboo\n##uchi\nresurrection\nposterior\nbowls\nvaguely\n##thi\nthou\npreserving\ntensed\noffence\n##inas\nmeyrick\ncallum\nridden\nwatt\nlangdon\ntying\nlowland\nsnorted\ndaring\ntruman\n##hale\n##girl\naura\noverly\nfiling\nweighing\ngoa\ninfections\nphilanthropist\nsaunders\neponymous\n##owski\nlatitude\nperspectives\nreviewing\nmets\ncommandant\nradial\n##kha\nflashlight\nreliability\nkoch\nvowels\namazed\nada\nelaine\nsupper\n##rth\n##encies\npredator\ndebated\nsoviets\ncola\n##boards\n##nah\ncompartment\ncrooked\narbitrary\nfourteenth\n##ctive\nhavana\nmajors\nsteelers\nclips\nprofitable\nambush\nexited\npackers\n##tile\nnude\ncracks\nfungi\n##е\nlimb\ntrousers\njosie\nshelby\ntens\nfrederic\n##ος\ndefinite\nsmoothly\nconstellation\ninsult\nbaton\ndiscs\nlingering\n##nco\nconclusions\nlent\nstaging\nbecker\ngrandpa\nshaky\n##tron\neinstein\nobstacles\nsk\nadverse\nelle\neconomically\n##moto\nmccartney\nthor\ndismissal\nmotions\nreadings\nnostrils\ntreatise\n##pace\nsqueezing\nevidently\nprolonged\n1783\nvenezuelan\nje\nmarguerite\nbeirut\ntakeover\nshareholders\n##vent\ndenise\ndigit\nairplay\nnorse\n##bbling\nimaginary\npills\nhubert\nblaze\nvacated\neliminating\n##ello\nvine\nmansfield\n##tty\nretrospective\nbarrow\nborne\nclutch\nbail\nforensic\nweaving\n##nett\n##witz\ndesktop\ncitadel\npromotions\nworrying\ndorset\nieee\nsubdivided\n##iating\nmanned\nexpeditionary\npickup\nsynod\nchuckle\n185\nbarney\n##rz\n##ffin\nfunctionality\nkarachi\nlitigation\nmeanings\nuc\nlick\nturbo\nanders\n##ffed\nexecute\ncurl\noppose\nankles\ntyphoon\n##د\n##ache\n##asia\nlinguistics\ncompassion\npressures\ngrazing\nperfection\n##iting\nimmunity\nmonopoly\nmuddy\nbackgrounds\n136\nnamibia\nfrancesca\nmonitors\nattracting\nstunt\ntuition\n##ии\nvegetable\n##mates\n##quent\nmgm\njen\ncomplexes\nforts\n##ond\ncellar\nbites\nseventeenth\nroyals\nflemish\nfailures\nmast\ncharities\n##cular\nperuvian\ncapitals\nmacmillan\nipswich\noutward\nfrigate\npostgraduate\nfolds\nemploying\n##ouse\nconcurrently\nfiery\n##tai\ncontingent\nnightmares\nmonumental\nnicaragua\n##kowski\nlizard\nmal\nfielding\ngig\nreject\n##pad\nharding\n##ipe\ncoastline\n##cin\n##nos\nbeethoven\nhumphrey\ninnovations\n##tam\n##nge\nnorris\ndoris\nsolicitor\nhuang\nobey\n141\n##lc\nniagara\n##tton\nshelves\naug\nbourbon\ncurry\nnightclub\nspecifications\nhilton\n##ndo\ncentennial\ndispersed\nworm\nneglected\nbriggs\nsm\nfont\nkuala\nuneasy\nplc\n##nstein\n##bound\n##aking\n##burgh\nawaiting\npronunciation\n##bbed\n##quest\neh\noptimal\nzhu\nraped\ngreens\npresided\nbrenda\nworries\n##life\nvenetian\nmarxist\nturnout\n##lius\nrefined\nbraced\nsins\ngrasped\nsunderland\nnickel\nspeculated\nlowell\ncyrillic\ncommunism\nfundraising\nresembling\ncolonists\nmutant\nfreddie\nusc\n##mos\ngratitude\n##run\nmural\n##lous\nchemist\nwi\nreminds\n28th\nsteals\ntess\npietro\n##ingen\npromoter\nri\nmicrophone\nhonoured\nrai\nsant\n##qui\nfeather\n##nson\nburlington\nkurdish\nterrorists\ndeborah\nsickness\n##wed\n##eet\nhazard\nirritated\ndesperation\nveil\nclarity\n##rik\njewels\nxv\n##gged\n##ows\n##cup\nberkshire\nunfair\nmysteries\norchid\nwinced\nexhaustion\nrenovations\nstranded\nobe\ninfinity\n##nies\nadapt\nredevelopment\nthanked\nregistry\nolga\ndomingo\nnoir\ntudor\nole\n##atus\ncommenting\nbehaviors\n##ais\ncrisp\npauline\nprobable\nstirling\nwigan\n##bian\nparalympics\npanting\nsurpassed\n##rew\nluca\nbarred\npony\nfamed\n##sters\ncassandra\nwaiter\ncarolyn\nexported\n##orted\nandres\ndestructive\ndeeds\njonah\ncastles\nvacancy\nsuv\n##glass\n1788\norchard\nyep\nfamine\nbelarusian\nsprang\n##forth\nskinny\n##mis\nadministrators\nrotterdam\nzambia\nzhao\nboiler\ndiscoveries\n##ride\n##physics\nlucius\ndisappointing\noutreach\nspoon\n##frame\nqualifications\nunanimously\nenjoys\nregency\n##iidae\nstade\nrealism\nveterinary\nrodgers\ndump\nalain\nchestnut\ncastile\ncensorship\nrumble\ngibbs\n##itor\ncommunion\nreggae\ninactivated\nlogs\nloads\n##houses\nhomosexual\n##iano\nale\ninforms\n##cas\nphrases\nplaster\nlinebacker\nambrose\nkaiser\nfascinated\n850\nlimerick\nrecruitment\nforge\nmastered\n##nding\nleinster\nrooted\nthreaten\n##strom\nborneo\n##hes\nsuggestions\nscholarships\npropeller\ndocumentaries\npatronage\ncoats\nconstructing\ninvest\nneurons\ncomet\nentirety\nshouts\nidentities\nannoying\nunchanged\nwary\n##antly\n##ogy\nneat\noversight\n##kos\nphillies\nreplay\nconstance\n##kka\nincarnation\nhumble\nskies\nminus\n##acy\nsmithsonian\n##chel\nguerrilla\njar\ncadets\n##plate\nsurplus\naudit\n##aru\ncracking\njoanna\nlouisa\npacing\n##lights\nintentionally\n##iri\ndiner\nnwa\nimprint\naustralians\ntong\nunprecedented\nbunker\nnaive\nspecialists\nark\nnichols\nrailing\nleaked\npedal\n##uka\nshrub\nlonging\nroofs\nv8\ncaptains\nneural\ntuned\n##ntal\n##jet\nemission\nmedina\nfrantic\ncodex\ndefinitive\nsid\nabolition\nintensified\nstocks\nenrique\nsustain\ngenoa\noxide\n##written\nclues\ncha\n##gers\ntributaries\nfragment\nvenom\n##rity\n##ente\n##sca\nmuffled\nvain\nsire\nlaos\n##ingly\n##hana\nhastily\nsnapping\nsurfaced\nsentiment\nmotive\n##oft\ncontests\napproximate\nmesa\nluckily\ndinosaur\nexchanges\npropelled\naccord\nbourne\nrelieve\ntow\nmasks\noffended\n##ues\ncynthia\n##mmer\nrains\nbartender\nzinc\nreviewers\nlois\n##sai\nlegged\narrogant\nrafe\nrosie\ncomprise\nhandicap\nblockade\ninlet\nlagoon\ncopied\ndrilling\nshelley\npetals\n##inian\nmandarin\nobsolete\n##inated\nonward\narguably\nproductivity\ncindy\npraising\nseldom\nbusch\ndiscusses\nraleigh\nshortage\nranged\nstanton\nencouragement\nfirstly\nconceded\novers\ntemporal\n##uke\ncbe\n##bos\nwoo\ncertainty\npumps\n##pton\nstalked\n##uli\nlizzie\nperiodic\nthieves\nweaker\n##night\ngases\nshoving\nchooses\nwc\n##chemical\nprompting\nweights\n##kill\nrobust\nflanked\nsticky\nhu\ntuberculosis\n##eb\n##eal\nchristchurch\nresembled\nwallet\nreese\ninappropriate\npictured\ndistract\nfixing\nfiddle\ngiggled\nburger\nheirs\nhairy\nmechanic\ntorque\napache\nobsessed\nchiefly\ncheng\nlogging\n##tag\nextracted\nmeaningful\nnumb\n##vsky\ngloucestershire\nreminding\n##bay\nunite\n##lit\nbreeds\ndiminished\nclown\nglove\n1860s\n##ن\n##ug\narchibald\nfocal\nfreelance\nsliced\ndepiction\n##yk\norganism\nswitches\nsights\nstray\ncrawling\n##ril\nlever\nleningrad\ninterpretations\nloops\nanytime\nreel\nalicia\ndelighted\n##ech\ninhaled\nxiv\nsuitcase\nbernie\nvega\nlicenses\nnorthampton\nexclusion\ninduction\nmonasteries\nracecourse\nhomosexuality\n##right\n##sfield\n##rky\ndimitri\nmichele\nalternatives\nions\ncommentators\ngenuinely\nobjected\npork\nhospitality\nfencing\nstephan\nwarships\nperipheral\nwit\ndrunken\nwrinkled\nquentin\nspends\ndeparting\nchung\nnumerical\nspokesperson\n##zone\njohannesburg\ncaliber\nkillers\n##udge\nassumes\nneatly\ndemographic\nabigail\nbloc\n##vel\nmounting\n##lain\nbentley\nslightest\nxu\nrecipients\n##jk\nmerlin\n##writer\nseniors\nprisons\nblinking\nhindwings\nflickered\nkappa\n##hel\n80s\nstrengthening\nappealing\nbrewing\ngypsy\nmali\nlashes\nhulk\nunpleasant\nharassment\nbio\ntreaties\npredict\ninstrumentation\npulp\ntroupe\nboiling\nmantle\n##ffe\nins\n##vn\ndividing\nhandles\nverbs\n##onal\ncoconut\nsenegal\n340\nthorough\ngum\nmomentarily\n##sto\ncocaine\npanicked\ndestined\n##turing\nteatro\ndenying\nweary\ncaptained\nmans\n##hawks\n##code\nwakefield\nbollywood\nthankfully\n##16\ncyril\n##wu\namendments\n##bahn\nconsultation\nstud\nreflections\nkindness\n1787\ninternally\n##ovo\ntex\nmosaic\ndistribute\npaddy\nseeming\n143\n##hic\npiers\n##15\n##mura\n##verse\npopularly\nwinger\nkang\nsentinel\nmccoy\n##anza\ncovenant\n##bag\nverge\nfireworks\nsuppress\nthrilled\ndominate\n##jar\nswansea\n##60\n142\nreconciliation\n##ndi\nstiffened\ncue\ndorian\n##uf\ndamascus\namor\nida\nforemost\n##aga\nporsche\nunseen\ndir\n##had\n##azi\nstony\nlexi\nmelodies\n##nko\nangular\ninteger\npodcast\nants\ninherent\njaws\njustify\npersona\n##olved\njosephine\n##nr\n##ressed\ncustomary\nflashes\ngala\ncyrus\nglaring\nbackyard\nariel\nphysiology\ngreenland\nhtml\nstir\navon\natletico\nfinch\nmethodology\nked\n##lent\nmas\ncatholicism\ntownsend\nbranding\nquincy\nfits\ncontainers\n1777\nashore\naragon\n##19\nforearm\npoisoning\n##sd\nadopting\nconquer\ngrinding\namnesty\nkeller\nfinances\nevaluate\nforged\nlankan\ninstincts\n##uto\nguam\nbosnian\nphotographed\nworkplace\ndesirable\nprotector\n##dog\nallocation\nintently\nencourages\nwilly\n##sten\nbodyguard\nelectro\nbrighter\n##ν\nbihar\n##chev\nlasts\nopener\namphibious\nsal\nverde\narte\n##cope\ncaptivity\nvocabulary\nyields\n##tted\nagreeing\ndesmond\npioneered\n##chus\nstrap\ncampaigned\nrailroads\n##ович\nemblem\n##dre\nstormed\n501\n##ulous\nmarijuana\nnorthumberland\n##gn\n##nath\nbowen\nlandmarks\nbeaumont\n##qua\ndanube\n##bler\nattorneys\nth\nge\nflyers\ncritique\nvillains\ncass\nmutation\nacc\n##0s\ncolombo\nmckay\nmotif\nsampling\nconcluding\nsyndicate\n##rell\nneon\nstables\nds\nwarnings\nclint\nmourning\nwilkinson\n##tated\nmerrill\nleopard\nevenings\nexhaled\nemil\nsonia\nezra\ndiscrete\nstove\nfarrell\nfifteenth\nprescribed\nsuperhero\n##rier\nworms\nhelm\nwren\n##duction\n##hc\nexpo\n##rator\nhq\nunfamiliar\nantony\nprevents\nacceleration\nfiercely\nmari\npainfully\ncalculations\ncheaper\nign\nclifton\nirvine\ndavenport\nmozambique\n##np\npierced\n##evich\nwonders\n##wig\n##cate\n##iling\ncrusade\nware\n##uel\nenzymes\nreasonably\nmls\n##coe\nmater\nambition\nbunny\neliot\nkernel\n##fin\nasphalt\nheadmaster\ntorah\naden\nlush\npins\nwaived\n##care\n##yas\njoao\nsubstrate\nenforce\n##grad\n##ules\nalvarez\nselections\nepidemic\ntempted\n##bit\nbremen\ntranslates\nensured\nwaterfront\n29th\nforrest\nmanny\nmalone\nkramer\nreigning\ncookies\nsimpler\nabsorption\n205\nengraved\n##ffy\nevaluated\n1778\nhaze\n146\ncomforting\ncrossover\n##abe\nthorn\n##rift\n##imo\n##pop\nsuppression\nfatigue\ncutter\n##tr\n201\nwurttemberg\n##orf\nenforced\nhovering\nproprietary\ngb\nsamurai\nsyllable\nascent\nlacey\ntick\nlars\ntractor\nmerchandise\nrep\nbouncing\ndefendants\n##yre\nhuntington\n##ground\n##oko\nstandardized\n##hor\n##hima\nassassinated\nnu\npredecessors\nrainy\nliar\nassurance\nlyrical\n##uga\nsecondly\nflattened\nios\nparameter\nundercover\n##mity\nbordeaux\npunish\nridges\nmarkers\nexodus\ninactive\nhesitate\ndebbie\nnyc\npledge\nsavoy\nnagar\noffset\norganist\n##tium\nhesse\nmarin\nconverting\n##iver\ndiagram\npropulsion\npu\nvalidity\nreverted\nsupportive\n##dc\nministries\nclans\nresponds\nproclamation\n##inae\n##ø\n##rea\nein\npleading\npatriot\nsf\nbirch\nislanders\nstrauss\nhates\n##dh\nbrandenburg\nconcession\nrd\n##ob\n1900s\nkillings\ntextbook\nantiquity\ncinematography\nwharf\nembarrassing\nsetup\ncreed\nfarmland\ninequality\ncentred\nsignatures\nfallon\n370\n##ingham\n##uts\nceylon\ngazing\ndirective\nlaurie\n##tern\nglobally\n##uated\n##dent\nallah\nexcavation\nthreads\n##cross\n148\nfrantically\nicc\nutilize\ndetermines\nrespiratory\nthoughtful\nreceptions\n##dicate\nmerging\nchandra\nseine\n147\nbuilders\nbuilds\ndiagnostic\ndev\nvisibility\ngoddamn\nanalyses\ndhaka\ncho\nproves\nchancel\nconcurrent\ncuriously\ncanadians\npumped\nrestoring\n1850s\nturtles\njaguar\nsinister\nspinal\ntraction\ndeclan\nvows\n1784\nglowed\ncapitalism\nswirling\ninstall\nuniversidad\n##lder\n##oat\nsoloist\n##genic\n##oor\ncoincidence\nbeginnings\nnissan\ndip\nresorts\ncaucasus\ncombustion\ninfectious\n##eno\npigeon\nserpent\n##itating\nconclude\nmasked\nsalad\njew\n##gr\nsurreal\ntoni\n##wc\nharmonica\n151\n##gins\n##etic\n##coat\nfishermen\nintending\nbravery\n##wave\nklaus\ntitan\nwembley\ntaiwanese\nransom\n40th\nincorrect\nhussein\neyelids\njp\ncooke\ndramas\nutilities\n##etta\n##print\neisenhower\nprincipally\ngranada\nlana\n##rak\nopenings\nconcord\n##bl\nbethany\nconnie\nmorality\nsega\n##mons\n##nard\nearnings\n##kara\n##cine\nwii\ncommunes\n##rel\ncoma\ncomposing\nsoftened\nsevered\ngrapes\n##17\nnguyen\nanalyzed\nwarlord\nhubbard\nheavenly\nbehave\nslovenian\n##hit\n##ony\nhailed\nfilmmakers\ntrance\ncaldwell\nskye\nunrest\ncoward\nlikelihood\n##aging\nbern\nsci\ntaliban\nhonolulu\npropose\n##wang\n1700\nbrowser\nimagining\ncobra\ncontributes\ndukes\ninstinctively\nconan\nviolinist\n##ores\naccessories\ngradual\n##amp\nquotes\nsioux\n##dating\nundertake\nintercepted\nsparkling\ncompressed\n139\nfungus\ntombs\nhaley\nimposing\nrests\ndegradation\nlincolnshire\nretailers\nwetlands\ntulsa\ndistributor\ndungeon\nnun\ngreenhouse\nconvey\natlantis\naft\nexits\noman\ndresser\nlyons\n##sti\njoking\neddy\njudgement\nomitted\ndigits\n##cts\n##game\njuniors\n##rae\ncents\nstricken\nune\n##ngo\nwizards\nweir\nbreton\nnan\ntechnician\nfibers\nliking\nroyalty\n##cca\n154\npersia\nterribly\nmagician\n##rable\n##unt\nvance\ncafeteria\nbooker\ncamille\nwarmer\n##static\nconsume\ncavern\ngaps\ncompass\ncontemporaries\nfoyer\nsoothing\ngraveyard\nmaj\nplunged\nblush\n##wear\ncascade\ndemonstrates\nordinance\n##nov\nboyle\n##lana\nrockefeller\nshaken\nbanjo\nizzy\n##ense\nbreathless\nvines\n##32\n##eman\nalterations\nchromosome\ndwellings\nfeudal\nmole\n153\ncatalonia\nrelics\ntenant\nmandated\n##fm\nfridge\nhats\nhonesty\npatented\nraul\nheap\ncruisers\naccusing\nenlightenment\ninfants\nwherein\nchatham\ncontractors\nzen\naffinity\nhc\nosborne\npiston\n156\ntraps\nmaturity\n##rana\nlagos\n##zal\npeering\n##nay\nattendant\ndealers\nprotocols\nsubset\nprospects\nbiographical\n##cre\nartery\n##zers\ninsignia\nnuns\nendured\n##eration\nrecommend\nschwartz\nserbs\nberger\ncromwell\ncrossroads\n##ctor\nenduring\nclasped\ngrounded\n##bine\nmarseille\ntwitched\nabel\nchoke\nhttps\ncatalyst\nmoldova\nitalians\n##tist\ndisastrous\nwee\n##oured\n##nti\nwwf\nnope\n##piration\n##asa\nexpresses\nthumbs\n167\n##nza\ncoca\n1781\ncheating\n##ption\nskipped\nsensory\nheidelberg\nspies\nsatan\ndangers\nsemifinal\n202\nbohemia\nwhitish\nconfusing\nshipbuilding\nrelies\nsurgeons\nlandings\nravi\nbaku\nmoor\nsuffix\nalejandro\n##yana\nlitre\nupheld\n##unk\nrajasthan\n##rek\ncoaster\ninsists\nposture\nscenarios\netienne\nfavoured\nappoint\ntransgender\nelephants\npoked\ngreenwood\ndefences\nfulfilled\nmilitant\nsomali\n1758\nchalk\npotent\n##ucci\nmigrants\nwink\nassistants\nnos\nrestriction\nactivism\nniger\n##ario\ncolon\nshaun\n##sat\ndaphne\n##erated\nswam\ncongregations\nreprise\nconsiderations\nmagnet\nplayable\nxvi\n##р\noverthrow\ntobias\nknob\nchavez\ncoding\n##mers\npropped\nkatrina\norient\nnewcomer\n##suke\ntemperate\n##pool\nfarmhouse\ninterrogation\n##vd\ncommitting\n##vert\nforthcoming\nstrawberry\njoaquin\nmacau\nponds\nshocking\nsiberia\n##cellular\nchant\ncontributors\n##nant\n##ologists\nsped\nabsorb\nhail\n1782\nspared\n##hore\nbarbados\nkarate\nopus\noriginates\nsaul\n##xie\nevergreen\nleaped\n##rock\ncorrelation\nexaggerated\nweekday\nunification\nbump\ntracing\nbrig\nafb\npathways\nutilizing\n##ners\nmod\nmb\ndisturbance\nkneeling\n##stad\n##guchi\n100th\npune\n##thy\ndecreasing\n168\nmanipulation\nmiriam\nacademia\necosystem\noccupational\nrbi\n##lem\nrift\n##14\nrotary\nstacked\nincorporation\nawakening\ngenerators\nguerrero\nracist\n##omy\ncyber\nderivatives\nculminated\nallie\nannals\npanzer\nsainte\nwikipedia\npops\nzu\naustro\n##vate\nalgerian\npolitely\nnicholson\nmornings\neducate\ntastes\nthrill\ndartmouth\n##gating\ndb\n##jee\nregan\ndiffering\nconcentrating\nchoreography\ndivinity\n##media\npledged\nalexandre\nrouting\ngregor\nmadeline\n##idal\napocalypse\n##hora\ngunfire\nculminating\nelves\nfined\nliang\nlam\nprogrammed\ntar\nguessing\ntransparency\ngabrielle\n##gna\ncancellation\nflexibility\n##lining\naccession\nshea\nstronghold\nnets\nspecializes\n##rgan\nabused\nhasan\nsgt\nling\nexceeding\n##₄\nadmiration\nsupermarket\n##ark\nphotographers\nspecialised\ntilt\nresonance\nhmm\nperfume\n380\nsami\nthreatens\ngarland\nbotany\nguarding\nboiled\ngreet\npuppy\nrusso\nsupplier\nwilmington\nvibrant\nvijay\n##bius\nparalympic\ngrumbled\npaige\nfaa\nlicking\nmargins\nhurricanes\n##gong\nfest\ngrenade\nripping\n##uz\ncounseling\nweigh\n##sian\nneedles\nwiltshire\nedison\ncostly\n##not\nfulton\ntramway\nredesigned\nstaffordshire\ncache\ngasping\nwatkins\nsleepy\ncandidacy\n##group\nmonkeys\ntimeline\nthrobbing\n##bid\n##sos\nberth\nuzbekistan\nvanderbilt\nbothering\noverturned\nballots\ngem\n##iger\nsunglasses\nsubscribers\nhooker\ncompelling\nang\nexceptionally\nsaloon\nstab\n##rdi\ncarla\nterrifying\nrom\n##vision\ncoil\n##oids\nsatisfying\nvendors\n31st\nmackay\ndeities\noverlooked\nambient\nbahamas\nfelipe\nolympia\nwhirled\nbotanist\nadvertised\ntugging\n##dden\ndisciples\nmorales\nunionist\nrites\nfoley\nmorse\nmotives\ncreepy\n##₀\nsoo\n##sz\nbargain\nhighness\nfrightening\nturnpike\ntory\nreorganization\n##cer\ndepict\nbiographer\n##walk\nunopposed\nmanifesto\n##gles\ninstitut\nemile\naccidental\nkapoor\n##dam\nkilkenny\ncortex\nlively\n##13\nromanesque\njain\nshan\ncannons\n##ood\n##ske\npetrol\nechoing\namalgamated\ndisappears\ncautious\nproposes\nsanctions\ntrenton\n##ر\nflotilla\naus\ncontempt\ntor\ncanary\ncote\ntheirs\n##hun\nconceptual\ndeleted\nfascinating\npaso\nblazing\nelf\nhonourable\nhutchinson\n##eiro\n##outh\n##zin\nsurveyor\ntee\namidst\nwooded\nreissue\nintro\n##ono\ncobb\nshelters\nnewsletter\nhanson\nbrace\nencoding\nconfiscated\ndem\ncaravan\nmarino\nscroll\nmelodic\ncows\nimam\n##adi\n##aneous\nnorthward\nsearches\nbiodiversity\ncora\n310\nroaring\n##bers\nconnell\ntheologian\nhalo\ncompose\npathetic\nunmarried\ndynamo\n##oot\naz\ncalculation\ntoulouse\ndeserves\nhumour\nnr\nforgiveness\ntam\nundergone\nmartyr\npamela\nmyths\nwhore\ncounselor\nhicks\n290\nheavens\nbattleship\nelectromagnetic\n##bbs\nstellar\nestablishments\npresley\nhopped\n##chin\ntemptation\n90s\nwills\nnas\n##yuan\nnhs\n##nya\nseminars\n##yev\nadaptations\ngong\nasher\nlex\nindicator\nsikh\ntobago\ncites\ngoin\n##yte\nsatirical\n##gies\ncharacterised\ncorrespond\nbubbles\nlure\nparticipates\n##vid\neruption\nskate\ntherapeutic\n1785\ncanals\nwholesale\ndefaulted\nsac\n460\npetit\n##zzled\nvirgil\nleak\nravens\n256\nportraying\n##yx\nghetto\ncreators\ndams\nportray\nvicente\n##rington\nfae\nnamesake\nbounty\n##arium\njoachim\n##ota\n##iser\naforementioned\naxle\nsnout\ndepended\ndismantled\nreuben\n480\n##ibly\ngallagher\n##lau\n##pd\nearnest\n##ieu\n##iary\ninflicted\nobjections\n##llar\nasa\ngritted\n##athy\njericho\n##sea\n##was\nflick\nunderside\nceramics\nundead\nsubstituted\n195\neastward\nundoubtedly\nwheeled\nchimney\n##iche\nguinness\ncb\n##ager\nsiding\n##bell\ntraitor\nbaptiste\ndisguised\ninauguration\n149\ntipperary\nchoreographer\nperched\nwarmed\nstationary\neco\n##ike\n##ntes\nbacterial\n##aurus\nflores\nphosphate\n##core\nattacker\ninvaders\nalvin\nintersects\na1\nindirectly\nimmigrated\nbusinessmen\ncornelius\nvalves\nnarrated\npill\nsober\nul\nnationale\nmonastic\napplicants\nscenery\n##jack\n161\nmotifs\nconstitutes\ncpu\n##osh\njurisdictions\nsd\ntuning\nirritation\nwoven\n##uddin\nfertility\ngao\n##erie\nantagonist\nimpatient\nglacial\nhides\nboarded\ndenominations\ninterception\n##jas\ncookie\nnicola\n##tee\nalgebraic\nmarquess\nbahn\nparole\nbuyers\nbait\nturbines\npaperwork\nbestowed\nnatasha\nrenee\noceans\npurchases\n157\nvaccine\n215\n##tock\nfixtures\nplayhouse\nintegrate\njai\noswald\nintellectuals\n##cky\nbooked\nnests\nmortimer\n##isi\nobsession\nsept\n##gler\n##sum\n440\nscrutiny\nsimultaneous\nsquinted\n##shin\ncollects\noven\nshankar\npenned\nremarkably\n##я\nslips\nluggage\nspectral\n1786\ncollaborations\nlouie\nconsolidation\n##ailed\n##ivating\n420\nhoover\nblackpool\nharness\nignition\nvest\ntails\nbelmont\nmongol\nskinner\n##nae\nvisually\nmage\nderry\n##tism\n##unce\nstevie\ntransitional\n##rdy\nredskins\ndrying\nprep\nprospective\n##21\nannoyance\noversee\n##loaded\nfills\n##books\n##iki\nannounces\nfda\nscowled\nrespects\nprasad\nmystic\ntucson\n##vale\nrevue\nspringer\nbankrupt\n1772\naristotle\nsalvatore\nhabsburg\n##geny\ndal\nnatal\nnut\npod\nchewing\ndarts\nmoroccan\nwalkover\nrosario\nlenin\npunjabi\n##ße\ngrossed\nscattering\nwired\ninvasive\nhui\npolynomial\ncorridors\nwakes\ngina\nportrays\n##cratic\narid\nretreating\nerich\nirwin\nsniper\n##dha\nlinen\nlindsey\nmaneuver\nbutch\nshutting\nsocio\nbounce\ncommemorative\npostseason\njeremiah\npines\n275\nmystical\nbeads\nbp\nabbas\nfurnace\nbidding\nconsulted\nassaulted\nempirical\nrubble\nenclosure\nsob\nweakly\ncancel\npolly\nyielded\n##emann\ncurly\nprediction\nbattered\n70s\nvhs\njacqueline\nrender\nsails\nbarked\ndetailing\ngrayson\nriga\nsloane\nraging\n##yah\nherbs\nbravo\n##athlon\nalloy\ngiggle\nimminent\nsuffers\nassumptions\nwaltz\n##itate\naccomplishments\n##ited\nbathing\nremixed\ndeception\nprefix\n##emia\ndeepest\n##tier\n##eis\nbalkan\nfrogs\n##rong\nslab\n##pate\nphilosophers\npeterborough\ngrains\nimports\ndickinson\nrwanda\n##atics\n1774\ndirk\nlan\ntablets\n##rove\nclone\n##rice\ncaretaker\nhostilities\nmclean\n##gre\nregimental\ntreasures\nnorms\nimpose\ntsar\ntango\ndiplomacy\nvariously\ncomplain\n192\nrecognise\narrests\n1779\ncelestial\npulitzer\n##dus\nbing\nlibretto\n##moor\nadele\nsplash\n##rite\nexpectation\nlds\nconfronts\n##izer\nspontaneous\nharmful\nwedge\nentrepreneurs\nbuyer\n##ope\nbilingual\ntranslate\nrugged\nconner\ncirculated\nuae\neaton\n##gra\n##zzle\nlingered\nlockheed\nvishnu\nreelection\nalonso\n##oom\njoints\nyankee\nheadline\ncooperate\nheinz\nlaureate\ninvading\n##sford\nechoes\nscandinavian\n##dham\nhugging\nvitamin\nsalute\nmicah\nhind\ntrader\n##sper\nradioactive\n##ndra\nmilitants\npoisoned\nratified\nremark\ncampeonato\ndeprived\nwander\nprop\n##dong\noutlook\n##tani\n##rix\n##eye\nchiang\ndarcy\n##oping\nmandolin\nspice\nstatesman\nbabylon\n182\nwalled\nforgetting\nafro\n##cap\n158\ngiorgio\nbuffer\n##polis\nplanetary\n##gis\noverlap\nterminals\nkinda\ncentenary\n##bir\narising\nmanipulate\nelm\nke\n1770\nak\n##tad\nchrysler\nmapped\nmoose\npomeranian\nquad\nmacarthur\nassemblies\nshoreline\nrecalls\nstratford\n##rted\nnoticeable\n##evic\nimp\n##rita\n##sque\naccustomed\nsupplying\ntents\ndisgusted\nvogue\nsipped\nfilters\nkhz\nreno\nselecting\nluftwaffe\nmcmahon\ntyne\nmasterpiece\ncarriages\ncollided\ndunes\nexercised\nflare\nremembers\nmuzzle\n##mobile\nheck\n##rson\nburgess\nlunged\nmiddleton\nboycott\nbilateral\n##sity\nhazardous\nlumpur\nmultiplayer\nspotlight\njackets\ngoldman\nliege\nporcelain\nrag\nwaterford\nbenz\nattracts\nhopeful\nbattling\nottomans\nkensington\nbaked\nhymns\ncheyenne\nlattice\nlevine\nborrow\npolymer\nclashes\nmichaels\nmonitored\ncommitments\ndenounced\n##25\n##von\ncavity\n##oney\nhobby\nakin\n##holders\nfutures\nintricate\ncornish\npatty\n##oned\nillegally\ndolphin\n##lag\nbarlow\nyellowish\nmaddie\napologized\nluton\nplagued\n##puram\nnana\n##rds\nsway\nfanny\nłodz\n##rino\npsi\nsuspicions\nhanged\n##eding\ninitiate\ncharlton\n##por\nnak\ncompetent\n235\nanalytical\nannex\nwardrobe\nreservations\n##rma\nsect\n162\nfairfax\nhedge\npiled\nbuckingham\nuneven\nbauer\nsimplicity\nsnyder\ninterpret\naccountability\ndonors\nmoderately\nbyrd\ncontinents\n##cite\n##max\ndisciple\nhr\njamaican\nping\nnominees\n##uss\nmongolian\ndiver\nattackers\neagerly\nideological\npillows\nmiracles\napartheid\nrevolver\nsulfur\nclinics\nmoran\n163\n##enko\nile\nkaty\nrhetoric\n##icated\nchronology\nrecycling\n##hrer\nelongated\nmughal\npascal\nprofiles\nvibration\ndatabases\ndomination\n##fare\n##rant\nmatthias\ndigest\nrehearsal\npolling\nweiss\ninitiation\nreeves\nclinging\nflourished\nimpress\nngo\n##hoff\n##ume\nbuckley\nsymposium\nrhythms\nweed\nemphasize\ntransforming\n##taking\n##gence\n##yman\naccountant\nanalyze\nflicker\nfoil\npriesthood\nvoluntarily\ndecreases\n##80\n##hya\nslater\nsv\ncharting\nmcgill\n##lde\nmoreno\n##iu\nbesieged\nzur\nrobes\n##phic\nadmitting\napi\ndeported\nturmoil\npeyton\nearthquakes\n##ares\nnationalists\nbeau\nclair\nbrethren\ninterrupt\nwelch\ncurated\ngalerie\nrequesting\n164\n##ested\nimpending\nsteward\nviper\n##vina\ncomplaining\nbeautifully\nbrandy\nfoam\nnl\n1660\n##cake\nalessandro\npunches\nlaced\nexplanations\n##lim\nattribute\nclit\nreggie\ndiscomfort\n##cards\nsmoothed\nwhales\n##cene\nadler\ncountered\nduffy\ndisciplinary\nwidening\nrecipe\nreliance\nconducts\ngoats\ngradient\npreaching\n##shaw\nmatilda\nquasi\nstriped\nmeridian\ncannabis\ncordoba\ncertificates\n##agh\n##tering\ngraffiti\nhangs\npilgrims\nrepeats\n##ych\nrevive\nurine\netat\n##hawk\nfueled\nbelts\nfuzzy\nsusceptible\n##hang\nmauritius\nsalle\nsincere\nbeers\nhooks\n##cki\narbitration\nentrusted\nadvise\nsniffed\nseminar\njunk\ndonnell\nprocessors\nprincipality\nstrapped\ncelia\nmendoza\neverton\nfortunes\nprejudice\nstarving\nreassigned\nsteamer\n##lund\ntuck\nevenly\nforeman\n##ffen\ndans\n375\nenvisioned\nslit\n##xy\nbaseman\nliberia\nrosemary\n##weed\nelectrified\nperiodically\npotassium\nstride\ncontexts\nsperm\nslade\nmariners\ninflux\nbianca\nsubcommittee\n##rane\nspilling\nicao\nestuary\n##nock\ndelivers\niphone\n##ulata\nisa\nmira\nbohemian\ndessert\n##sbury\nwelcoming\nproudly\nslowing\n##chs\nmusee\nascension\nruss\n##vian\nwaits\n##psy\nafricans\nexploit\n##morphic\ngov\neccentric\ncrab\npeck\n##ull\nentrances\nformidable\nmarketplace\ngroom\nbolted\nmetabolism\npatton\nrobbins\ncourier\npayload\nendure\n##ifier\nandes\nrefrigerator\n##pr\nornate\n##uca\nruthless\nillegitimate\nmasonry\nstrasbourg\nbikes\nadobe\n##³\napples\nquintet\nwillingly\nniche\nbakery\ncorpses\nenergetic\n##cliffe\n##sser\n##ards\n177\ncentimeters\ncentro\nfuscous\ncretaceous\nrancho\n##yde\nandrei\ntelecom\ntottenham\noasis\nordination\nvulnerability\npresiding\ncorey\ncp\npenguins\nsims\n##pis\nmalawi\npiss\n##48\ncorrection\n##cked\n##ffle\n##ryn\ncountdown\ndetectives\npsychiatrist\npsychedelic\ndinosaurs\nblouse\n##get\nchoi\nvowed\n##oz\nrandomly\n##pol\n49ers\nscrub\nblanche\nbruins\ndusseldorf\n##using\nunwanted\n##ums\n212\ndominique\nelevations\nheadlights\nom\nlaguna\n##oga\n1750\nfamously\nignorance\nshrewsbury\n##aine\najax\nbreuning\nche\nconfederacy\ngreco\noverhaul\n##screen\npaz\nskirts\ndisagreement\ncruelty\njagged\nphoebe\nshifter\nhovered\nviruses\n##wes\nmandy\n##lined\n##gc\nlandlord\nsquirrel\ndashed\n##ι\nornamental\ngag\nwally\ngrange\nliteral\nspurs\nundisclosed\nproceeding\nyin\n##text\nbillie\norphan\nspanned\nhumidity\nindy\nweighted\npresentations\nexplosions\nlucian\n##tary\nvaughn\nhindus\n##anga\n##hell\npsycho\n171\ndaytona\nprotects\nefficiently\nrematch\nsly\ntandem\n##oya\nrebranded\nimpaired\nhee\nmetropolis\npeach\ngodfrey\ndiaspora\nethnicity\nprosperous\ngleaming\ndar\ngrossing\nplayback\n##rden\nstripe\npistols\n##tain\nbirths\nlabelled\n##cating\n172\nrudy\nalba\n##onne\naquarium\nhostility\n##gb\n##tase\nshudder\nsumatra\nhardest\nlakers\nconsonant\ncreeping\ndemos\nhomicide\ncapsule\nzeke\nliberties\nexpulsion\npueblo\n##comb\ntrait\ntransporting\n##ddin\n##neck\n##yna\ndepart\ngregg\nmold\nledge\nhangar\noldham\nplayboy\ntermination\nanalysts\ngmbh\nromero\n##itic\ninsist\ncradle\nfilthy\nbrightness\nslash\nshootout\ndeposed\nbordering\n##truct\nisis\nmicrowave\ntumbled\nsheltered\ncathy\nwerewolves\nmessy\nandersen\nconvex\nclapped\nclinched\nsatire\nwasting\nedo\nvc\nrufus\n##jak\nmont\n##etti\npoznan\n##keeping\nrestructuring\ntransverse\n##rland\nazerbaijani\nslovene\ngestures\nroommate\nchoking\nshear\n##quist\nvanguard\noblivious\n##hiro\ndisagreed\nbaptism\n##lich\ncoliseum\n##aceae\nsalvage\nsociete\ncory\nlocke\nrelocation\nrelying\nversailles\nahl\nswelling\n##elo\ncheerful\n##word\n##edes\ngin\nsarajevo\nobstacle\ndiverted\n##nac\nmessed\nthoroughbred\nfluttered\nutrecht\nchewed\nacquaintance\nassassins\ndispatch\nmirza\n##wart\nnike\nsalzburg\nswell\nyen\n##gee\nidle\nligue\nsamson\n##nds\n##igh\nplayful\nspawned\n##cise\ntease\n##case\nburgundy\n##bot\nstirring\nskeptical\ninterceptions\nmarathi\n##dies\nbedrooms\naroused\npinch\n##lik\npreferences\ntattoos\nbuster\ndigitally\nprojecting\nrust\n##ital\nkitten\npriorities\naddison\npseudo\n##guard\ndusk\nicons\nsermon\n##psis\n##iba\nbt\n##lift\n##xt\nju\ntruce\nrink\n##dah\n##wy\ndefects\npsychiatry\noffences\ncalculate\nglucose\n##iful\n##rized\n##unda\nfrancaise\n##hari\nrichest\nwarwickshire\ncarly\n1763\npurity\nredemption\nlending\n##cious\nmuse\nbruises\ncerebral\naero\ncarving\n##name\npreface\nterminology\ninvade\nmonty\n##int\nanarchist\nblurred\n##iled\nrossi\ntreats\nguts\nshu\nfoothills\nballads\nundertaking\npremise\ncecilia\naffiliates\nblasted\nconditional\nwilder\nminors\ndrone\nrudolph\nbuffy\nswallowing\nhorton\nattested\n##hop\nrutherford\nhowell\nprimetime\nlivery\npenal\n##bis\nminimize\nhydro\nwrecked\nwrought\npalazzo\n##gling\ncans\nvernacular\nfriedman\nnobleman\nshale\nwalnut\ndanielle\n##ection\n##tley\nsears\n##kumar\nchords\nlend\nflipping\nstreamed\npor\ndracula\ngallons\nsacrifices\ngamble\norphanage\n##iman\nmckenzie\n##gible\nboxers\ndaly\n##balls\n##ان\n208\n##ific\n##rative\n##iq\nexploited\nslated\n##uity\ncircling\nhillary\npinched\ngoldberg\nprovost\ncampaigning\nlim\npiles\nironically\njong\nmohan\nsuccessors\nusaf\n##tem\n##ught\nautobiographical\nhaute\npreserves\n##ending\nacquitted\ncomparisons\n203\nhydroelectric\ngangs\ncypriot\ntorpedoes\nrushes\nchrome\nderive\nbumps\ninstability\nfiat\npets\n##mbe\nsilas\ndye\nreckless\nsettler\n##itation\ninfo\nheats\n##writing\n176\ncanonical\nmaltese\nfins\nmushroom\nstacy\naspen\navid\n##kur\n##loading\nvickers\ngaston\nhillside\nstatutes\nwilde\ngail\nkung\nsabine\ncomfortably\nmotorcycles\n##rgo\n169\npneumonia\nfetch\n##sonic\naxel\nfaintly\nparallels\n##oop\nmclaren\nspouse\ncompton\ninterdisciplinary\nminer\n##eni\n181\nclamped\n##chal\n##llah\nseparates\nversa\n##mler\nscarborough\nlabrador\n##lity\n##osing\nrutgers\nhurdles\ncomo\n166\nburt\ndivers\n##100\nwichita\ncade\ncoincided\n##erson\nbruised\nmla\n##pper\nvineyard\n##ili\n##brush\nnotch\nmentioning\njase\nhearted\nkits\ndoe\n##acle\npomerania\n##ady\nronan\nseizure\npavel\nproblematic\n##zaki\ndomenico\n##ulin\ncatering\npenelope\ndependence\nparental\nemilio\nministerial\natkinson\n##bolic\nclarkson\nchargers\ncolby\ngrill\npeeked\narises\nsummon\n##aged\nfools\n##grapher\nfaculties\nqaeda\n##vial\ngarner\nrefurbished\n##hwa\ngeelong\ndisasters\nnudged\nbs\nshareholder\nlori\nalgae\nreinstated\nrot\n##ades\n##nous\ninvites\nstainless\n183\ninclusive\n##itude\ndiocesan\ntil\n##icz\ndenomination\n##xa\nbenton\nfloral\nregisters\n##ider\n##erman\n##kell\nabsurd\nbrunei\nguangzhou\nhitter\nretaliation\n##uled\n##eve\nblanc\nnh\nconsistency\ncontamination\n##eres\n##rner\ndire\npalermo\nbroadcasters\ndiaries\ninspire\nvols\nbrewer\ntightening\nky\nmixtape\nhormone\n##tok\nstokes\n##color\n##dly\n##ssi\npg\n##ometer\n##lington\nsanitation\n##tility\nintercontinental\napps\n##adt\n¹⁄₂\ncylinders\neconomies\nfavourable\nunison\ncroix\ngertrude\nodyssey\nvanity\ndangling\n##logists\nupgrades\ndice\nmiddleweight\npractitioner\n##ight\n206\nhenrik\nparlor\norion\nangered\nlac\npython\nblurted\n##rri\nsensual\nintends\nswings\nangled\n##phs\nhusky\nattain\npeerage\nprecinct\ntextiles\ncheltenham\nshuffled\ndai\nconfess\ntasting\nbhutan\n##riation\ntyrone\nsegregation\nabrupt\nruiz\n##rish\nsmirked\nblackwell\nconfidential\nbrowning\namounted\n##put\nvase\nscarce\nfabulous\nraided\nstaple\nguyana\nunemployed\nglider\nshay\n##tow\ncarmine\ntroll\nintervene\nsquash\nsuperstar\n##uce\ncylindrical\nlen\nroadway\nresearched\nhandy\n##rium\n##jana\nmeta\nlao\ndeclares\n##rring\n##tadt\n##elin\n##kova\nwillem\nshrubs\nnapoleonic\nrealms\nskater\nqi\nvolkswagen\n##ł\ntad\nhara\narchaeologist\nawkwardly\neerie\n##kind\nwiley\n##heimer\n##24\ntitus\norganizers\ncfl\ncrusaders\nlama\nusb\nvent\nenraged\nthankful\noccupants\nmaximilian\n##gaard\npossessing\ntextbooks\n##oran\ncollaborator\nquaker\n##ulo\navalanche\nmono\nsilky\nstraits\nisaiah\nmustang\nsurged\nresolutions\npotomac\ndescend\ncl\nkilograms\nplato\nstrains\nsaturdays\n##olin\nbernstein\n##ype\nholstein\nponytail\n##watch\nbelize\nconversely\nheroine\nperpetual\n##ylus\ncharcoal\npiedmont\nglee\nnegotiating\nbackdrop\nprologue\n##jah\n##mmy\npasadena\nclimbs\nramos\nsunni\n##holm\n##tner\n##tri\nanand\ndeficiency\nhertfordshire\nstout\n##avi\naperture\norioles\n##irs\ndoncaster\nintrigued\nbombed\ncoating\notis\n##mat\ncocktail\n##jit\n##eto\namir\narousal\nsar\n##proof\n##act\n##ories\ndixie\npots\n##bow\nwhereabouts\n159\n##fted\ndrains\nbullying\ncottages\nscripture\ncoherent\nfore\npoe\nappetite\n##uration\nsampled\n##ators\n##dp\nderrick\nrotor\njays\npeacock\ninstallment\n##rro\nadvisors\n##coming\nrodeo\nscotch\n##mot\n##db\n##fen\n##vant\nensued\nrodrigo\ndictatorship\nmartyrs\ntwenties\n##н\ntowed\nincidence\nmarta\nrainforest\nsai\nscaled\n##cles\noceanic\nqualifiers\nsymphonic\nmcbride\ndislike\ngeneralized\naubrey\ncolonization\n##iation\n##lion\n##ssing\ndisliked\nlublin\nsalesman\n##ulates\nspherical\nwhatsoever\nsweating\navalon\ncontention\npunt\nseverity\nalderman\natari\n##dina\n##grant\n##rop\nscarf\nseville\nvertices\nannexation\nfairfield\nfascination\ninspiring\nlaunches\npalatinate\nregretted\n##rca\nferal\n##iom\nelk\nnap\nolsen\nreddy\nyong\n##leader\n##iae\ngarment\ntransports\nfeng\ngracie\noutrage\nviceroy\ninsides\n##esis\nbreakup\ngrady\norganizer\nsofter\ngrimaced\n222\nmurals\ngalicia\narranging\nvectors\n##rsten\nbas\n##sb\n##cens\nsloan\n##eka\nbitten\nara\nfender\nnausea\nbumped\nkris\nbanquet\ncomrades\ndetector\npersisted\n##llan\nadjustment\nendowed\ncinemas\n##shot\nsellers\n##uman\npeek\nepa\nkindly\nneglect\nsimpsons\ntalon\nmausoleum\nrunaway\nhangul\nlookout\n##cic\nrewards\ncoughed\nacquainted\nchloride\n##ald\nquicker\naccordion\nneolithic\n##qa\nartemis\ncoefficient\nlenny\npandora\ntx\n##xed\necstasy\nlitter\nsegunda\nchairperson\ngemma\nhiss\nrumor\nvow\nnasal\nantioch\ncompensate\npatiently\ntransformers\n##eded\njudo\nmorrow\npenis\nposthumous\nphilips\nbandits\nhusbands\ndenote\nflaming\n##any\n##phones\nlangley\nyorker\n1760\nwalters\n##uo\n##kle\ngubernatorial\nfatty\nsamsung\nleroy\noutlaw\n##nine\nunpublished\npoole\njakob\n##ᵢ\n##ₙ\ncrete\ndistorted\nsuperiority\n##dhi\nintercept\ncrust\nmig\nclaus\ncrashes\npositioning\n188\nstallion\n301\nfrontal\narmistice\n##estinal\nelton\naj\nencompassing\ncamel\ncommemorated\nmalaria\nwoodward\ncalf\ncigar\npenetrate\n##oso\nwillard\n##rno\n##uche\nillustrate\namusing\nconvergence\nnoteworthy\n##lma\n##rva\njourneys\nrealise\nmanfred\n##sable\n410\n##vocation\nhearings\nfiance\n##posed\neducators\nprovoked\nadjusting\n##cturing\nmodular\nstockton\npaterson\nvlad\nrejects\nelectors\nselena\nmaureen\n##tres\nuber\n##rce\nswirled\n##num\nproportions\nnanny\npawn\nnaturalist\nparma\napostles\nawoke\nethel\nwen\n##bey\nmonsoon\noverview\n##inating\nmccain\nrendition\nrisky\nadorned\n##ih\nequestrian\ngermain\nnj\nconspicuous\nconfirming\n##yoshi\nshivering\n##imeter\nmilestone\nrumours\nflinched\nbounds\nsmacked\ntoken\n##bei\nlectured\nautomobiles\n##shore\nimpacted\n##iable\nnouns\nnero\n##leaf\nismail\nprostitute\ntrams\n##lace\nbridget\nsud\nstimulus\nimpressions\nreins\nrevolves\n##oud\n##gned\ngiro\nhoneymoon\n##swell\ncriterion\n##sms\n##uil\nlibyan\nprefers\n##osition\n211\npreview\nsucks\naccusation\nbursts\nmetaphor\ndiffusion\ntolerate\nfaye\nbetting\ncinematographer\nliturgical\nspecials\nbitterly\nhumboldt\n##ckle\nflux\nrattled\n##itzer\narchaeologists\nodor\nauthorised\nmarshes\ndiscretion\n##ов\nalarmed\narchaic\ninverse\n##leton\nexplorers\n##pine\ndrummond\ntsunami\nwoodlands\n##minate\n##tland\nbooklet\ninsanity\nowning\ninsert\ncrafted\ncalculus\n##tore\nreceivers\n##bt\nstung\n##eca\n##nched\nprevailing\ntravellers\neyeing\nlila\ngraphs\n##borne\n178\njulien\n##won\nmorale\nadaptive\ntherapist\nerica\ncw\nlibertarian\nbowman\npitches\nvita\n##ional\ncrook\n##ads\n##entation\ncaledonia\nmutiny\n##sible\n1840s\nautomation\n##ß\nflock\n##pia\nironic\npathology\n##imus\nremarried\n##22\njoker\nwithstand\nenergies\n##att\nshropshire\nhostages\nmadeleine\ntentatively\nconflicting\nmateo\nrecipes\neuros\nol\nmercenaries\nnico\n##ndon\nalbuquerque\naugmented\nmythical\nbel\nfreud\n##child\ncough\n##lica\n365\nfreddy\nlillian\ngenetically\nnuremberg\ncalder\n209\nbonn\noutdoors\npaste\nsuns\nurgency\nvin\nrestraint\ntyson\n##cera\n##selle\nbarrage\nbethlehem\nkahn\n##par\nmounts\nnippon\nbarony\nhappier\nryu\nmakeshift\nsheldon\nblushed\ncastillo\nbarking\nlistener\ntaped\nbethel\nfluent\nheadlines\npornography\nrum\ndisclosure\nsighing\nmace\ndoubling\ngunther\nmanly\n##plex\nrt\ninterventions\nphysiological\nforwards\nemerges\n##tooth\n##gny\ncompliment\nrib\nrecession\nvisibly\nbarge\nfaults\nconnector\nexquisite\nprefect\n##rlin\npatio\n##cured\nelevators\nbrandt\nitalics\npena\n173\nwasp\nsatin\nea\nbotswana\ngraceful\nrespectable\n##jima\n##rter\n##oic\nfranciscan\ngenerates\n##dl\nalfredo\ndisgusting\n##olate\n##iously\nsherwood\nwarns\ncod\npromo\ncheryl\nsino\n##ة\n##escu\ntwitch\n##zhi\nbrownish\nthom\nortiz\n##dron\ndensely\n##beat\ncarmel\nreinforce\n##bana\n187\nanastasia\ndownhill\nvertex\ncontaminated\nremembrance\nharmonic\nhomework\n##sol\nfiancee\ngears\nolds\nangelica\nloft\nramsay\nquiz\ncolliery\nsevens\n##cape\nautism\n##hil\nwalkway\n##boats\nruben\nabnormal\nounce\nkhmer\n##bbe\nzachary\nbedside\nmorphology\npunching\n##olar\nsparrow\nconvinces\n##35\nhewitt\nqueer\nremastered\nrods\nmabel\nsolemn\nnotified\nlyricist\nsymmetric\n##xide\n174\nencore\npassports\nwildcats\n##uni\nbaja\n##pac\nmildly\n##ease\nbleed\ncommodity\nmounds\nglossy\norchestras\n##omo\ndamian\nprelude\nambitions\n##vet\nawhile\nremotely\n##aud\nasserts\nimply\n##iques\ndistinctly\nmodelling\nremedy\n##dded\nwindshield\ndani\nxiao\n##endra\naudible\npowerplant\n1300\ninvalid\nelemental\nacquisitions\n##hala\nimmaculate\nlibby\nplata\nsmuggling\nventilation\ndenoted\nminh\n##morphism\n430\ndiffered\ndion\nkelley\nlore\nmocking\nsabbath\nspikes\nhygiene\ndrown\nrunoff\nstylized\ntally\nliberated\naux\ninterpreter\nrighteous\naba\nsiren\nreaper\npearce\nmillie\n##cier\n##yra\ngaius\n##iso\ncaptures\n##ttering\ndorm\nclaudio\n##sic\nbenches\nknighted\nblackness\n##ored\ndiscount\nfumble\noxidation\nrouted\n##ς\nnovak\nperpendicular\nspoiled\nfracture\nsplits\n##urt\npads\ntopology\n##cats\naxes\nfortunate\noffenders\nprotestants\nesteem\n221\nbroadband\nconvened\nfrankly\nhound\nprototypes\nisil\nfacilitated\nkeel\n##sher\nsahara\nawaited\nbubba\norb\nprosecutors\n186\nhem\n520\n##xing\nrelaxing\nremnant\nromney\nsorted\nslalom\nstefano\nulrich\n##active\nexemption\nfolder\npauses\nfoliage\nhitchcock\nepithet\n204\ncriticisms\n##aca\nballistic\nbrody\nhinduism\nchaotic\nyouths\nequals\n##pala\npts\nthicker\nanalogous\ncapitalist\nimprovised\noverseeing\nsinatra\nascended\nbeverage\n##tl\nstraightforward\n##kon\ncurran\n##west\nbois\n325\ninduce\nsurveying\nemperors\nsax\nunpopular\n##kk\ncartoonist\nfused\n##mble\nunto\n##yuki\nlocalities\n##cko\n##ln\ndarlington\nslain\nacademie\nlobbying\nsediment\npuzzles\n##grass\ndefiance\ndickens\nmanifest\ntongues\nalumnus\narbor\ncoincide\n184\nappalachian\nmustafa\nexaminer\ncabaret\ntraumatic\nyves\nbracelet\ndraining\nheroin\nmagnum\nbaths\nodessa\nconsonants\nmitsubishi\n##gua\nkellan\nvaudeville\n##fr\njoked\nnull\nstraps\nprobation\n##ław\nceded\ninterfaces\n##pas\n##zawa\nblinding\nviet\n224\nrothschild\nmuseo\n640\nhuddersfield\n##vr\ntactic\n##storm\nbrackets\ndazed\nincorrectly\n##vu\nreg\nglazed\nfearful\nmanifold\nbenefited\nirony\n##sun\nstumbling\n##rte\nwillingness\nbalkans\nmei\nwraps\n##aba\ninjected\n##lea\ngu\nsyed\nharmless\n##hammer\nbray\ntakeoff\npoppy\ntimor\ncardboard\nastronaut\npurdue\nweeping\nsouthbound\ncursing\nstalls\ndiagonal\n##neer\nlamar\nbryce\ncomte\nweekdays\nharrington\n##uba\nnegatively\n##see\nlays\ngrouping\n##cken\n##henko\naffirmed\nhalle\nmodernist\n##lai\nhodges\nsmelling\naristocratic\nbaptized\ndismiss\njustification\noilers\n##now\ncoupling\nqin\nsnack\nhealer\n##qing\ngardener\nlayla\nbattled\nformulated\nstephenson\ngravitational\n##gill\n##jun\n1768\ngranny\ncoordinating\nsuites\n##cd\n##ioned\nmonarchs\n##cote\n##hips\nsep\nblended\napr\nbarrister\ndeposition\nfia\nmina\npolicemen\nparanoid\n##pressed\nchurchyard\ncovert\ncrumpled\ncreep\nabandoning\ntr\ntransmit\nconceal\nbarr\nunderstands\nreadiness\nspire\n##cology\n##enia\n##erry\n610\nstartling\nunlock\nvida\nbowled\nslots\n##nat\n##islav\nspaced\ntrusting\nadmire\nrig\n##ink\nslack\n##70\nmv\n207\ncasualty\n##wei\nclassmates\n##odes\n##rar\n##rked\namherst\nfurnished\nevolve\nfoundry\nmenace\nmead\n##lein\nflu\nwesleyan\n##kled\nmonterey\nwebber\n##vos\nwil\n##mith\n##на\nbartholomew\njustices\nrestrained\n##cke\namenities\n191\nmediated\nsewage\ntrenches\nml\nmainz\n##thus\n1800s\n##cula\n##inski\ncaine\nbonding\n213\nconverts\nspheres\nsuperseded\nmarianne\ncrypt\nsweaty\nensign\nhistoria\n##br\nspruce\n##post\n##ask\nforks\nthoughtfully\nyukon\npamphlet\names\n##uter\nkarma\n##yya\nbryn\nnegotiation\nsighs\nincapable\n##mbre\n##ntial\nactresses\ntaft\n##mill\nluce\nprevailed\n##amine\n1773\nmotionless\nenvoy\ntestify\ninvesting\nsculpted\ninstructors\nprovence\nkali\ncullen\nhorseback\n##while\ngoodwin\n##jos\ngaa\nnorte\n##ldon\nmodify\nwavelength\nabd\n214\nskinned\nsprinter\nforecast\nscheduling\nmarries\nsquared\ntentative\n##chman\nboer\n##isch\nbolts\nswap\nfisherman\nassyrian\nimpatiently\nguthrie\nmartins\nmurdoch\n194\ntanya\nnicely\ndolly\nlacy\nmed\n##45\nsyn\ndecks\nfashionable\nmillionaire\n##ust\nsurfing\n##ml\n##ision\nheaved\ntammy\nconsulate\nattendees\nroutinely\n197\nfuse\nsaxophonist\nbackseat\nmalaya\n##lord\nscowl\ntau\n##ishly\n193\nsighted\nsteaming\n##rks\n303\n911\n##holes\n##hong\nching\n##wife\nbless\nconserved\njurassic\nstacey\nunix\nzion\nchunk\nrigorous\nblaine\n198\npeabody\nslayer\ndismay\nbrewers\nnz\n##jer\ndet\n##glia\nglover\npostwar\nint\npenetration\nsylvester\nimitation\nvertically\nairlift\nheiress\nknoxville\nviva\n##uin\n390\nmacon\n##rim\n##fighter\n##gonal\njanice\n##orescence\n##wari\nmarius\nbelongings\nleicestershire\n196\nblanco\ninverted\npreseason\nsanity\nsobbing\n##due\n##elt\n##dled\ncollingwood\nregeneration\nflickering\nshortest\n##mount\n##osi\nfeminism\n##lat\nsherlock\ncabinets\nfumbled\nnorthbound\nprecedent\nsnaps\n##mme\nresearching\n##akes\nguillaume\ninsights\nmanipulated\nvapor\nneighbour\nsap\ngangster\nfrey\nf1\nstalking\nscarcely\ncallie\nbarnett\ntendencies\naudi\ndoomed\nassessing\nslung\npanchayat\nambiguous\nbartlett\n##etto\ndistributing\nviolating\nwolverhampton\n##hetic\nswami\nhistoire\n##urus\nliable\npounder\ngroin\nhussain\nlarsen\npopping\nsurprises\n##atter\nvie\ncurt\n##station\nmute\nrelocate\nmusicals\nauthorization\nrichter\n##sef\nimmortality\ntna\nbombings\n##press\ndeteriorated\nyiddish\n##acious\nrobbed\ncolchester\ncs\npmid\nao\nverified\nbalancing\napostle\nswayed\nrecognizable\noxfordshire\nretention\nnottinghamshire\ncontender\njudd\ninvitational\nshrimp\nuhf\n##icient\ncleaner\nlongitudinal\ntanker\n##mur\nacronym\nbroker\nkoppen\nsundance\nsuppliers\n##gil\n4000\nclipped\nfuels\npetite\n##anne\nlandslide\nhelene\ndiversion\npopulous\nlandowners\nauspices\nmelville\nquantitative\n##xes\nferries\nnicky\n##llus\ndoo\nhaunting\nroche\ncarver\ndowned\nunavailable\n##pathy\napproximation\nhiroshima\n##hue\ngarfield\nvalle\ncomparatively\nkeyboardist\ntraveler\n##eit\ncongestion\ncalculating\nsubsidiaries\n##bate\nserb\nmodernization\nfairies\ndeepened\nville\naverages\n##lore\ninflammatory\ntonga\n##itch\nco₂\nsquads\n##hea\ngigantic\nserum\nenjoyment\nretailer\nverona\n35th\ncis\n##phobic\nmagna\ntechnicians\n##vati\narithmetic\n##sport\nlevin\n##dation\namtrak\nchow\nsienna\n##eyer\nbackstage\nentrepreneurship\n##otic\nlearnt\ntao\n##udy\nworcestershire\nformulation\nbaggage\nhesitant\nbali\nsabotage\n##kari\nbarren\nenhancing\nmurmur\npl\nfreshly\nputnam\nsyntax\naces\nmedicines\nresentment\nbandwidth\n##sier\ngrins\nchili\nguido\n##sei\nframing\nimplying\ngareth\nlissa\ngenevieve\npertaining\nadmissions\ngeo\nthorpe\nproliferation\nsato\nbela\nanalyzing\nparting\n##gor\nawakened\n##isman\nhuddled\nsecrecy\n##kling\nhush\ngentry\n540\ndungeons\n##ego\ncoasts\n##utz\nsacrificed\n##chule\nlandowner\nmutually\nprevalence\nprogrammer\nadolescent\ndisrupted\nseaside\ngee\ntrusts\nvamp\ngeorgie\n##nesian\n##iol\nschedules\nsindh\n##market\netched\nhm\nsparse\nbey\nbeaux\nscratching\ngliding\nunidentified\n216\ncollaborating\ngems\njesuits\noro\naccumulation\nshaping\nmbe\nanal\n##xin\n231\nenthusiasts\nnewscast\n##egan\njanata\ndewey\nparkinson\n179\nankara\nbiennial\ntowering\ndd\ninconsistent\n950\n##chet\nthriving\nterminate\ncabins\nfuriously\neats\nadvocating\ndonkey\nmarley\nmuster\nphyllis\nleiden\n##user\ngrassland\nglittering\niucn\nloneliness\n217\nmemorandum\narmenians\n##ddle\npopularized\nrhodesia\n60s\nlame\n##illon\nsans\nbikini\nheader\norbits\n##xx\n##finger\n##ulator\nsharif\nspines\nbiotechnology\nstrolled\nnaughty\nyates\n##wire\nfremantle\nmilo\n##mour\nabducted\nremoves\n##atin\nhumming\nwonderland\n##chrome\n##ester\nhume\npivotal\n##rates\narmand\ngrams\nbelievers\nelector\nrte\napron\nbis\nscraped\n##yria\nendorsement\ninitials\n##llation\neps\ndotted\nhints\nbuzzing\nemigration\nnearer\n##tom\nindicators\n##ulu\ncoarse\nneutron\nprotectorate\n##uze\ndirectional\nexploits\npains\nloire\n1830s\nproponents\nguggenheim\nrabbits\nritchie\n305\nhectare\ninputs\nhutton\n##raz\nverify\n##ako\nboilers\nlongitude\n##lev\nskeletal\nyer\nemilia\ncitrus\ncompromised\n##gau\npokemon\nprescription\nparagraph\neduard\ncadillac\nattire\ncategorized\nkenyan\nweddings\ncharley\n##bourg\nentertain\nmonmouth\n##lles\nnutrients\ndavey\nmesh\nincentive\npractised\necosystems\nkemp\nsubdued\noverheard\n##rya\nbodily\nmaxim\n##nius\napprenticeship\nursula\n##fight\nlodged\nrug\nsilesian\nunconstitutional\npatel\ninspected\ncoyote\nunbeaten\n##hak\n34th\ndisruption\nconvict\nparcel\n##cl\n##nham\ncollier\nimplicated\nmallory\n##iac\n##lab\nsusannah\nwinkler\n##rber\nshia\nphelps\nsediments\ngraphical\nrobotic\n##sner\nadulthood\nmart\nsmoked\n##isto\nkathryn\nclarified\n##aran\ndivides\nconvictions\noppression\npausing\nburying\n##mt\nfederico\nmathias\neileen\n##tana\nkite\nhunched\n##acies\n189\n##atz\ndisadvantage\nliza\nkinetic\ngreedy\nparadox\nyokohama\ndowager\ntrunks\nventured\n##gement\ngupta\nvilnius\nolaf\n##thest\ncrimean\nhopper\n##ej\nprogressively\narturo\nmouthed\narrondissement\n##fusion\nrubin\nsimulcast\noceania\n##orum\n##stra\n##rred\nbusiest\nintensely\nnavigator\ncary\n##vine\n##hini\n##bies\nfife\nrowe\nrowland\nposing\ninsurgents\nshafts\nlawsuits\nactivate\nconor\ninward\nculturally\ngarlic\n265\n##eering\neclectic\n##hui\n##kee\n##nl\nfurrowed\nvargas\nmeteorological\nrendezvous\n##aus\nculinary\ncommencement\n##dition\nquota\n##notes\nmommy\nsalaries\noverlapping\nmule\n##iology\n##mology\nsums\nwentworth\n##isk\n##zione\nmainline\nsubgroup\n##illy\nhack\nplaintiff\nverdi\nbulb\ndifferentiation\nengagements\nmultinational\nsupplemented\nbertrand\ncaller\nregis\n##naire\n##sler\n##arts\n##imated\nblossom\npropagation\nkilometer\nviaduct\nvineyards\n##uate\nbeckett\noptimization\ngolfer\nsongwriters\nseminal\nsemitic\nthud\nvolatile\nevolving\nridley\n##wley\ntrivial\ndistributions\nscandinavia\njiang\n##ject\nwrestled\ninsistence\n##dio\nemphasizes\nnapkin\n##ods\nadjunct\nrhyme\n##ricted\n##eti\nhopeless\nsurrounds\ntremble\n32nd\nsmoky\n##ntly\noils\nmedicinal\npadded\nsteer\nwilkes\n219\n255\nconcessions\nhue\nuniquely\nblinded\nlandon\nyahoo\n##lane\nhendrix\ncommemorating\ndex\nspecify\nchicks\n##ggio\nintercity\n1400\nmorley\n##torm\nhighlighting\n##oting\npang\noblique\nstalled\n##liner\nflirting\nnewborn\n1769\nbishopric\nshaved\n232\ncurrie\n##ush\ndharma\nspartan\n##ooped\nfavorites\nsmug\nnovella\nsirens\nabusive\ncreations\nespana\n##lage\nparadigm\nsemiconductor\nsheen\n##rdo\n##yen\n##zak\nnrl\nrenew\n##pose\n##tur\nadjutant\nmarches\nnorma\n##enity\nineffective\nweimar\ngrunt\n##gat\nlordship\nplotting\nexpenditure\ninfringement\nlbs\nrefrain\nav\nmimi\nmistakenly\npostmaster\n1771\n##bara\nras\nmotorsports\ntito\n199\nsubjective\n##zza\nbully\nstew\n##kaya\nprescott\n1a\n##raphic\n##zam\nbids\nstyling\nparanormal\nreeve\nsneaking\nexploding\nkatz\nakbar\nmigrant\nsyllables\nindefinitely\n##ogical\ndestroys\nreplaces\napplause\n##phine\npest\n##fide\n218\narticulated\nbertie\n##thing\n##cars\n##ptic\ncourtroom\ncrowley\naesthetics\ncummings\ntehsil\nhormones\ntitanic\ndangerously\n##ibe\nstadion\njaenelle\nauguste\nciudad\n##chu\nmysore\npartisans\n##sio\nlucan\nphilipp\n##aly\ndebating\nhenley\ninteriors\n##rano\n##tious\nhomecoming\nbeyonce\nusher\nhenrietta\nprepares\nweeds\n##oman\nely\nplucked\n##pire\n##dable\nluxurious\n##aq\nartifact\npassword\npasture\njuno\nmaddy\nminsk\n##dder\n##ologies\n##rone\nassessments\nmartian\nroyalist\n1765\nexamines\n##mani\n##rge\nnino\n223\nparry\nscooped\nrelativity\n##eli\n##uting\n##cao\ncongregational\nnoisy\ntraverse\n##agawa\nstrikeouts\nnickelodeon\nobituary\ntransylvania\nbinds\ndepictions\npolk\ntrolley\n##yed\n##lard\nbreeders\n##under\ndryly\nhokkaido\n1762\nstrengths\nstacks\nbonaparte\nconnectivity\nneared\nprostitutes\nstamped\nanaheim\ngutierrez\nsinai\n##zzling\nbram\nfresno\nmadhya\n##86\nproton\n##lena\n##llum\n##phon\nreelected\nwanda\n##anus\n##lb\nample\ndistinguishing\n##yler\ngrasping\nsermons\ntomato\nbland\nstimulation\navenues\n##eux\nspreads\nscarlett\nfern\npentagon\nassert\nbaird\nchesapeake\nir\ncalmed\ndistortion\nfatalities\n##olis\ncorrectional\npricing\n##astic\n##gina\nprom\ndammit\nying\ncollaborate\n##chia\nwelterweight\n33rd\npointer\nsubstitution\nbonded\numpire\ncommunicating\nmultitude\npaddle\n##obe\nfederally\nintimacy\n##insky\nbetray\nssr\n##lett\n##lean\n##lves\n##therapy\nairbus\n##tery\nfunctioned\nud\nbearer\nbiomedical\nnetflix\n##hire\n##nca\ncondom\nbrink\nik\n##nical\nmacy\n##bet\nflap\ngma\nexperimented\njelly\nlavender\n##icles\n##ulia\nmunro\n##mian\n##tial\nrye\n##rle\n60th\ngigs\nhottest\nrotated\npredictions\nfuji\nbu\n##erence\n##omi\nbarangay\n##fulness\n##sas\nclocks\n##rwood\n##liness\ncereal\nroe\nwight\ndecker\nuttered\nbabu\nonion\nxml\nforcibly\n##df\npetra\nsarcasm\nhartley\npeeled\nstorytelling\n##42\n##xley\n##ysis\n##ffa\nfibre\nkiel\nauditor\nfig\nharald\ngreenville\n##berries\ngeographically\nnell\nquartz\n##athic\ncemeteries\n##lr\ncrossings\nnah\nholloway\nreptiles\nchun\nsichuan\nsnowy\n660\ncorrections\n##ivo\nzheng\nambassadors\nblacksmith\nfielded\nfluids\nhardcover\nturnover\nmedications\nmelvin\nacademies\n##erton\nro\nroach\nabsorbing\nspaniards\ncolton\n##founded\noutsider\nespionage\nkelsey\n245\nedible\n##ulf\ndora\nestablishes\n##sham\n##tries\ncontracting\n##tania\ncinematic\ncostello\nnesting\n##uron\nconnolly\nduff\n##nology\nmma\n##mata\nfergus\nsexes\ngi\noptics\nspectator\nwoodstock\nbanning\n##hee\n##fle\ndifferentiate\noutfielder\nrefinery\n226\n312\ngerhard\nhorde\nlair\ndrastically\n##udi\nlandfall\n##cheng\nmotorsport\nodi\n##achi\npredominant\nquay\nskins\n##ental\nedna\nharshly\ncomplementary\nmurdering\n##aves\nwreckage\n##90\nono\noutstretched\nlennox\nmunitions\ngalen\nreconcile\n470\nscalp\nbicycles\ngillespie\nquestionable\nrosenberg\nguillermo\nhostel\njarvis\nkabul\nvolvo\nopium\nyd\n##twined\nabuses\ndecca\noutpost\n##cino\nsensible\nneutrality\n##64\nponce\nanchorage\natkins\nturrets\ninadvertently\ndisagree\nlibre\nvodka\nreassuring\nweighs\n##yal\nglide\njumper\nceilings\nrepertory\nouts\nstain\n##bial\nenvy\n##ucible\nsmashing\nheightened\npolicing\nhyun\nmixes\nlai\nprima\n##ples\nceleste\n##bina\nlucrative\nintervened\nkc\nmanually\n##rned\nstature\nstaffed\nbun\nbastards\nnairobi\npriced\n##auer\nthatcher\n##kia\ntripped\ncomune\n##ogan\n##pled\nbrasil\nincentives\nemanuel\nhereford\nmusica\n##kim\nbenedictine\nbiennale\n##lani\neureka\ngardiner\nrb\nknocks\nsha\n##ael\n##elled\n##onate\nefficacy\nventura\nmasonic\nsanford\nmaize\nleverage\n##feit\ncapacities\nsantana\n##aur\nnovelty\nvanilla\n##cter\n##tour\nbenin\n##oir\n##rain\nneptune\ndrafting\ntallinn\n##cable\nhumiliation\n##boarding\nschleswig\nfabian\nbernardo\nliturgy\nspectacle\nsweeney\npont\nroutledge\n##tment\ncosmos\nut\nhilt\nsleek\nuniversally\n##eville\n##gawa\ntyped\n##dry\nfavors\nallegheny\nglaciers\n##rly\nrecalling\naziz\n##log\nparasite\nrequiem\nauf\n##berto\n##llin\nillumination\n##breaker\n##issa\nfestivities\nbows\ngovern\nvibe\nvp\n333\nsprawled\nlarson\npilgrim\nbwf\nleaping\n##rts\n##ssel\nalexei\ngreyhound\nhoarse\n##dler\n##oration\nseneca\n##cule\ngaping\n##ulously\n##pura\ncinnamon\n##gens\n##rricular\ncraven\nfantasies\nhoughton\nengined\nreigned\ndictator\nsupervising\n##oris\nbogota\ncommentaries\nunnatural\nfingernails\nspirituality\ntighten\n##tm\ncanadiens\nprotesting\nintentional\ncheers\nsparta\n##ytic\n##iere\n##zine\nwiden\nbelgarath\ncontrollers\ndodd\niaaf\nnavarre\n##ication\ndefect\nsquire\nsteiner\nwhisky\n##mins\n560\ninevitably\ntome\n##gold\nchew\n##uid\n##lid\nelastic\n##aby\nstreaked\nalliances\njailed\nregal\n##ined\n##phy\nczechoslovak\nnarration\nabsently\n##uld\nbluegrass\nguangdong\nquran\ncriticizing\nhose\nhari\n##liest\n##owa\nskier\nstreaks\ndeploy\n##lom\nraft\nbose\ndialed\nhuff\n##eira\nhaifa\nsimplest\nbursting\nendings\nib\nsultanate\n##titled\nfranks\nwhitman\nensures\nsven\n##ggs\ncollaborators\nforster\norganising\nui\nbanished\nnapier\ninjustice\nteller\nlayered\nthump\n##otti\nroc\nbattleships\nevidenced\nfugitive\nsadie\nrobotics\n##roud\nequatorial\ngeologist\n##iza\nyielding\n##bron\n##sr\ninternationale\nmecca\n##diment\nsbs\nskyline\ntoad\nuploaded\nreflective\nundrafted\nlal\nleafs\nbayern\n##dai\nlakshmi\nshortlisted\n##stick\n##wicz\ncamouflage\ndonate\naf\nchristi\nlau\n##acio\ndisclosed\nnemesis\n1761\nassemble\nstraining\nnorthamptonshire\ntal\n##asi\nbernardino\npremature\nheidi\n42nd\ncoefficients\ngalactic\nreproduce\nbuzzed\nsensations\nzionist\nmonsieur\nmyrtle\n##eme\narchery\nstrangled\nmusically\nviewpoint\nantiquities\nbei\ntrailers\nseahawks\ncured\npee\npreferring\ntasmanian\nlange\nsul\n##mail\n##working\ncolder\noverland\nlucivar\nmassey\ngatherings\nhaitian\n##smith\ndisapproval\nflaws\n##cco\n##enbach\n1766\nnpr\n##icular\nboroughs\ncreole\nforums\ntechno\n1755\ndent\nabdominal\nstreetcar\n##eson\n##stream\nprocurement\ngemini\npredictable\n##tya\nacheron\nchristoph\nfeeder\nfronts\nvendor\nbernhard\njammu\ntumors\nslang\n##uber\ngoaltender\ntwists\ncurving\nmanson\nvuelta\nmer\npeanut\nconfessions\npouch\nunpredictable\nallowance\ntheodor\nvascular\n##factory\nbala\nauthenticity\nmetabolic\ncoughing\nnanjing\n##cea\npembroke\n##bard\nsplendid\n36th\nff\nhourly\n##ahu\nelmer\nhandel\n##ivate\nawarding\nthrusting\ndl\nexperimentation\n##hesion\n##46\ncaressed\nentertained\nsteak\n##rangle\nbiologist\norphans\nbaroness\noyster\nstepfather\n##dridge\nmirage\nreefs\nspeeding\n##31\nbarons\n1764\n227\ninhabit\npreached\nrepealed\n##tral\nhonoring\nboogie\ncaptives\nadminister\njohanna\n##imate\ngel\nsuspiciously\n1767\nsobs\n##dington\nbackbone\nhayward\ngarry\n##folding\n##nesia\nmaxi\n##oof\n##ppe\nellison\ngalileo\n##stand\ncrimea\nfrenzy\namour\nbumper\nmatrices\nnatalia\nbaking\ngarth\npalestinians\n##grove\nsmack\nconveyed\nensembles\ngardening\n##manship\n##rup\n##stituting\n1640\nharvesting\ntopography\njing\nshifters\ndormitory\n##carriage\n##lston\nist\nskulls\n##stadt\ndolores\njewellery\nsarawak\n##wai\n##zier\nfences\nchristy\nconfinement\ntumbling\ncredibility\nfir\nstench\n##bria\n##plication\n##nged\n##sam\nvirtues\n##belt\nmarjorie\npba\n##eem\n##made\ncelebrates\nschooner\nagitated\nbarley\nfulfilling\nanthropologist\n##pro\nrestrict\nnovi\nregulating\n##nent\npadres\n##rani\n##hesive\nloyola\ntabitha\nmilky\nolson\nproprietor\ncrambidae\nguarantees\nintercollegiate\nljubljana\nhilda\n##sko\nignorant\nhooded\n##lts\nsardinia\n##lidae\n##vation\nfrontman\nprivileged\nwitchcraft\n##gp\njammed\nlaude\npoking\n##than\nbracket\namazement\nyunnan\n##erus\nmaharaja\nlinnaeus\n264\ncommissioning\nmilano\npeacefully\n##logies\nakira\nrani\nregulator\n##36\ngrasses\n##rance\nluzon\ncrows\ncompiler\ngretchen\nseaman\nedouard\ntab\nbuccaneers\nellington\nhamlets\nwhig\nsocialists\n##anto\ndirectorial\neaston\nmythological\n##kr\n##vary\nrhineland\nsemantic\ntaut\ndune\ninventions\nsucceeds\n##iter\nreplication\nbranched\n##pired\njul\nprosecuted\nkangaroo\npenetrated\n##avian\nmiddlesbrough\ndoses\nbleak\nmadam\npredatory\nrelentless\n##vili\nreluctance\n##vir\nhailey\ncrore\nsilvery\n1759\nmonstrous\nswimmers\ntransmissions\nhawthorn\ninforming\n##eral\ntoilets\ncaracas\ncrouch\nkb\n##sett\n295\ncartel\nhadley\n##aling\nalexia\nyvonne\n##biology\ncinderella\neton\nsuperb\nblizzard\nstabbing\nindustrialist\nmaximus\n##gm\n##orus\ngroves\nmaud\nclade\noversized\ncomedic\n##bella\nrosen\nnomadic\nfulham\nmontane\nbeverages\ngalaxies\nredundant\nswarm\n##rot\n##folia\n##llis\nbuckinghamshire\nfen\nbearings\nbahadur\n##rom\ngilles\nphased\ndynamite\nfaber\nbenoit\nvip\n##ount\n##wd\nbooking\nfractured\ntailored\nanya\nspices\nwestwood\ncairns\nauditions\ninflammation\nsteamed\n##rocity\n##acion\n##urne\nskyla\nthereof\nwatford\ntorment\narchdeacon\ntransforms\nlulu\ndemeanor\nfucked\nserge\n##sor\nmckenna\nminas\nentertainer\n##icide\ncaress\noriginate\nresidue\n##sty\n1740\n##ilised\n##org\nbeech\n##wana\nsubsidies\n##ghton\nemptied\ngladstone\nru\nfirefighters\nvoodoo\n##rcle\nhet\nnightingale\ntamara\nedmond\ningredient\nweaknesses\nsilhouette\n285\ncompatibility\nwithdrawing\nhampson\n##mona\nanguish\ngiggling\n##mber\nbookstore\n##jiang\nsouthernmost\ntilting\n##vance\nbai\neconomical\nrf\nbriefcase\ndreadful\nhinted\nprojections\nshattering\ntotaling\n##rogate\nanalogue\nindicted\nperiodical\nfullback\n##dman\nhaynes\n##tenberg\n##ffs\n##ishment\n1745\nthirst\nstumble\npenang\nvigorous\n##ddling\n##kor\n##lium\noctave\n##ove\n##enstein\n##inen\n##ones\nsiberian\n##uti\ncbn\nrepeal\nswaying\n##vington\nkhalid\ntanaka\nunicorn\notago\nplastered\nlobe\nriddle\n##rella\nperch\n##ishing\ncroydon\nfiltered\ngraeme\ntripoli\n##ossa\ncrocodile\n##chers\nsufi\nmined\n##tung\ninferno\nlsu\n##phi\nswelled\nutilizes\n£2\ncale\nperiodicals\nstyx\nhike\ninformally\ncoop\nlund\n##tidae\nala\nhen\nqui\ntransformations\ndisposed\nsheath\nchickens\n##cade\nfitzroy\nsas\nsilesia\nunacceptable\nodisha\n1650\nsabrina\npe\nspokane\nratios\nathena\nmassage\nshen\ndilemma\n##drum\n##riz\n##hul\ncorona\ndoubtful\nniall\n##pha\n##bino\nfines\ncite\nacknowledging\nbangor\nballard\nbathurst\n##resh\nhuron\nmustered\nalzheimer\ngarments\nkinase\ntyre\nwarship\n##cp\nflashback\npulmonary\nbraun\ncheat\nkamal\ncyclists\nconstructions\ngrenades\nndp\ntraveller\nexcuses\nstomped\nsignalling\ntrimmed\nfutsal\nmosques\nrelevance\n##wine\nwta\n##23\n##vah\n##lter\nhoc\n##riding\noptimistic\n##´s\ndeco\nsim\ninteracting\nrejecting\nmoniker\nwaterways\n##ieri\n##oku\nmayors\ngdansk\noutnumbered\npearls\n##ended\n##hampton\nfairs\ntotals\ndominating\n262\nnotions\nstairway\ncompiling\npursed\ncommodities\ngrease\nyeast\n##jong\ncarthage\ngriffiths\nresidual\namc\ncontraction\nlaird\nsapphire\n##marine\n##ivated\namalgamation\ndissolve\ninclination\nlyle\npackaged\naltitudes\nsuez\ncanons\ngraded\nlurched\nnarrowing\nboasts\nguise\nwed\nenrico\n##ovsky\nrower\nscarred\nbree\ncub\niberian\nprotagonists\nbargaining\nproposing\ntrainers\nvoyages\nvans\nfishes\n##aea\n##ivist\n##verance\nencryption\nartworks\nkazan\nsabre\ncleopatra\nhepburn\nrotting\nsupremacy\nmecklenburg\n##brate\nburrows\nhazards\noutgoing\nflair\norganizes\n##ctions\nscorpion\n##usions\nboo\n234\nchevalier\ndunedin\nslapping\n##34\nineligible\npensions\n##38\n##omic\nmanufactures\nemails\nbismarck\n238\nweakening\nblackish\nding\nmcgee\nquo\n##rling\nnorthernmost\nxx\nmanpower\ngreed\nsampson\nclicking\n##ange\n##horpe\n##inations\n##roving\ntorre\n##eptive\n##moral\nsymbolism\n38th\nasshole\nmeritorious\noutfits\nsplashed\nbiographies\nsprung\nastros\n##tale\n302\n737\nfilly\nraoul\nnw\ntokugawa\nlinden\nclubhouse\n##apa\ntracts\nromano\n##pio\nputin\ntags\n##note\nchained\ndickson\ngunshot\nmoe\ngunn\nrashid\n##tails\nzipper\n##bas\n##nea\ncontrasted\n##ply\n##udes\nplum\npharaoh\n##pile\naw\ncomedies\ningrid\nsandwiches\nsubdivisions\n1100\nmariana\nnokia\nkamen\nhz\ndelaney\nveto\nherring\n##words\npossessive\noutlines\n##roup\nsiemens\nstairwell\nrc\ngallantry\nmessiah\npalais\nyells\n233\nzeppelin\n##dm\nbolivar\n##cede\nsmackdown\nmckinley\n##mora\n##yt\nmuted\ngeologic\nfinely\nunitary\navatar\nhamas\nmaynard\nrees\nbog\ncontrasting\n##rut\nliv\nchico\ndisposition\npixel\n##erate\nbecca\ndmitry\nyeshiva\nnarratives\n##lva\n##ulton\nmercenary\nsharpe\ntempered\nnavigate\nstealth\namassed\nkeynes\n##lini\nuntouched\n##rrie\nhavoc\nlithium\n##fighting\nabyss\ngraf\nsouthward\nwolverine\nballoons\nimplements\nngos\ntransitions\n##icum\nambushed\nconcacaf\ndormant\neconomists\n##dim\ncosting\ncsi\nrana\nuniversite\nboulders\nverity\n##llon\ncollin\nmellon\nmisses\ncypress\nfluorescent\nlifeless\nspence\n##ulla\ncrewe\nshepard\npak\nrevelations\n##م\njolly\ngibbons\npaw\n##dro\n##quel\nfreeing\n##test\nshack\nfries\npalatine\n##51\n##hiko\naccompaniment\ncruising\nrecycled\n##aver\nerwin\nsorting\nsynthesizers\ndyke\nrealities\nsg\nstrides\nenslaved\nwetland\n##ghan\ncompetence\ngunpowder\ngrassy\nmaroon\nreactors\nobjection\n##oms\ncarlson\ngearbox\nmacintosh\nradios\nshelton\n##sho\nclergyman\nprakash\n254\nmongols\ntrophies\noricon\n228\nstimuli\ntwenty20\ncantonese\ncortes\nmirrored\n##saurus\nbhp\ncristina\nmelancholy\n##lating\nenjoyable\nnuevo\n##wny\ndownfall\nschumacher\n##ind\nbanging\nlausanne\nrumbled\nparamilitary\nreflex\nax\namplitude\nmigratory\n##gall\n##ups\nmidi\nbarnard\nlastly\nsherry\n##hp\n##nall\nkeystone\n##kra\ncarleton\nslippery\n##53\ncoloring\nfoe\nsocket\notter\n##rgos\nmats\n##tose\nconsultants\nbafta\nbison\ntopping\n##km\n490\nprimal\nabandonment\ntransplant\natoll\nhideous\nmort\npained\nreproduced\ntae\nhowling\n##turn\nunlawful\nbillionaire\nhotter\npoised\nlansing\n##chang\ndinamo\nretro\nmessing\nnfc\ndomesday\n##mina\nblitz\ntimed\n##athing\n##kley\nascending\ngesturing\n##izations\nsignaled\ntis\nchinatown\nmermaid\nsavanna\njameson\n##aint\ncatalina\n##pet\n##hers\ncochrane\ncy\nchatting\n##kus\nalerted\ncomputation\nmused\nnoelle\nmajestic\nmohawk\ncampo\noctagonal\n##sant\n##hend\n241\naspiring\n##mart\ncomprehend\niona\nparalyzed\nshimmering\nswindon\nrhone\n##eley\nreputed\nconfigurations\npitchfork\nagitation\nfrancais\ngillian\nlipstick\n##ilo\noutsiders\npontifical\nresisting\nbitterness\nsewer\nrockies\n##edd\n##ucher\nmisleading\n1756\nexiting\ngalloway\n##nging\nrisked\n##heart\n246\ncommemoration\nschultz\n##rka\nintegrating\n##rsa\nposes\nshrieked\n##weiler\nguineas\ngladys\njerking\nowls\ngoldsmith\nnightly\npenetrating\n##unced\nlia\n##33\nignited\nbetsy\n##aring\n##thorpe\nfollower\nvigorously\n##rave\ncoded\nkiran\nknit\nzoology\ntbilisi\n##28\n##bered\nrepository\ngovt\ndeciduous\ndino\ngrowling\n##bba\nenhancement\nunleashed\nchanting\npussy\nbiochemistry\n##eric\nkettle\nrepression\ntoxicity\nnrhp\n##arth\n##kko\n##bush\nernesto\ncommended\noutspoken\n242\nmca\nparchment\nsms\nkristen\n##aton\nbisexual\nraked\nglamour\nnavajo\na2\nconditioned\nshowcased\n##hma\nspacious\nyouthful\n##esa\nusl\nappliances\njunta\nbrest\nlayne\nconglomerate\nenchanted\nchao\nloosened\npicasso\ncirculating\ninspect\nmontevideo\n##centric\n##kti\npiazza\nspurred\n##aith\nbari\nfreedoms\npoultry\nstamford\nlieu\n##ect\nindigo\nsarcastic\nbahia\nstump\nattach\ndvds\nfrankenstein\nlille\napprox\nscriptures\npollen\n##script\nnmi\noverseen\n##ivism\ntides\nproponent\nnewmarket\ninherit\nmilling\n##erland\ncentralized\n##rou\ndistributors\ncredentials\ndrawers\nabbreviation\n##lco\n##xon\ndowning\nuncomfortably\nripe\n##oes\nerase\nfranchises\n##ever\npopulace\n##bery\n##khar\ndecomposition\npleas\n##tet\ndaryl\nsabah\n##stle\n##wide\nfearless\ngenie\nlesions\nannette\n##ogist\noboe\nappendix\nnair\ndripped\npetitioned\nmaclean\nmosquito\nparrot\nrpg\nhampered\n1648\noperatic\nreservoirs\n##tham\nirrelevant\njolt\nsummarized\n##fp\nmedallion\n##taff\n##−\nclawed\nharlow\nnarrower\ngoddard\nmarcia\nbodied\nfremont\nsuarez\naltering\ntempest\nmussolini\nporn\n##isms\nsweetly\noversees\nwalkers\nsolitude\ngrimly\nshrines\nhk\nich\nsupervisors\nhostess\ndietrich\nlegitimacy\nbrushes\nexpressive\n##yp\ndissipated\n##rse\nlocalized\nsystemic\n##nikov\ngettysburg\n##js\n##uaries\ndialogues\nmuttering\n251\nhousekeeper\nsicilian\ndiscouraged\n##frey\nbeamed\nkaladin\nhalftime\nkidnap\n##amo\n##llet\n1754\nsynonymous\ndepleted\ninstituto\ninsulin\nreprised\n##opsis\nclashed\n##ctric\ninterrupting\nradcliffe\ninsisting\nmedici\n1715\nejected\nplayfully\nturbulent\n##47\nstarvation\n##rini\nshipment\nrebellious\npetersen\nverification\nmerits\n##rified\ncakes\n##charged\n1757\nmilford\nshortages\nspying\nfidelity\n##aker\nemitted\nstorylines\nharvested\nseismic\n##iform\ncheung\nkilda\ntheoretically\nbarbie\nlynx\n##rgy\n##tius\ngoblin\nmata\npoisonous\n##nburg\nreactive\nresidues\nobedience\n##евич\nconjecture\n##rac\n401\nhating\nsixties\nkicker\nmoaning\nmotown\n##bha\nemancipation\nneoclassical\n##hering\nconsoles\nebert\nprofessorship\n##tures\nsustaining\nassaults\nobeyed\naffluent\nincurred\ntornadoes\n##eber\n##zow\nemphasizing\nhighlanders\ncheated\nhelmets\n##ctus\ninternship\nterence\nbony\nexecutions\nlegislators\nberries\npeninsular\ntinged\n##aco\n1689\namplifier\ncorvette\nribbons\nlavish\npennant\n##lander\nworthless\n##chfield\n##forms\nmariano\npyrenees\nexpenditures\n##icides\nchesterfield\nmandir\ntailor\n39th\nsergey\nnestled\nwilled\naristocracy\ndevotees\ngoodnight\nraaf\nrumored\nweaponry\nremy\nappropriations\nharcourt\nburr\nriaa\n##lence\nlimitation\nunnoticed\nguo\nsoaking\nswamps\n##tica\ncollapsing\ntatiana\ndescriptive\nbrigham\npsalm\n##chment\nmaddox\n##lization\npatti\ncaliph\n##aja\nakron\ninjuring\nserra\n##ganj\nbasins\n##sari\nastonished\nlauncher\n##church\nhilary\nwilkins\nsewing\n##sf\nstinging\n##fia\n##ncia\nunderwood\nstartup\n##ition\ncompilations\nvibrations\nembankment\njurist\n##nity\nbard\njuventus\ngroundwater\nkern\npalaces\nhelium\nboca\ncramped\nmarissa\nsoto\n##worm\njae\nprincely\n##ggy\nfaso\nbazaar\nwarmly\n##voking\n229\npairing\n##lite\n##grate\n##nets\nwien\nfreaked\nulysses\nrebirth\n##alia\n##rent\nmummy\nguzman\njimenez\nstilled\n##nitz\ntrajectory\ntha\nwoken\narchival\nprofessions\n##pts\n##pta\nhilly\nshadowy\nshrink\n##bolt\nnorwood\nglued\nmigrate\nstereotypes\ndevoid\n##pheus\n625\nevacuate\nhorrors\ninfancy\ngotham\nknowles\noptic\ndownloaded\nsachs\nkingsley\nparramatta\ndarryl\nmor\n##onale\nshady\ncommence\nconfesses\nkan\n##meter\n##placed\nmarlborough\nroundabout\nregents\nfrigates\nio\n##imating\ngothenburg\nrevoked\ncarvings\nclockwise\nconvertible\nintruder\n##sche\nbanged\n##ogo\nvicky\nbourgeois\n##mony\ndupont\nfooting\n##gum\npd\n##real\nbuckle\nyun\npenthouse\nsane\n720\nserviced\nstakeholders\nneumann\nbb\n##eers\ncomb\n##gam\ncatchment\npinning\nrallies\ntyping\n##elles\nforefront\nfreiburg\nsweetie\ngiacomo\nwidowed\ngoodwill\nworshipped\naspirations\nmidday\n##vat\nfishery\n##trick\nbournemouth\nturk\n243\nhearth\nethanol\nguadalajara\nmurmurs\nsl\n##uge\nafforded\nscripted\n##hta\nwah\n##jn\ncoroner\ntranslucent\n252\nmemorials\npuck\nprogresses\nclumsy\n##race\n315\ncandace\nrecounted\n##27\n##slin\n##uve\nfiltering\n##mac\nhowl\nstrata\nheron\nleveled\n##ays\ndubious\n##oja\n##т\n##wheel\ncitations\nexhibiting\n##laya\n##mics\n##pods\nturkic\n##lberg\ninjunction\n##ennial\n##mit\nantibodies\n##44\norganise\n##rigues\ncardiovascular\ncushion\ninverness\n##zquez\ndia\ncocoa\nsibling\n##tman\n##roid\nexpanse\nfeasible\ntunisian\nalgiers\n##relli\nrus\nbloomberg\ndso\nwestphalia\nbro\ntacoma\n281\ndownloads\n##ours\nkonrad\nduran\n##hdi\ncontinuum\njett\ncompares\nlegislator\nsecession\n##nable\n##gues\n##zuka\ntranslating\nreacher\n##gley\n##ła\naleppo\n##agi\ntc\norchards\ntrapping\nlinguist\nversatile\ndrumming\npostage\ncalhoun\nsuperiors\n##mx\nbarefoot\nleary\n##cis\nignacio\nalfa\nkaplan\n##rogen\nbratislava\nmori\n##vot\ndisturb\nhaas\n313\ncartridges\ngilmore\nradiated\nsalford\ntunic\nhades\n##ulsive\narcheological\ndelilah\nmagistrates\nauditioned\nbrewster\ncharters\nempowerment\nblogs\ncappella\ndynasties\niroquois\nwhipping\n##krishna\nraceway\ntruths\nmyra\nweaken\njudah\nmcgregor\n##horse\nmic\nrefueling\n37th\nburnley\nbosses\nmarkus\npremio\nquery\n##gga\ndunbar\n##economic\ndarkest\nlyndon\nsealing\ncommendation\nreappeared\n##mun\naddicted\nezio\nslaughtered\nsatisfactory\nshuffle\n##eves\n##thic\n##uj\nfortification\nwarrington\n##otto\nresurrected\nfargo\nmane\n##utable\n##lei\n##space\nforeword\nox\n##aris\n##vern\nabrams\nhua\n##mento\nsakura\n##alo\nuv\nsentimental\n##skaya\nmidfield\n##eses\nsturdy\nscrolls\nmacleod\n##kyu\nentropy\n##lance\nmitochondrial\ncicero\nexcelled\nthinner\nconvoys\nperceive\n##oslav\n##urable\nsystematically\ngrind\nburkina\n287\n##tagram\nops\n##aman\nguantanamo\n##cloth\n##tite\nforcefully\nwavy\n##jou\npointless\n##linger\n##tze\nlayton\nportico\nsuperficial\nclerical\noutlaws\n##hism\nburials\nmuir\n##inn\ncreditors\nhauling\nrattle\n##leg\ncalais\nmonde\narchers\nreclaimed\ndwell\nwexford\nhellenic\nfalsely\nremorse\n##tek\ndough\nfurnishings\n##uttered\ngabon\nneurological\nnovice\n##igraphy\ncontemplated\npulpit\nnightstand\nsaratoga\n##istan\ndocumenting\npulsing\ntaluk\n##firmed\nbusted\nmarital\n##rien\ndisagreements\nwasps\n##yes\nhodge\nmcdonnell\nmimic\nfran\npendant\ndhabi\nmusa\n##nington\ncongratulations\nargent\ndarrell\nconcussion\nlosers\nregrets\nthessaloniki\nreversal\ndonaldson\nhardwood\nthence\nachilles\nritter\n##eran\ndemonic\njurgen\nprophets\ngoethe\neki\nclassmate\nbuff\n##cking\nyank\nirrational\n##inging\nperished\nseductive\nqur\nsourced\n##crat\n##typic\nmustard\nravine\nbarre\nhorizontally\ncharacterization\nphylogenetic\nboise\n##dit\n##runner\n##tower\nbrutally\nintercourse\nseduce\n##bbing\nfay\nferris\nogden\namar\nnik\nunarmed\n##inator\nevaluating\nkyrgyzstan\nsweetness\n##lford\n##oki\nmccormick\nmeiji\nnotoriety\nstimulate\ndisrupt\nfiguring\ninstructional\nmcgrath\n##zoo\ngroundbreaking\n##lto\nflinch\nkhorasan\nagrarian\nbengals\nmixer\nradiating\n##sov\ningram\npitchers\nnad\ntariff\n##cript\ntata\n##codes\n##emi\n##ungen\nappellate\nlehigh\n##bled\n##giri\nbrawl\nduct\ntexans\n##ciation\n##ropolis\nskipper\nspeculative\nvomit\ndoctrines\nstresses\n253\ndavy\ngraders\nwhitehead\njozef\ntimely\ncumulative\nharyana\npaints\nappropriately\nboon\ncactus\n##ales\n##pid\ndow\nlegions\n##pit\nperceptions\n1730\npicturesque\n##yse\nperiphery\nrune\nwr\n##aha\nceltics\nsentencing\nwhoa\n##erin\nconfirms\nvariance\n425\nmoines\nmathews\nspade\nrave\nm1\nfronted\nfx\nblending\nalleging\nreared\n##gl\n237\n##paper\ngrassroots\neroded\n##free\n##physical\ndirects\nordeal\n##sław\naccelerate\nhacker\nrooftop\n##inia\nlev\nbuys\ncebu\ndevote\n##lce\nspecialising\n##ulsion\nchoreographed\nrepetition\nwarehouses\n##ryl\npaisley\ntuscany\nanalogy\nsorcerer\nhash\nhuts\nshards\ndescends\nexclude\nnix\nchaplin\ngaga\nito\nvane\n##drich\ncauseway\nmisconduct\nlimo\norchestrated\nglands\njana\n##kot\nu2\n##mple\n##sons\nbranching\ncontrasts\nscoop\nlonged\n##virus\nchattanooga\n##75\nsyrup\ncornerstone\n##tized\n##mind\n##iaceae\ncareless\nprecedence\nfrescoes\n##uet\nchilled\nconsult\nmodelled\nsnatch\npeat\n##thermal\ncaucasian\nhumane\nrelaxation\nspins\ntemperance\n##lbert\noccupations\nlambda\nhybrids\nmoons\nmp3\n##oese\n247\nrolf\nsocietal\nyerevan\nness\n##ssler\nbefriended\nmechanized\nnominate\ntrough\nboasted\ncues\nseater\n##hom\nbends\n##tangle\nconductors\nemptiness\n##lmer\neurasian\nadriatic\ntian\n##cie\nanxiously\nlark\npropellers\nchichester\njock\nev\n2a\n##holding\ncredible\nrecounts\ntori\nloyalist\nabduction\n##hoot\n##redo\nnepali\n##mite\nventral\ntempting\n##ango\n##crats\nsteered\n##wice\njavelin\ndipping\nlaborers\nprentice\nlooming\ntitanium\n##ː\nbadges\nemir\ntensor\n##ntation\negyptians\nrash\ndenies\nhawthorne\nlombard\nshowers\nwehrmacht\ndietary\ntrojan\n##reus\nwelles\nexecuting\nhorseshoe\nlifeboat\n##lak\nelsa\ninfirmary\nnearing\nroberta\nboyer\nmutter\ntrillion\njoanne\n##fine\n##oked\nsinks\nvortex\nuruguayan\nclasp\nsirius\n##block\naccelerator\nprohibit\nsunken\nbyu\nchronological\ndiplomats\nochreous\n510\nsymmetrical\n1644\nmaia\n##tology\nsalts\nreigns\natrocities\n##ия\nhess\nbared\nissn\n##vyn\ncater\nsaturated\n##cycle\n##isse\nsable\nvoyager\ndyer\nyusuf\n##inge\nfountains\nwolff\n##39\n##nni\nengraving\nrollins\natheist\nominous\n##ault\nherr\nchariot\nmartina\nstrung\n##fell\n##farlane\nhorrific\nsahib\ngazes\nsaetan\nerased\nptolemy\n##olic\nflushing\nlauderdale\nanalytic\n##ices\n530\nnavarro\nbeak\ngorilla\nherrera\nbroom\nguadalupe\nraiding\nsykes\n311\nbsc\ndeliveries\n1720\ninvasions\ncarmichael\ntajikistan\nthematic\necumenical\nsentiments\nonstage\n##rians\n##brand\n##sume\ncatastrophic\nflanks\nmolten\n##arns\nwaller\naimee\nterminating\n##icing\nalternately\n##oche\nnehru\nprinters\noutraged\n##eving\nempires\ntemplate\nbanners\nrepetitive\nza\n##oise\nvegetarian\n##tell\nguiana\nopt\ncavendish\nlucknow\nsynthesized\n##hani\n##mada\nfinalized\n##ctable\nfictitious\nmayoral\nunreliable\n##enham\nembracing\npeppers\nrbis\n##chio\n##neo\ninhibition\nslashed\ntogo\norderly\nembroidered\nsafari\nsalty\n236\nbarron\nbenito\ntotaled\n##dak\npubs\nsimulated\ncaden\ndevin\ntolkien\nmomma\nwelding\nsesame\n##ept\ngottingen\nhardness\n630\nshaman\ntemeraire\n620\nadequately\npediatric\n##kit\nck\nassertion\nradicals\ncomposure\ncadence\nseafood\nbeaufort\nlazarus\nmani\nwarily\ncunning\nkurdistan\n249\ncantata\n##kir\nares\n##41\n##clusive\nnape\ntownland\ngeared\ninsulted\nflutter\nboating\nviolate\ndraper\ndumping\nmalmo\n##hh\n##romatic\nfirearm\nalta\nbono\nobscured\n##clave\nexceeds\npanorama\nunbelievable\n##train\npreschool\n##essed\ndisconnected\ninstalling\nrescuing\nsecretaries\naccessibility\n##castle\n##drive\n##ifice\n##film\nbouts\nslug\nwaterway\nmindanao\n##buro\n##ratic\nhalves\n##ل\ncalming\nliter\nmaternity\nadorable\nbragg\nelectrification\nmcc\n##dote\nroxy\nschizophrenia\n##body\nmunoz\nkaye\nwhaling\n239\nmil\ntingling\ntolerant\n##ago\nunconventional\nvolcanoes\n##finder\ndeportivo\n##llie\nrobson\nkaufman\nneuroscience\nwai\ndeportation\nmasovian\nscraping\nconverse\n##bh\nhacking\nbulge\n##oun\nadministratively\nyao\n580\namp\nmammoth\nbooster\nclaremont\nhooper\nnomenclature\npursuits\nmclaughlin\nmelinda\n##sul\ncatfish\nbarclay\nsubstrates\ntaxa\nzee\noriginals\nkimberly\npackets\npadma\n##ality\nborrowing\nostensibly\nsolvent\n##bri\n##genesis\n##mist\nlukas\nshreveport\nveracruz\n##ь\n##lou\n##wives\ncheney\ntt\nanatolia\nhobbs\n##zyn\ncyclic\nradiant\nalistair\ngreenish\nsiena\ndat\nindependents\n##bation\nconform\npieter\nhyper\napplicant\nbradshaw\nspores\ntelangana\nvinci\ninexpensive\nnuclei\n322\njang\nnme\nsoho\nspd\n##ign\ncradled\nreceptionist\npow\n##43\n##rika\nfascism\n##ifer\nexperimenting\n##ading\n##iec\n##region\n345\njocelyn\nmaris\nstair\nnocturnal\ntoro\nconstabulary\nelgin\n##kker\nmsc\n##giving\n##schen\n##rase\ndoherty\ndoping\nsarcastically\nbatter\nmaneuvers\n##cano\n##apple\n##gai\n##git\nintrinsic\n##nst\n##stor\n1753\nshowtime\ncafes\ngasps\nlviv\nushered\n##thed\nfours\nrestart\nastonishment\ntransmitting\nflyer\nshrugs\n##sau\nintriguing\ncones\ndictated\nmushrooms\nmedial\n##kovsky\n##elman\nescorting\ngaped\n##26\ngodfather\n##door\n##sell\ndjs\nrecaptured\ntimetable\nvila\n1710\n3a\naerodrome\nmortals\nscientology\n##orne\nangelina\nmag\nconvection\nunpaid\ninsertion\nintermittent\nlego\n##nated\nendeavor\nkota\npereira\n##lz\n304\nbwv\nglamorgan\ninsults\nagatha\nfey\n##cend\nfleetwood\nmahogany\nprotruding\nsteamship\nzeta\n##arty\nmcguire\nsuspense\n##sphere\nadvising\nurges\n##wala\nhurriedly\nmeteor\ngilded\ninline\narroyo\nstalker\n##oge\nexcitedly\nrevered\n##cure\nearle\nintroductory\n##break\n##ilde\nmutants\npuff\npulses\nreinforcement\n##haling\ncurses\nlizards\nstalk\ncorrelated\n##fixed\nfallout\nmacquarie\n##unas\nbearded\ndenton\nheaving\n802\n##ocation\nwinery\nassign\ndortmund\n##lkirk\neverest\ninvariant\ncharismatic\nsusie\n##elling\nbled\nlesley\ntelegram\nsumner\nbk\n##ogen\n##к\nwilcox\nneedy\ncolbert\nduval\n##iferous\n##mbled\nallotted\nattends\nimperative\n##hita\nreplacements\nhawker\n##inda\ninsurgency\n##zee\n##eke\ncasts\n##yla\n680\nives\ntransitioned\n##pack\n##powering\nauthoritative\nbaylor\nflex\ncringed\nplaintiffs\nwoodrow\n##skie\ndrastic\nape\naroma\nunfolded\ncommotion\nnt\npreoccupied\ntheta\nroutines\nlasers\nprivatization\nwand\ndomino\nek\nclenching\nnsa\nstrategically\nshowered\nbile\nhandkerchief\npere\nstoring\nchristophe\ninsulting\n316\nnakamura\nromani\nasiatic\nmagdalena\npalma\ncruises\nstripping\n405\nkonstantin\nsoaring\n##berman\ncolloquially\nforerunner\nhavilland\nincarcerated\nparasites\nsincerity\n##utus\ndisks\nplank\nsaigon\n##ining\ncorbin\nhomo\nornaments\npowerhouse\n##tlement\nchong\nfastened\nfeasibility\nidf\nmorphological\nusable\n##nish\n##zuki\naqueduct\njaguars\nkeepers\n##flies\naleksandr\nfaust\nassigns\newing\nbacterium\nhurled\ntricky\nhungarians\nintegers\nwallis\n321\nyamaha\n##isha\nhushed\noblivion\naviator\nevangelist\nfriars\n##eller\nmonograph\node\n##nary\nairplanes\nlabourers\ncharms\n##nee\n1661\nhagen\ntnt\nrudder\nfiesta\ntranscript\ndorothea\nska\ninhibitor\nmaccabi\nretorted\nraining\nencompassed\nclauses\nmenacing\n1642\nlineman\n##gist\nvamps\n##ape\n##dick\ngloom\n##rera\ndealings\neasing\nseekers\n##nut\n##pment\nhelens\nunmanned\n##anu\n##isson\nbasics\n##amy\n##ckman\nadjustments\n1688\nbrutality\nhorne\n##zell\nsui\n##55\n##mable\naggregator\n##thal\nrhino\n##drick\n##vira\ncounters\nzoom\n##01\n##rting\nmn\nmontenegrin\npackard\n##unciation\n##♭\n##kki\nreclaim\nscholastic\nthugs\npulsed\n##icia\nsyriac\nquan\nsaddam\nbanda\nkobe\nblaming\nbuddies\ndissent\n##lusion\n##usia\ncorbett\njaya\ndelle\nerratic\nlexie\n##hesis\n435\namiga\nhermes\n##pressing\n##leen\nchapels\ngospels\njamal\n##uating\ncompute\nrevolving\nwarp\n##sso\n##thes\narmory\n##eras\n##gol\nantrim\nloki\n##kow\n##asian\n##good\n##zano\nbraid\nhandwriting\nsubdistrict\nfunky\npantheon\n##iculate\nconcurrency\nestimation\nimproper\njuliana\n##his\nnewcomers\njohnstone\nstaten\ncommunicated\n##oco\n##alle\nsausage\nstormy\n##stered\n##tters\nsuperfamily\n##grade\nacidic\ncollateral\ntabloid\n##oped\n##rza\nbladder\nausten\n##ellant\nmcgraw\n##hay\nhannibal\nmein\naquino\nlucifer\nwo\nbadger\nboar\ncher\nchristensen\ngreenberg\ninterruption\n##kken\njem\n244\nmocked\nbottoms\ncambridgeshire\n##lide\nsprawling\n##bbly\neastwood\nghent\nsynth\n##buck\nadvisers\n##bah\nnominally\nhapoel\nqu\ndaggers\nestranged\nfabricated\ntowels\nvinnie\nwcw\nmisunderstanding\nanglia\nnothin\nunmistakable\n##dust\n##lova\nchilly\nmarquette\ntruss\n##edge\n##erine\nreece\n##lty\n##chemist\n##connected\n272\n308\n41st\nbash\nraion\nwaterfalls\n##ump\n##main\nlabyrinth\nqueue\ntheorist\n##istle\nbharatiya\nflexed\nsoundtracks\nrooney\nleftist\npatrolling\nwharton\nplainly\nalleviate\neastman\nschuster\ntopographic\nengages\nimmensely\nunbearable\nfairchild\n1620\ndona\nlurking\nparisian\noliveira\nia\nindictment\nhahn\nbangladeshi\n##aster\nvivo\n##uming\n##ential\nantonia\nexpects\nindoors\nkildare\nharlan\n##logue\n##ogenic\n##sities\nforgiven\n##wat\nchildish\ntavi\n##mide\n##orra\nplausible\ngrimm\nsuccessively\nscooted\n##bola\n##dget\n##rith\nspartans\nemery\nflatly\nazure\nepilogue\n##wark\nflourish\n##iny\n##tracted\n##overs\n##oshi\nbestseller\ndistressed\nreceipt\nspitting\nhermit\ntopological\n##cot\ndrilled\nsubunit\nfrancs\n##layer\neel\n##fk\n##itas\noctopus\nfootprint\npetitions\nufo\n##say\n##foil\ninterfering\nleaking\npalo\n##metry\nthistle\nvaliant\n##pic\nnarayan\nmcpherson\n##fast\ngonzales\n##ym\n##enne\ndustin\nnovgorod\nsolos\n##zman\ndoin\n##raph\n##patient\n##meyer\nsoluble\nashland\ncuffs\ncarole\npendleton\nwhistling\nvassal\n##river\ndeviation\nrevisited\nconstituents\nrallied\nrotate\nloomed\n##eil\n##nting\namateurs\naugsburg\nauschwitz\ncrowns\nskeletons\n##cona\nbonnet\n257\ndummy\nglobalization\nsimeon\nsleeper\nmandal\ndifferentiated\n##crow\n##mare\nmilne\nbundled\nexasperated\ntalmud\nowes\nsegregated\n##feng\n##uary\ndentist\npiracy\nprops\n##rang\ndevlin\n##torium\nmalicious\npaws\n##laid\ndependency\n##ergy\n##fers\n##enna\n258\npistons\nrourke\njed\ngrammatical\ntres\nmaha\nwig\n512\nghostly\njayne\n##achal\n##creen\n##ilis\n##lins\n##rence\ndesignate\n##with\narrogance\ncambodian\nclones\nshowdown\nthrottle\ntwain\n##ception\nlobes\nmetz\nnagoya\n335\nbraking\n##furt\n385\nroaming\n##minster\namin\ncrippled\n##37\n##llary\nindifferent\nhoffmann\nidols\nintimidating\n1751\n261\ninfluenza\nmemo\nonions\n1748\nbandage\nconsciously\n##landa\n##rage\nclandestine\nobserves\nswiped\ntangle\n##ener\n##jected\n##trum\n##bill\n##lta\nhugs\ncongresses\njosiah\nspirited\n##dek\nhumanist\nmanagerial\nfilmmaking\ninmate\nrhymes\ndebuting\ngrimsby\nur\n##laze\nduplicate\nvigor\n##tf\nrepublished\nbolshevik\nrefurbishment\nantibiotics\nmartini\nmethane\nnewscasts\nroyale\nhorizons\nlevant\niain\nvisas\n##ischen\npaler\n##around\nmanifestation\nsnuck\nalf\nchop\nfutile\npedestal\nrehab\n##kat\nbmg\nkerman\nres\nfairbanks\njarrett\nabstraction\nsaharan\n##zek\n1746\nprocedural\nclearer\nkincaid\nsash\nluciano\n##ffey\ncrunch\nhelmut\n##vara\nrevolutionaries\n##tute\ncreamy\nleach\n##mmon\n1747\npermitting\nnes\nplight\nwendell\n##lese\ncontra\nts\nclancy\nipa\nmach\nstaples\nautopsy\ndisturbances\nnueva\nkarin\npontiac\n##uding\nproxy\nvenerable\nhaunt\nleto\nbergman\nexpands\n##helm\nwal\n##pipe\ncanning\nceline\ncords\nobesity\n##enary\nintrusion\nplanner\n##phate\nreasoned\nsequencing\n307\nharrow\n##chon\n##dora\nmarred\nmcintyre\nrepay\ntarzan\ndarting\n248\nharrisburg\nmargarita\nrepulsed\n##hur\n##lding\nbelinda\nhamburger\nnovo\ncompliant\nrunways\nbingham\nregistrar\nskyscraper\nic\ncuthbert\nimprovisation\nlivelihood\n##corp\n##elial\nadmiring\n##dened\nsporadic\nbeliever\ncasablanca\npopcorn\n##29\nasha\nshovel\n##bek\n##dice\ncoiled\ntangible\n##dez\ncasper\nelsie\nresin\ntenderness\nrectory\n##ivision\navail\nsonar\n##mori\nboutique\n##dier\nguerre\nbathed\nupbringing\nvaulted\nsandals\nblessings\n##naut\n##utnant\n1680\n306\nfoxes\npia\ncorrosion\nhesitantly\nconfederates\ncrystalline\nfootprints\nshapiro\ntirana\nvalentin\ndrones\n45th\nmicroscope\nshipments\ntexted\ninquisition\nwry\nguernsey\nunauthorized\nresigning\n760\nripple\nschubert\nstu\nreassure\nfelony\n##ardo\nbrittle\nkoreans\n##havan\n##ives\ndun\nimplicit\ntyres\n##aldi\n##lth\nmagnolia\n##ehan\n##puri\n##poulos\naggressively\nfei\ngr\nfamiliarity\n##poo\nindicative\n##trust\nfundamentally\njimmie\noverrun\n395\nanchors\nmoans\n##opus\nbritannia\narmagh\n##ggle\npurposely\nseizing\n##vao\nbewildered\nmundane\navoidance\ncosmopolitan\ngeometridae\nquartermaster\ncaf\n415\nchatter\nengulfed\ngleam\npurge\n##icate\njuliette\njurisprudence\nguerra\nrevisions\n##bn\ncasimir\nbrew\n##jm\n1749\nclapton\ncloudy\nconde\nhermitage\n278\nsimulations\ntorches\nvincenzo\nmatteo\n##rill\nhidalgo\nbooming\nwestbound\naccomplishment\ntentacles\nunaffected\n##sius\nannabelle\nflopped\nsloping\n##litz\ndreamer\ninterceptor\nvu\n##loh\nconsecration\ncopying\nmessaging\nbreaker\nclimates\nhospitalized\n1752\ntorino\nafternoons\nwinfield\nwitnessing\n##teacher\nbreakers\nchoirs\nsawmill\ncoldly\n##ege\nsipping\nhaste\nuninhabited\nconical\nbibliography\npamphlets\nsevern\nedict\n##oca\ndeux\nillnesses\ngrips\n##pl\nrehearsals\nsis\nthinkers\ntame\n##keepers\n1690\nacacia\nreformer\n##osed\n##rys\nshuffling\n##iring\n##shima\neastbound\nionic\nrhea\nflees\nlittered\n##oum\nrocker\nvomiting\ngroaning\nchamp\noverwhelmingly\ncivilizations\npaces\nsloop\nadoptive\n##tish\nskaters\n##vres\naiding\nmango\n##joy\nnikola\nshriek\n##ignon\npharmaceuticals\n##mg\ntuna\ncalvert\ngustavo\nstocked\nyearbook\n##urai\n##mana\ncomputed\nsubsp\nriff\nhanoi\nkelvin\nhamid\nmoors\npastures\nsummons\njihad\nnectar\n##ctors\nbayou\nuntitled\npleasing\nvastly\nrepublics\nintellect\n##η\n##ulio\n##tou\ncrumbling\nstylistic\nsb\n##ی\nconsolation\nfrequented\nh₂o\nwalden\nwidows\n##iens\n404\n##ignment\nchunks\nimproves\n288\ngrit\nrecited\n##dev\nsnarl\nsociological\n##arte\n##gul\ninquired\n##held\nbruise\nclube\nconsultancy\nhomogeneous\nhornets\nmultiplication\npasta\nprick\nsavior\n##grin\n##kou\n##phile\nyoon\n##gara\ngrimes\nvanishing\ncheering\nreacting\nbn\ndistillery\n##quisite\n##vity\ncoe\ndockyard\nmassif\n##jord\nescorts\nvoss\n##valent\nbyte\nchopped\nhawke\nillusions\nworkings\nfloats\n##koto\n##vac\nkv\nannapolis\nmadden\n##onus\nalvaro\nnoctuidae\n##cum\n##scopic\navenge\nsteamboat\nforte\nillustrates\nerika\n##trip\n570\ndew\nnationalities\nbran\nmanifested\nthirsty\ndiversified\nmuscled\nreborn\n##standing\narson\n##lessness\n##dran\n##logram\n##boys\n##kushima\n##vious\nwilloughby\n##phobia\n286\nalsace\ndashboard\nyuki\n##chai\ngranville\nmyspace\npublicized\ntricked\n##gang\nadjective\n##ater\nrelic\nreorganisation\nenthusiastically\nindications\nsaxe\n##lassified\nconsolidate\niec\npadua\nhelplessly\nramps\nrenaming\nregulars\npedestrians\naccents\nconvicts\ninaccurate\nlowers\nmana\n##pati\nbarrie\nbjp\noutta\nsomeplace\nberwick\nflanking\ninvoked\nmarrow\nsparsely\nexcerpts\nclothed\nrei\n##ginal\nwept\n##straße\n##vish\nalexa\nexcel\n##ptive\nmembranes\naquitaine\ncreeks\ncutler\nsheppard\nimplementations\nns\n##dur\nfragrance\nbudge\nconcordia\nmagnesium\nmarcelo\n##antes\ngladly\nvibrating\n##rral\n##ggles\nmontrose\n##omba\nlew\nseamus\n1630\ncocky\n##ament\n##uen\nbjorn\n##rrick\nfielder\nfluttering\n##lase\nmethyl\nkimberley\nmcdowell\nreductions\nbarbed\n##jic\n##tonic\naeronautical\ncondensed\ndistracting\n##promising\nhuffed\n##cala\n##sle\nclaudius\ninvincible\nmissy\npious\nbalthazar\nci\n##lang\nbutte\ncombo\norson\n##dication\nmyriad\n1707\nsilenced\n##fed\n##rh\ncoco\nnetball\nyourselves\n##oza\nclarify\nheller\npeg\ndurban\netudes\noffender\nroast\nblackmail\ncurvature\n##woods\nvile\n309\nillicit\nsuriname\n##linson\noverture\n1685\nbubbling\ngymnast\ntucking\n##mming\n##ouin\nmaldives\n##bala\ngurney\n##dda\n##eased\n##oides\nbackside\npinto\njars\nracehorse\ntending\n##rdial\nbaronetcy\nwiener\nduly\n##rke\nbarbarian\ncupping\nflawed\n##thesis\nbertha\npleistocene\npuddle\nswearing\n##nob\n##tically\nfleeting\nprostate\namulet\neducating\n##mined\n##iti\n##tler\n75th\njens\nrespondents\nanalytics\ncavaliers\npapacy\nraju\n##iente\n##ulum\n##tip\nfunnel\n271\ndisneyland\n##lley\nsociologist\n##iam\n2500\nfaulkner\nlouvre\nmenon\n##dson\n276\n##ower\nafterlife\nmannheim\npeptide\nreferees\ncomedians\nmeaningless\n##anger\n##laise\nfabrics\nhurley\nrenal\nsleeps\n##bour\n##icle\nbreakout\nkristin\nroadside\nanimator\nclover\ndisdain\nunsafe\nredesign\n##urity\nfirth\nbarnsley\nportage\nreset\nnarrows\n268\ncommandos\nexpansive\nspeechless\ntubular\n##lux\nessendon\neyelashes\nsmashwords\n##yad\n##bang\n##claim\ncraved\nsprinted\nchet\nsomme\nastor\nwrocław\norton\n266\nbane\n##erving\n##uing\nmischief\n##amps\n##sund\nscaling\nterre\n##xious\nimpairment\noffenses\nundermine\nmoi\nsoy\ncontiguous\narcadia\ninuit\nseam\n##tops\nmacbeth\nrebelled\n##icative\n##iot\n590\nelaborated\nfrs\nuniformed\n##dberg\n259\npowerless\npriscilla\nstimulated\n980\nqc\narboretum\nfrustrating\ntrieste\nbullock\n##nified\nenriched\nglistening\nintern\n##adia\nlocus\nnouvelle\nollie\nike\nlash\nstarboard\nee\ntapestry\nheadlined\nhove\nrigged\n##vite\npollock\n##yme\nthrive\nclustered\ncas\nroi\ngleamed\nolympiad\n##lino\npressured\nregimes\n##hosis\n##lick\nripley\n##ophone\nkickoff\ngallon\nrockwell\n##arable\ncrusader\nglue\nrevolutions\nscrambling\n1714\ngrover\n##jure\nenglishman\naztec\n263\ncontemplating\ncoven\nipad\npreach\ntriumphant\ntufts\n##esian\nrotational\n##phus\n328\nfalkland\n##brates\nstrewn\nclarissa\nrejoin\nenvironmentally\nglint\nbanded\ndrenched\nmoat\nalbanians\njohor\nrr\nmaestro\nmalley\nnouveau\nshaded\ntaxonomy\nv6\nadhere\nbunk\nairfields\n##ritan\n1741\nencompass\nremington\ntran\n##erative\namelie\nmazda\nfriar\nmorals\npassions\n##zai\nbreadth\nvis\n##hae\nargus\nburnham\ncaressing\ninsider\nrudd\n##imov\n##mini\n##rso\nitalianate\nmurderous\ntextual\nwainwright\narmada\nbam\nweave\ntimer\n##taken\n##nh\nfra\n##crest\nardent\nsalazar\ntaps\ntunis\n##ntino\nallegro\ngland\nphilanthropic\n##chester\nimplication\n##optera\nesq\njudas\nnoticeably\nwynn\n##dara\ninched\nindexed\ncrises\nvilliers\nbandit\nroyalties\npatterned\ncupboard\ninterspersed\naccessory\nisla\nkendrick\nentourage\nstitches\n##esthesia\nheadwaters\n##ior\ninterlude\ndistraught\ndraught\n1727\n##basket\nbiased\nsy\ntransient\ntriad\nsubgenus\nadapting\nkidd\nshortstop\n##umatic\ndimly\nspiked\nmcleod\nreprint\nnellie\npretoria\nwindmill\n##cek\nsingled\n##mps\n273\nreunite\n##orous\n747\nbankers\noutlying\n##omp\n##ports\n##tream\napologies\ncosmetics\npatsy\n##deh\n##ocks\n##yson\nbender\nnantes\nserene\n##nad\nlucha\nmmm\n323\n##cius\n##gli\ncmll\ncoinage\nnestor\njuarez\n##rook\nsmeared\nsprayed\ntwitching\nsterile\nirina\nembodied\njuveniles\nenveloped\nmiscellaneous\ncancers\ndq\ngulped\nluisa\ncrested\nswat\ndonegal\nref\n##anov\n##acker\nhearst\nmercantile\n##lika\ndoorbell\nua\nvicki\n##alla\n##som\nbilbao\npsychologists\nstryker\nsw\nhorsemen\nturkmenistan\nwits\n##national\nanson\nmathew\nscreenings\n##umb\nrihanna\n##agne\n##nessy\naisles\n##iani\n##osphere\nhines\nkenton\nsaskatoon\ntasha\ntruncated\n##champ\n##itan\nmildred\nadvises\nfredrik\ninterpreting\ninhibitors\n##athi\nspectroscopy\n##hab\n##kong\nkarim\npanda\n##oia\n##nail\n##vc\nconqueror\nkgb\nleukemia\n##dity\narrivals\ncheered\npisa\nphosphorus\nshielded\n##riated\nmammal\nunitarian\nurgently\nchopin\nsanitary\n##mission\nspicy\ndrugged\nhinges\n##tort\ntipping\ntrier\nimpoverished\nwestchester\n##caster\n267\nepoch\nnonstop\n##gman\n##khov\naromatic\ncentrally\ncerro\n##tively\n##vio\nbillions\nmodulation\nsedimentary\n283\nfacilitating\noutrageous\ngoldstein\n##eak\n##kt\nld\nmaitland\npenultimate\npollard\n##dance\nfleets\nspaceship\nvertebrae\n##nig\nalcoholism\nals\nrecital\n##bham\n##ference\n##omics\nm2\n##bm\ntrois\n##tropical\n##в\ncommemorates\n##meric\nmarge\n##raction\n1643\n670\ncosmetic\nravaged\n##ige\ncatastrophe\neng\n##shida\nalbrecht\narterial\nbellamy\ndecor\nharmon\n##rde\nbulbs\nsynchronized\nvito\neasiest\nshetland\nshielding\nwnba\n##glers\n##ssar\n##riam\nbrianna\ncumbria\n##aceous\n##rard\ncores\nthayer\n##nsk\nbrood\nhilltop\nluminous\ncarts\nkeynote\nlarkin\nlogos\n##cta\n##ا\n##mund\n##quay\nlilith\ntinted\n277\nwrestle\nmobilization\n##uses\nsequential\nsiam\nbloomfield\ntakahashi\n274\n##ieving\npresenters\nringo\nblazed\nwitty\n##oven\n##ignant\ndevastation\nhaydn\nharmed\nnewt\ntherese\n##peed\ngershwin\nmolina\nrabbis\nsudanese\n001\ninnate\nrestarted\n##sack\n##fus\nslices\nwb\n##shah\nenroll\nhypothetical\nhysterical\n1743\nfabio\nindefinite\nwarped\n##hg\nexchanging\n525\nunsuitable\n##sboro\ngallo\n1603\nbret\ncobalt\nhomemade\n##hunter\nmx\noperatives\n##dhar\nterraces\ndurable\nlatch\npens\nwhorls\n##ctuated\n##eaux\nbilling\nligament\nsuccumbed\n##gly\nregulators\nspawn\n##brick\n##stead\nfilmfare\nrochelle\n##nzo\n1725\ncircumstance\nsaber\nsupplements\n##nsky\n##tson\ncrowe\nwellesley\ncarrot\n##9th\n##movable\nprimate\ndrury\nsincerely\ntopical\n##mad\n##rao\ncallahan\nkyiv\nsmarter\ntits\nundo\n##yeh\nannouncements\nanthologies\nbarrio\nnebula\n##islaus\n##shaft\n##tyn\nbodyguards\n2021\nassassinate\nbarns\nemmett\nscully\n##mah\n##yd\n##eland\n##tino\n##itarian\ndemoted\ngorman\nlashed\nprized\nadventist\nwrit\n##gui\nalla\ninvertebrates\n##ausen\n1641\namman\n1742\nalign\nhealy\nredistribution\n##gf\n##rize\ninsulation\n##drop\nadherents\nhezbollah\nvitro\nferns\nyanking\n269\nphp\nregistering\nuppsala\ncheerleading\nconfines\nmischievous\ntully\n##ross\n49th\ndocked\nroam\nstipulated\npumpkin\n##bry\nprompt\n##ezer\nblindly\nshuddering\ncraftsmen\nfrail\nscented\nkatharine\nscramble\nshaggy\nsponge\nhelix\nzaragoza\n279\n##52\n43rd\nbacklash\nfontaine\nseizures\nposse\ncowan\nnonfiction\ntelenovela\nwwii\nhammered\nundone\n##gpur\nencircled\nirs\n##ivation\nartefacts\noneself\nsearing\nsmallpox\n##belle\n##osaurus\nshandong\nbreached\nupland\nblushing\nrankin\ninfinitely\npsyche\ntolerated\ndocking\nevicted\n##col\nunmarked\n##lving\ngnome\nlettering\nlitres\nmusique\n##oint\nbenevolent\n##jal\nblackened\n##anna\nmccall\nracers\ntingle\n##ocene\n##orestation\nintroductions\nradically\n292\n##hiff\n##باد\n1610\n1739\nmunchen\nplead\n##nka\ncondo\nscissors\n##sight\n##tens\napprehension\n##cey\n##yin\nhallmark\nwatering\nformulas\nsequels\n##llas\naggravated\nbae\ncommencing\n##building\nenfield\nprohibits\nmarne\nvedic\ncivilized\neuclidean\njagger\nbeforehand\nblasts\ndumont\n##arney\n##nem\n740\nconversions\nhierarchical\nrios\nsimulator\n##dya\n##lellan\nhedges\noleg\nthrusts\nshadowed\ndarby\nmaximize\n1744\ngregorian\n##nded\n##routed\nsham\nunspecified\n##hog\nemory\nfactual\n##smo\n##tp\nfooled\n##rger\nortega\nwellness\nmarlon\n##oton\n##urance\ncasket\nkeating\nley\nenclave\n##ayan\nchar\ninfluencing\njia\n##chenko\n412\nammonia\nerebidae\nincompatible\nviolins\ncornered\n##arat\ngrooves\nastronauts\ncolumbian\nrampant\nfabrication\nkyushu\nmahmud\nvanish\n##dern\nmesopotamia\n##lete\nict\n##rgen\ncaspian\nkenji\npitted\n##vered\n999\ngrimace\nroanoke\ntchaikovsky\ntwinned\n##analysis\n##awan\nxinjiang\narias\nclemson\nkazakh\nsizable\n1662\n##khand\n##vard\nplunge\ntatum\nvittorio\n##nden\ncholera\n##dana\n##oper\nbracing\nindifference\nprojectile\nsuperliga\n##chee\nrealises\nupgrading\n299\nporte\nretribution\n##vies\nnk\nstil\n##resses\nama\nbureaucracy\nblackberry\nbosch\ntestosterone\ncollapses\ngreer\n##pathic\nioc\nfifties\nmalls\n##erved\nbao\nbaskets\nadolescents\nsiegfried\n##osity\n##tosis\nmantra\ndetecting\nexistent\nfledgling\n##cchi\ndissatisfied\ngan\ntelecommunication\nmingled\nsobbed\n6000\ncontroversies\noutdated\ntaxis\n##raus\nfright\nslams\n##lham\n##fect\n##tten\ndetectors\nfetal\ntanned\n##uw\nfray\ngoth\nolympian\nskipping\nmandates\nscratches\nsheng\nunspoken\nhyundai\ntracey\nhotspur\nrestrictive\n##buch\namericana\nmundo\n##bari\nburroughs\ndiva\nvulcan\n##6th\ndistinctions\nthumping\n##ngen\nmikey\nsheds\nfide\nrescues\nspringsteen\nvested\nvaluation\n##ece\n##ely\npinnacle\nrake\nsylvie\n##edo\nalmond\nquivering\n##irus\nalteration\nfaltered\n##wad\n51st\nhydra\nticked\n##kato\nrecommends\n##dicated\nantigua\narjun\nstagecoach\nwilfred\ntrickle\npronouns\n##pon\naryan\nnighttime\n##anian\ngall\npea\nstitch\n##hei\nleung\nmilos\n##dini\neritrea\nnexus\nstarved\nsnowfall\nkant\nparasitic\ncot\ndiscus\nhana\nstrikers\nappleton\nkitchens\n##erina\n##partisan\n##itha\n##vius\ndisclose\nmetis\n##channel\n1701\ntesla\n##vera\nfitch\n1735\nblooded\n##tila\ndecimal\n##tang\n##bai\ncyclones\neun\nbottled\npeas\npensacola\nbasha\nbolivian\ncrabs\nboil\nlanterns\npartridge\nroofed\n1645\nnecks\n##phila\nopined\npatting\n##kla\n##lland\nchuckles\nvolta\nwhereupon\n##nche\ndevout\neuroleague\nsuicidal\n##dee\ninherently\ninvoluntary\nknitting\nnasser\n##hide\npuppets\ncolourful\ncourageous\nsouthend\nstills\nmiraculous\nhodgson\nricher\nrochdale\nethernet\ngreta\nuniting\nprism\numm\n##haya\n##itical\n##utation\ndeterioration\npointe\nprowess\n##ropriation\nlids\nscranton\nbillings\nsubcontinent\n##koff\n##scope\nbrute\nkellogg\npsalms\ndegraded\n##vez\nstanisław\n##ructured\nferreira\npun\nastonishing\ngunnar\n##yat\narya\nprc\ngottfried\n##tight\nexcursion\n##ographer\ndina\n##quil\n##nare\nhuffington\nillustrious\nwilbur\ngundam\nverandah\n##zard\nnaacp\n##odle\nconstructive\nfjord\nkade\n##naud\ngenerosity\nthrilling\nbaseline\ncayman\nfrankish\nplastics\naccommodations\nzoological\n##fting\ncedric\nqb\nmotorized\n##dome\n##otted\nsquealed\ntackled\ncanucks\nbudgets\nsitu\nasthma\ndail\ngabled\ngrasslands\nwhimpered\nwrithing\njudgments\n##65\nminnie\npv\n##carbon\nbananas\ngrille\ndomes\nmonique\nodin\nmaguire\nmarkham\ntierney\n##estra\n##chua\nlibel\npoke\nspeedy\natrium\nlaval\nnotwithstanding\n##edly\nfai\nkala\n##sur\nrobb\n##sma\nlistings\nluz\nsupplementary\ntianjin\n##acing\nenzo\njd\nric\nscanner\ncroats\ntranscribed\n##49\narden\ncv\n##hair\n##raphy\n##lver\n##uy\n357\nseventies\nstaggering\nalam\nhorticultural\nhs\nregression\ntimbers\nblasting\n##ounded\nmontagu\nmanipulating\n##cit\ncatalytic\n1550\ntroopers\n##meo\ncondemnation\nfitzpatrick\n##oire\n##roved\ninexperienced\n1670\ncastes\n##lative\nouting\n314\ndubois\nflicking\nquarrel\nste\nlearners\n1625\niq\nwhistled\n##class\n282\nclassify\ntariffs\ntemperament\n355\nfolly\nliszt\n##yles\nimmersed\njordanian\nceasefire\napparel\nextras\nmaru\nfished\n##bio\nharta\nstockport\nassortment\ncraftsman\nparalysis\ntransmitters\n##cola\nblindness\n##wk\nfatally\nproficiency\nsolemnly\n##orno\nrepairing\namore\ngroceries\nultraviolet\n##chase\nschoolhouse\n##tua\nresurgence\nnailed\n##otype\n##×\nruse\nsaliva\ndiagrams\n##tructing\nalbans\nrann\nthirties\n1b\nantennas\nhilarious\ncougars\npaddington\nstats\n##eger\nbreakaway\nipod\nreza\nauthorship\nprohibiting\nscoffed\n##etz\n##ttle\nconscription\ndefected\ntrondheim\n##fires\nivanov\nkeenan\n##adan\n##ciful\n##fb\n##slow\nlocating\n##ials\n##tford\ncadiz\nbasalt\nblankly\ninterned\nrags\nrattling\n##tick\ncarpathian\nreassured\nsync\nbum\nguildford\niss\nstaunch\n##onga\nastronomers\nsera\nsofie\nemergencies\nsusquehanna\n##heard\nduc\nmastery\nvh1\nwilliamsburg\nbayer\nbuckled\ncraving\n##khan\n##rdes\nbloomington\n##write\nalton\nbarbecue\n##bians\njustine\n##hri\n##ndt\ndelightful\nsmartphone\nnewtown\nphoton\nretrieval\npeugeot\nhissing\n##monium\n##orough\nflavors\nlighted\nrelaunched\ntainted\n##games\n##lysis\nanarchy\nmicroscopic\nhopping\nadept\nevade\nevie\n##beau\ninhibit\nsinn\nadjustable\nhurst\nintuition\nwilton\ncisco\n44th\nlawful\nlowlands\nstockings\nthierry\n##dalen\n##hila\n##nai\nfates\nprank\ntb\nmaison\nlobbied\nprovocative\n1724\n4a\nutopia\n##qual\ncarbonate\ngujarati\npurcell\n##rford\ncurtiss\n##mei\novergrown\narenas\nmediation\nswallows\n##rnik\nrespectful\nturnbull\n##hedron\n##hope\nalyssa\nozone\n##ʻi\nami\ngestapo\njohansson\nsnooker\ncanteen\ncuff\ndeclines\nempathy\nstigma\n##ags\n##iner\n##raine\ntaxpayers\ngui\nvolga\n##wright\n##copic\nlifespan\novercame\ntattooed\nenactment\ngiggles\n##ador\n##camp\nbarrington\nbribe\nobligatory\norbiting\npeng\n##enas\nelusive\nsucker\n##vating\ncong\nhardship\nempowered\nanticipating\nestrada\ncryptic\ngreasy\ndetainees\nplanck\nsudbury\nplaid\ndod\nmarriott\nkayla\n##ears\n##vb\n##zd\nmortally\n##hein\ncognition\nradha\n319\nliechtenstein\nmeade\nrichly\nargyle\nharpsichord\nliberalism\ntrumpets\nlauded\ntyrant\nsalsa\ntiled\nlear\npromoters\nreused\nslicing\ntrident\n##chuk\n##gami\n##lka\ncantor\ncheckpoint\n##points\ngaul\nleger\nmammalian\n##tov\n##aar\n##schaft\ndoha\nfrenchman\nnirvana\n##vino\ndelgado\nheadlining\n##eron\n##iography\njug\ntko\n1649\nnaga\nintersections\n##jia\nbenfica\nnawab\n##suka\nashford\ngulp\n##deck\n##vill\n##rug\nbrentford\nfrazier\npleasures\ndunne\npotsdam\nshenzhen\ndentistry\n##tec\nflanagan\n##dorff\n##hear\nchorale\ndinah\nprem\nquezon\n##rogated\nrelinquished\nsutra\nterri\n##pani\nflaps\n##rissa\npoly\n##rnet\nhomme\naback\n##eki\nlinger\nwomb\n##kson\n##lewood\ndoorstep\northodoxy\nthreaded\nwestfield\n##rval\ndioceses\nfridays\nsubsided\n##gata\nloyalists\n##biotic\n##ettes\nletterman\nlunatic\nprelate\ntenderly\ninvariably\nsouza\nthug\nwinslow\n##otide\nfurlongs\ngogh\njeopardy\n##runa\npegasus\n##umble\nhumiliated\nstandalone\ntagged\n##roller\nfreshmen\nklan\n##bright\nattaining\ninitiating\ntransatlantic\nlogged\nviz\n##uance\n1723\ncombatants\nintervening\nstephane\nchieftain\ndespised\ngrazed\n317\ncdc\ngalveston\ngodzilla\nmacro\nsimulate\n##planes\nparades\n##esses\n960\n##ductive\n##unes\nequator\noverdose\n##cans\n##hosh\n##lifting\njoshi\nepstein\nsonora\ntreacherous\naquatics\nmanchu\nresponsive\n##sation\nsupervisory\n##christ\n##llins\n##ibar\n##balance\n##uso\nkimball\nkarlsruhe\nmab\n##emy\nignores\nphonetic\nreuters\nspaghetti\n820\nalmighty\ndanzig\nrumbling\ntombstone\ndesignations\nlured\noutset\n##felt\nsupermarkets\n##wt\ngrupo\nkei\nkraft\nsusanna\n##blood\ncomprehension\ngenealogy\n##aghan\n##verted\nredding\n##ythe\n1722\nbowing\n##pore\n##roi\nlest\nsharpened\nfulbright\nvalkyrie\nsikhs\n##unds\nswans\nbouquet\nmerritt\n##tage\n##venting\ncommuted\nredhead\nclerks\nleasing\ncesare\ndea\nhazy\n##vances\nfledged\ngreenfield\nservicemen\n##gical\narmando\nblackout\ndt\nsagged\ndownloadable\nintra\npotion\npods\n##4th\n##mism\nxp\nattendants\ngambia\nstale\n##ntine\nplump\nasteroids\nrediscovered\nbuds\nflea\nhive\n##neas\n1737\nclassifications\ndebuts\n##eles\nolympus\nscala\n##eurs\n##gno\n##mute\nhummed\nsigismund\nvisuals\nwiggled\nawait\npilasters\nclench\nsulfate\n##ances\nbellevue\nenigma\ntrainee\nsnort\n##sw\nclouded\ndenim\n##rank\n##rder\nchurning\nhartman\nlodges\nriches\nsima\n##missible\naccountable\nsocrates\nregulates\nmueller\n##cr\n1702\navoids\nsolids\nhimalayas\nnutrient\npup\n##jevic\nsquat\nfades\nnec\n##lates\n##pina\n##rona\n##ου\nprivateer\ntequila\n##gative\n##mpton\napt\nhornet\nimmortals\n##dou\nasturias\ncleansing\ndario\n##rries\n##anta\netymology\nservicing\nzhejiang\n##venor\n##nx\nhorned\nerasmus\nrayon\nrelocating\n£10\n##bags\nescalated\npromenade\nstubble\n2010s\nartisans\naxial\nliquids\nmora\nsho\nyoo\n##tsky\nbundles\noldies\n##nally\nnotification\nbastion\n##ths\nsparkle\n##lved\n1728\nleash\npathogen\nhighs\n##hmi\nimmature\n880\ngonzaga\nignatius\nmansions\nmonterrey\nsweets\nbryson\n##loe\npolled\nregatta\nbrightest\npei\nrosy\nsquid\nhatfield\npayroll\naddict\nmeath\ncornerback\nheaviest\nlodging\n##mage\ncapcom\nrippled\n##sily\nbarnet\nmayhem\nymca\nsnuggled\nrousseau\n##cute\nblanchard\n284\nfragmented\nleighton\nchromosomes\nrisking\n##md\n##strel\n##utter\ncorinne\ncoyotes\ncynical\nhiroshi\nyeomanry\n##ractive\nebook\ngrading\nmandela\nplume\nagustin\nmagdalene\n##rkin\nbea\nfemme\ntrafford\n##coll\n##lun\n##tance\n52nd\nfourier\nupton\n##mental\ncamilla\ngust\niihf\nislamabad\nlongevity\n##kala\nfeldman\nnetting\n##rization\nendeavour\nforaging\nmfa\norr\n##open\ngreyish\ncontradiction\ngraz\n##ruff\nhandicapped\nmarlene\ntweed\noaxaca\nspp\ncampos\nmiocene\npri\nconfigured\ncooks\npluto\ncozy\npornographic\n##entes\n70th\nfairness\nglided\njonny\nlynne\nrounding\nsired\n##emon\n##nist\nremade\nuncover\n##mack\ncomplied\nlei\nnewsweek\n##jured\n##parts\n##enting\n##pg\n293\nfiner\nguerrillas\nathenian\ndeng\ndisused\nstepmother\naccuse\ngingerly\nseduction\n521\nconfronting\n##walker\n##going\ngora\nnostalgia\nsabres\nvirginity\nwrenched\n##minated\nsyndication\nwielding\neyre\n##56\n##gnon\n##igny\nbehaved\ntaxpayer\nsweeps\n##growth\nchildless\ngallant\n##ywood\namplified\ngeraldine\nscrape\n##ffi\nbabylonian\nfresco\n##rdan\n##kney\n##position\n1718\nrestricting\ntack\nfukuoka\nosborn\nselector\npartnering\n##dlow\n318\ngnu\nkia\ntak\nwhitley\ngables\n##54\n##mania\nmri\nsoftness\nimmersion\n##bots\n##evsky\n1713\nchilling\ninsignificant\npcs\n##uis\nelites\nlina\npurported\nsupplemental\nteaming\n##americana\n##dding\n##inton\nproficient\nrouen\n##nage\n##rret\nniccolo\nselects\n##bread\nfluffy\n1621\ngruff\nknotted\nmukherjee\npolgara\nthrash\nnicholls\nsecluded\nsmoothing\nthru\ncorsica\nloaf\nwhitaker\ninquiries\n##rrier\n##kam\nindochina\n289\nmarlins\nmyles\npeking\n##tea\nextracts\npastry\nsuperhuman\nconnacht\nvogel\n##ditional\n##het\n##udged\n##lash\ngloss\nquarries\nrefit\nteaser\n##alic\n##gaon\n20s\nmaterialized\nsling\ncamped\npickering\ntung\ntracker\npursuant\n##cide\ncranes\nsoc\n##cini\n##typical\n##viere\nanhalt\noverboard\nworkout\nchores\nfares\norphaned\nstains\n##logie\nfenton\nsurpassing\njoyah\ntriggers\n##itte\ngrandmaster\n##lass\n##lists\nclapping\nfraudulent\nledger\nnagasaki\n##cor\n##nosis\n##tsa\neucalyptus\ntun\n##icio\n##rney\n##tara\ndax\nheroism\nina\nwrexham\nonboard\nunsigned\n##dates\nmoshe\ngalley\nwinnie\ndroplets\nexiles\npraises\nwatered\nnoodles\n##aia\nfein\nadi\nleland\nmulticultural\nstink\nbingo\ncomets\nerskine\nmodernized\ncanned\nconstraint\ndomestically\nchemotherapy\nfeatherweight\nstifled\n##mum\ndarkly\nirresistible\nrefreshing\nhasty\nisolate\n##oys\nkitchener\nplanners\n##wehr\ncages\nyarn\nimplant\ntoulon\nelects\nchildbirth\nyue\n##lind\n##lone\ncn\nrightful\nsportsman\njunctions\nremodeled\nspecifies\n##rgh\n291\n##oons\ncomplimented\n##urgent\nlister\not\n##logic\nbequeathed\ncheekbones\nfontana\ngabby\n##dial\namadeus\ncorrugated\nmaverick\nresented\ntriangles\n##hered\n##usly\nnazareth\ntyrol\n1675\nassent\npoorer\nsectional\naegean\n##cous\n296\nnylon\nghanaian\n##egorical\n##weig\ncushions\nforbid\nfusiliers\nobstruction\nsomerville\n##scia\ndime\nearrings\nelliptical\nleyte\noder\npolymers\ntimmy\natm\nmidtown\npiloted\nsettles\ncontinual\nexternally\nmayfield\n##uh\nenrichment\nhenson\nkeane\npersians\n1733\nbenji\nbraden\npep\n324\n##efe\ncontenders\npepsi\nvalet\n##isches\n298\n##asse\n##earing\ngoofy\nstroll\n##amen\nauthoritarian\noccurrences\nadversary\nahmedabad\ntangent\ntoppled\ndorchester\n1672\nmodernism\nmarxism\nislamist\ncharlemagne\nexponential\nracks\nunicode\nbrunette\nmbc\npic\nskirmish\n##bund\n##lad\n##powered\n##yst\nhoisted\nmessina\nshatter\n##ctum\njedi\nvantage\n##music\n##neil\nclemens\nmahmoud\ncorrupted\nauthentication\nlowry\nnils\n##washed\nomnibus\nwounding\njillian\n##itors\n##opped\nserialized\nnarcotics\nhandheld\n##arm\n##plicity\nintersecting\nstimulating\n##onis\ncrate\nfellowships\nhemingway\ncasinos\nclimatic\nfordham\ncopeland\ndrip\nbeatty\nleaflets\nrobber\nbrothel\nmadeira\n##hedral\nsphinx\nultrasound\n##vana\nvalor\nforbade\nleonid\nvillas\n##aldo\nduane\nmarquez\n##cytes\ndisadvantaged\nforearms\nkawasaki\nreacts\nconsular\nlax\nuncles\nuphold\n##hopper\nconcepcion\ndorsey\nlass\n##izan\narching\npassageway\n1708\nresearches\ntia\ninternationals\n##graphs\n##opers\ndistinguishes\njavanese\ndivert\n##uven\nplotted\n##listic\n##rwin\n##erik\n##tify\naffirmative\nsignifies\nvalidation\n##bson\nkari\nfelicity\ngeorgina\nzulu\n##eros\n##rained\n##rath\novercoming\n##dot\nargyll\n##rbin\n1734\nchiba\nratification\nwindy\nearls\nparapet\n##marks\nhunan\npristine\nastrid\npunta\n##gart\nbrodie\n##kota\n##oder\nmalaga\nminerva\nrouse\n##phonic\nbellowed\npagoda\nportals\nreclamation\n##gur\n##odies\n##⁄₄\nparentheses\nquoting\nallergic\npalette\nshowcases\nbenefactor\nheartland\nnonlinear\n##tness\nbladed\ncheerfully\nscans\n##ety\n##hone\n1666\ngirlfriends\npedersen\nhiram\nsous\n##liche\n##nator\n1683\n##nery\n##orio\n##umen\nbobo\nprimaries\nsmiley\n##cb\nunearthed\nuniformly\nfis\nmetadata\n1635\nind\n##oted\nrecoil\n##titles\n##tura\n##ια\n406\nhilbert\njamestown\nmcmillan\ntulane\nseychelles\n##frid\nantics\ncoli\nfated\nstucco\n##grants\n1654\nbulky\naccolades\narrays\ncaledonian\ncarnage\noptimism\npuebla\n##tative\n##cave\nenforcing\nrotherham\nseo\ndunlop\naeronautics\nchimed\nincline\nzoning\narchduke\nhellenistic\n##oses\n##sions\ncandi\nthong\n##ople\nmagnate\nrustic\n##rsk\nprojective\nslant\n##offs\ndanes\nhollis\nvocalists\n##ammed\ncongenital\ncontend\ngesellschaft\n##ocating\n##pressive\ndouglass\nquieter\n##cm\n##kshi\nhowled\nsalim\nspontaneously\ntownsville\nbuena\nsouthport\n##bold\nkato\n1638\nfaerie\nstiffly\n##vus\n##rled\n297\nflawless\nrealising\ntaboo\n##7th\nbytes\nstraightening\n356\njena\n##hid\n##rmin\ncartwright\nberber\nbertram\nsoloists\n411\nnoses\n417\ncoping\nfission\nhardin\ninca\n##cen\n1717\nmobilized\nvhf\n##raf\nbiscuits\ncurate\n##85\n##anial\n331\ngaunt\nneighbourhoods\n1540\n##abas\nblanca\nbypassed\nsockets\nbehold\ncoincidentally\n##bane\nnara\nshave\nsplinter\nterrific\n##arion\n##erian\ncommonplace\njuris\nredwood\nwaistband\nboxed\ncaitlin\nfingerprints\njennie\nnaturalized\n##ired\nbalfour\ncraters\njody\nbungalow\nhugely\nquilt\nglitter\npigeons\nundertaker\nbulging\nconstrained\ngoo\n##sil\n##akh\nassimilation\nreworked\n##person\npersuasion\n##pants\nfelicia\n##cliff\n##ulent\n1732\nexplodes\n##dun\n##inium\n##zic\nlyman\nvulture\nhog\noverlook\nbegs\nnorthwards\now\nspoil\n##urer\nfatima\nfavorably\naccumulate\nsargent\nsorority\ncorresponded\ndispersal\nkochi\ntoned\n##imi\n##lita\ninternacional\nnewfound\n##agger\n##lynn\n##rigue\nbooths\npeanuts\n##eborg\nmedicare\nmuriel\nnur\n##uram\ncrates\nmillennia\npajamas\nworsened\n##breakers\njimi\nvanuatu\nyawned\n##udeau\ncarousel\n##hony\nhurdle\n##ccus\n##mounted\n##pod\nrv\n##eche\nairship\nambiguity\ncompulsion\nrecapture\n##claiming\narthritis\n##osomal\n1667\nasserting\nngc\nsniffing\ndade\ndiscontent\nglendale\nported\n##amina\ndefamation\nrammed\n##scent\nfling\nlivingstone\n##fleet\n875\n##ppy\napocalyptic\ncomrade\nlcd\n##lowe\ncessna\neine\npersecuted\nsubsistence\ndemi\nhoop\nreliefs\n710\ncoptic\nprogressing\nstemmed\nperpetrators\n1665\npriestess\n##nio\ndobson\nebony\nrooster\nitf\ntortricidae\n##bbon\n##jian\ncleanup\n##jean\n##øy\n1721\neighties\ntaxonomic\nholiness\n##hearted\n##spar\nantilles\nshowcasing\nstabilized\n##nb\ngia\nmascara\nmichelangelo\ndawned\n##uria\n##vinsky\nextinguished\nfitz\ngrotesque\n£100\n##fera\n##loid\n##mous\nbarges\nneue\nthrobbed\ncipher\njohnnie\n##a1\n##mpt\noutburst\n##swick\nspearheaded\nadministrations\nc1\nheartbreak\npixels\npleasantly\n##enay\nlombardy\nplush\n##nsed\nbobbie\n##hly\nreapers\ntremor\nxiang\nminogue\nsubstantive\nhitch\nbarak\n##wyl\nkwan\n##encia\n910\nobscene\nelegance\nindus\nsurfer\nbribery\nconserve\n##hyllum\n##masters\nhoratio\n##fat\napes\nrebound\npsychotic\n##pour\niteration\n##mium\n##vani\nbotanic\nhorribly\nantiques\ndispose\npaxton\n##hli\n##wg\ntimeless\n1704\ndisregard\nengraver\nhounds\n##bau\n##version\nlooted\nuno\nfacilitates\ngroans\nmasjid\nrutland\nantibody\ndisqualification\ndecatur\nfootballers\nquake\nslacks\n48th\nrein\nscribe\nstabilize\ncommits\nexemplary\ntho\n##hort\n##chison\npantry\ntraversed\n##hiti\ndisrepair\nidentifiable\nvibrated\nbaccalaureate\n##nnis\ncsa\ninterviewing\n##iensis\n##raße\ngreaves\nwealthiest\n343\nclassed\njogged\n£5\n##58\n##atal\nilluminating\nknicks\nrespecting\n##uno\nscrubbed\n##iji\n##dles\nkruger\nmoods\ngrowls\nraider\nsilvia\nchefs\nkam\nvr\ncree\npercival\n##terol\ngunter\ncounterattack\ndefiant\nhenan\nze\n##rasia\n##riety\nequivalence\nsubmissions\n##fra\n##thor\nbautista\nmechanically\n##heater\ncornice\nherbal\ntemplar\n##mering\noutputs\nruining\nligand\nrenumbered\nextravagant\nmika\nblockbuster\neta\ninsurrection\n##ilia\ndarkening\nferocious\npianos\nstrife\nkinship\n##aer\nmelee\n##anor\n##iste\n##may\n##oue\ndecidedly\nweep\n##jad\n##missive\n##ppel\n354\npuget\nunease\n##gnant\n1629\nhammering\nkassel\nob\nwessex\n##lga\nbromwich\negan\nparanoia\nutilization\n##atable\n##idad\ncontradictory\nprovoke\n##ols\n##ouring\n##tangled\nknesset\n##very\n##lette\nplumbing\n##sden\n##¹\ngreensboro\noccult\nsniff\n338\nzev\nbeaming\ngamer\nhaggard\nmahal\n##olt\n##pins\nmendes\nutmost\nbriefing\ngunnery\n##gut\n##pher\n##zh\n##rok\n1679\nkhalifa\nsonya\n##boot\nprincipals\nurbana\nwiring\n##liffe\n##minating\n##rrado\ndahl\nnyu\nskepticism\nnp\ntownspeople\nithaca\nlobster\nsomethin\n##fur\n##arina\n##−1\nfreighter\nzimmerman\nbiceps\ncontractual\n##herton\namend\nhurrying\nsubconscious\n##anal\n336\nmeng\nclermont\nspawning\n##eia\n##lub\ndignitaries\nimpetus\nsnacks\nspotting\ntwigs\n##bilis\n##cz\n##ouk\nlibertadores\nnic\nskylar\n##aina\n##firm\ngustave\nasean\n##anum\ndieter\nlegislatures\nflirt\nbromley\ntrolls\numar\n##bbies\n##tyle\nblah\nparc\nbridgeport\ncrank\nnegligence\n##nction\n46th\nconstantin\nmolded\nbandages\nseriousness\n00pm\nsiegel\ncarpets\ncompartments\nupbeat\nstatehood\n##dner\n##edging\nmarko\n730\nplatt\n##hane\npaving\n##iy\n1738\nabbess\nimpatience\nlimousine\nnbl\n##talk\n441\nlucille\nmojo\nnightfall\nrobbers\n##nais\nkarel\nbrisk\ncalves\nreplicate\nascribed\ntelescopes\n##olf\nintimidated\n##reen\nballast\nspecialization\n##sit\naerodynamic\ncaliphate\nrainer\nvisionary\n##arded\nepsilon\n##aday\n##onte\naggregation\nauditory\nboosted\nreunification\nkathmandu\nloco\nrobyn\n402\nacknowledges\nappointing\nhumanoid\nnewell\nredeveloped\nrestraints\n##tained\nbarbarians\nchopper\n1609\nitaliana\n##lez\n##lho\ninvestigates\nwrestlemania\n##anies\n##bib\n690\n##falls\ncreaked\ndragoons\ngravely\nminions\nstupidity\nvolley\n##harat\n##week\nmusik\n##eries\n##uously\nfungal\nmassimo\nsemantics\nmalvern\n##ahl\n##pee\ndiscourage\nembryo\nimperialism\n1910s\nprofoundly\n##ddled\njiangsu\nsparkled\nstat\n##holz\nsweatshirt\ntobin\n##iction\nsneered\n##cheon\n##oit\nbrit\ncausal\nsmyth\n##neuve\ndiffuse\nperrin\nsilvio\n##ipes\n##recht\ndetonated\niqbal\nselma\n##nism\n##zumi\nroasted\n##riders\ntay\n##ados\n##mament\n##mut\n##rud\n840\ncompletes\nnipples\ncfa\nflavour\nhirsch\n##laus\ncalderon\nsneakers\nmoravian\n##ksha\n1622\nrq\n294\n##imeters\nbodo\n##isance\n##pre\n##ronia\nanatomical\nexcerpt\n##lke\ndh\nkunst\n##tablished\n##scoe\nbiomass\npanted\nunharmed\ngael\nhousemates\nmontpellier\n##59\ncoa\nrodents\ntonic\nhickory\nsingleton\n##taro\n451\n1719\naldo\nbreaststroke\ndempsey\noch\nrocco\n##cuit\nmerton\ndissemination\nmidsummer\nserials\n##idi\nhaji\npolynomials\n##rdon\ngs\nenoch\nprematurely\nshutter\ntaunton\n£3\n##grating\n##inates\narchangel\nharassed\n##asco\n326\narchway\ndazzling\n##ecin\n1736\nsumo\nwat\n##kovich\n1086\nhonneur\n##ently\n##nostic\n##ttal\n##idon\n1605\n403\n1716\nblogger\nrents\n##gnan\nhires\n##ikh\n##dant\nhowie\n##rons\nhandler\nretracted\nshocks\n1632\narun\nduluth\nkepler\ntrumpeter\n##lary\npeeking\nseasoned\ntrooper\n##mara\nlaszlo\n##iciencies\n##rti\nheterosexual\n##inatory\n##ssion\nindira\njogging\n##inga\n##lism\nbeit\ndissatisfaction\nmalice\n##ately\nnedra\npeeling\n##rgeon\n47th\nstadiums\n475\nvertigo\n##ains\niced\nrestroom\n##plify\n##tub\nillustrating\npear\n##chner\n##sibility\ninorganic\nrappers\nreceipts\nwatery\n##kura\nlucinda\n##oulos\nreintroduced\n##8th\n##tched\ngracefully\nsaxons\nnutritional\nwastewater\nrained\nfavourites\nbedrock\nfisted\nhallways\nlikeness\nupscale\n##lateral\n1580\nblinds\nprequel\n##pps\n##tama\ndeter\nhumiliating\nrestraining\ntn\nvents\n1659\nlaundering\nrecess\nrosary\ntractors\ncoulter\nfederer\n##ifiers\n##plin\npersistence\n##quitable\ngeschichte\npendulum\nquakers\n##beam\nbassett\npictorial\nbuffet\nkoln\n##sitor\ndrills\nreciprocal\nshooters\n##57\n##cton\n##tees\nconverge\npip\ndmitri\ndonnelly\nyamamoto\naqua\nazores\ndemographics\nhypnotic\nspitfire\nsuspend\nwryly\nroderick\n##rran\nsebastien\n##asurable\nmavericks\n##fles\n##200\nhimalayan\nprodigy\n##iance\ntransvaal\ndemonstrators\nhandcuffs\ndodged\nmcnamara\nsublime\n1726\ncrazed\n##efined\n##till\nivo\npondered\nreconciled\nshrill\nsava\n##duk\nbal\ncad\nheresy\njaipur\ngoran\n##nished\n341\nlux\nshelly\nwhitehall\n##hre\nisraelis\npeacekeeping\n##wled\n1703\ndemetrius\nousted\n##arians\n##zos\nbeale\nanwar\nbackstroke\nraged\nshrinking\ncremated\n##yck\nbenign\ntowing\nwadi\ndarmstadt\nlandfill\nparana\nsoothe\ncolleen\nsidewalks\nmayfair\ntumble\nhepatitis\nferrer\nsuperstructure\n##gingly\n##urse\n##wee\nanthropological\ntranslators\n##mies\ncloseness\nhooves\n##pw\nmondays\n##roll\n##vita\nlandscaping\n##urized\npurification\nsock\nthorns\nthwarted\njalan\ntiberius\n##taka\nsaline\n##rito\nconfidently\nkhyber\nsculptors\n##ij\nbrahms\nhammersmith\ninspectors\nbattista\nfivb\nfragmentation\nhackney\n##uls\narresting\nexercising\nantoinette\nbedfordshire\n##zily\ndyed\n##hema\n1656\nracetrack\nvariability\n##tique\n1655\naustrians\ndeteriorating\nmadman\ntheorists\naix\nlehman\nweathered\n1731\ndecreed\neruptions\n1729\nflaw\nquinlan\nsorbonne\nflutes\nnunez\n1711\nadored\ndownwards\nfable\nrasped\n1712\nmoritz\nmouthful\nrenegade\nshivers\nstunts\ndysfunction\nrestrain\ntranslit\n327\npancakes\n##avio\n##cision\n##tray\n351\nvial\n##lden\nbain\n##maid\n##oxide\nchihuahua\nmalacca\nvimes\n##rba\n##rnier\n1664\ndonnie\nplaques\n##ually\n337\nbangs\nfloppy\nhuntsville\nloretta\nnikolay\n##otte\neater\nhandgun\nubiquitous\n##hett\neras\nzodiac\n1634\n##omorphic\n1820s\n##zog\ncochran\n##bula\n##lithic\nwarring\n##rada\ndalai\nexcused\nblazers\nmcconnell\nreeling\nbot\neste\n##abi\ngeese\nhoax\ntaxon\n##bla\nguitarists\n##icon\ncondemning\nhunts\ninversion\nmoffat\ntaekwondo\n##lvis\n1624\nstammered\n##rest\n##rzy\nsousa\nfundraiser\nmarylebone\nnavigable\nuptown\ncabbage\ndaniela\nsalman\nshitty\nwhimper\n##kian\n##utive\nprogrammers\nprotections\nrm\n##rmi\n##rued\nforceful\n##enes\nfuss\n##tao\n##wash\nbrat\noppressive\nreykjavik\nspartak\nticking\n##inkles\n##kiewicz\nadolph\nhorst\nmaui\nprotege\nstraighten\ncpc\nlandau\nconcourse\nclements\nresultant\n##ando\nimaginative\njoo\nreactivated\n##rem\n##ffled\n##uising\nconsultative\n##guide\nflop\nkaitlyn\nmergers\nparenting\nsomber\n##vron\nsupervise\nvidhan\n##imum\ncourtship\nexemplified\nharmonies\nmedallist\nrefining\n##rrow\n##ка\namara\n##hum\n780\ngoalscorer\nsited\novershadowed\nrohan\ndispleasure\nsecretive\nmultiplied\nosman\n##orth\nengravings\npadre\n##kali\n##veda\nminiatures\nmis\n##yala\nclap\npali\nrook\n##cana\n1692\n57th\nantennae\nastro\noskar\n1628\nbulldog\ncrotch\nhackett\nyucatan\n##sure\namplifiers\nbrno\nferrara\nmigrating\n##gree\nthanking\nturing\n##eza\nmccann\nting\nandersson\nonslaught\ngaines\nganga\nincense\nstandardization\n##mation\nsentai\nscuba\nstuffing\nturquoise\nwaivers\nalloys\n##vitt\nregaining\nvaults\n##clops\n##gizing\ndigger\nfurry\nmemorabilia\nprobing\n##iad\npayton\nrec\ndeutschland\nfilippo\nopaque\nseamen\nzenith\nafrikaans\n##filtration\ndisciplined\ninspirational\n##merie\nbanco\nconfuse\ngrafton\ntod\n##dgets\nchampioned\nsimi\nanomaly\nbiplane\n##ceptive\nelectrode\n##para\n1697\ncleavage\ncrossbow\nswirl\ninformant\n##lars\n##osta\nafi\nbonfire\nspec\n##oux\nlakeside\nslump\n##culus\n##lais\n##qvist\n##rrigan\n1016\nfacades\nborg\ninwardly\ncervical\nxl\npointedly\n050\nstabilization\n##odon\nchests\n1699\nhacked\nctv\northogonal\nsuzy\n##lastic\ngaulle\njacobite\nrearview\n##cam\n##erted\nashby\n##drik\n##igate\n##mise\n##zbek\naffectionately\ncanine\ndisperse\nlatham\n##istles\n##ivar\nspielberg\n##orin\n##idium\nezekiel\ncid\n##sg\ndurga\nmiddletown\n##cina\ncustomized\nfrontiers\nharden\n##etano\n##zzy\n1604\nbolsheviks\n##66\ncoloration\nyoko\n##bedo\nbriefs\nslabs\ndebra\nliquidation\nplumage\n##oin\nblossoms\ndementia\nsubsidy\n1611\nproctor\nrelational\njerseys\nparochial\nter\n##ici\nesa\npeshawar\ncavalier\nloren\ncpi\nidiots\nshamrock\n1646\ndutton\nmalabar\nmustache\n##endez\n##ocytes\nreferencing\nterminates\nmarche\nyarmouth\n##sop\nacton\nmated\nseton\nsubtly\nbaptised\nbeige\nextremes\njolted\nkristina\ntelecast\n##actic\nsafeguard\nwaldo\n##baldi\n##bular\nendeavors\nsloppy\nsubterranean\n##ensburg\n##itung\ndelicately\npigment\ntq\n##scu\n1626\n##ound\ncollisions\ncoveted\nherds\n##personal\n##meister\n##nberger\nchopra\n##ricting\nabnormalities\ndefective\ngalician\nlucie\n##dilly\nalligator\nlikened\n##genase\nburundi\nclears\ncomplexion\nderelict\ndeafening\ndiablo\nfingered\nchampaign\ndogg\nenlist\nisotope\nlabeling\nmrna\n##erre\nbrilliance\nmarvelous\n##ayo\n1652\ncrawley\nether\nfooted\ndwellers\ndeserts\nhamish\nrubs\nwarlock\nskimmed\n##lizer\n870\nbuick\nembark\nheraldic\nirregularities\n##ajan\nkiara\n##kulam\n##ieg\nantigen\nkowalski\n##lge\noakley\nvisitation\n##mbit\nvt\n##suit\n1570\nmurderers\n##miento\n##rites\nchimneys\n##sling\ncondemn\ncuster\nexchequer\nhavre\n##ghi\nfluctuations\n##rations\ndfb\nhendricks\nvaccines\n##tarian\nnietzsche\nbiking\njuicy\n##duced\nbrooding\nscrolling\nselangor\n##ragan\n352\nannum\nboomed\nseminole\nsugarcane\n##dna\ndepartmental\ndismissing\ninnsbruck\narteries\nashok\nbatavia\ndaze\nkun\novertook\n##rga\n##tlan\nbeheaded\ngaddafi\nholm\nelectronically\nfaulty\ngalilee\nfractures\nkobayashi\n##lized\ngunmen\nmagma\naramaic\nmala\neastenders\ninference\nmessengers\nbf\n##qu\n407\nbathrooms\n##vere\n1658\nflashbacks\nideally\nmisunderstood\n##jali\n##weather\nmendez\n##grounds\n505\nuncanny\n##iii\n1709\nfriendships\n##nbc\nsacrament\naccommodated\nreiterated\nlogistical\npebbles\nthumped\n##escence\nadministering\ndecrees\ndrafts\n##flight\n##cased\n##tula\nfuturistic\npicket\nintimidation\nwinthrop\n##fahan\ninterfered\n339\nafar\nfrancoise\nmorally\nuta\ncochin\ncroft\ndwarfs\n##bruck\n##dents\n##nami\nbiker\n##hner\n##meral\nnano\n##isen\n##ometric\n##pres\n##ан\nbrightened\nmeek\nparcels\nsecurely\ngunners\n##jhl\n##zko\nagile\nhysteria\n##lten\n##rcus\nbukit\nchamps\nchevy\ncuckoo\nleith\nsadler\ntheologians\nwelded\n##section\n1663\njj\nplurality\nxander\n##rooms\n##formed\nshredded\ntemps\nintimately\npau\ntormented\n##lok\n##stellar\n1618\ncharred\nems\nessen\n##mmel\nalarms\nspraying\nascot\nblooms\ntwinkle\n##abia\n##apes\ninternment\nobsidian\n##chaft\nsnoop\n##dav\n##ooping\nmalibu\n##tension\nquiver\n##itia\nhays\nmcintosh\ntravers\nwalsall\n##ffie\n1623\nbeverley\nschwarz\nplunging\nstructurally\nm3\nrosenthal\nvikram\n##tsk\n770\nghz\n##onda\n##tiv\nchalmers\ngroningen\npew\nreckon\nunicef\n##rvis\n55th\n##gni\n1651\nsulawesi\navila\ncai\nmetaphysical\nscrewing\nturbulence\n##mberg\naugusto\nsamba\n56th\nbaffled\nmomentary\ntoxin\n##urian\n##wani\naachen\ncondoms\ndali\nsteppe\n##3d\n##app\n##oed\n##year\nadolescence\ndauphin\nelectrically\ninaccessible\nmicroscopy\nnikita\n##ega\natv\n##cel\n##enter\n##oles\n##oteric\n##ы\naccountants\npunishments\nwrongly\nbribes\nadventurous\nclinch\nflinders\nsouthland\n##hem\n##kata\ngough\n##ciency\nlads\nsoared\n##ה\nundergoes\ndeformation\noutlawed\nrubbish\n##arus\n##mussen\n##nidae\n##rzburg\narcs\n##ingdon\n##tituted\n1695\nwheelbase\nwheeling\nbombardier\ncampground\nzebra\n##lices\n##oj\n##bain\nlullaby\n##ecure\ndonetsk\nwylie\ngrenada\n##arding\n##ης\nsquinting\neireann\nopposes\n##andra\nmaximal\nrunes\n##broken\n##cuting\n##iface\n##ror\n##rosis\nadditive\nbritney\nadultery\ntriggering\n##drome\ndetrimental\naarhus\ncontainment\njc\nswapped\nvichy\n##ioms\nmadly\n##oric\n##rag\nbrant\n##ckey\n##trix\n1560\n1612\nbroughton\nrustling\n##stems\n##uder\nasbestos\nmentoring\n##nivorous\nfinley\nleaps\n##isan\napical\npry\nslits\nsubstitutes\n##dict\nintuitive\nfantasia\ninsistent\nunreasonable\n##igen\n##vna\ndomed\nhannover\nmargot\nponder\n##zziness\nimpromptu\njian\nlc\nrampage\nstemming\n##eft\nandrey\ngerais\nwhichever\namnesia\nappropriated\nanzac\nclicks\nmodifying\nultimatum\ncambrian\nmaids\nverve\nyellowstone\n##mbs\nconservatoire\n##scribe\nadherence\ndinners\nspectra\nimperfect\nmysteriously\nsidekick\ntatar\ntuba\n##aks\n##ifolia\ndistrust\n##athan\n##zle\nc2\nronin\nzac\n##pse\ncelaena\ninstrumentalist\nscents\nskopje\n##mbling\ncomical\ncompensated\nvidal\ncondor\nintersect\njingle\nwavelengths\n##urrent\nmcqueen\n##izzly\ncarp\nweasel\n422\nkanye\nmilitias\npostdoctoral\neugen\ngunslinger\n##ɛ\nfaux\nhospice\n##for\nappalled\nderivation\ndwarves\n##elis\ndilapidated\n##folk\nastoria\nphilology\n##lwyn\n##otho\n##saka\ninducing\nphilanthropy\n##bf\n##itative\ngeek\nmarkedly\nsql\n##yce\nbessie\nindices\nrn\n##flict\n495\nfrowns\nresolving\nweightlifting\ntugs\ncleric\ncontentious\n1653\nmania\nrms\n##miya\n##reate\n##ruck\n##tucket\nbien\neels\nmarek\n##ayton\n##cence\ndiscreet\nunofficially\n##ife\nleaks\n##bber\n1705\n332\ndung\ncompressor\nhillsborough\npandit\nshillings\ndistal\n##skin\n381\n##tat\n##you\nnosed\n##nir\nmangrove\nundeveloped\n##idia\ntextures\n##inho\n##500\n##rise\nae\nirritating\nnay\namazingly\nbancroft\napologetic\ncompassionate\nkata\nsymphonies\n##lovic\nairspace\n##lch\n930\ngifford\nprecautions\nfulfillment\nsevilla\nvulgar\nmartinique\n##urities\nlooting\npiccolo\ntidy\n##dermott\nquadrant\narmchair\nincomes\nmathematicians\nstampede\nnilsson\n##inking\n##scan\nfoo\nquarterfinal\n##ostal\nshang\nshouldered\nsquirrels\n##owe\n344\nvinegar\n##bner\n##rchy\n##systems\ndelaying\n##trics\nars\ndwyer\nrhapsody\nsponsoring\n##gration\nbipolar\ncinder\nstarters\n##olio\n##urst\n421\nsignage\n##nty\naground\nfigurative\nmons\nacquaintances\nduets\nerroneously\nsoyuz\nelliptic\nrecreated\n##cultural\n##quette\n##ssed\n##tma\n##zcz\nmoderator\nscares\n##itaire\n##stones\n##udence\njuniper\nsighting\n##just\n##nsen\nbritten\ncalabria\nry\nbop\ncramer\nforsyth\nstillness\n##л\nairmen\ngathers\nunfit\n##umber\n##upt\ntaunting\n##rip\nseeker\nstreamlined\n##bution\nholster\nschumann\ntread\nvox\n##gano\n##onzo\nstrive\ndil\nreforming\ncovent\nnewbury\npredicting\n##orro\ndecorate\ntre\n##puted\nandover\nie\nasahi\ndept\ndunkirk\ngills\n##tori\nburen\nhuskies\n##stis\n##stov\nabstracts\nbets\nloosen\n##opa\n1682\nyearning\n##glio\n##sir\nberman\neffortlessly\nenamel\nnapoli\npersist\n##peration\n##uez\nattache\nelisa\nb1\ninvitations\n##kic\naccelerating\nreindeer\nboardwalk\nclutches\nnelly\npolka\nstarbucks\n##kei\nadamant\nhuey\nlough\nunbroken\nadventurer\nembroidery\ninspecting\nstanza\n##ducted\nnaia\ntaluka\n##pone\n##roids\nchases\ndeprivation\nflorian\n##jing\n##ppet\nearthly\n##lib\n##ssee\ncolossal\nforeigner\nvet\nfreaks\npatrice\nrosewood\ntriassic\nupstate\n##pkins\ndominates\nata\nchants\nks\nvo\n##400\n##bley\n##raya\n##rmed\n555\nagra\ninfiltrate\n##ailing\n##ilation\n##tzer\n##uppe\n##werk\nbinoculars\nenthusiast\nfujian\nsqueak\n##avs\nabolitionist\nalmeida\nboredom\nhampstead\nmarsden\nrations\n##ands\ninflated\n334\nbonuses\nrosalie\npatna\n##rco\n329\ndetachments\npenitentiary\n54th\nflourishing\nwoolf\n##dion\n##etched\npapyrus\n##lster\n##nsor\n##toy\nbobbed\ndismounted\nendelle\ninhuman\nmotorola\ntbs\nwince\nwreath\n##ticus\nhideout\ninspections\nsanjay\ndisgrace\ninfused\npudding\nstalks\n##urbed\narsenic\nleases\n##hyl\n##rrard\ncollarbone\n##waite\n##wil\ndowry\n##bant\n##edance\ngenealogical\nnitrate\nsalamanca\nscandals\nthyroid\nnecessitated\n##!\n##\"\n###\n##$\n##%\n##&\n##'\n##(\n##)\n##*\n##+\n##,\n##-\n##.\n##/\n##:\n##;\n##<\n##=\n##>\n##?\n##@\n##[\n##\\\n##]\n##^\n##_\n##`\n##{\n##|\n##}\n##~\n##¡\n##¢\n##£\n##¤\n##¥\n##¦\n##§\n##¨\n##©\n##ª\n##«\n##¬\n##®\n##±\n##´\n##µ\n##¶\n##·\n##º\n##»\n##¼\n##¾\n##¿\n##æ\n##ð\n##÷\n##þ\n##đ\n##ħ\n##ŋ\n##œ\n##ƒ\n##ɐ\n##ɑ\n##ɒ\n##ɔ\n##ɕ\n##ə\n##ɡ\n##ɣ\n##ɨ\n##ɪ\n##ɫ\n##ɬ\n##ɯ\n##ɲ\n##ɴ\n##ɹ\n##ɾ\n##ʀ\n##ʁ\n##ʂ\n##ʃ\n##ʉ\n##ʊ\n##ʋ\n##ʌ\n##ʎ\n##ʐ\n##ʑ\n##ʒ\n##ʔ\n##ʰ\n##ʲ\n##ʳ\n##ʷ\n##ʸ\n##ʻ\n##ʼ\n##ʾ\n##ʿ\n##ˈ\n##ˡ\n##ˢ\n##ˣ\n##ˤ\n##β\n##γ\n##δ\n##ε\n##ζ\n##θ\n##κ\n##λ\n##μ\n##ξ\n##ο\n##π\n##ρ\n##σ\n##τ\n##υ\n##φ\n##χ\n##ψ\n##ω\n##б\n##г\n##д\n##ж\n##з\n##м\n##п\n##с\n##у\n##ф\n##х\n##ц\n##ч\n##ш\n##щ\n##ъ\n##э\n##ю\n##ђ\n##є\n##і\n##ј\n##љ\n##њ\n##ћ\n##ӏ\n##ա\n##բ\n##գ\n##դ\n##ե\n##թ\n##ի\n##լ\n##կ\n##հ\n##մ\n##յ\n##ն\n##ո\n##պ\n##ս\n##վ\n##տ\n##ր\n##ւ\n##ք\n##־\n##א\n##ב\n##ג\n##ד\n##ו\n##ז\n##ח\n##ט\n##י\n##ך\n##כ\n##ל\n##ם\n##מ\n##ן\n##נ\n##ס\n##ע\n##ף\n##פ\n##ץ\n##צ\n##ק\n##ר\n##ש\n##ת\n##،\n##ء\n##ب\n##ت\n##ث\n##ج\n##ح\n##خ\n##ذ\n##ز\n##س\n##ش\n##ص\n##ض\n##ط\n##ظ\n##ع\n##غ\n##ـ\n##ف\n##ق\n##ك\n##و\n##ى\n##ٹ\n##پ\n##چ\n##ک\n##گ\n##ں\n##ھ\n##ہ\n##ے\n##अ\n##आ\n##उ\n##ए\n##क\n##ख\n##ग\n##च\n##ज\n##ट\n##ड\n##ण\n##त\n##थ\n##द\n##ध\n##न\n##प\n##ब\n##भ\n##म\n##य\n##र\n##ल\n##व\n##श\n##ष\n##स\n##ह\n##ा\n##ि\n##ी\n##ो\n##।\n##॥\n##ং\n##অ\n##আ\n##ই\n##উ\n##এ\n##ও\n##ক\n##খ\n##গ\n##চ\n##ছ\n##জ\n##ট\n##ড\n##ণ\n##ত\n##থ\n##দ\n##ধ\n##ন\n##প\n##ব\n##ভ\n##ম\n##য\n##র\n##ল\n##শ\n##ষ\n##স\n##হ\n##া\n##ি\n##ী\n##ে\n##க\n##ச\n##ட\n##த\n##ந\n##ன\n##ப\n##ம\n##ய\n##ர\n##ல\n##ள\n##வ\n##ா\n##ி\n##ு\n##ே\n##ை\n##ನ\n##ರ\n##ಾ\n##ක\n##ය\n##ර\n##ල\n##ව\n##ා\n##ก\n##ง\n##ต\n##ท\n##น\n##พ\n##ม\n##ย\n##ร\n##ล\n##ว\n##ส\n##อ\n##า\n##เ\n##་\n##།\n##ག\n##ང\n##ད\n##ན\n##པ\n##བ\n##མ\n##འ\n##ར\n##ལ\n##ས\n##မ\n##ა\n##ბ\n##გ\n##დ\n##ე\n##ვ\n##თ\n##ი\n##კ\n##ლ\n##მ\n##ნ\n##ო\n##რ\n##ს\n##ტ\n##უ\n##ᄀ\n##ᄂ\n##ᄃ\n##ᄅ\n##ᄆ\n##ᄇ\n##ᄉ\n##ᄊ\n##ᄋ\n##ᄌ\n##ᄎ\n##ᄏ\n##ᄐ\n##ᄑ\n##ᄒ\n##ᅡ\n##ᅢ\n##ᅥ\n##ᅦ\n##ᅧ\n##ᅩ\n##ᅪ\n##ᅭ\n##ᅮ\n##ᅯ\n##ᅲ\n##ᅳ\n##ᅴ\n##ᅵ\n##ᆨ\n##ᆫ\n##ᆯ\n##ᆷ\n##ᆸ\n##ᆼ\n##ᴬ\n##ᴮ\n##ᴰ\n##ᴵ\n##ᴺ\n##ᵀ\n##ᵃ\n##ᵇ\n##ᵈ\n##ᵉ\n##ᵍ\n##ᵏ\n##ᵐ\n##ᵒ\n##ᵖ\n##ᵗ\n##ᵘ\n##ᵣ\n##ᵤ\n##ᵥ\n##ᶜ\n##ᶠ\n##‐\n##‑\n##‒\n##–\n##—\n##―\n##‖\n##‘\n##’\n##‚\n##“\n##”\n##„\n##†\n##‡\n##•\n##…\n##‰\n##′\n##″\n##›\n##‿\n##⁄\n##⁰\n##ⁱ\n##⁴\n##⁵\n##⁶\n##⁷\n##⁸\n##⁹\n##⁻\n##ⁿ\n##₅\n##₆\n##₇\n##₈\n##₉\n##₊\n##₍\n##₎\n##ₐ\n##ₑ\n##ₒ\n##ₓ\n##ₕ\n##ₖ\n##ₗ\n##ₘ\n##ₚ\n##ₛ\n##ₜ\n##₤\n##₩\n##€\n##₱\n##₹\n##ℓ\n##№\n##ℝ\n##™\n##⅓\n##⅔\n##←\n##↑\n##→\n##↓\n##↔\n##↦\n##⇄\n##⇌\n##⇒\n##∂\n##∅\n##∆\n##∇\n##∈\n##∗\n##∘\n##√\n##∞\n##∧\n##∨\n##∩\n##∪\n##≈\n##≡\n##≤\n##≥\n##⊂\n##⊆\n##⊕\n##⊗\n##⋅\n##─\n##│\n##■\n##▪\n##●\n##★\n##☆\n##☉\n##♠\n##♣\n##♥\n##♦\n##♯\n##⟨\n##⟩\n##ⱼ\n##⺩\n##⺼\n##⽥\n##、\n##。\n##〈\n##〉\n##《\n##》\n##「\n##」\n##『\n##』\n##〜\n##あ\n##い\n##う\n##え\n##お\n##か\n##き\n##く\n##け\n##こ\n##さ\n##し\n##す\n##せ\n##そ\n##た\n##ち\n##っ\n##つ\n##て\n##と\n##な\n##に\n##ぬ\n##ね\n##の\n##は\n##ひ\n##ふ\n##へ\n##ほ\n##ま\n##み\n##む\n##め\n##も\n##や\n##ゆ\n##よ\n##ら\n##り\n##る\n##れ\n##ろ\n##を\n##ん\n##ァ\n##ア\n##ィ\n##イ\n##ウ\n##ェ\n##エ\n##オ\n##カ\n##キ\n##ク\n##ケ\n##コ\n##サ\n##シ\n##ス\n##セ\n##タ\n##チ\n##ッ\n##ツ\n##テ\n##ト\n##ナ\n##ニ\n##ノ\n##ハ\n##ヒ\n##フ\n##ヘ\n##ホ\n##マ\n##ミ\n##ム\n##メ\n##モ\n##ャ\n##ュ\n##ョ\n##ラ\n##リ\n##ル\n##レ\n##ロ\n##ワ\n##ン\n##・\n##ー\n##一\n##三\n##上\n##下\n##不\n##世\n##中\n##主\n##久\n##之\n##也\n##事\n##二\n##五\n##井\n##京\n##人\n##亻\n##仁\n##介\n##代\n##仮\n##伊\n##会\n##佐\n##侍\n##保\n##信\n##健\n##元\n##光\n##八\n##公\n##内\n##出\n##分\n##前\n##劉\n##力\n##加\n##勝\n##北\n##区\n##十\n##千\n##南\n##博\n##原\n##口\n##古\n##史\n##司\n##合\n##吉\n##同\n##名\n##和\n##囗\n##四\n##国\n##國\n##土\n##地\n##坂\n##城\n##堂\n##場\n##士\n##夏\n##外\n##大\n##天\n##太\n##夫\n##奈\n##女\n##子\n##学\n##宀\n##宇\n##安\n##宗\n##定\n##宣\n##宮\n##家\n##宿\n##寺\n##將\n##小\n##尚\n##山\n##岡\n##島\n##崎\n##川\n##州\n##巿\n##帝\n##平\n##年\n##幸\n##广\n##弘\n##張\n##彳\n##後\n##御\n##德\n##心\n##忄\n##志\n##忠\n##愛\n##成\n##我\n##戦\n##戸\n##手\n##扌\n##政\n##文\n##新\n##方\n##日\n##明\n##星\n##春\n##昭\n##智\n##曲\n##書\n##月\n##有\n##朝\n##木\n##本\n##李\n##村\n##東\n##松\n##林\n##森\n##楊\n##樹\n##橋\n##歌\n##止\n##正\n##武\n##比\n##氏\n##民\n##水\n##氵\n##氷\n##永\n##江\n##沢\n##河\n##治\n##法\n##海\n##清\n##漢\n##瀬\n##火\n##版\n##犬\n##王\n##生\n##田\n##男\n##疒\n##発\n##白\n##的\n##皇\n##目\n##相\n##省\n##真\n##石\n##示\n##社\n##神\n##福\n##禾\n##秀\n##秋\n##空\n##立\n##章\n##竹\n##糹\n##美\n##義\n##耳\n##良\n##艹\n##花\n##英\n##華\n##葉\n##藤\n##行\n##街\n##西\n##見\n##訁\n##語\n##谷\n##貝\n##貴\n##車\n##軍\n##辶\n##道\n##郎\n##郡\n##部\n##都\n##里\n##野\n##金\n##鈴\n##镇\n##長\n##門\n##間\n##阝\n##阿\n##陳\n##陽\n##雄\n##青\n##面\n##風\n##食\n##香\n##馬\n##高\n##龍\n##龸\n##fi\n##fl\n##！\n##（\n##）\n##，\n##－\n##．\n##／\n##：\n##？\n##～\n" }, { "path": "src/examples/tensorflow/huggingface_bert/huggingface_bert.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e91cf83b\",\n \"metadata\": {},\n \"source\": [\n \"# Running Huggingface DistilBERT with TensorFlow-Neuron\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"71394e1e\",\n \"metadata\": {},\n \"source\": [\n \"In this tutorial you will compile and deploy DistilBERT version of HuggingFace 🤗 Transformers BERT for Inferentia using TensorFlow-Neuron. The full list of HuggingFace's pretrained BERT models can be found in the BERT section on this page https://huggingface.co/transformers/pretrained_models.html. you can also read about HuggingFace's pipeline feature here: https://huggingface.co/transformers/main_classes/pipelines.html\\n\",\n \"\\n\",\n \"This Jupyter notebook should be run on an instance which is inf1.6xlarge or larger, but in real life scenario the compilation should be done on a compute instance and the deployment on inf1 instance to save costs.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"828ef9bd\",\n \"metadata\": {},\n \"source\": [\n \"## Setup\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5becc549\",\n \"metadata\": {},\n \"source\": [\n \"To run this tutorial please follow the instructions for [TensorFlow-Neuron Setup](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/tensorflow-neuron.html#setup-tensorflow-neuron) and the [Jupyter Notebook Quickstart](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html) and set your kernel to \\\"Python (tensorflow-neuron)\\\" .\\n\",\n \"\\n\",\n \"Next, install some additional dependencies.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ee1a3b84\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"!pip install transformers==4.30.2\\n\",\n \"!pip install ipywidgets\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"c301cfce\",\n \"metadata\": {},\n \"source\": [\n \"## Download From Huggingface and Compile for AWS-Neuron\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"92e8050d\",\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import tensorflow as tf\\n\",\n \"import tensorflow_neuron as tfn\\n\",\n \"from transformers import DistilBertTokenizer, TFDistilBertModel\\n\",\n \"\\n\",\n \"# Create a wrapper for the roberta model that will accept inputs as a list\\n\",\n \"# instead of a dictionary. This will allow the compiled model to be saved\\n\",\n \"# to disk with the model.save() fucntion.\\n\",\n \"class DistilBertWrapper(tf.keras.Model):\\n\",\n \" def __init__(self, model):\\n\",\n \" super().__init__()\\n\",\n \" self.model = model\\n\",\n \" def __call__(self, example_inputs):\\n\",\n \" return self.model({'input_ids' : example_inputs[0], 'attention_mask' : example_inputs[1]})\\n\",\n \" \\n\",\n \"\\n\",\n \"tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')\\n\",\n \"model = DistilBertWrapper(TFDistilBertModel.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english'))\\n\",\n \"\\n\",\n \"batch_size = 16\\n\",\n \"\\n\",\n \"# create example inputs with a batch size of 16\\n\",\n \"text = [\\\"Paris is the of France.\\\"] * batch_size\\n\",\n \"encoded_input = tokenizer(text, return_tensors='tf', padding='max_length', max_length=64)\\n\",\n \"\\n\",\n \"# turn inputs into a list\\n\",\n \"example_input = [encoded_input['input_ids'], encoded_input['attention_mask']]\\n\",\n \"\\n\",\n \"#compile\\n\",\n \"model_neuron = tfn.trace(model, example_input)\\n\",\n \"\\n\",\n \"print(\\\"Running on neuron:\\\", model_neuron(example_input))\\n\",\n \"\\n\",\n \"# save the model to disk to save recompilation time for next usage\\n\",\n \"model_neuron.save('./distilbert-neuron-b16')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0f2e159a\",\n \"metadata\": {},\n \"source\": [\n \"## Run Basic Inference Benchmarking\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ccf22e74\",\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import numpy as np\\n\",\n \"import concurrent.futures\\n\",\n \"import time\\n\",\n \"\\n\",\n \"reloaded_neuron_model = tf.keras.models.load_model('./distilbert-neuron-b16')\\n\",\n \"print(\\\"Reloaded model running on neuron:\\\", reloaded_neuron_model(example_input))\\n\",\n \"\\n\",\n \"num_threads = 4\\n\",\n \"num_inferences = 1000\\n\",\n \"\\n\",\n \"latency_list = []\\n\",\n \"def inference_with_latency_calculation(example_input):\\n\",\n \" global latency_list\\n\",\n \" start = time.time()\\n\",\n \" result = reloaded_neuron_model(example_input)\\n\",\n \" end = time.time()\\n\",\n \" latency_list.append((end-start) * 1000)\\n\",\n \" return result\\n\",\n \"\\n\",\n \"start = time.time()\\n\",\n \"with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:\\n\",\n \" futures = []\\n\",\n \" for i in range(num_inferences):\\n\",\n \" futures.append(executor.submit(inference_with_latency_calculation, example_input))\\n\",\n \" for future in concurrent.futures.as_completed(futures):\\n\",\n \" get_result = future.result()\\n\",\n \"end = time.time()\\n\",\n \"\\n\",\n \"total_time = end - start\\n\",\n \"throughput = (num_inferences * batch_size)/total_time\\n\",\n \"\\n\",\n \"print(f\\\"Throughput was {throughput} samples per second.\\\")\\n\",\n \"print(f\\\"Latency p50 was {np.percentile(latency_list, 50)} ms\\\")\\n\",\n \"print(f\\\"Latency p90 was {np.percentile(latency_list, 90)} ms\\\")\\n\",\n \"print(f\\\"Latency p95 was {np.percentile(latency_list, 95)} ms\\\")\\n\",\n \"print(f\\\"Latency p99 was {np.percentile(latency_list, 99)} ms\\\")\\n\",\n \"assert(throughput >= 1930.0)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"b31b82fc\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.0\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/tensorflow/k8s_bert_demo/Dockerfile.tfserving_example", "content": "From ubuntu:16.04\nRUN apt-get update\nRUN apt-get install -y wget apt-transport-https ca-certificates awscli\nRUN echo \"deb https://apt.repos.neuron.amazonaws.com xenial main\" > /etc/apt/sources.list.d/neuron.list\nRUN wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | apt-key add -\n\nRUN apt-get update\nRUN apt-get install -y tensorflow-model-server-neuron" }, { "path": "src/examples/tensorflow/k8s_bert_demo/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/tensorflow/k8s_bert_demo/bert_client.py", "content": "import numpy as np\nimport grpc\nimport tensorflow as tf\nfrom tensorflow_serving.apis import predict_pb2\nfrom tensorflow_serving.apis import prediction_service_pb2_grpc\nimport time\n\nif __name__ == '__main__':\n channel = grpc.insecure_channel('localhost:9000')\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n request = predict_pb2.PredictRequest()\n request.model_spec.name = 'bert_mrpc_hc_gelus_b4_l24_0926_02'\n i = np.zeros([1, 128], dtype=np.int32)\n request.inputs['input_ids'].CopyFrom(tf.contrib.util.make_tensor_proto(i, shape=i.shape))\n request.inputs['input_mask'].CopyFrom(tf.contrib.util.make_tensor_proto(i, shape=i.shape))\n request.inputs['segment_ids'].CopyFrom(tf.contrib.util.make_tensor_proto(i, shape=i.shape))\n\n latencies = []\n for i in range(100):\n start = time.time()\n result = stub.Predict(request)\n latencies.append(time.time() - start)\n print(\"Inference successful: {}\".format(i))\n\n print (\"Ran {} inferences successfully. Latency average = {}\".format(len(latencies), np.average(latencies)))\n" }, { "path": "src/examples/tensorflow/k8s_bert_demo/bert_service.yml", "content": "---\nkind: Service\napiVersion: v1\nmetadata:\n name: inf-k8s-test\n labels:\n app: inf-k8s-test\nspec:\n ports:\n - name: http-tf-serving\n port: 8500\n targetPort: 8500\n - name: grpc-tf-serving\n port: 9000\n targetPort: 9000\n selector:\n app: inf-k8s-test\n role: master\n type: ClusterIP\n---\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: inf-k8s-test\n labels:\n app: inf-k8s-test\n role: master\nspec:\n replicas: 1 # Number of desired replicas. Increase to desired number.\n selector:\n matchLabels:\n app: inf-k8s-test\n role: master\n template:\n metadata:\n labels:\n app: inf-k8s-test\n role: master\n spec:\n volumes:\n - name: sock\n emptyDir: {}\n containers:\n - name: inf-k8s-test\n image: tf-serving-ctr\n imagePullPolicy: IfNotPresent\n command: [\"/bin/sh\",\"-c\"]\n\n # Pull model from s3, then start tensorflow_model_server_neuron with the model.\n args:\n - \"aws s3 sync s3:///bert /tmp/bert && \\\n tensorflow_model_server_neuron --port=9000 --rest_api_port=8500 --model_name=bert_mrpc_hc_gelus_b4_l24_0926_02 --model_base_path=/tmp//bert/\"\n\n # Open grpc and rest API ports\n ports:\n - containerPort: 8500\n - containerPort: 9000\n\n # Informs tensorflow_model_server_neuron of UDS socket location\n env:\n - name: NEURON_RTD_ADDRESS\n value: unix:/sock/neuron.sock\n\n # Arbitrary resource requirements\n resources:\n limits:\n cpu: 4\n memory: 4Gi\n requests:\n cpu: \"1\"\n memory: 1Gi\n\n # Shared volume mount, for UDS socket\n volumeMounts:\n - name: sock\n mountPath: /sock\n\n # Neuron-rtd container\n - name: neuron-rtd\n image: 790709498068.dkr.ecr.us-east-1.amazonaws.com/neuron-rtd:latest # neuron-rtd image.\n imagePullPolicy: IfNotPresent\n\n # Neuron-rtd required capabilities\n securityContext:\n capabilities:\n add:\n - SYS_ADMIN\n - IPC_LOCK\n\n # Shared volume mount, for UDS socket\n volumeMounts:\n - name: sock\n mountPath: /sock\n\n resources:\n limits:\n hugepages-2Mi: 256Mi # configure to 256 * desired number of Inferentia devices.\n aws.amazon.com/neuron: 1 # desired number of Inferentia devices.\n requests:\n memory: 1024Mi # Desired amount of memory. Should be larger than hugepages-2Mi limit.\n" }, { "path": "src/examples/tensorflow/keras_resnet50/LICENSE", "content": "Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n \nPermission is hereby granted, free of charge, to any person obtaining a copy of this\nsoftware and associated documentation files (the \"Software\"), to deal in the Software\nwithout restriction, including without limitation the rights to use, copy, modify,\nmerge, publish, distribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,\nINCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A\nPARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT\nHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\nSOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n" }, { "path": "src/examples/tensorflow/keras_resnet50/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/tensorflow/keras_resnet50/fp32tofp16.py", "content": "\"\"\" Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n\"\"\"\n\nimport re\nimport argparse\nimport tensorflow as tf\nimport numpy as np\n\nfrom google.protobuf import text_format\nfrom tensorflow.core.framework import graph_pb2\nfrom tensorflow.core.framework import node_def_pb2\nfrom tensorflow.python.platform import gfile\n\nfrom tensorflow.core.framework import attr_value_pb2\nfrom tensorflow.python.framework import tensor_util\n\ndef ConvertFP32ToOther(graphdef):\n \"\"\"Converts an FP32 network by casting all constants (weights) to a lower\n precision floating point type (FP16) and updating the dtypes\n everywhere.\"\"\"\n cast_type = \"float16\"\n sess = tf.Session(graph=tf.import_graph_def(graphdef))\n output_graph_def = graph_pb2.GraphDef()\n dummy_tensor = sess.run(tf.constant([0.1]))\n dummy_tensor_proto = tensor_util.make_tensor_proto(dummy_tensor, \\\n dtype=cast_type, shape=dummy_tensor.shape)\n dummy_tensor32 = sess.run(tf.constant([0.1]))\n dummy_tensor_proto32 = tensor_util.make_tensor_proto(dummy_tensor, \\\n dtype=tf.float32, shape=dummy_tensor.shape)\n dt_float_type_attr = attr_value_pb2.AttrValue(type=dummy_tensor_proto32.dtype)\n dt_half_type_attr = attr_value_pb2.AttrValue(type=dummy_tensor_proto.dtype)\n for node in graphdef.node:\n output_node = node_def_pb2.NodeDef()\n output_node.CopyFrom(node)\n if (node.op == \"Const\"):\n if (node.attr[\"dtype\"] == dt_float_type_attr):\n a = tensor_util.MakeNdarray(node.attr[\"value\"].tensor)\n a = tf.cast(a, cast_type)\n a = sess.run(a)\n output_node.attr[\"dtype\"].CopyFrom(dt_half_type_attr)\n output_node.attr[\"value\"].CopyFrom(\n attr_value_pb2.AttrValue(\n tensor=tensor_util.make_tensor_proto(a,\\\n dtype=cast_type, shape=a.shape)))\n else:\n if (\"T\" in node.attr.keys()):\n if (output_node.attr[\"T\"] == dt_float_type_attr):\n output_node.attr[\"T\"].CopyFrom(dt_half_type_attr)\n if (\"Tparams\" in node.attr.keys()):\n if (output_node.attr[\"Tparams\"] == dt_float_type_attr):\n output_node.attr[\"Tparams\"].CopyFrom(dt_half_type_attr)\n if (\"dtype\" in node.attr.keys()):\n if (node.attr[\"dtype\"] == dt_float_type_attr):\n output_node.attr[\"dtype\"].CopyFrom(dt_half_type_attr)\n if (\"SrcT\" in node.attr.keys()):\n if (node.attr[\"SrcT\"] == dt_float_type_attr):\n output_node.attr[\"SrcT\"].CopyFrom(dt_half_type_attr)\n if (\"DstT\" in node.attr.keys()):\n if (node.attr[\"DstT\"] == dt_float_type_attr):\n output_node.attr[\"DstT\"].CopyFrom(dt_half_type_attr)\n output_graph_def.node.extend([output_node])\n return output_graph_def\n\ndef load_graph(model_file):\n graph_def = tf.GraphDef()\n\n with open(model_file, \"rb\") as f:\n graph_def.ParseFromString(f.read())\n\n return graph_def\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--graph\", help=\"graph/model to be executed\",\n required=True)\n parser.add_argument(\"--out_graph\", help=\"graph/model to be generated\",\n required=True)\n args = parser.parse_args()\n\n graph_f32 = load_graph(args.graph)\n graph_f16 = ConvertFP32ToOther(graph_f32)\n output_xformed_graph_name = args.out_graph\n with gfile.GFile(output_xformed_graph_name, \"wb\") as f:\n f.write(graph_f16.SerializeToString())\n #with gfile.GFile(output_xformed_graph_name+\"txt\", 'w') as f:\n # f.write(text_format.MessageToString(graph_f16))\n" }, { "path": "src/examples/tensorflow/keras_resnet50/full_sweep", "content": "#!/usr/bin/env bash\n\n##########################################################################\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n##########################################################################\n\necho \"\" > full_sweep.log\necho \"\" > full_sweep_results.txt\n\nresults=()\nfor b in $(seq 1 5); do \n for i in 1 2 4 8 12 16; do \n python pb2sm_compile.py --batch_size=$b --neuroncore-pipeline-cores=$i | tee -a full_sweep.log;\n results[$b]+=\", \"`tail -1 full_sweep.log`\n done\ndone\n\nhead=\"batch\"\nfor i in 1 2 4 8 12 16; do\n head+=\", nc${i}\"\ndone \necho $head | tee -a full_sweep_results.txt\nfor b in $(seq 1 5); do \n echo $b${results[$b]} | tee -a full_sweep_results.txt\ndone\n" }, { "path": "src/examples/tensorflow/keras_resnet50/gen_resnet50_keras.py", "content": "\"\"\" Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n\"\"\"\n\nimport re\nimport argparse\nimport tensorflow as tf\nimport numpy as np\n\nfrom tensorflow.keras.applications.resnet50 import ResNet50\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions\n\nfrom google.protobuf import text_format\nimport tensorflow.python.saved_model\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--fp16\", action='store_true', help=\"use float16 parameters and operations\")\n args = parser.parse_args()\n\n # set Keras global configurations\n tf.keras.backend.set_learning_phase(0)\n tf.keras.backend.set_image_data_format('channels_last')\n if (args.fp16):\n float_type = 'float16'\n float_type2 = 'fp16'\n else:\n float_type = 'float32'\n float_type2 = 'fp32'\n tf.keras.backend.set_floatx(float_type)\n\n # load pre-trained model using Keras\n model_name = 'resnet50_%s_keras'%float_type2\n model = ResNet50(weights='imagenet')\n\n # various save files\n frozen_file = model_name + '.pb'\n opt_file = model_name + '_opt.pb'\n\n # obtain parameters\n model_input = model.input.name.replace(':0', '')\n model_output = model.output.name.replace(':0', '')\n batch, height, width, channels = model.input.shape\n\n print (\"model, frozen file, optimized file, input size, input node, output node,\")\n print (\"%s, %s, %s, %dx%dx%d, %s, %s\" %(model_name, frozen_file, opt_file, width, height, channels, model_input, model_output) ) \n\n # obtain the TF session\n sess = tf.compat.v1.keras.backend.get_session()\n\n # save checkpoint files for freeze_graph\n ckpt_file = '/tmp/' + model_name + '/' + model_name + '.ckpt'\n graph_file = '/tmp/' + model_name + '/' + model_name + '.pb'\n tf.compat.v1.train.Saver().save(sess, ckpt_file)\n tf.io.write_graph(sess.graph.as_graph_def(), logdir='.', name=graph_file, as_text=False)\n\n print(model_output)\n with tf.compat.v1.Session(graph=tf.Graph()) as sess:\n saver = tf.compat.v1.train.import_meta_graph(ckpt_file + '.meta')\n saver.restore(sess, ckpt_file)\n output_graph_def = tf.compat.v1.graph_util.convert_variables_to_constants(\n sess, tf.compat.v1.get_default_graph().as_graph_def(), [model_output])\n output_graph_def = tf.compat.v1.graph_util.remove_training_nodes(\n output_graph_def, protected_nodes=[model_output])\n with open(frozen_file, 'wb') as f:\n f.write(output_graph_def.SerializeToString())\n\n" }, { "path": "src/examples/tensorflow/keras_resnet50/infer_resnet50_keras.py", "content": "\"\"\" Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n\"\"\"\n\nimport os\nimport time\nimport shutil\nimport argparse\n\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications import resnet50\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"--graph\", default=\"resnet50_fp32_keras.pb\", help=\"Graph to use for inference\", required=True)\nparser.add_argument(\"--input\", default=\"input_1\", help=\"Input of graph\")\nparser.add_argument(\"--output\", default=\"probs/Softmax\", help=\"Output of graph\")\nargs = parser.parse_args()\n\ntf.keras.backend.set_image_data_format('channels_last')\n\ndef pb_to_saved_model(pb_path, input_names, output_names, model_dir):\n graph_def = tf.compat.v1.GraphDef()\n graph_def.ParseFromString(open(pb_path, 'rb').read())\n with tf.compat.v1.Session(graph=tf.Graph()) as sess:\n tf.import_graph_def(graph_def, name='')\n inputs = {name: sess.graph.get_tensor_by_name(ts_name) for name, ts_name in input_names.items()}\n outputs = {name: sess.graph.get_tensor_by_name(ts_name) for name, ts_name in output_names.items()}\n tf.saved_model.simple_save(sess, model_dir, inputs, outputs)\n\nSAVED_MODEL_DIR = './rn50_fp16'\nshutil.rmtree(SAVED_MODEL_DIR, ignore_errors=True)\ninput_tname=\"{}:0\".format(args.input)\noutput_tname=\"{}:0\".format(args.output)\npb_to_saved_model(args.graph, {input_tname : input_tname}, {output_tname : output_tname}, SAVED_MODEL_DIR)\n\n# Create input from image\nimg_sgl = image.load_img('kitten_small.jpg', target_size=(224, 224))\nimg_arr = image.img_to_array(img_sgl)\nimg_arr2 = np.expand_dims(img_arr, axis=0)\nimg_arr3 = resnet50.preprocess_input(np.repeat(img_arr2, 1, axis=0))\n\n# Load model\npredictor_host = tf.contrib.predictor.from_saved_model(SAVED_MODEL_DIR)\n\n# Run inference\nmodel_feed_dict={'input_1:0': img_arr3}\ninfa_rslts = predictor_host(model_feed_dict);\nprint(resnet50.decode_predictions(infa_rslts[output_tname], top=5)[0])\n" }, { "path": "src/examples/tensorflow/keras_resnet50/infer_resnet50_keras_loadtest.py", "content": "\"\"\" Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n\"\"\"\n\nimport shutil\nimport tensorflow as tf\nimport os\nimport time\nfrom concurrent import futures\nimport numpy as np\nimport statistics\nimport argparse\nimport requests\nimport tensorflow as tf\nimport tensorflow.neuron\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications import resnet50\nimport warnings\nimport subprocess\nimport json\n\ntf.keras.backend.set_image_data_format('channels_last')\n\narg_parser = argparse.ArgumentParser()\narg_parser.add_argument('--batch_size', type=int, default=5, choices=range(1, 6), help='Batch size of model as it was compiled')\narg_parser.add_argument('--neuroncore-pipeline-cores', type=int, default=1, choices=range(1, 17), help='Number of NeuronCores limit for each partitioned graph')\nargs = arg_parser.parse_args()\n\nneuron_ls_output = subprocess.run([\"neuron-ls\",\"-j\"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True, encoding=\"utf-8\")\nneuron_ls_json = json.loads(neuron_ls_output.stdout)\navail_neuroncores = neuron_ls_json[0][\"nc_count\"]\n\nUSER_BATCH_SIZE = 2 * args.batch_size\nNUM_LOOPS_PER_THREAD = 400\nCOMPILED_MODEL_DIR = \"./rn50_fp16_compiled_b\" + str(args.batch_size) + \"_nc\" + str(args.neuroncore_pipeline_cores) + \"/1\"\n\n# Ensure there's enough buffer capacity to hold in-flight requests in runtime\nNUM_INFERS_IN_FLIGHT = args.neuroncore_pipeline_cores + 3\nos.environ['NEURON_MAX_NUM_INFERS'] = str(NUM_INFERS_IN_FLIGHT)\n\nnum_groups = avail_neuroncores // args.neuroncore_pipeline_cores\ngroup_sizes = [str(args.neuroncore_pipeline_cores)] * num_groups\nwarnings.warn(\"NEURONCORE_GROUP_SIZES is being deprecated, if your application is using NEURONCORE_GROUP_SIZES please \\\nsee https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/deprecation.html#announcing-end-of-support-for-neuroncore-group-sizes \\\nfor more details.\", DeprecationWarning)\nos.environ['NEURONCORE_GROUP_SIZES'] = ','.join(group_sizes)\n\n# Create input from image\nimg_sgl = image.load_img('kitten_small.jpg', target_size=(224, 224))\nimg_arr = image.img_to_array(img_sgl, dtype='float16')\nimg_arr2 = np.expand_dims(img_arr, axis=0)\nimg_arr3 = np.repeat(img_arr2, USER_BATCH_SIZE, axis=0)\n\n# Load model\nNUM_THREADS_PER_PREDICTOR = args.neuroncore_pipeline_cores\npred_list = [tf.contrib.predictor.from_saved_model(COMPILED_MODEL_DIR) for _ in range(num_groups)]\npred_list = pred_list * NUM_THREADS_PER_PREDICTOR\nnum_threads = len(pred_list)\n\nnum_infer_per_thread = []\ntot_latency_per_thread = []\nthread_active = []\nlatency_list = []\nfor i in range(num_threads):\n num_infer_per_thread.append(0)\n tot_latency_per_thread.append(0)\n thread_active.append(0)\n\ndef one_thread(pred, model_feed_dict, index):\n global num_infer_per_thread\n thread_active[index] = 1\n for i in range(NUM_LOOPS_PER_THREAD):\n start = time.time()\n result = pred(model_feed_dict)\n delta = time.time() - start\n latency_list.append(delta)\n # skip first warmup run\n if i > 0:\n tot_latency_per_thread[index] += delta\n num_infer_per_thread[index] += USER_BATCH_SIZE\n #print(num_infer_per_thread[index])\n thread_active[index] = 0\n\ndef current_throughput():\n global num_infer_per_thread\n global args\n iteration = 0\n num_infer = 0\n last_num_infer = num_infer\n throughput_stats = []\n print(\"Run with {} NeuronCores\".format(avail_neuroncores))\n print(\"NEURON_MAX_NUM_INFERS (env): \" + os.environ.get('NEURON_MAX_NUM_INFERS', ''))\n print(\"NEURONCORE_GROUP_SIZES (env): \" + os.environ.get('NEURONCORE_GROUP_SIZES', ''))\n print(\"NUM THREADS: \", num_threads)\n print(\"NUM_LOOPS_PER_THREAD: \", NUM_LOOPS_PER_THREAD)\n print(\"USER_BATCH_SIZE: \", USER_BATCH_SIZE)\n while num_infer < NUM_LOOPS_PER_THREAD * USER_BATCH_SIZE * num_threads:\n num_infer = 0\n total_thread_cnt = 0\n for i in range(num_threads):\n num_infer = num_infer + num_infer_per_thread[i]\n total_thread_cnt = total_thread_cnt + thread_active[i]\n current_num_infer = num_infer\n throughput = current_num_infer - last_num_infer\n #print('Active threads: {}, current throughput: {} images/sec'.format(total_thread_cnt, throughput))\n # track throughput over time, after warmup\n if iteration > 4 and total_thread_cnt == num_threads:\n throughput_stats.append(throughput)\n last_num_infer = current_num_infer\n iteration += 1\n time.sleep(1.0)\n time.sleep(1.0)\n tot_latency = 0\n for i in range(num_threads):\n tot_latency += tot_latency_per_thread[i]\n # adjust loop count to remove the first warmup run\n print(\"Throughput values collected:\")\n print(throughput_stats)\n\n print(\"\\nCompiled batch size {:}, user batch size {:}, Throughput stats (images/sec): Avg={:0.0f} Max={:}, Latency stats (msec/user-batch): P50={:0.1f} P90={:0.1f} P95={:0.1f} P99={:0.1f} \\n\".format( args.batch_size, USER_BATCH_SIZE, np.mean(throughput_stats), np.max(throughput_stats), \n (np.percentile(latency_list, 50))*1000.0, (np.percentile(latency_list, 90))*1000.0, (np.percentile(latency_list, 95))*1000.0, (np.percentile(latency_list, 99))*1000.0)\n )\n\n\nprint(\"\\n*** Compiled batch size {}, user batch size {}, num NeuronCores {} (input shape: {}, saved model dir: {}) ***\\n\".format(args.batch_size, USER_BATCH_SIZE, args.neuroncore_pipeline_cores, img_arr3.shape, COMPILED_MODEL_DIR))\n\n# Run inference\nmodel_feed_dict={'input_1:0': img_arr3}\n\nexecutor = futures.ThreadPoolExecutor(max_workers = num_threads + 1)\nexecutor.submit(current_throughput)\nfor i,pred in enumerate(pred_list):\n executor.submit(one_thread, pred, model_feed_dict, i)\n" }, { "path": "src/examples/tensorflow/keras_resnet50/keras_resnet50.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"spectacular-payroll\",\n \"metadata\": {},\n \"source\": [\n \"# Tensorflow ResNet 50 Optimization Tutorial\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"equivalent-stack\",\n \"metadata\": {},\n \"source\": [\n \"## Note: this tutorial runs on tensorflow-neuron 1.x only\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"alpine-aside\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction: \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"In this tutorial we provide three main sections:\\n\",\n \"\\n\",\n \"* Take a Resnet 50 model and perform optimizations on it\\n\",\n \"\\n\",\n \"* Compile the model with different batch sizes and Neuroncore Group sizes (read about Neuroncore Group sizes here: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-runtime/nrt-theory-of-operation.html#neuron-core-group)\\n\",\n \"\\n\",\n \"* Run inference on our multiple compiled models to see which has the best throughput\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [Tensorflow Installation Guide](../../../../frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install.html#install-neuron-tensorflow). You can select the Kernel from the “Kernel -> Change Kernel” option on the top of this Jupyter notebook page.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"opened-forty\",\n \"metadata\": {},\n \"source\": [\n \"## Install Dependencies\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"meaningful-algebra\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install pillow requests # Necessary for loading images\\n\",\n \"!pip install tensorflow_neuron==1.15.5.2.8.9.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com/\\n\",\n \"!pip install neuron_cc==1.13.5.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"remarkable-exercise\",\n \"metadata\": {},\n \"source\": [\n \"## Compile\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"consecutive-right\",\n \"metadata\": {},\n \"source\": [\n \"The following example shows how to compile a FP16 ResNet50 network using various batching parameters to find the optimal solution. On inf1.6xlarge, run through the following steps to get a optimized Resnet 50 model.\\n\",\n \"First, extract Keras ResNet50 FP32 (resnet50_fp32_keras.pb will be generated):\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"vertical-finland\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import re\\n\",\n \"import argparse\\n\",\n \"import tensorflow as tf\\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"from tensorflow.keras.applications.resnet50 import ResNet50\\n\",\n \"from tensorflow.keras.preprocessing import image\\n\",\n \"from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions\\n\",\n \"\\n\",\n \"from google.protobuf import text_format\\n\",\n \"import tensorflow.python.saved_model\\n\",\n \"\\n\",\n \"# set Keras global configurations\\n\",\n \"tf.keras.backend.set_learning_phase(0)\\n\",\n \"tf.keras.backend.set_image_data_format('channels_last')\\n\",\n \"\\n\",\n \"float_type = 'float32'\\n\",\n \"float_type2 = 'fp32'\\n\",\n \"tf.keras.backend.set_floatx(float_type)\\n\",\n \"\\n\",\n \"# load pre-trained model using Keras\\n\",\n \"model_name = 'resnet50_%s_keras'%float_type2\\n\",\n \"model = ResNet50(weights='imagenet')\\n\",\n \"\\n\",\n \"# various save files\\n\",\n \"frozen_file = model_name + '.pb'\\n\",\n \"opt_file = model_name + '_opt.pb'\\n\",\n \"\\n\",\n \"# obtain parameters\\n\",\n \"model_input = model.input.name.replace(':0', '')\\n\",\n \"model_output = model.output.name.replace(':0', '')\\n\",\n \"batch, height, width, channels = model.input.shape\\n\",\n \"\\n\",\n \"print (\\\"model, frozen file, optimized file, input size, input node, output node,\\\")\\n\",\n \"print (\\\"%s, %s, %s, %dx%dx%d, %s, %s\\\" %(model_name, frozen_file, opt_file, width, height, channels, model_input, model_output) ) \\n\",\n \"\\n\",\n \"# obtain the TF session\\n\",\n \"sess = tf.compat.v1.keras.backend.get_session()\\n\",\n \"\\n\",\n \"# save checkpoint files for freeze_graph\\n\",\n \"ckpt_file = '/tmp/' + model_name + '/' + model_name + '.ckpt'\\n\",\n \"graph_file = '/tmp/' + model_name + '/' + model_name + '.pb'\\n\",\n \"tf.compat.v1.train.Saver().save(sess, ckpt_file)\\n\",\n \"tf.io.write_graph(sess.graph.as_graph_def(), logdir='.', name=graph_file, as_text=False)\\n\",\n \"\\n\",\n \"print(model_output)\\n\",\n \"with tf.compat.v1.Session(graph=tf.Graph()) as sess:\\n\",\n \" saver = tf.compat.v1.train.import_meta_graph(ckpt_file + '.meta')\\n\",\n \" saver.restore(sess, ckpt_file)\\n\",\n \" output_graph_def = tf.compat.v1.graph_util.convert_variables_to_constants(\\n\",\n \" sess, tf.compat.v1.get_default_graph().as_graph_def(), [model_output])\\n\",\n \" output_graph_def = tf.compat.v1.graph_util.remove_training_nodes(\\n\",\n \" output_graph_def, protected_nodes=[model_output])\\n\",\n \" with open(frozen_file, 'wb') as f:\\n\",\n \" f.write(output_graph_def.SerializeToString())\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"romance-cyprus\",\n \"metadata\": {},\n \"source\": [\n \"Optimize the extracted Keras ResNet50 FP32 graph for inference before casting (resnet50_fp32_keras_opt.pb will be generated) with the following transformations to the graph:\\n\",\n \"\\n\",\n \"* Remove Identity and CheckNumerics nodes\\n\",\n \"* Fold FusedBatchNorm constants into previous Conv2D weights\\n\",\n \"* Fold other constants\\n\",\n \"* Strip unused nodes\\n\",\n \"* Sort by execution order\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"higher-grant\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import copy\\n\",\n \"import string\\n\",\n \"\\n\",\n \"from google.protobuf import text_format\\n\",\n \"from tensorflow.core.framework import node_def_pb2\\n\",\n \"from tensorflow.core.framework import attr_value_pb2\\n\",\n \"from tensorflow.python.framework import tensor_util\\n\",\n \"from tensorflow.tools.graph_transforms import TransformGraph\\n\",\n \"\\n\",\n \"def clear_input(node):\\n\",\n \" for i in range(len(node.input)):\\n\",\n \" node.input.pop()\\n\",\n \"\\n\",\n \"def replace_name(node, name):\\n\",\n \" node.name = name\\n\",\n \" \\n\",\n \"def replace_input(node, input_name, new_name):\\n\",\n \" # node.input.replace(input_name, new_name)\\n\",\n \" temp = []\\n\",\n \" for i in node.input:\\n\",\n \" temp.extend([new_name if i == input_name else i])\\n\",\n \" clear_input(node)\\n\",\n \" for i in temp:\\n\",\n \" node.input.extend([i])\\n\",\n \"\\n\",\n \"def swap_names(node1, node2):\\n\",\n \" temp = node2.name\\n\",\n \" node2.name = node1.name\\n\",\n \" node1.name = temp\\n\",\n \"\\n\",\n \"def get_const_node(const_node_name, const_by_name):\\n\",\n \" name = re.sub(\\\"/read$\\\", \\\"\\\", const_node_name)\\n\",\n \" return const_by_name[name]\\n\",\n \"\\n\",\n \"def get_const_ndarray(const_node_name, const_by_name):\\n\",\n \" name = re.sub(\\\"/read$\\\", \\\"\\\", const_node_name)\\n\",\n \" node = const_by_name[name]\\n\",\n \" return tf.make_ndarray(node.attr.get(\\\"value\\\").tensor)\\n\",\n \"\\n\",\n \"def adjust_bias_values(bias_node, fbn_node, const_by_name):\\n\",\n \" bias_val = get_const_ndarray(bias_node.input[1], const_by_name) \\n\",\n \" gamma_val = get_const_ndarray(fbn_node.input[1], const_by_name) \\n\",\n \" mean_val = get_const_ndarray(fbn_node.input[3], const_by_name) \\n\",\n \" variance_val = get_const_ndarray(fbn_node.input[4], const_by_name) \\n\",\n \" new_bias = bias_val * gamma_val / np.sqrt(variance_val)\\n\",\n \" new_tensor = tensor_util.make_tensor_proto(new_bias, new_bias.dtype, new_bias.shape)\\n\",\n \" bias_const_node = get_const_node(bias_node.input[1], const_by_name)\\n\",\n \" bias_const_node.attr[\\\"value\\\"].CopyFrom(attr_value_pb2.AttrValue(tensor=new_tensor))\\n\",\n \"\\n\",\n \"def MoveBiasAddAfterFusedBatchNorm(graphdef):\\n\",\n \" \\\"\\\"\\\"fold_batch_norm function of TransformGraph is unable to fold Keras ResNet50\\n\",\n \" because of BiasAdd between Conv2D and FusedBatchNorm (BiasAdd is not needed\\n\",\n \" if FusedBatchNorm is used, but it exists in Keras ResNet50). Here, we \\n\",\n \" move BiasAdd to after FusedBatchNorm, and adjust bias value by gamma/sqrt(variance).\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" sess = tf.compat.v1.Session(graph=tf.import_graph_def(graphdef))\\n\",\n \" output_graph_def = tf.compat.v1.GraphDef()\\n\",\n \" node_by_name = {}\\n\",\n \" const_by_name = {}\\n\",\n \" for node in graphdef.node:\\n\",\n \" # Hack: use FusedBatchNormV2 so fold_batch_norm can recognize\\n\",\n \" if node.op == \\\"FusedBatchNormV3\\\":\\n\",\n \" node.op = \\\"FusedBatchNorm\\\"\\n\",\n \" del(node.attr[\\\"U\\\"])\\n\",\n \" #import pdb; pdb.set_trace()\\n\",\n \" copied_node = node_def_pb2.NodeDef()\\n\",\n \" copied_node.CopyFrom(node)\\n\",\n \" node_by_name[node.name] = copied_node\\n\",\n \" skip_add_node = False\\n\",\n \" # Switch Mul/BiasAdd in Keras RN50 so fold_batch_norm transform would work\\n\",\n \" if node.op == \\\"Const\\\":\\n\",\n \" const_by_name[node.name] = copied_node \\n\",\n \" elif node.op.startswith(\\\"FusedBatchNorm\\\"):\\n\",\n \" inputs = node.input\\n\",\n \" for i in inputs:\\n\",\n \" input_node = node_by_name[i]\\n\",\n \" if input_node.op == \\\"BiasAdd\\\":\\n\",\n \" output_graph_def.node.remove(input_node)\\n\",\n \" input_node_input0 = input_node.input[0]\\n\",\n \" # Adjust bias values (multiply by scale/sqrt(variance))\\n\",\n \" adjust_bias_values(input_node, node, const_by_name)\\n\",\n \" # Hack: swap names to avoid changing input of activation\\n\",\n \" swap_names(copied_node, input_node)\\n\",\n \" # Fix inputs for these two ops\\n\",\n \" replace_input(copied_node, i, input_node_input0)\\n\",\n \" replace_input(input_node, input_node_input0, copied_node.name)\\n\",\n \" # Fix order in node list\\n\",\n \" output_graph_def.node.extend([copied_node])\\n\",\n \" output_graph_def.node.extend([input_node])\\n\",\n \" skip_add_node = True\\n\",\n \" # Add maybe-modified nodes if not already done\\n\",\n \" if not skip_add_node:\\n\",\n \" output_graph_def.node.extend([copied_node])\\n\",\n \" return output_graph_def\\n\",\n \"\\n\",\n \"def FoldFusedBatchNorm(graph_def):\\n\",\n \" \\\"\\\"\\\"Optimize training graph for inference:\\n\",\n \" - Remove Identity and CheckNumerics nodes\\n\",\n \" - Fold FusedBatchNorm constants into previous Conv2D weights\\n\",\n \" - Fold other constants\\n\",\n \" - Strip unused nodes\\n\",\n \" - Sort by execution order\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" transformed_graph_def = TransformGraph (\\n\",\n \" graph_def,\\n\",\n \" ['input_1'],\\n\",\n \" ['probs/Softmax'],\\n\",\n \" [\\n\",\n \" 'add_default_attributes',\\n\",\n \" 'remove_nodes(op=Identity, op=CheckNumerics)',\\n\",\n \" 'fold_constants(ignore_errors=true)',\\n\",\n \" 'fold_batch_norms',\\n\",\n \" 'fold_old_batch_norms',\\n\",\n \" 'strip_unused_nodes',\\n\",\n \" 'sort_by_execution_order',\\n\",\n \" ])\\n\",\n \" return transformed_graph_def\\n\",\n \"\\n\",\n \"def load_graph(model_file):\\n\",\n \" graph_def = tf.compat.v1.GraphDef()\\n\",\n \"\\n\",\n \" with open(model_file, \\\"rb\\\") as f:\\n\",\n \" graph_def.ParseFromString(f.read())\\n\",\n \" return graph_def\\n\",\n \"\\n\",\n \"\\n\",\n \"graph_orig = load_graph('resnet50_fp32_keras.pb')\\n\",\n \"graph_mod = MoveBiasAddAfterFusedBatchNorm(graph_orig)\\n\",\n \"graph_mod2 = FoldFusedBatchNorm(graph_mod)\\n\",\n \"with tf.io.gfile.GFile('resnet50_fp32_keras_opt.pb', \\\"wb\\\") as f:\\n\",\n \" f.write(graph_mod2.SerializeToString())\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"corresponding-acquisition\",\n \"metadata\": {},\n \"source\": [\n \"Convert full graph to FP16 (resnet50_fp16_keras_opt.pb will be generated.\\n\",\n \"This will take about a minute.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"detected-training\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from tensorflow.core.framework import graph_pb2\\n\",\n \"from tensorflow.python.platform import gfile\\n\",\n \"\\n\",\n \"def ConvertFP32ToOther(graphdef):\\n\",\n \" \\\"\\\"\\\"Converts an FP32 network by casting all constants (weights) to a lower\\n\",\n \" precision floating point type (FP16) and updating the dtypes\\n\",\n \" everywhere.\\\"\\\"\\\"\\n\",\n \" cast_type = \\\"float16\\\"\\n\",\n \" sess = tf.Session(graph=tf.import_graph_def(graphdef))\\n\",\n \" output_graph_def = graph_pb2.GraphDef()\\n\",\n \" dummy_tensor = sess.run(tf.constant([0.1]))\\n\",\n \" dummy_tensor_proto = tensor_util.make_tensor_proto(dummy_tensor, \\\\\\n\",\n \" dtype=cast_type, shape=dummy_tensor.shape)\\n\",\n \" dummy_tensor32 = sess.run(tf.constant([0.1]))\\n\",\n \" dummy_tensor_proto32 = tensor_util.make_tensor_proto(dummy_tensor, \\\\\\n\",\n \" dtype=tf.float32, shape=dummy_tensor.shape)\\n\",\n \" dt_float_type_attr = attr_value_pb2.AttrValue(type=dummy_tensor_proto32.dtype)\\n\",\n \" dt_half_type_attr = attr_value_pb2.AttrValue(type=dummy_tensor_proto.dtype)\\n\",\n \" for node in graphdef.node:\\n\",\n \" output_node = node_def_pb2.NodeDef()\\n\",\n \" output_node.CopyFrom(node)\\n\",\n \" if (node.op == \\\"Const\\\"):\\n\",\n \" if (node.attr[\\\"dtype\\\"] == dt_float_type_attr):\\n\",\n \" a = tensor_util.MakeNdarray(node.attr[\\\"value\\\"].tensor)\\n\",\n \" a = tf.cast(a, cast_type)\\n\",\n \" a = sess.run(a)\\n\",\n \" output_node.attr[\\\"dtype\\\"].CopyFrom(dt_half_type_attr)\\n\",\n \" output_node.attr[\\\"value\\\"].CopyFrom(\\n\",\n \" attr_value_pb2.AttrValue(\\n\",\n \" tensor=tensor_util.make_tensor_proto(a,\\\\\\n\",\n \" dtype=cast_type, shape=a.shape)))\\n\",\n \" else:\\n\",\n \" if (\\\"T\\\" in node.attr.keys()):\\n\",\n \" if (output_node.attr[\\\"T\\\"] == dt_float_type_attr):\\n\",\n \" output_node.attr[\\\"T\\\"].CopyFrom(dt_half_type_attr)\\n\",\n \" if (\\\"Tparams\\\" in node.attr.keys()):\\n\",\n \" if (output_node.attr[\\\"Tparams\\\"] == dt_float_type_attr):\\n\",\n \" output_node.attr[\\\"Tparams\\\"].CopyFrom(dt_half_type_attr)\\n\",\n \" if (\\\"dtype\\\" in node.attr.keys()):\\n\",\n \" if (node.attr[\\\"dtype\\\"] == dt_float_type_attr):\\n\",\n \" output_node.attr[\\\"dtype\\\"].CopyFrom(dt_half_type_attr)\\n\",\n \" if (\\\"SrcT\\\" in node.attr.keys()):\\n\",\n \" if (node.attr[\\\"SrcT\\\"] == dt_float_type_attr):\\n\",\n \" output_node.attr[\\\"SrcT\\\"].CopyFrom(dt_half_type_attr)\\n\",\n \" if (\\\"DstT\\\" in node.attr.keys()):\\n\",\n \" if (node.attr[\\\"DstT\\\"] == dt_float_type_attr):\\n\",\n \" output_node.attr[\\\"DstT\\\"].CopyFrom(dt_half_type_attr)\\n\",\n \" output_graph_def.node.extend([output_node])\\n\",\n \" return output_graph_def\\n\",\n \"\\n\",\n \"def load_graph(model_file):\\n\",\n \" graph_def = tf.GraphDef()\\n\",\n \"\\n\",\n \" with open(model_file, \\\"rb\\\") as f:\\n\",\n \" graph_def.ParseFromString(f.read())\\n\",\n \"\\n\",\n \" return graph_def\\n\",\n \"\\n\",\n \"graph_f32 = load_graph('resnet50_fp32_keras_opt.pb')\\n\",\n \"graph_f16 = ConvertFP32ToOther(graph_f32)\\n\",\n \"output_xformed_graph_name = 'resnet50_fp16_keras_opt.pb'\\n\",\n \"with gfile.GFile(output_xformed_graph_name, \\\"wb\\\") as f:\\n\",\n \" f.write(graph_f16.SerializeToString())\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"correct-travel\",\n \"metadata\": {},\n \"source\": [\n \"Run the compilation script to sweep through various batch sizes up to 5 and several NeuronCore Group sizes up to 16. The script calls the compilation script pb2sm_compile.py which tries to perform compilation. Some error messages are expected due to known issues (see Known Issues section in the tutorial). If you run all the configurations it will take about 45 minutes.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"shared-ratio\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%bash\\n\",\n \"#!/usr/bin/env bash\\n\",\n \"\\n\",\n \"echo \\\"\\\" > full_sweep.log\\n\",\n \"echo \\\"\\\" > full_sweep_results.txt\\n\",\n \"\\n\",\n \"results=()\\n\",\n \"for b in $(seq 1 5); do \\n\",\n \" for i in 1 2 4 8 12 16; do \\n\",\n \" python pb2sm_compile.py --batch_size=$b --neuroncore-pipeline-cores=$i | tee -a full_sweep.log;\\n\",\n \" results[$b]+=\\\", \\\"`tail -1 full_sweep.log`\\n\",\n \" done\\n\",\n \"done\\n\",\n \"\\n\",\n \"head=\\\"batch\\\"\\n\",\n \"for i in 1 2 4 8 12 16; do\\n\",\n \" head+=\\\", nc${i}\\\"\\n\",\n \"done \\n\",\n \"echo $head | tee -a full_sweep_results.txt\\n\",\n \"for b in $(seq 1 5); do \\n\",\n \" echo $b${results[$b]} | tee -a full_sweep_results.txt\\n\",\n \"done\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"attached-austin\",\n \"metadata\": {},\n \"source\": [\n \"You should see some output like this:\\n\",\n \"```\\n\",\n \"INFO: Compilation finished in 95 seconds with 99.5% operations placed on Inferentia\\n\",\n \"\\n\",\n \"1\\n\",\n \"\\n\",\n \"*** Batch size 1, num NeuronCores 2 (input shape: (1, 224, 224, 3), saved model dir: rn50_fp16_compiled_b1_nc2) ***\\n\",\n \"\\n\",\n \"INFO: Compilation finished in 95 seconds with 99.5% operations placed on Inferentia\\n\",\n \"\\n\",\n \"1\\n\",\n \"\\n\",\n \"*** Batch size 1, num NeuronCores 4 (input shape: (1, 224, 224, 3), saved model dir: rn50_fp16_compiled_b1_nc4) ***\\n\",\n \"\\n\",\n \"INFO: Compilation finished in 95 seconds with 99.5% operations placed on Inferentia\\n\",\n \"\\n\",\n \"1\\n\",\n \"\\n\",\n \"... (outputs removed)\\n\",\n \"\\n\",\n \"*** Batch size 5, num NeuronCores 16 (input shape: (5, 224, 224, 3), saved model dir: rn50_fp16_compiled_b5_nc16) ***\\n\",\n \"\\n\",\n \"ERROR: Compilation finished in 120 seconds with less than 50% operations placed on Inferentia (0.0%)\\n\",\n \"\\n\",\n \"INFO: Retry compilation without static weights\\n\",\n \"\\n\",\n \"ERROR: Retry compilation finished in 137 seconds with less than 50% operations placed on Inferentia (0.0%)\\n\",\n \"\\n\",\n \"0\\n\",\n \"\\n\",\n \"The file full_sweep_results.txt shows a summary of the sweep results with latest Neuron 1/27/20 release (0 means compilation unsuccessful and 0 ops mapped to Inferentia, 1 means most ops mapped to Inferentia and non-static weights, 2 means most ops mapped to Inferentia and using static weights):\\n\",\n \"\\n\",\n \"batch, nc1, nc2, nc4, nc8, nc12, nc16\\n\",\n \"1, 1, 1, 1, 2, 2, 2\\n\",\n \"2, 1, 1, 0, 1, 2, 2\\n\",\n \"3, 1, 1, 1, 1, 1, 1\\n\",\n \"4, 1, 1, 0, 1, 1, 1\\n\",\n \"5, 1, 1, 0, 0, 0, 0\\n\",\n \"```\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"surprised-abortion\",\n \"metadata\": {},\n \"source\": [\n \"## Inference\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"departmental-surprise\",\n \"metadata\": {},\n \"source\": [\n \"Run inference over different batch sizes and Neuroncore groups to obtain throughput and latency results for ResNet50. To apply dynamic batching, the user batch size is set to 10x the compiled batch size, in order to keep input queue full and to amortize framework-to-Neuron overhead.\\n\",\n \"\\n\",\n \"Note: The results are based on the Neuron v1.12.2 (Mar 4th 2021) release. These will continue improve as we increase Neuron performance.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"requested-inspiration\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!cd ~/aws-neuron-sdk/src/examples/tensorflow/keras_resnet50/\\n\",\n \"!echo \\\"\\\" > batch.log\\n\",\n \"!for i in $(seq 1 5); do python infer_resnet50_keras_loadtest.py --batch_size=$i --neuroncore-pipeline-cores=1 | tee -a batch.log; done\\n\",\n \"!for i in $(seq 1 5); do python infer_resnet50_keras_loadtest.py --batch_size=$i --neuroncore-pipeline-cores=2 | tee -a batch.log; done\\n\",\n \"!for i in $(seq 1 5); do python infer_resnet50_keras_loadtest.py --batch_size=$i --neuroncore-pipeline-cores=4 | tee -a batch.log; done\\n\",\n \"!for i in $(seq 1 5); do python infer_resnet50_keras_loadtest.py --batch_size=$i --neuroncore-pipeline-cores=8 | tee -a batch.log; done\\n\",\n \"!for i in $(seq 1 5); do python infer_resnet50_keras_loadtest.py --batch_size=$i --neuroncore-pipeline-cores=12 | tee -a batch.log; done\\n\",\n \"!for i in $(seq 1 5); do python infer_resnet50_keras_loadtest.py --batch_size=$i --neuroncore-pipeline-cores=16 | tee -a batch.log; done\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"split-genesis\",\n \"metadata\": {},\n \"source\": [\n \"The file batch.log now contains the results for each batch size. We can look at the throughput values to get an idea of which models are performing well. The output should look something like this:\\n\",\n \"\\n\",\n \"The model best model configuration for throughput (if you run on an Inf1.6xlarge as suggested in the tutorial) is batch size 5 NeuronCore group size 2. Increasing batch size usually helps to increase throughput (up to a certain extent).\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"filled-township\",\n \"metadata\": {},\n \"source\": [\n \"```\\n\",\n \"*** Compiled batch size 5, user batch size 10, num NeuronCores 2 (input shape: (10, 224, 224, 3), saved model dir: ./rn50_fp16_compiled_b5_nc2/1) ***\\n\",\n \"\\n\",\n \"Instance type inf1.6xlarge with 16 NeuronCores\\n\",\n \"NEURON_MAX_NUM_INFERS (env): 5\\n\",\n \"NEURONCORE_GROUP_SIZES (env): 2,2,2,2,2,2,2,2\\n\",\n \"NUM THREADS: 16\\n\",\n \"NUM_LOOPS_PER_THREAD: 400\\n\",\n \"USER_BATCH_SIZE: 10\\n\",\n \"Throughput values collected:\\n\",\n \"[10680, 10700, 10660]\\n\",\n \"\\n\",\n \"(rest of outputs removed)\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"189c4f0e-1a4e-4067-921f-95449c45dedd\",\n \"metadata\": {},\n \"source\": [\n \"## Known Issues\\n\",\n \"\\n\",\n \"### Unable to compile with batch and num NeuronCores combination\\n\",\n \"\\n\",\n \"For some combination of batch and number of NeuronCores setting, you may\\n\",\n \"see an internal compiler error as below. Please see the sweep result\\n\",\n \"above for Neuron 1/27/20 release. Furthermore, if using auto-casting to\\n\",\n \"bfloat16 from FP32 network and batch size is larger than 1 would result\\n\",\n \"in the same error.\\n\",\n \"\\n\",\n \"\\n\",\n \"```bash\\n\",\n \"\\n\",\n \"INFO:tensorflow:fusing subgraph neuron_op_a73aed4b95ca5d5b with neuron-cc; log file is at /home/ubuntu/keras_fp16_benchmarking_db/compiler_workdir/neuron_op_a73aed4b95ca5d5b/graph_def.neuron-cc.log\\n\",\n \" WARNING:tensorflow:Failed to fuse subgraph neuron_op_a73aed4b95ca5d5b with '/home/ubuntu/test_venv/bin/neuron-cc compile /home/ubuntu/keras_fp16_benchmarking_db/compiler_workdir/neuron_op_a73aed4b95ca5d5b/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /home/ubuntu/keras_fp16_benchmarking_db/compiler_workdir/neuron_op_a73aed4b95ca5d5b/graph_def.neff --io-config \\\"{\\\\\\\"inputs\\\\\\\": {\\\\\\\"input_10/_0:0\\\\\\\": [[6, 224, 224, 3], \\\\\\\"float16\\\\\\\"]}, \\\\\\\"outputs\\\\\\\": [\\\\\\\"probs/Softmax:0\\\\\\\"]}\\\" --batching_en --rematerialization_en --sb_size 120 --spill_dis --enable-replication True'\\n\",\n \" WARNING:tensorflow:neuron-cc error message:\\n\",\n \" WARNING:tensorflow:01/23/2020 01:15:40 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: ***************************************************************\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: An Internal Compiler Error has occurred\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: ***************************************************************\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Please contact Customer Support and provide the following details.\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Error message: Non-zero exit status (134) for command: /home/ubuntu/test_venv/lib/python3.6/site-packages/neuroncc/starfish/bin/list_sch --hhir hh-tr-external-move.json --verbose 0 --sb_size 120 --arith_intensity_target 2300 --sb_watermark_low 0.250000 --sb_watermark_high 0.750000 --sb_size_tol 1 --alloc simple1 --alloc_opt --depth_diff 0.100000 --verbose_start_cycle 0 --tt_dist --mm_meet_cnt 1 --load_speed_factor 0.300000 --schir sch_tmp.json --spill_depth_limit 5 --spill_dis --true_dep --mm_order --batching_en --rematerialization_en\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Error class: CompilerInternalError\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Error location: job.Scheduler.3\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Command line: /home/ubuntu/test_venv/bin/neuron-cc compile /home/ubuntu/keras_fp16_benchmarking_db/compiler_workdir/neuron_op_a73aed4b95ca5d5b/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /home/ubuntu/keras_fp16_benchmarking_db/compiler_workdir/neuron_op_a73aed4b95ca5d5b/graph_def.neff --io-config '{\\\"inputs\\\": {\\\"input_10/_0:0\\\": [[6, 224, 224, 3], \\\"float16\\\"]}, \\\"outputs\\\": [\\\"probs/Softmax:0\\\"]}' --batching_en --rematerialization_en --sb_size 120 --spill_dis --enable-replication True\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Internal details:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: File \\\"neuroncc/driver/Job.py\\\", line 207, in neuroncc.driver.Job.runSingleInputFn\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: File \\\"neuroncc/driver/jobs/Scheduler.py\\\", line 58, in neuroncc.driver.jobs.Scheduler.Scheduler.runSingleInput\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: File \\\"neuroncc/driver/Job.py\\\", line 145, in neuroncc.driver.Job.Job.shellCommand\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:40 AM ERROR [neuron-cc]: Version information:\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: Neuron Compiler version 1.0.6632.0+6001610955\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]:\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: HWM version 1.0.839.0-6001300654\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: NEFF version 0.6\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: TVM version 1.0.1589.0+6001610955\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: NumPy version 1.16.5\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: MXNet not available\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]: TF version 1.15.0\\n\",\n \" 01/23/2020 01:15:41 AM ERROR [neuron-cc]:\\n\",\n \"\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"gentle-census\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/tensorflow/keras_resnet50/optimize_for_inference.py", "content": "\"\"\" Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n\"\"\"\n\nimport re\nimport copy\nimport argparse\nimport tensorflow as tf\nimport numpy as np\nimport string\n\nfrom google.protobuf import text_format\nfrom tensorflow.core.framework import node_def_pb2\nfrom tensorflow.core.framework import attr_value_pb2\nfrom tensorflow.python.framework import tensor_util\nfrom tensorflow.tools.graph_transforms import TransformGraph\n\ndef clear_input(node):\n for i in range(len(node.input)):\n node.input.pop()\n\ndef replace_name(node, name):\n node.name = name\n \ndef replace_input(node, input_name, new_name):\n # node.input.replace(input_name, new_name)\n temp = []\n for i in node.input:\n temp.extend([new_name if i == input_name else i])\n clear_input(node)\n for i in temp:\n node.input.extend([i])\n\ndef swap_names(node1, node2):\n temp = node2.name\n node2.name = node1.name\n node1.name = temp\n\ndef get_const_node(const_node_name, const_by_name):\n name = re.sub(\"/read$\", \"\", const_node_name)\n return const_by_name[name]\n\ndef get_const_ndarray(const_node_name, const_by_name):\n name = re.sub(\"/read$\", \"\", const_node_name)\n node = const_by_name[name]\n return tf.make_ndarray(node.attr.get(\"value\").tensor)\n\ndef adjust_bias_values(bias_node, fbn_node, const_by_name):\n bias_val = get_const_ndarray(bias_node.input[1], const_by_name) \n gamma_val = get_const_ndarray(fbn_node.input[1], const_by_name) \n mean_val = get_const_ndarray(fbn_node.input[3], const_by_name) \n variance_val = get_const_ndarray(fbn_node.input[4], const_by_name) \n new_bias = bias_val * gamma_val / np.sqrt(variance_val)\n new_tensor = tensor_util.make_tensor_proto(new_bias, new_bias.dtype, new_bias.shape)\n bias_const_node = get_const_node(bias_node.input[1], const_by_name)\n bias_const_node.attr[\"value\"].CopyFrom(attr_value_pb2.AttrValue(tensor=new_tensor))\n\ndef MoveBiasAddAfterFusedBatchNorm(graphdef):\n \"\"\"fold_batch_norm function of TransformGraph is unable to fold Keras ResNet50\n because of BiasAdd between Conv2D and FusedBatchNorm (BiasAdd is not needed\n if FusedBatchNorm is used, but it exists in Keras ResNet50). Here, we \n move BiasAdd to after FusedBatchNorm, and adjust bias value by gamma/sqrt(variance).\n \"\"\"\n sess = tf.compat.v1.Session(graph=tf.import_graph_def(graphdef))\n output_graph_def = tf.compat.v1.GraphDef()\n node_by_name = {}\n const_by_name = {}\n for node in graphdef.node:\n # Hack: use FusedBatchNormV2 so fold_batch_norm can recognize\n if node.op == \"FusedBatchNormV3\":\n node.op = \"FusedBatchNorm\"\n del(node.attr[\"U\"])\n #import pdb; pdb.set_trace()\n copied_node = node_def_pb2.NodeDef()\n copied_node.CopyFrom(node)\n node_by_name[node.name] = copied_node\n skip_add_node = False\n # Switch Mul/BiasAdd in Keras RN50 so fold_batch_norm transform would work\n if node.op == \"Const\":\n const_by_name[node.name] = copied_node \n elif node.op.startswith(\"FusedBatchNorm\"):\n inputs = node.input\n for i in inputs:\n input_node = node_by_name[i]\n if input_node.op == \"BiasAdd\":\n output_graph_def.node.remove(input_node)\n input_node_input0 = input_node.input[0]\n # Adjust bias values (multiply by scale/sqrt(variance))\n adjust_bias_values(input_node, node, const_by_name)\n # Hack: swap names to avoid changing input of activation\n swap_names(copied_node, input_node)\n # Fix inputs for these two ops\n replace_input(copied_node, i, input_node_input0)\n replace_input(input_node, input_node_input0, copied_node.name)\n # Fix order in node list\n output_graph_def.node.extend([copied_node])\n output_graph_def.node.extend([input_node])\n skip_add_node = True\n # Add maybe-modified nodes if not already done\n if not skip_add_node:\n output_graph_def.node.extend([copied_node])\n return output_graph_def\n\ndef FoldFusedBatchNorm(graph_def):\n \"\"\"Optimize training graph for inference:\n - Remove Identity and CheckNumerics nodes\n - Fold FusedBatchNorm constants into previous Conv2D weights\n - Fold other constants\n - Strip unused nodes\n - Sort by execution order\n \"\"\"\n transformed_graph_def = TransformGraph (\n graph_def,\n ['input_1'],\n ['probs/Softmax'],\n [\n 'add_default_attributes',\n 'remove_nodes(op=Identity, op=CheckNumerics)',\n 'fold_constants(ignore_errors=true)',\n 'fold_batch_norms',\n 'fold_old_batch_norms',\n 'strip_unused_nodes',\n 'sort_by_execution_order',\n ])\n return transformed_graph_def\n\ndef load_graph(model_file):\n graph_def = tf.compat.v1.GraphDef()\n\n with open(model_file, \"rb\") as f:\n graph_def.ParseFromString(f.read())\n return graph_def\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--graph\", help=\"graph/model to be executed\",\n required=True)\n parser.add_argument(\"--out_graph\", help=\"graph/model to be generated\",\n required=True)\n args = parser.parse_args()\n\n graph_orig = load_graph(args.graph)\n graph_mod = MoveBiasAddAfterFusedBatchNorm(graph_orig)\n graph_mod2 = FoldFusedBatchNorm(graph_mod)\n with tf.io.gfile.GFile(args.out_graph, \"wb\") as f:\n f.write(graph_mod2.SerializeToString())\n #with tf.io.gfile.GFile(args.out_graph + \"txt\", 'w') as f:\n # f.write(text_format.MessageToString(graph_mod2))\n" }, { "path": "src/examples/tensorflow/keras_resnet50/pb2sm_compile.py", "content": "\"\"\" Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n SPDX-License-Identifier: MIT-0\n\"\"\"\n\nimport time\nimport shutil\nimport numpy as np\nimport argparse\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications import resnet50\nimport tensorflow.neuron as tfn\n\ntf.keras.backend.set_image_data_format('channels_last')\n\narg_parser = argparse.ArgumentParser()\narg_parser.add_argument('--batch_size', type=int, default=5, choices=range(1, 6), help='Input data batch size for compilation of model')\narg_parser.add_argument('--neuroncore-pipeline-cores', type=int, default=1, choices=range(1, 17), help='Number of NeuronCores limit for each partitioned graph')\narg_parser.add_argument('--debug_args', type=str, default=\"\", help='Optional Compiler debug args')\narg_parser.add_argument('--workdir', type=str, default=\"compiler_workdir\", help='Compiler work directory')\n\nargs = arg_parser.parse_args()\n\ndef pb_to_saved_model(pb_path, input_names, output_names, model_dir):\n graph_def = tf.GraphDef()\n graph_def.ParseFromString(open(pb_path, 'rb').read())\n with tf.Session(graph=tf.Graph()) as sess:\n tf.import_graph_def(graph_def, name='')\n inputs = {name: sess.graph.get_tensor_by_name(ts_name) for name, ts_name in input_names.items()}\n outputs = {name: sess.graph.get_tensor_by_name(ts_name) for name, ts_name in output_names.items()}\n tf.saved_model.simple_save(sess, model_dir, inputs, outputs)\n\nsaved_model_dir = \"rn50_fp16\"\n\nshutil.rmtree(saved_model_dir, ignore_errors=True)\n\npb_to_saved_model(\"resnet50_fp16_keras_opt.pb\", {\"input_1:0\": \"input_1:0\"}, {\"probs/Softmax:0\" : \"probs/Softmax:0\"}, saved_model_dir)\n\nbatch_size = args.batch_size\nimg_arr = np.zeros([batch_size, 224, 224, 3], dtype='float16')\ncompiled_saved_model_dir = saved_model_dir + \"_compiled_b\" + str(batch_size) + \"_nc\" + str(args.neuroncore_pipeline_cores)\nshutil.rmtree(compiled_saved_model_dir + \"/1\", ignore_errors=True)\n\nprint(\"\\n*** Batch size {}, num NeuronCores {} (input shape: {}, saved model dir: {}) ***\\n\".format(batch_size, args.neuroncore_pipeline_cores, img_arr.shape, compiled_saved_model_dir))\ncompiler_args = ['--neuroncore-pipeline-cores', str(args.neuroncore_pipeline_cores)]\nif args.debug_args:\n compiler_args.extend(args.debug_args.split(\" \"))\n\nstatic_weights = False\nif args.neuroncore_pipeline_cores >= 8:\n static_weights = True\n\nshutil.rmtree(args.workdir, ignore_errors=True)\nstart = time.time()\nrslts = tfn.saved_model.compile(saved_model_dir, compiled_saved_model_dir + \"/1\",\n model_feed_dict={'input_1:0' : img_arr},\n compiler_workdir=args.workdir,\n dynamic_batch_size=True,\n compiler_args = compiler_args)\ndelta = time.time() - start\nperc_on_inf = rslts['OnNeuronRatio'] * 100\n\ncompile_success = False\nif perc_on_inf < 50:\n print(\"\\nERROR: Compilation finished in {:.0f} seconds with less than 50% operations placed on Inferentia ({:.1f}%)\\n\".format(delta, perc_on_inf))\n if '--static-weights' in compiler_args:\n print(\"INFO: Retry compilation without static weights\")\n compiler_args.remove('--static-weights')\n static_weights = False\n shutil.rmtree(compiled_saved_model_dir + \"/1\", ignore_errors=True)\n shutil.rmtree('compiler_workdir2', ignore_errors=True)\n start = time.time()\n rslts = tfn.saved_model.compile(saved_model_dir, compiled_saved_model_dir + \"/1\",\n model_feed_dict={'input_1:0' : img_arr},\n compiler_workdir='compiler_workdir2',\n dynamic_batch_size=True,\n compiler_args = compiler_args)\n delta = time.time() - start\n perc_on_inf = rslts['OnNeuronRatio'] * 100\n if perc_on_inf < 50:\n print(\"\\nERROR: Retry compilation finished in {:.0f} seconds with less than 50% operations placed on Inferentia ({:.1f}%)\\n\".format(delta, perc_on_inf))\n else:\n print(\"\\nINFO: Retry compilation finished in {:.0f} seconds with {:.1f}% operations placed on Inferentia\\n\".format(delta, perc_on_inf))\n compile_success = True\nelse:\n print(\"\\nINFO: Compilation finished in {:.0f} seconds with {:.1f}% operations placed on Inferentia\\n\".format(delta, perc_on_inf))\n compile_success = True\n\n# Prepare SavedModel for uploading to Inf1 instance\ncompletion_code = 0\nif compile_success:\n shutil.make_archive('./' + compiled_saved_model_dir, 'zip', './', compiled_saved_model_dir)\n completion_code = 1 + int(static_weights)\n\nprint(completion_code)\n\nexit(int(not compile_success))\n" }, { "path": "src/examples/tensorflow/keras_resnet50/run_all", "content": "#!/usr/bin/env bash\n\n##########################################################################\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n##########################################################################\n\npip install pillow\n\n# Extract Keras ResNet50 FP32 and check inference\npython gen_resnet50_keras.py\npython infer_resnet50_keras.py --graph resnet50_fp32_keras.pb\n# Optimize fp32 graph for inference before casting\npython optimize_for_inference.py --graph resnet50_fp32_keras.pb --out_graph resnet50_fp32_keras_opt.pb\npython infer_resnet50_keras.py --graph resnet50_fp32_keras_opt.pb\n# Cast full graph to FP16\npython fp32tofp16.py --graph resnet50_fp32_keras_opt.pb --out_graph resnet50_fp16_keras_opt.pb\npython infer_resnet50_keras.py --graph resnet50_fp16_keras_opt.pb\n# Compile\npython pb2sm_compile.py\n# Infer\npython infer_resnet50_keras_loadtest.py\n\n" }, { "path": "src/examples/tensorflow/openpose_demo/openpose.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"caff04ba\",\n \"metadata\": {},\n \"source\": [\n \"# Running OpenPose on Inferentia\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"09b2919a\",\n \"metadata\": {},\n \"source\": [\n \"## Note: this tutorial runs on tensorflow-neuron 1.x only\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"4dcf9bb1\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction:\\n\",\n \"\\n\",\n \"In this tutorial we will compile and deploy Openpose model for Inferentia. This jupyter notebook should run on an inf1.6xlarge instance for compilation and inference. The inference part of this tutorial requires inf1.6xlarge and not the compilation itself. For simplicity we will run this tutorial on a single instance but in real life scenario the compilation can be done on a compute c5.4xlarge instance and the deployment on the inf1 instance family.\\n\",\n \"\\n\",\n \"In this tutorial we provide two main sections:\\n\",\n \"1. Compile the OpenPose model on inf1x6large.\\n\",\n \"2. Infer the same compiled model on inf1x6large.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [Tensorflow Installation Guide](../../../../frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install.html#install-neuron-tensorflow). You can select the Kernel from the “Kernel -> Change Kernel” option on the top of this Jupyter notebook page.\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"04ae0838\",\n \"metadata\": {},\n \"source\": [\n \"## Acknowledgement:\\n\",\n \"\\n\",\n \"Many thanks to https://github.com/ildoonet for providing pretrained model as well as the image preprocessing/pose estimating infrastructure.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"d0d6d08e\",\n \"metadata\": {},\n \"source\": [\n \"## Download tensorflow pose net frozen graph.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"1926d4e3\",\n \"metadata\": {\n \"scrolled\": false\n },\n \"outputs\": [],\n \"source\": [\n \"!wget -c --tries=2 $( wget -q -O - http://www.mediafire.com/file/qlzzr20mpocnpa3/graph_opt.pb | grep -o 'http*://download[^\\\"]*' | tail -n 1 ) -O graph_opt.pb\\n\",\n \"\\n\",\n \"!pip install tensorflow_neuron==1.15.5.2.8.9.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com/\\n\",\n \"!pip install neuron_cc==1.13.5.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"83eb578b\",\n \"metadata\": {},\n \"source\": [\n \"## Compile\\n\",\n \"Compile the pose net frozen graph into AWS Neuron compatible form. Network input image resolution is adjustable with argument --net_resolution (e. g., --net_resolution=656x368). The compiled model can accept arbitrary batch size input at runtime.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"362f322e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"\\\"\\\"\\\"\\n\",\n \"Usage: python convert_graph_opt.py /path/to/graph_opt.pb /path/to/graph_opt_neuron.pb\\n\",\n \"\\\"\\\"\\\"\\n\",\n \"#import argparse\\n\",\n \"import numpy as np\\n\",\n \"import tensorflow as tf\\n\",\n \"from tensorflow.core.framework.tensor_shape_pb2 import TensorShapeProto\\n\",\n \"import tensorflow.neuron as tfn\\n\",\n \"\\n\",\n \"\\n\",\n \"def compile():\\n\",\n \" #parser = argparse.ArgumentParser()\\n\",\n \" #parser.add_argument('input_pb_path', help='Input serialized GraphDef protobuf')\\n\",\n \" #parser.add_argument('output_pb_path', help='Ouput serialized GraphDef protobuf')\\n\",\n \" #parser.add_argument('--net_resolution', default='656x368', help='Network resolution in WxH format, e. g., --net_resolution=656x368')\\n\",\n \" #parser.add_argument('--debug_verify', action='store_true')\\n\",\n \" #args = parser.parse_args()\\n\",\n \" \\n\",\n \" input_pb_path = './graph_opt.pb'\\n\",\n \" net_resolution = '656x368'\\n\",\n \" output_pb_path = './graph_opt_neuron_' + net_resolution + '.pb'\\n\",\n \" \\n\",\n \" debug_verify = 'store_true'\\n\",\n \" dim_w, dim_h = net_resolution.split('x')\\n\",\n \" dim_w = int(dim_w)\\n\",\n \" dim_h = int(dim_h)\\n\",\n \" graph_def = tf.GraphDef()\\n\",\n \" with open(input_pb_path, 'rb') as f:\\n\",\n \" graph_def.ParseFromString(f.read())\\n\",\n \"\\n\",\n \" if debug_verify:\\n\",\n \" np.random.seed(0)\\n\",\n \" feed_dict = {'image:0': np.random.rand(1, dim_h, dim_w, 3)}\\n\",\n \" output_name = 'Openpose/concat_stage7:0'\\n\",\n \" with tf.Session(graph=tf.Graph()) as sess:\\n\",\n \" tf.import_graph_def(graph_def, name='')\\n\",\n \" result_reference = sess.run(output_name, feed_dict)\\n\",\n \"\\n\",\n \" preprocessing_ops = {'preprocess_divide', 'preprocess_divide/y', 'preprocess_subtract', 'preprocess_subtract/y'}\\n\",\n \" graph_def = nhwc_to_nchw(graph_def, preprocessing_ops)\\n\",\n \" graph_def = inline_float32_to_float16(graph_def, preprocessing_ops)\\n\",\n \" with tf.Session(graph=tf.Graph()) as sess:\\n\",\n \" tf.import_graph_def(graph_def, name='')\\n\",\n \" no_fuse_ops = preprocessing_ops.union({'Openpose/concat_stage7'})\\n\",\n \" infer_graph = tfn.graph_util.inference_graph_from_session(\\n\",\n \" sess, shape_feed_dict={'image:0': [1, dim_h, dim_w, 3]}, output_tensors=['Openpose/concat_stage7:0'],\\n\",\n \" no_fuse_ops=no_fuse_ops, dynamic_batch_size=True,\\n\",\n \" )\\n\",\n \" with open(output_pb_path, 'wb') as f:\\n\",\n \" f.write(infer_graph.as_graph_def().SerializeToString())\\n\",\n \"\\n\",\n \" if debug_verify:\\n\",\n \" with tf.Session(graph=infer_graph) as sess:\\n\",\n \" result_compiled = sess.run(output_name, feed_dict)\\n\",\n \" np.testing.assert_allclose(result_compiled, result_reference, rtol=1e-2, atol=1e-3)\\n\",\n \"\\n\",\n \"\\n\",\n \"def inline_float32_to_float16(graph_def, preprocessing_ops):\\n\",\n \" float32_enum = tf.float32.as_datatype_enum\\n\",\n \" float16_enum = tf.float16.as_datatype_enum\\n\",\n \" graph = tf.Graph()\\n\",\n \" with graph.as_default():\\n\",\n \" tf.import_graph_def(graph_def, name='')\\n\",\n \" graph_def = graph.as_graph_def()\\n\",\n \" for node in graph_def.node:\\n\",\n \" if node.name in preprocessing_ops or node.op == 'Placeholder':\\n\",\n \" cast_input_node_name = node.name\\n\",\n \" continue\\n\",\n \" if node.op == 'Const':\\n\",\n \" if node.attr['dtype'].type == float32_enum:\\n\",\n \" node.attr['dtype'].type = float16_enum\\n\",\n \" tensor_def = node.attr['value'].tensor\\n\",\n \" tensor_def.dtype = float16_enum\\n\",\n \" if tensor_def.tensor_content:\\n\",\n \" const_np = np.frombuffer(tensor_def.tensor_content, dtype=np.float32).astype(np.float16)\\n\",\n \" tensor_def.tensor_content = const_np.tobytes()\\n\",\n \" elif len(tensor_def.float_val):\\n\",\n \" const_np = np.array(tensor_def.float_val).astype(np.float16).view(np.uint16)\\n\",\n \" tensor_def.float_val[:] = []\\n\",\n \" tensor_def.half_val[:] = list(const_np)\\n\",\n \" else:\\n\",\n \" raise NotImplementedError\\n\",\n \" elif 'T' in node.attr and node.attr['T'].type == float32_enum:\\n\",\n \" node.attr['T'].type = float16_enum\\n\",\n \" for node in graph_def.node:\\n\",\n \" if node.name == cast_input_node_name:\\n\",\n \" node.name = '{}_PreCastFloat32ToFlot16'.format(node.name)\\n\",\n \" input_node = node\\n\",\n \" break\\n\",\n \" cast_input_node = _gen_cast_node_def(cast_input_node_name, tf.float16, input_node)\\n\",\n \"\\n\",\n \" output_node = graph_def.node[-1]\\n\",\n \" cast_output_node_name = output_node.name\\n\",\n \" output_node.name = '{}_PreCastFloat16ToFlot32'.format(output_node.name)\\n\",\n \" cast_output_node = _gen_cast_node_def(cast_output_node_name, tf.float32, output_node)\\n\",\n \"\\n\",\n \" preprocessing_ops.add(input_node.name)\\n\",\n \" new_graph_def = tf.GraphDef()\\n\",\n \" new_graph_def.node.extend(graph_def.node)\\n\",\n \" new_graph_def.node.append(cast_input_node)\\n\",\n \" new_graph_def.node.append(cast_output_node)\\n\",\n \" graph = tf.Graph()\\n\",\n \" with graph.as_default():\\n\",\n \" tf.import_graph_def(new_graph_def, name='')\\n\",\n \" return graph.as_graph_def()\\n\",\n \"\\n\",\n \"\\n\",\n \"def nhwc_to_nchw(graph_def, preprocessing_ops):\\n\",\n \" graph = tf.Graph()\\n\",\n \" with graph.as_default():\\n\",\n \" tf.import_graph_def(graph_def, name='')\\n\",\n \" graph_def = graph.as_graph_def()\\n\",\n \" node_name_to_node = {node.name: node for node in graph_def.node}\\n\",\n \" for node in graph_def.node:\\n\",\n \" if node.name in preprocessing_ops or node.op == 'Placeholder':\\n\",\n \" transpose_input_node_name = node.name\\n\",\n \" continue\\n\",\n \" if node.op == 'Conv2D':\\n\",\n \" node.attr['data_format'].s = b'NCHW'\\n\",\n \" strides = node.attr['strides'].list.i\\n\",\n \" strides[:] = [strides[0], strides[3], strides[1], strides[2]]\\n\",\n \" elif node.op == 'BiasAdd':\\n\",\n \" if node.name != 'probs/BiasAdd':\\n\",\n \" node.attr['data_format'].s = b'NCHW'\\n\",\n \" elif node.op == 'MaxPool':\\n\",\n \" node.attr['data_format'].s = b'NCHW'\\n\",\n \" ksize = node.attr['ksize'].list.i\\n\",\n \" ksize[:] = [ksize[0], ksize[3], ksize[1], ksize[2]]\\n\",\n \" strides = node.attr['strides'].list.i\\n\",\n \" strides[:] = [strides[0], strides[3], strides[1], strides[2]]\\n\",\n \" elif node.op in {'Concat', 'ConcatV2'}:\\n\",\n \" node_axes = node_name_to_node[node.input[-1]]\\n\",\n \" node_axes.attr['value'].tensor.int_val[:] = [1]\\n\",\n \" for node in graph_def.node:\\n\",\n \" if node.name == transpose_input_node_name:\\n\",\n \" node.name = '{}_PreTransposeNHWC2NCHW'.format(node.name)\\n\",\n \" input_node = node\\n\",\n \" break\\n\",\n \" transpose_input_node, transpose_input_perm_node = _gen_transpose_def(transpose_input_node_name, [0, 3, 1, 2], input_node)\\n\",\n \"\\n\",\n \" output_node = graph_def.node[-1]\\n\",\n \" transpose_output_node_name = output_node.name\\n\",\n \" output_node.name = '{}_PreTransposeNCHW2NHWC'.format(output_node.name)\\n\",\n \" transpose_output_node, transpose_output_perm_node = _gen_transpose_def(transpose_output_node_name, [0, 2, 3, 1], output_node)\\n\",\n \"\\n\",\n \" preprocessing_ops.add(input_node.name)\\n\",\n \" preprocessing_ops.add(transpose_input_perm_node.name)\\n\",\n \" new_graph_def = tf.GraphDef()\\n\",\n \" new_graph_def.node.extend(graph_def.node)\\n\",\n \" new_graph_def.node.append(transpose_input_perm_node)\\n\",\n \" new_graph_def.node.append(transpose_input_node)\\n\",\n \" new_graph_def.node.append(transpose_output_perm_node)\\n\",\n \" new_graph_def.node.append(transpose_output_node)\\n\",\n \" graph = tf.Graph()\\n\",\n \" with graph.as_default():\\n\",\n \" tf.import_graph_def(new_graph_def, name='')\\n\",\n \" return graph.as_graph_def()\\n\",\n \"\\n\",\n \"\\n\",\n \"def _gen_cast_node_def(name, target_dtype, input_node):\\n\",\n \" cast_node = tf.NodeDef(name=name, op='Cast')\\n\",\n \" cast_node.input.append(input_node.name)\\n\",\n \" cast_node.attr['DstT'].type = target_dtype.as_datatype_enum\\n\",\n \" cast_node.attr['SrcT'].type = input_node.attr['T'].type\\n\",\n \" cast_node.attr['Truncate'].b = False\\n\",\n \" return cast_node\\n\",\n \"\\n\",\n \"\\n\",\n \"def _gen_transpose_def(name, perm, input_node):\\n\",\n \" perm_node = tf.NodeDef(name='{}/perm'.format(name), op='Const')\\n\",\n \" perm_node.attr['dtype'].type = tf.int32.as_datatype_enum\\n\",\n \" tensor_def = perm_node.attr['value'].tensor\\n\",\n \" tensor_def.dtype = tf.int32.as_datatype_enum\\n\",\n \" tensor_def.tensor_shape.dim.append(TensorShapeProto.Dim(size=4))\\n\",\n \" tensor_def.tensor_content = np.array(perm, dtype=np.int32).tobytes()\\n\",\n \" transpose_node = tf.NodeDef(name=name, op='Transpose')\\n\",\n \" transpose_node.input.append(input_node.name)\\n\",\n \" transpose_node.input.append(perm_node.name)\\n\",\n \" transpose_node.attr['T'].type = input_node.attr['T'].type\\n\",\n \" transpose_node.attr['Tperm'].type = tf.int32.as_datatype_enum\\n\",\n \" return transpose_node, perm_node\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"88c41e01\",\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"compile()\\n\",\n \"\\n\",\n \"# Sample output will look like below:\\n\",\n \"# WARNING:tensorflow:From :47: inference_graph_from_session (from tensorflow_neuron.python.graph_util) is deprecated and will be removed in a future version.\\n\",\n \"# Instructions for updating:\\n\",\n \"# Please refer to AWS documentation on Neuron integrated TensorFlow 2.0.\\n\",\n \"# INFO:tensorflow:Froze 0 variables.\\n\",\n \"# INFO:tensorflow:Converted 0 variables to const ops.\\n\",\n \"# INFO:tensorflow:fusing subgraph {subgraph neuron_op_ed41d2deb8c54255 with input tensors [\\\"\\\"], output tensors [\\\"\\\"]} with neuron-cc\\n\",\n \"# INFO:tensorflow:Number of operations in TensorFlow session: 474\\n\",\n \"# INFO:tensorflow:Number of operations after tf.neuron optimizations: 474\\n\",\n \"# INFO:tensorflow:Number of operations placed on Neuron runtime: 465\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5a9af0c7\",\n \"metadata\": {},\n \"source\": [\n \"## Deploy\\n\",\n \"Using same instance to deploy the model.\\n\",\n \"In case of different deployment instance, launch a deployment inf1 instance and copy the AWS Neuron optimized tensorflow frozen graph graph_opt_neuron_656x368.pb to the deployment inf1 instance. The smallest instance type inf1.xlarge is sufficient for this demo.\\n\",\n \"\\n\",\n \"Your graph_opt_neuron_656x368.pb can now be plugged into https://github.com/ildoonet seemlessly if you have tensorflow-neuron installed. When it is used at runtime, please ensure that the image resolution is the same as compile-time image resolution, i. e., 656x368.\\n\",\n \"\\n\",\n \"Measure performance on the compiled frozen graph using dummy inputs.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"0481d049\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"\\\"\\\"\\\"\\n\",\n \"Copyright (C) 2020, Amazon.com. All Rights Reserved\\n\",\n \"\\\"\\\"\\\"\\n\",\n \"import os\\n\",\n \"import atexit\\n\",\n \"import time\\n\",\n \"import math\\n\",\n \"import json\\n\",\n \"from collections import OrderedDict, Counter\\n\",\n \"from contextlib import contextmanager, ContextDecorator\\n\",\n \"from functools import wraps\\n\",\n \"from tensorflow.python.client import session\\n\",\n \"from tensorflow.python.platform import tf_logging as logging\\n\",\n \"\\n\",\n \"\\n\",\n \"class measure_performance(ContextDecorator):\\n\",\n \" \\\"\\\"\\\"Convenient tool for performance measurements.\\n\",\n \" Can be apply on tensorflow session.run, tf-serving unary gRPC calls, or a given custom function.\\n\",\n \" Usage:\\n\",\n \" To generate performance report for the entire Python or gRPC-client process, insert\\n\",\n \" the following function call before running inferences:\\n\",\n \" `tfn.measure_performance()`\\n\",\n \" Then latency/throughput report will be generated when the process terminates.\\n\",\n \" Alternatively, it is possible to use `tfn.measure_performance` programmatically\\n\",\n \" as a context manager. Performance measurement will be done for all inferences\\n\",\n \" happening under this context. Report will be displayed as INFO level log when exiting\\n\",\n \" the context. It is also possible to obtain a JSON format report in Python.\\n\",\n \" For example:\\n\",\n \" ```\\n\",\n \" with tfn.measure_performance() as perf:\\n\",\n \" ... (run some inferences) ...\\n\",\n \" report_json = perf.report()\\n\",\n \" report_full_json = perf.report(verbosity=1)\\n\",\n \" ```\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" def __init__(self, func=None, window_size=1):\\n\",\n \" self.perf_tracker = PerformanceTracker(window_size)\\n\",\n \" atexit.register(self.perf_tracker.report)\\n\",\n \" self._original_run = session.Session.run\\n\",\n \" self._original_grpc_call = None\\n\",\n \" if callable(func):\\n\",\n \" self.perf_tracker.register_func(self._track_performance(func))\\n\",\n \" else:\\n\",\n \" session.Session.run = self._track_performance(session.Session.run)\\n\",\n \" try:\\n\",\n \" import grpc\\n\",\n \" from tensorflow_serving.apis import prediction_service_pb2_grpc\\n\",\n \" dummy_stub = prediction_service_pb2_grpc.PredictionServiceStub(grpc.insecure_channel(''))\\n\",\n \" self._grpc_callable_type = type(dummy_stub.Predict)\\n\",\n \" self._original_grpc_call = self._grpc_callable_type.__call__\\n\",\n \" except ImportError:\\n\",\n \" pass\\n\",\n \" if callable(self._original_grpc_call):\\n\",\n \" self._grpc_callable_type.__call__ = self._track_performance(\\n\",\n \" grpc._channel._UnaryUnaryMultiCallable.__call__\\n\",\n \" )\\n\",\n \"\\n\",\n \" def __enter__(self):\\n\",\n \" return self.perf_tracker\\n\",\n \"\\n\",\n \" def __exit__(self, *exc):\\n\",\n \" atexit.unregister(self.perf_tracker.report)\\n\",\n \" self.perf_tracker.report()\\n\",\n \" session.Session.run = self._original_run\\n\",\n \" if self._original_grpc_call is not None:\\n\",\n \" self._grpc_callable_type.__call__ = self._original_grpc_call\\n\",\n \" return False\\n\",\n \"\\n\",\n \" def _track_performance(self, func):\\n\",\n \" @wraps(func)\\n\",\n \" def wrapper(*args, **kwargs):\\n\",\n \" start = time.time()\\n\",\n \" result = func(*args, **kwargs)\\n\",\n \" end = time.time()\\n\",\n \" self.perf_tracker.add_timestamps(start, end)\\n\",\n \" return result\\n\",\n \" return wrapper\\n\",\n \"\\n\",\n \"\\n\",\n \"class PerformanceTracker(ContextDecorator):\\n\",\n \"\\n\",\n \" description = (\\n\",\n \" \\\"Latency unit: second. Throughput unit: number of batched inferences per second. \\\"\\n\",\n \" \\\"Reported throughput is a lower bound of the actual throughput as inferences \\\"\\n\",\n \" \\\"spanning across window boundaries are not counted towards any of the windows. \\\"\\n\",\n \" \\\"'Quiet' periods (i. e., window buckets where the inference function is not called) \\\"\\n\",\n \" \\\"are not counted towards the reported average throughput.\\\"\\n\",\n \" )\\n\",\n \"\\n\",\n \" def __init__(self, window_size):\\n\",\n \" self.window_size = window_size\\n\",\n \" self.timestamps_list = []\\n\",\n \" self._func = None\\n\",\n \"\\n\",\n \" def __call__(self, *args, **kwargs):\\n\",\n \" return self._func(*args, **kwargs)\\n\",\n \"\\n\",\n \" def register_func(self, func):\\n\",\n \" self._func = func\\n\",\n \"\\n\",\n \" def add_timestamps(self, start, end):\\n\",\n \" self.timestamps_list.append([start, end])\\n\",\n \"\\n\",\n \" def report(self, verbosity=0):\\n\",\n \" if self.timestamps_list:\\n\",\n \" latency_list = [end - start for start, end in self.timestamps_list]\\n\",\n \" latency_json = {\\n\",\n \" 'p50': percentile(latency_list, 50),\\n\",\n \" 'p90': percentile(latency_list, 90),\\n\",\n \" 'p99': percentile(latency_list, 99),\\n\",\n \" 'p100': percentile(latency_list, 100),\\n\",\n \" }\\n\",\n \" bucketed_timestamps = [self._get_bucket(start, end) for start, end in self.timestamps_list]\\n\",\n \" counted_buckets = Counter(item for item in bucketed_timestamps if item is not None)\\n\",\n \" bucket_throughputs = [(key, value / self.window_size) for key, value in sorted(counted_buckets.items())]\\n\",\n \" busy_throughputs = list(OrderedDict((key, value) for key, value in bucket_throughputs).values())\\n\",\n \" throughput_json = {\\n\",\n \" 'peak': max(busy_throughputs),\\n\",\n \" 'median': percentile(busy_throughputs, 50),\\n\",\n \" 'average': sum(busy_throughputs) / len(busy_throughputs),\\n\",\n \" }\\n\",\n \" if verbosity > 0:\\n\",\n \" throughput_json['trend'] = busy_throughputs\\n\",\n \" report_json = {\\n\",\n \" 'pid': os.getpid(),\\n\",\n \" 'throughput': throughput_json,\\n\",\n \" 'latency': latency_json,\\n\",\n \" 'description': PerformanceTracker.description,\\n\",\n \" }\\n\",\n \" with _logging_show_info():\\n\",\n \" logging.info('performance report:\\\\n{}'.format(json.dumps(report_json, indent=4)))\\n\",\n \" return report_json\\n\",\n \"\\n\",\n \" def _get_bucket(self, start, end):\\n\",\n \" bucketed_start = math.floor(start / self.window_size) * self.window_size\\n\",\n \" bucketed_end = math.ceil(end / self.window_size) * self.window_size\\n\",\n \" if bucketed_end - bucketed_start == self.window_size:\\n\",\n \" return bucketed_start\\n\",\n \" else:\\n\",\n \" return None\\n\",\n \"\\n\",\n \"\\n\",\n \"def percentile(number_list, percent):\\n\",\n \" pos_float = len(number_list) * percent / 100\\n\",\n \" max_pos = len(number_list) - 1\\n\",\n \" pos_floor = min(math.floor(pos_float), max_pos)\\n\",\n \" pos_ceil = min(math.ceil(pos_float), max_pos)\\n\",\n \" number_list = sorted(number_list)\\n\",\n \" return number_list[pos_ceil] if pos_float - pos_floor > 0.5 else number_list[pos_floor]\\n\",\n \"\\n\",\n \"\\n\",\n \"@contextmanager\\n\",\n \"def _logging_show_info():\\n\",\n \" try:\\n\",\n \" verbosity = logging.get_verbosity()\\n\",\n \" logging.set_verbosity(logging.INFO)\\n\",\n \" yield\\n\",\n \" finally:\\n\",\n \" logging.set_verbosity(verbosity)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"960c6aa9\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"\\\"\\\"\\\"\\n\",\n \"Below are the inputs for compiled frozen graph \\n\",\n \"\\n\",\n \"pb_path is a /path/graph_opt_neuron_656x368.pb\\n\",\n \"num_thread = 8 ( Number of threads that work on each tensorflow session ) \\n\",\n \"batch_size =1 ( batch_size )\\n\",\n \"net_resolution ,default=656x368\\n\",\n \"num_inferences = 200\\n\",\n \"\\\"\\\"\\\"\\n\",\n \"import os\\n\",\n \"from concurrent import futures\\n\",\n \"import numpy as np\\n\",\n \"import tensorflow as tf\\n\",\n \"import tensorflow.neuron as tfn\\n\",\n \"\\n\",\n \"def run_with_dummy(sess, dummy_feed_dict, num_inferences):\\n\",\n \" for _ in range(num_inferences):\\n\",\n \" sess.run('Openpose/concat_stage7:0', dummy_feed_dict)\\n\",\n \" \\n\",\n \"def main():\\n\",\n \" NUM_NEURON_CORES = 16\\n\",\n \" pb_path = './graph_opt_neuron_656x368.pb'\\n\",\n \" num_thread = 8\\n\",\n \" batch_size = 1\\n\",\n \" net_resolution = '656x368'\\n\",\n \" num_inferences = 200\\n\",\n \" dim_w, dim_h = net_resolution.split('x')\\n\",\n \" dim_w = int(dim_w)\\n\",\n \" dim_h = int(dim_h)\\n\",\n \" graph_def = tf.GraphDef()\\n\",\n \" with open(pb_path, 'rb') as f:\\n\",\n \" graph_def.ParseFromString(f.read())\\n\",\n \" \\n\",\n \" graph_def = tfn.graph_util.tag_multicore(graph_def, NUM_NEURON_CORES)\\n\",\n \" \\n\",\n \" with tfn.measure_performance() as perf:\\n\",\n \" with tf.Session(graph=tf.Graph()) as sess:\\n\",\n \" tf.import_graph_def(graph_def, name='')\\n\",\n \" input_name = 'image:0'\\n\",\n \" input_shape = sess.graph.get_tensor_by_name(input_name).shape.as_list()\\n\",\n \" input_shape[0] = batch_size\\n\",\n \" input_shape[1] = dim_h\\n\",\n \" input_shape[2] = dim_w\\n\",\n \" dummy_feed_dict = {input_name: np.zeros(input_shape).astype(np.float32)}\\n\",\n \" with futures.ThreadPoolExecutor(max_workers=num_thread) as executor:\\n\",\n \" fut_list = [executor.submit(run_with_dummy, sess, dummy_feed_dict, num_inferences) for _ in range(num_thread)]\\n\",\n \" res_list = [fut.result() for fut in fut_list] \\n\",\n \"\\n\",\n \"main()\\n\",\n \"\\n\",\n \"# Sample output will look like below:\\n\",\n \"# INFO:tensorflow:performance report:\\n\",\n \"# {\\n\",\n \"# \\\"pid\\\": 17713,\\n\",\n \"# \\\"throughput\\\": {\\n\",\n \"# \\\"peak\\\": 66.0,\\n\",\n \"# \\\"median\\\": 64.0,\\n\",\n \"# \\\"average\\\": 61.56521739130435\\n\",\n \"# },\\n\",\n \"# \\\"latency\\\": {\\n\",\n \"# \\\"p50\\\": 0.1106414794921875,\\n\",\n \"# \\\"p90\\\": 0.11212301254272461,\\n\",\n \"# \\\"p99\\\": 0.11337876319885254,\\n\",\n \"# \\\"p100\\\": 7.08282732963562\\n\",\n \"# },\\n\",\n \"# \\\"description\\\": \\\"Latency unit: second. Throughput unit: number of batched inferences per second. Reported throughput is a lower bound of the actual throughput as inferences spanning across window boundaries are not counted towards any of the windows. 'Quiet' periods (i. e., window buckets where the inference function is not called) are not counted towards the reported average throughput.\\\"\\n\",\n \"# }\"\n ]\n },\n {\n \"cell_type\": \"raw\",\n \"id\": \"4f15e776\",\n \"metadata\": {},\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/tensorflow/ssd300_demo/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/tensorflow/ssd300_demo/ssd300_detection.py", "content": "import argparse\nimport json\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport numpy as np\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport tensorflow as tf\nimport tensorflow.neuron as tfn\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--image', required=True, help='Path to image that is to be detected. Support jpeg and png format.')\n parser.add_argument('--image_with_detections', required=True, help='Path to save image after detection (with bounding boxes drawn). Png format.')\n parser.add_argument('--saved_model', required=True, help='TensorFlow SSD300 SavedModel')\n parser.add_argument('--score_threshold', type=float, default=0.15, help='Minimum required score for drawing a bounding box')\n parser.add_argument('--instances_val2017_json', default=None, help='Json file that contains labeling information')\n parser.add_argument('--save_results', default=None)\n parser.add_argument('--disable_version_check', action='store_true')\n args = parser.parse_args()\n if not args.disable_version_check:\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version < LooseVersion('1.15.0.1.0.1333.0'):\n raise RuntimeError(\n 'tensorflow-neuron version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U tensorflow-neuron --extra-index-url=https://pip.repos.neuron.amazonaws.com\"'.format(tfn_version))\n\n with open(args.image, 'rb') as f:\n img_jpg_bytes = f.read()\n model_feed_dict = {'batch_image': [img_jpg_bytes]}\n\n predictor = tf.contrib.predictor.from_saved_model(args.saved_model)\n results = predictor(model_feed_dict)\n if args.save_results is not None:\n np.savez(args.save_results, **results)\n boxes_np = results['boxes']\n scores_np = results['scores']\n classes_np = results['classes']\n\n if args.instances_val2017_json is not None:\n with open(args.instances_val2017_json) as f:\n annotate_json = json.load(f)\n label_info = {idx+1: cat['name'] for idx, cat in enumerate(annotate_json['categories'])}\n\n plt.switch_backend('agg')\n fig, ax = plt.subplots(1)\n ax.imshow(Image.open(args.image).convert('RGB'))\n\n wanted = scores_np[0] > args.score_threshold\n for xywh, label_no_bg in zip(boxes_np[0][wanted], classes_np[0][wanted]):\n rect = patches.Rectangle((xywh[0], xywh[1]), xywh[2], xywh[3], linewidth=1, edgecolor='g', facecolor='none')\n ax.add_patch(rect)\n rx, ry = rect.get_xy()\n rx = rx + rect.get_width() / 2.0\n if args.instances_val2017_json is not None:\n ax.annotate(label_info[label_no_bg + 1], (rx, ry), color='w', backgroundcolor='g', fontsize=10,\n ha='center', va='center', bbox=dict(boxstyle='square,pad=0.01', fc='g', ec='none', alpha=0.5))\n plt.savefig(args.image_with_detections)\n plt.close(fig)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/ssd300_demo/ssd300_evaluation.py", "content": "import argparse\nimport os\nimport json\nimport glob\nfrom concurrent import futures\nimport time\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow.neuron as tfn\nfrom pycocotools.cocoeval import COCOeval\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.coco import COCO\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import dboxes300_coco\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import SSDTransformer\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import COCODetection\n\n\ndef get_val_dataset(val_annotate, val_coco_root):\n dboxes = dboxes300_coco()\n val_trans = SSDTransformer(dboxes, (300, 300), val=True)\n val_coco = COCODetection(val_coco_root, val_annotate, val_trans)\n return val_coco\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--saved_model', required=True, help='TensorFlow SSD300 SavedModel')\n parser.add_argument('--val2017', required=True, help='Path to COCO 2017 validation dataset')\n parser.add_argument('--instances_val2017_json', required=True, help='Json file that contains labeling information')\n parser.add_argument('--num_sessions', type=int, default=1, help='Number of tensorflow sessions')\n parser.add_argument('--num_threads', type=int, default=4, help='Number of threads')\n parser.add_argument('--throughput_interval', type=int, default=10, help='Interval for counting throughput')\n parser.add_argument('--save_results', default=None)\n parser.add_argument('--disable_version_check', action='store_true')\n args = parser.parse_args()\n if not args.disable_version_check:\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version < LooseVersion('1.15.0.1.0.1333.0'):\n raise RuntimeError(\n 'tensorflow-neuron version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U tensorflow-neuron --extra-index-url=https://pip.repos.neuron.amazonaws.com\"'.format(tfn_version))\n predictor_list = [tf.contrib.predictor.from_saved_model(args.saved_model) for _ in range(args.num_sessions)]\n\n val_dataset = get_val_dataset(args.instances_val2017_json, args.val2017)\n inv_map = {v: k for k, v in val_dataset.label_map.items()}\n model_feed_dict_list = []\n for img_id in val_dataset.img_keys:\n img_path = os.path.join(args.val2017, val_dataset.images[img_id][0])\n with open(img_path, 'rb') as f:\n img_jpg_bytes = f.read()\n model_feed_dict_list.append({'batch_image': [img_jpg_bytes]})\n\n latency_list = []\n throughput_list = []\n def predict(pred, model_feed_dict):\n start = time.time()\n result = pred(model_feed_dict)\n latency_list.append(time.time() - start)\n return result\n\n def performance():\n last_num_infer = len(latency_list)\n while len(latency_list) < len(model_feed_dict_list):\n current_num_infer = len(latency_list)\n throughput = (current_num_infer - last_num_infer) / args.throughput_interval\n throughput_list.append(throughput)\n p50 = 0.0\n p90 = 0.0\n if latency_list:\n p50 = np.percentile(latency_list, 50)\n p90 = np.percentile(latency_list, 90)\n print('pid {}: current throughput {}, latency p50={:.3f} p90={:.3f}'.format(os.getpid(), throughput, p50, p90))\n last_num_infer = current_num_infer\n time.sleep(args.throughput_interval)\n\n executor = futures.ThreadPoolExecutor(max_workers=(args.num_sessions*args.num_threads)+1)\n performance_future = executor.submit(performance)\n eval_futures = []\n for idx, model_feed_dict in enumerate(model_feed_dict_list):\n eval_fut = executor.submit(predict, predictor_list[idx%len(predictor_list)], model_feed_dict)\n eval_futures.append(eval_fut)\n waited_results = []\n for idx, eval_fut in enumerate(eval_futures):\n if idx % 100 == 0:\n print('evaluating image {}/{}'.format(idx, len(eval_futures)))\n waited_results.append(eval_fut.result())\n eval_results = []\n for idx, (img_id, results) in enumerate(zip(val_dataset.img_keys, waited_results)):\n boxes = results['boxes']\n for box, label, prob in zip(results['boxes'][0], results['classes'][0], results['scores'][0]):\n res = [img_id, box[0], box[1], box[2], box[3], prob, inv_map[label+1]] # +1 to account for background\n eval_results.append(res)\n performance_future.result()\n\n coco_gt = COCO(annotation_file=args.instances_val2017_json)\n coco_dt = coco_gt.loadRes(np.array(eval_results).astype(np.float32))\n coco_eval = COCOeval(coco_gt, coco_dt, iouType='bbox')\n coco_eval.evaluate()\n coco_eval.accumulate()\n coco_eval.summarize()\n if args.save_results is not None:\n np.save(args.save_results, coco_eval.stats)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/ssd300_demo/ssd300_evaluation_client.py", "content": "import argparse\nimport os\nimport json\nimport glob\nfrom concurrent import futures\nimport time\nimport subprocess\nfrom distutils.version import LooseVersion\nimport numpy as np\nimport tensorflow as tf\nimport grpc\nfrom tensorflow_serving.apis import predict_pb2\nfrom tensorflow_serving.apis import prediction_service_pb2_grpc\nfrom pycocotools.cocoeval import COCOeval\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.coco import COCO\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import dboxes300_coco\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import SSDTransformer\nfrom DeepLearningExamples.PyTorch.Detection.SSD.src.utils import COCODetection\n\n\ndef get_val_dataset(val_annotate, val_coco_root):\n dboxes = dboxes300_coco()\n val_trans = SSDTransformer(dboxes, (300, 300), val=True)\n val_coco = COCODetection(val_coco_root, val_annotate, val_trans)\n return val_coco\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--server_address', default='localhost:8500', help='tensorflow-model-server-neuron grpc address')\n parser.add_argument('--model_name', default='default', help='Serving model name')\n parser.add_argument('--val2017', required=True, help='Path to COCO 2017 validation dataset')\n parser.add_argument('--instances_val2017_json', required=True, help='Json file that contains labeling information')\n parser.add_argument('--num_threads', type=int, default=4, help='Number of threads')\n parser.add_argument('--throughput_interval', type=int, default=10, help='Interval for counting throughput')\n parser.add_argument('--save_results', default=None)\n args = parser.parse_args()\n\n channel = grpc.insecure_channel(args.server_address)\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n\n val_dataset = get_val_dataset(args.instances_val2017_json, args.val2017)\n inv_map = {v: k for k, v in val_dataset.label_map.items()}\n request_list = []\n for img_id in val_dataset.img_keys:\n img_path = os.path.join(args.val2017, val_dataset.images[img_id][0])\n with open(img_path, 'rb') as f:\n img_jpg_bytes = f.read()\n data = np.array([img_jpg_bytes], dtype=object)\n data = tf.contrib.util.make_tensor_proto(data, shape=data.shape)\n request = predict_pb2.PredictRequest()\n request.model_spec.name = args.model_name\n request.inputs['batch_image'].CopyFrom(data)\n request_list.append(request)\n\n latency_list = []\n throughput_list = []\n def predict(request):\n start = time.time()\n result = stub.Predict(request).outputs\n latency_list.append(time.time() - start)\n return result\n\n def performance():\n last_num_infer = len(latency_list)\n while len(latency_list) < len(request_list):\n current_num_infer = len(latency_list)\n throughput = (current_num_infer - last_num_infer) / args.throughput_interval\n throughput_list.append(throughput)\n p50 = 0.0\n p90 = 0.0\n if latency_list:\n p50 = np.percentile(latency_list, 50)\n p90 = np.percentile(latency_list, 90)\n print('pid {}: current throughput {}, latency p50={:.3f} p90={:.3f}'.format(os.getpid(), throughput, p50, p90))\n last_num_infer = current_num_infer\n time.sleep(args.throughput_interval)\n\n executor = futures.ThreadPoolExecutor(max_workers=args.num_threads+1)\n performance_future = executor.submit(performance)\n eval_futures = []\n for idx, request in enumerate(request_list):\n eval_fut = executor.submit(predict, request)\n eval_futures.append(eval_fut)\n waited_results = []\n for idx, eval_fut in enumerate(eval_futures):\n if idx % 100 == 0:\n print('evaluating image {}/{}'.format(idx, len(eval_futures)))\n waited_results.append(eval_fut.result())\n eval_results = []\n for idx, (img_id, results) in enumerate(zip(val_dataset.img_keys, waited_results)):\n results = {key: tf.make_ndarray(value) for key, value in results.items()}\n boxes = results['boxes']\n for box, label, prob in zip(results['boxes'][0], results['classes'][0], results['scores'][0]):\n res = [img_id, box[0], box[1], box[2], box[3], prob, inv_map[label+1]] # +1 to account for background\n eval_results.append(res)\n performance_future.result()\n\n coco_gt = COCO(annotation_file=args.instances_val2017_json)\n coco_dt = coco_gt.loadRes(np.array(eval_results).astype(np.float32))\n coco_eval = COCOeval(coco_gt, coco_dt, iouType='bbox')\n coco_eval.evaluate()\n coco_eval.accumulate()\n coco_eval.summarize()\n if args.save_results is not None:\n np.save(args.save_results, coco_eval.stats)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/ssd300_demo/ssd300_model.py", "content": "import sys\nimport os\nimport argparse\nimport time\nimport itertools\nfrom functools import partial\nfrom collections import Counter\nimport json\nimport shutil\nimport pkg_resources\nfrom distutils.version import LooseVersion\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.core.framework import attr_value_pb2\nimport tensorflow.neuron as tfn\nimport torch\n\n\ndef decode_jpeg_resize(input_tensor, image_size):\n # decode jpeg\n tensor = tf.image.decode_png(input_tensor, channels=3)\n\n # resize\n decoded_shape = tf.shape(tensor)\n tensor = tf.cast(tensor, tf.float32)\n decoded_shape_hw = decoded_shape[0:2]\n decoded_shape_hw_float32 = tf.cast(decoded_shape_hw, tf.float32)\n tensor = tf.image.resize(tensor, image_size)\n\n # normalize\n tensor -= np.array([0.485, 0.456, 0.406]).astype(np.float32) * 255.0\n return tensor, decoded_shape_hw_float32[::-1]\n\n\ndef preprocessor(input_tensor, image_size):\n with tf.name_scope('Preprocessor'):\n tensor, bbox_scale_hw = tf.map_fn(\n partial(decode_jpeg_resize, image_size=image_size), input_tensor,\n dtype=(tf.float32, tf.float32), back_prop=False, parallel_iterations=16)\n return tensor, bbox_scale_hw\n\n\ndef tf_Conv2d(input_tensor, module, first_conv=False):\n np_dtype = input_tensor.dtype.as_numpy_dtype\n kernel_np = module.weight.detach().numpy().transpose([2, 3, 1, 0])\n if first_conv:\n kernel_np /= (np.array([0.229, 0.224, 0.225]).astype(np.float32) * 255.0)[:, np.newaxis]\n kernel = tf.constant(kernel_np.astype(np_dtype))\n if any(module.padding):\n pad_h, pad_w = module.padding\n padding = [[0, 0], [pad_h, pad_h], [pad_w, pad_w], [0, 0]]\n input_tensor = tf.pad(input_tensor, padding)\n stride_h, stride_w = module.stride\n tensor = tf.nn.conv2d(input_tensor, kernel, strides=[1, stride_h, stride_w, 1], padding='VALID')\n if module.bias is not None:\n bias = tf.constant(module.bias.detach().numpy().astype(np_dtype))\n tensor = tf.nn.bias_add(tensor, bias)\n return tensor\n\ndef tf_BatchNorm2d(input_tensor, module):\n def _norm_np(ts):\n return ts.astype(input_tensor.dtype.as_numpy_dtype)\n mean = _norm_np(module.running_mean.detach().numpy())\n offset = _norm_np(module.bias.detach().numpy())\n inv_std = np.sqrt(module.running_var.detach().numpy() + module.eps)\n scale_inv_std = _norm_np(module.weight.detach().numpy() / inv_std)\n return scale_inv_std * (input_tensor - mean) + offset\n\ndef tf_MaxPool2d(input_tensor, module):\n pad = module.padding\n tensor = tf.pad(input_tensor, [[0, 0], [pad, pad], [pad, pad], [0, 0]])\n return tf.nn.max_pool2d(tensor, ksize=module.kernel_size, strides=module.stride, padding='VALID')\n\ndef tf_Bottleneck(input_tensor, module):\n tensor = tf_Conv2d(input_tensor, module.conv1)\n tensor = tf_BatchNorm2d(tensor, module.bn1)\n tensor = tf.nn.relu(tensor)\n tensor = tf_Conv2d(tensor, module.conv2)\n tensor = tf_BatchNorm2d(tensor, module.bn2)\n tensor = tf.nn.relu(tensor)\n tensor = tf_Conv2d(tensor, module.conv3)\n tensor = tf_BatchNorm2d(tensor, module.bn3)\n if module.downsample is not None:\n input_tensor = tf_Conv2d(input_tensor, module.downsample[0])\n input_tensor = tf_BatchNorm2d(input_tensor, module.downsample[1])\n return tf.nn.relu(input_tensor + tensor)\n\ndef tf_SequentialBottleneck(tensor, seq, resnet):\n with tf.name_scope('{}.Sequential'.format(seq)):\n for idx, module in enumerate(resnet[seq]):\n with tf.name_scope('{}.BasicBlock'.format(idx)):\n tensor = tf_Bottleneck(tensor, module)\n return tensor\n\ndef tf_bbox_view(detection_feed, modules, ndim):\n results = []\n for idx, (tensor, mod) in enumerate(zip(detection_feed, modules)):\n with tf.name_scope('branch{}'.format(idx)):\n tensor = tf_Conv2d(tensor, mod)\n tensor = tf.transpose(tensor, [0, 3, 1, 2])\n tensor = tf.cast(tensor, tf.float32)\n\n shape = tensor.shape.as_list()\n batch_size = -1 if shape[0] is None else shape[0]\n new_shape = [batch_size, ndim, np.prod(shape[1:]) // ndim]\n results.append(tf.reshape(tensor, new_shape))\n tensor = tf.concat(results, axis=-1)\n return tensor\n\n\ndef tf_feature_extractor(input_tensor, resnet):\n with tf.name_scope('FeatureExtractor'):\n with tf.name_scope('0.Conv2d'):\n tensor = tf_Conv2d(input_tensor, resnet[0], first_conv=True)\n with tf.name_scope('1.BatchNorm2d'):\n tensor = tf_BatchNorm2d(tensor, resnet[1])\n with tf.name_scope('2.ReLU'):\n tensor = tf.nn.relu(tensor)\n with tf.name_scope('3.MaxPool2d'):\n tensor = tf_MaxPool2d(tensor, resnet[3])\n tensor = tf_SequentialBottleneck(tensor, 4, resnet)\n tensor = tf_SequentialBottleneck(tensor, 5, resnet)\n tensor = tf_SequentialBottleneck(tensor, 6, resnet)\n tensor = tf.cast(tensor, tf.float16)\n return tensor\n\n\ndef tf_box_predictor(tensor, ssd300_torch):\n with tf.name_scope('BoxPredictor'):\n detection_feed = [tensor]\n for idx, block in enumerate(ssd300_torch.additional_blocks):\n with tf.name_scope('{}.Sequential'.format(idx)):\n tensor = tf_Conv2d(tensor, block[0])\n tensor = tf_BatchNorm2d(tensor, block[1])\n tensor = tf.nn.relu(tensor)\n tensor = tf_Conv2d(tensor, block[3])\n tensor = tf_BatchNorm2d(tensor, block[4])\n tensor = tf.nn.relu(tensor)\n detection_feed.append(tensor)\n with tf.name_scope('Boxes'):\n loc = tf_bbox_view(detection_feed, ssd300_torch.loc, ndim=4)\n with tf.name_scope('Probabilities'):\n conf = tf_bbox_view(detection_feed, ssd300_torch.conf, ndim=ssd300_torch.label_num)\n return loc, conf\n\n\n@tfn.fuse(batch_size=1, dynamic_batch_size=True)\ndef tf_ssd300(input_tensor, ssd300_torch):\n with tf.name_scope('SSD300'):\n tensor = tf_feature_extractor(input_tensor, ssd300_torch.feature_extractor.feature_extractor)\n loc, conf = tf_box_predictor(tensor, ssd300_torch)\n return loc, conf\n\n\ndef scale_back_batch(bboxes_in, scores_in, scale_xy, scale_wh, dboxes_xywh):\n \"\"\"\n Do scale and transform from xywh to ltrb\n suppose input Nx4xnum_bbox Nxlabel_numxnum_bbox\n \"\"\"\n with tf.name_scope('ScaleBackBatch'):\n bboxes_in = tf.transpose(bboxes_in, [0, 2, 1])\n scores_in = tf.transpose(scores_in, [0, 2, 1])\n\n bboxes_xy = bboxes_in[:, :, :2]\n bboxes_wh = bboxes_in[:, :, 2:]\n bboxes_xy *= scale_xy\n bboxes_wh *= scale_wh\n\n bboxes_xy = bboxes_xy * dboxes_xywh[:, :, 2:] + dboxes_xywh[:, :, :2]\n bboxes_wh = tf.exp(bboxes_wh) * dboxes_xywh[:, :, 2:]\n\n bboxes_wh_half = 0.5 * bboxes_wh\n bboxes_lt = bboxes_xy - bboxes_wh_half\n bboxes_rb = bboxes_xy + bboxes_wh_half\n\n bboxes_in = tf.concat([bboxes_lt, bboxes_rb], axis=-1)\n\n return bboxes_in, tf.nn.softmax(scores_in, axis=-1)\n\ndef select_nms_outputs(input_tensors):\n boxes_xywh, scores, classes, valid_detections = input_tensors\n return boxes_xywh[:valid_detections], scores[:valid_detections], classes[:valid_detections]\n\ndef postprocessor(ploc_ts, plabel_ts, bbox_scale_hw_ts, scale_xy, scale_wh, dboxes_xywh):\n with tf.name_scope('Postprocessor'):\n ploc_ts = tf.cast(ploc_ts, tf.float32)\n plabel_ts = tf.cast(plabel_ts, tf.float32)\n bboxes_ts, probs_ts = scale_back_batch(ploc_ts, plabel_ts, scale_xy, scale_wh, dboxes_xywh)\n bboxes_ts = bboxes_ts[:, :, tf.newaxis, :]\n probs_ts = probs_ts[:, :, 1:]\n nms_outputs = tf.image.combined_non_max_suppression(\n bboxes_ts,\n probs_ts,\n max_output_size_per_class=200,\n max_total_size=200,\n iou_threshold=0.5,\n score_threshold=0.05,\n pad_per_class=False,\n clip_boxes=False,\n name='CombinedNonMaxSuppression',\n )\n nmsed_boxes_x0y0x1y1, nmsed_scores, nmsed_classes, valid_detections = nms_outputs\n nmsed_boxes_x0y0 = nmsed_boxes_x0y0x1y1[..., :2]\n nmsed_boxes_x1y1 = nmsed_boxes_x0y0x1y1[..., 2:]\n bbox_scale_hw_ts = bbox_scale_hw_ts[:, tf.newaxis, :]\n nmsed_boxes_xy = nmsed_boxes_x0y0 * bbox_scale_hw_ts\n nmsed_boxes_wh = (nmsed_boxes_x1y1 - nmsed_boxes_x0y0) * bbox_scale_hw_ts\n nmsed_boxes_xywh = tf.concat([nmsed_boxes_xy, nmsed_boxes_wh], axis=-1)\n nmsed_boxes_xywh, nmsed_scores, nmsed_classes = tf.map_fn(\n select_nms_outputs, (nmsed_boxes_xywh, nmsed_scores, nmsed_classes, valid_detections),\n dtype=(tf.float32, tf.float32, tf.float32), back_prop=False, parallel_iterations=16)\n return nmsed_boxes_xywh, nmsed_scores, nmsed_classes\n\n\nclass DefaultBoxes(object):\n\n def __init__(self, fig_size, feat_size, steps, scales, aspect_ratios,\n scale_xy=0.1, scale_wh=0.2):\n\n self.feat_size = feat_size\n self.fig_size = fig_size\n\n self.scale_xy_ = scale_xy\n self.scale_wh_ = scale_wh\n\n # According to https://github.com/weiliu89/caffe\n # Calculation method slightly different from paper\n self.steps = steps\n self.scales = scales\n\n fk = fig_size/np.array(steps)\n self.aspect_ratios = aspect_ratios\n\n self.default_boxes = []\n # size of feature and number of feature\n for idx, sfeat in enumerate(self.feat_size):\n\n sk1 = scales[idx]/fig_size\n sk2 = scales[idx+1]/fig_size\n sk3 = np.sqrt(sk1*sk2)\n all_sizes = [(sk1, sk1), (sk3, sk3)]\n\n for alpha in aspect_ratios[idx]:\n w, h = sk1*np.sqrt(alpha), sk1/np.sqrt(alpha)\n all_sizes.append((w, h))\n all_sizes.append((h, w))\n for w, h in all_sizes:\n for i, j in itertools.product(range(sfeat), repeat=2):\n cx, cy = (j+0.5)/fk[idx], (i+0.5)/fk[idx]\n self.default_boxes.append((cx, cy, w, h))\n\n self.dboxes = np.array(self.default_boxes)\n self.dboxes = self.dboxes.clip(min=0, max=1)\n # For IoU calculation\n self.dboxes_ltrb = self.dboxes.copy()\n self.dboxes_ltrb[:, 0] = self.dboxes[:, 0] - 0.5 * self.dboxes[:, 2]\n self.dboxes_ltrb[:, 1] = self.dboxes[:, 1] - 0.5 * self.dboxes[:, 3]\n self.dboxes_ltrb[:, 2] = self.dboxes[:, 0] + 0.5 * self.dboxes[:, 2]\n self.dboxes_ltrb[:, 3] = self.dboxes[:, 1] + 0.5 * self.dboxes[:, 3]\n\n @property\n def scale_xy(self):\n return self.scale_xy_\n\n @property\n def scale_wh(self):\n return self.scale_wh_\n\n def __call__(self, order=\"ltrb\"):\n if order == \"ltrb\": return self.dboxes_ltrb\n if order == \"xywh\": return self.dboxes\n\n\ndef dboxes300_coco():\n figsize = 300\n feat_size = [38, 19, 10, 5, 3, 1]\n steps = [8, 16, 32, 64, 100, 300]\n # use the scales here: https://github.com/amdegroot/ssd.pytorch/blob/master/data/config.py\n scales = [21, 45, 99, 153, 207, 261, 315]\n aspect_ratios = [[2], [2, 3], [2, 3], [2, 3], [2], [2]]\n dboxes = DefaultBoxes(figsize, feat_size, steps, scales, aspect_ratios)\n return dboxes\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--torch_checkpoint', required=True, help='Path to PyTorch SSD300 model checkpoint')\n parser.add_argument('--output_saved_model', required=True, help='Output TensorFlow SavedModel that runs on Inferentia')\n parser.add_argument('--disable_version_check', action='store_true')\n args = parser.parse_args()\n if os.path.exists(args.output_saved_model):\n raise OSError('SavedModel dir {} already exists'.format(args.output_saved_model))\n\n if not args.disable_version_check:\n neuroncc_version = LooseVersion(pkg_resources.get_distribution('neuron-cc').version)\n if neuroncc_version < LooseVersion('1.0.18000'):\n raise RuntimeError(\n 'neuron-cc version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U neuron-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com\"'.format(neuroncc_version))\n tfn_version = LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version)\n if tfn_version < LooseVersion('1.15.3.1.0.1900.0'):\n raise RuntimeError(\n 'tensorflow-neuron version {} is too low for this demo. Please upgrade '\n 'by \"pip install -U tensorflow-neuron --extra-index-url=https://pip.repos.neuron.amazonaws.com\"'.format(tfn_version))\n\n sys.path.append(os.getcwd())\n from DeepLearningExamples.PyTorch.Detection.SSD.src import model as torch_ssd300_model\n ssd300_torch = torch_ssd300_model.SSD300()\n ckpt = torch.load(args.torch_checkpoint, map_location=torch.device('cpu'))\n ssd300_torch.load_state_dict(ckpt['model'])\n ssd300_torch.eval()\n\n input_tensor = tf.placeholder(tf.string, [None])\n image_tensor, bbox_scale_hw_tensor = preprocessor(input_tensor, [300, 300])\n\n dboxes = dboxes300_coco()\n dboxes_xywh = dboxes(order=\"xywh\")[np.newaxis, ...]\n\n ploc_tensor, plabel_tensor = tf_ssd300(image_tensor, ssd300_torch)\n boxes_tensor, scores_tensor, classes_tensor = postprocessor(\n ploc_tensor, plabel_tensor, bbox_scale_hw_tensor, dboxes.scale_xy, dboxes.scale_wh, dboxes_xywh)\n outputs = {\n 'boxes': boxes_tensor,\n 'scores': scores_tensor,\n 'classes': classes_tensor,\n }\n\n sess = tf.Session()\n try:\n sess.run(outputs)\n except:\n pass\n\n for op in sess.graph.get_operations():\n if op.type == 'NeuronOp':\n if not op.get_attr('executable'):\n raise AttributeError(\n 'Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly '\n '(\"pip install neuron-cc --force --extra-index-url=https://pip.repos.neuron.amazonaws.com\" '\n 'to force reinstall neuron-cc).')\n model_config = op.node_def.attr['model_config'].list\n if model_config.i:\n model_config.i[0] = 1\n else:\n model_config.i.extend([1, 1, 1, 10])\n op._set_attr('model_config', attr_value_pb2.AttrValue(list=model_config))\n tf.saved_model.simple_save(sess, args.output_saved_model, {'batch_image': input_tensor}, outputs)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/tensorflow-neuronx/tfneuronx-roberta-base-tutorial.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e91cf83b\",\n \"metadata\": {},\n \"source\": [\n \"# Running Huggingface Roberta-Base with TensorFlow-NeuronX\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"71394e1e\",\n \"metadata\": {},\n \"source\": [\n \"This tutorial demonstrates how to compile the Huggingface roberta-base model and infer on a trn1.2xlarge instance with \\n\",\n \"```tensorflow-neuronx```. To compile larger models like roberta-large, please consider using an inf2 instance.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"828ef9bd\",\n \"metadata\": {},\n \"source\": [\n \"## Setup\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5becc549\",\n \"metadata\": {},\n \"source\": [\n \"To run this tutorial please follow the instructions for [TensorFlow-NeuronX Setup](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/tensorflow/tensorflow-neuronx/setup/tensorflow-neuronx-install.html) and the [Jupyter Notebook Quickstart](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html) and set your kernel to \\\"Python (tensorflow-neuronx)\\\".\\n\",\n \"\\n\",\n \"Next, install some additional dependencies.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ee1a3b84\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%env TOKENIZERS_PARALLELISM=True #Supresses tokenizer warnings making errors easier to detect\\n\",\n \"!pip install transformers\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"c301cfce\",\n \"metadata\": {},\n \"source\": [\n \"## Download From Huggingface and Compile for AWS-Neuron\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"92e8050d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import tensorflow as tf\\n\",\n \"import tensorflow_neuronx as tfnx\\n\",\n \"from transformers import RobertaTokenizer, TFRobertaModel\\n\",\n \"from transformers import BertTokenizer, TFBertModel\\n\",\n \"\\n\",\n \"# Create a wrapper for the roberta model that will accept inputs as a list\\n\",\n \"# instead of a dictionary. This will allow the compiled model to be saved\\n\",\n \"# to disk with the model.save() fucntion.\\n\",\n \"class RobertaWrapper(tf.keras.Model):\\n\",\n \" def __init__(self, model):\\n\",\n \" super().__init__()\\n\",\n \" self.model = model\\n\",\n \" def __call__(self, example_inputs):\\n\",\n \" return self.model({'input_ids' : example_inputs[0], 'attention_mask' : example_inputs[1]})\\n\",\n \" \\n\",\n \"\\n\",\n \"tokenizer = RobertaTokenizer.from_pretrained('roberta-base')\\n\",\n \"model = RobertaWrapper(TFRobertaModel.from_pretrained('roberta-base'))\\n\",\n \"\\n\",\n \"batch_size = 16\\n\",\n \"\\n\",\n \"# create example inputs with a batch size of 16\\n\",\n \"text = [\\\"Paris is the of France.\\\"] * batch_size\\n\",\n \"encoded_input = tokenizer(text, return_tensors='tf', padding='max_length', max_length=64)\\n\",\n \"\\n\",\n \"# turn inputs into a list\\n\",\n \"example_input = [encoded_input['input_ids'], encoded_input['attention_mask']]\\n\",\n \"\\n\",\n \"#compile\\n\",\n \"model_neuron = tfnx.trace(model, example_input)\\n\",\n \"\\n\",\n \"print(\\\"Running on neuron:\\\", model_neuron(example_input))\\n\",\n \"\\n\",\n \"# save the model to disk to save recompilation time for next usage\\n\",\n \"model_neuron.save('./roberta-neuron-b16')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0f2e159a\",\n \"metadata\": {},\n \"source\": [\n \"## Run Basic Inference Benchmarking\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ccf22e74\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import numpy as np\\n\",\n \"import concurrent.futures\\n\",\n \"import time\\n\",\n \"\\n\",\n \"reloaded_neuron_model = tf.keras.models.load_model('./roberta-neuron-b16')\\n\",\n \"print(\\\"Reloaded model running on neuron:\\\", reloaded_neuron_model(example_input))\\n\",\n \"\\n\",\n \"num_threads = 4\\n\",\n \"num_inferences = 1000\\n\",\n \"\\n\",\n \"latency_list = []\\n\",\n \"def inference_with_latency_calculation(example_input):\\n\",\n \" global latency_list\\n\",\n \" start = time.time()\\n\",\n \" result = reloaded_neuron_model(example_input)\\n\",\n \" end = time.time()\\n\",\n \" latency_list.append((end-start) * 1000)\\n\",\n \" return result\\n\",\n \"\\n\",\n \"start = time.time()\\n\",\n \"with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:\\n\",\n \" futures = []\\n\",\n \" for i in range(num_inferences):\\n\",\n \" futures.append(executor.submit(inference_with_latency_calculation, example_input))\\n\",\n \" for future in concurrent.futures.as_completed(futures):\\n\",\n \" get_result = future.result()\\n\",\n \"end = time.time()\\n\",\n \"\\n\",\n \"total_time = end - start\\n\",\n \"\\n\",\n \"print(f\\\"Throughput was {(num_inferences * batch_size)/total_time} samples per second.\\\")\\n\",\n \"print(f\\\"Latency p50 was {np.percentile(latency_list, 50)} ms\\\")\\n\",\n \"print(f\\\"Latency p90 was {np.percentile(latency_list, 90)} ms\\\")\\n\",\n \"print(f\\\"Latency p99 was {np.percentile(latency_list, 99)} ms\\\")\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python (Neuron TensorFlow)\",\n \"language\": \"python\",\n \"name\": \"aws_neuron_venv_tf\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.10\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "src/examples/tensorflow/tensorflow_resnet50/resnet50.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"colab_type\": \"text\",\n \"id\": \"a3bskVXPvchm\"\n },\n \"source\": [\n \"# Running ResNet50 on Inferentia\\n\",\n \"## Note: this tutorial runs on tensorflow-neuron 1.x only\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"colab_type\": \"text\"\n },\n \"source\": [\n \"## Introduction:\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"colab_type\": \"text\",\n \"id\": \"Rb5rSpcZvYbX\"\n },\n \"source\": [\n \"In this tutorial we will compile and deploy ResNet50 model for Inferentia.\\n\",\n \"In this tutorial we provide two main sections:\\n\",\n \"1. Compile the ResNet50 model.\\n\",\n \"2. Infer the same compiled model.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [Tensorflow Installation Guide](../../../../frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install.html#install-neuron-tensorflow). You can select the Kernel from the “Kernel -> Change Kernel” option on the top of this Jupyter notebook page.\\n\",\n \"\\n\",\n \"Instructions of how to setup Neuron Tensorflow environment and run the tutorial as a Jupyter notebook are available in the [Tensorflow Quick Setup](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/tensorflow/tensorflow-neuron/tutorials/tensorflow-tutorial-setup.html#tensorflow-tutorial-setup)\\n\",\n \"\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"shellscript\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install tensorflow_neuron==1.15.5.2.8.9.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com/\\n\",\n \"!pip install neuron_cc==1.13.5.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"colab_type\": \"text\",\n \"id\": \"E8FhiMivhcYB\"\n },\n \"source\": [\n \"## Compile for Neuron\\n\",\n \"\\n\",\n \"A trained model must be compiled to Inferentia target before it can be deployed on Inferentia instances. In this step we compile the Keras ResNet50 model and export it as a SavedModel which is an interchange format for TensorFlow models.\\n\",\n \"At the end of compilation, the compiled SavedModel is saved in resnet50_neuron local directory:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import time\\n\",\n \"import shutil\\n\",\n \"import tensorflow as tf\\n\",\n \"import tensorflow.neuron as tfn\\n\",\n \"import tensorflow.compat.v1.keras as keras\\n\",\n \"from tensorflow.keras.applications.resnet50 import ResNet50\\n\",\n \"from tensorflow.keras.applications.resnet50 import preprocess_input\\n\",\n \"\\n\",\n \"# Create a workspace\\n\",\n \"WORKSPACE = './ws_resnet50'\\n\",\n \"os.makedirs(WORKSPACE, exist_ok=True)\\n\",\n \"\\n\",\n \"# Prepare export directory (old one removed)\\n\",\n \"model_dir = os.path.join(WORKSPACE, 'resnet50')\\n\",\n \"compiled_model_dir = os.path.join(WORKSPACE, 'resnet50_neuron')\\n\",\n \"shutil.rmtree(model_dir, ignore_errors=True)\\n\",\n \"shutil.rmtree(compiled_model_dir, ignore_errors=True)\\n\",\n \"\\n\",\n \"# Instantiate Keras ResNet50 model\\n\",\n \"keras.backend.set_learning_phase(0)\\n\",\n \"keras.backend.set_image_data_format('channels_last')\\n\",\n \"\\n\",\n \"model = ResNet50(weights='imagenet')\\n\",\n \"\\n\",\n \"# Export SavedModel\\n\",\n \"tf.saved_model.simple_save(\\n\",\n \" session = keras.backend.get_session(),\\n\",\n \" export_dir = model_dir,\\n\",\n \" inputs = {'input': model.inputs[0]},\\n\",\n \" outputs = {'output': model.outputs[0]})\\n\",\n \"\\n\",\n \"# Compile using Neuron\\n\",\n \"tfn.saved_model.compile(model_dir, compiled_model_dir)\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!ls\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"colab_type\": \"text\",\n \"id\": \"I52jQOyO8vAn\"\n },\n \"source\": [\n \"## Deploy on Inferentia\\n\",\n \"\\n\",\n \"Using same instance to deploy the model.\\n\",\n \"In case of different deployment instance, launch a deployment inf1 instance and copy compiled model to the deployment inf1 instance.\\n\",\n \"\\n\",\n \"Download the example image, and install pillow module for inference on deployement instance\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!curl -O https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg\\n\",\n \"!pip install pillow # Necessary for loading images\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### After downloading the example image, run the inference.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import time\\n\",\n \"import numpy as np\\n\",\n \"import tensorflow as tf\\n\",\n \"from tensorflow.keras.preprocessing import image\\n\",\n \"from tensorflow.keras.applications import resnet50\\n\",\n \"\\n\",\n \"tf.keras.backend.set_image_data_format('channels_last')\\n\",\n \"\\n\",\n \"# Create input from image\\n\",\n \"img_sgl = image.load_img('kitten_small.jpg', target_size=(224, 224))\\n\",\n \"img_arr = image.img_to_array(img_sgl)\\n\",\n \"img_arr2 = np.expand_dims(img_arr, axis=0)\\n\",\n \"img_arr3 = resnet50.preprocess_input(img_arr2)\\n\",\n \"\\n\",\n \"# Load model\\n\",\n \"COMPILED_MODEL_DIR = './ws_resnet50/resnet50_neuron/'\\n\",\n \"predictor_inferentia = tf.contrib.predictor.from_saved_model(COMPILED_MODEL_DIR)\\n\",\n \"\\n\",\n \"# Run inference\\n\",\n \"model_feed_dict={'input': img_arr3}\\n\",\n \"infa_rslts = predictor_inferentia(model_feed_dict);\\n\",\n \"\\n\",\n \"# Display results\\n\",\n \"print(resnet50.decode_predictions(infa_rslts[\\\"output\\\"], top=5)[0])\\n\",\n \"\\n\",\n \"# Sample output will look like below:\\n\",\n \"#[('n02123045', 'tabby', 0.68817204), ('n02127052', 'lynx', 0.12701613), ('n02123159', 'tiger_cat', 0.08736559), ('n02124075', 'Egyptian_cat', 0.063844085), ('n02128757', 'snow_leopard', 0.009240591)]\"\n ]\n }\n ],\n \"metadata\": {\n \"colab\": {\n \"default_view\": {},\n \"name\": \"Untitled\",\n \"provenance\": [],\n \"version\": \"0.3.2\",\n \"views\": {}\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.2\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n" }, { "path": "src/examples/tensorflow/tensorflow_serving_tutorial.rst", "content": ".. _tensorflow-serving-neuronrt-visible-cores:\n\nUsing NEURON_RT_VISIBLE_CORES with TensorFlow Serving\n=====================================================\n\nTensorFlow serving allows customers to scale-up inference workloads\nacross a network. TensorFlow Neuron Serving uses the same API as normal\nTensorFlow Serving with two differences: (a) the saved model must be\ncompiled for Inferentia and (b) the entry point is a different binary\nnamed ``tensorflow_model_server_neuron``. Follow the steps below \nto install the package using apt-get or yum. This will be pre-installed in a future relase.\n\nInstall TensorFlow Model Server and Serving API\n-----------------------------------------------\n\nFollow the steps in the :ref:`install-neuron-tensorflow`.\n\nThen ensure you install using either apt-get or yum.\nIf using TF 1.x, install the appropriate version (see above).:\n\n.. code:: bash\n\n sudo apt-get install tensorflow-model-server-neuron\n\nor\n\n.. code:: bash\n\n sudo dnf install tensorflow-model-server-neuron\n\nAlso, you would need TensorFlow Serving API (use --no-deps to prevent\ninstallation of regular tensorflow). Depending on the version of Tensorflow\nyou wish to use:\n\nFor Tensorflow 1.x:\n\n.. code:: bash\n\n pip install --no-deps tensorflow_serving_api==1.15\n\nFor Tensorflow 2.x:\n\n.. code:: bash\n\n pip install --no-deps tensorflow_serving_api\n\nFor the example image preprocessing using Keras preprocessing, the\nPython Imaging Library Pillow is required:\n\n.. code:: bash\n\n pip install pillow\n\nTo workaround h5py issue https://github.com/aws/aws-neuron-sdk/issues/220:\n\n.. code:: bash\n\n pip install \"h5py<3.0.0\"\n\n\nExport and Compile Saved Model\n------------------------------\n\nThe following example shows graph construction followed by the addition\nof Neuron compilation step before exporting to saved model.\n\nFor Tensorflow 1.x:\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow.neuron\n\n tf.keras.backend.set_learning_phase(0)\n tf.keras.backend.set_image_data_format('channels_last')\n model = tf.keras.applications.ResNet50(weights='imagenet')\n sess = tf.keras.backend.get_session()\n inputs = {'input': model.inputs[0]}\n outputs = {'output': model.outputs[0]}\n\n # save the model using tf.saved_model.simple_save\n modeldir = \"./resnet50/1\"\n tf.saved_model.simple_save(sess, modeldir, inputs, outputs)\n\n # compile the model for Inferentia\n neuron_modeldir = \"./resnet50_inf1/1\"\n tf.neuron.saved_model.compile(modeldir, neuron_modeldir, batch_size=1)\n\nFor Tensorflow 2.x:\n\n.. code:: python\n\n import tensorflow as tf\n import tensorflow.neuron as tfn\n import numpy as np\n\n tf.keras.backend.set_learning_phase(0)\n tf.keras.backend.set_image_data_format('channels_last')\n image_sizes = [224, 224]\n model = tf.keras.applications.ResNet50(weights='imagenet')\n example_inputs = tf.random.uniform([1, *image_sizes, 3], dtype=tf.float32)\n\n # run the model once to define the forward pass and allow for saving\n model_neuron(example_inputs)\n model_neuron = tfn.trace(model, example_inputs)\n tf.keras.models.save_model(model_neuron, './resnet50_inf1/1')\n\n\n\nServing Saved Model\n-------------------\n\nUser can now serve the saved model with the\ntensorflow_model_server_neuron binary. To utilize multiple NeuronCores,\nit is recommended to launch multiple tensorflow model servers that\nlisten to the same gRPC port:\n\n.. code:: bash\n\n export NEURON_RT_VISIBLE_CORES=0 # important to set this environment variable before launching model servers\n tensorflow_model_server_neuron --model_name=resnet50_inf1 \\\n --model_base_path=$(pwd)/resnet50_inf1/ --port=8500\n\n #then to run another server on a different neuron core open another\n #window and run this, except this time set NEURON_RT_VISIBLE_CORES=1\n #you can keep doing this up to the number of Neuron Cores on your machine\n\n export NEURON_RT_VISIBLE_CORES=1\n tensorflow_model_server_neuron --model_name=resnet50_inf1 \\\n --model_base_path=$(pwd)/resnet50_inf1/ --port=8500\n\nThe compiled model is staged in Inferentia DRAM by the server to prepare\nfor inference.\n\nGenerate inference requests to the model server\n-----------------------------------------------\n\nNow run inferences via GRPC as shown in the following sample client\ncode:\n\nFor Tensorflow 1.x:\n\n.. code:: python\n\n import numpy as np\n import grpc\n import tensorflow as tf\n from tensorflow.keras.preprocessing import image\n from tensorflow.keras.applications.resnet50 import preprocess_input\n from tensorflow.keras.applications.resnet50 import decode_predictions\n from tensorflow_serving.apis import predict_pb2\n from tensorflow_serving.apis import prediction_service_pb2_grpc\n\n if __name__ == '__main__':\n channel = grpc.insecure_channel('localhost:8500')\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n img_file = tf.keras.utils.get_file(\n \"./kitten_small.jpg\",\n \"https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg\")\n img = image.load_img(img_file, target_size=(224, 224))\n img_array = preprocess_input(image.img_to_array(img)[None, ...])\n request = predict_pb2.PredictRequest()\n request.model_spec.name = 'resnet50_inf1'\n request.inputs['input'].CopyFrom(\n tf.contrib.util.make_tensor_proto(img_array, shape=img_array.shape))\n result = stub.Predict(request)\n prediction = tf.make_ndarray(result.outputs['output'])\n print(decode_predictions(prediction))\n\nFor Tensorflow 2.x:\n\n.. code:: python\n\n import numpy as np\n import grpc\n import tensorflow as tf\n from tensorflow.keras.preprocessing import image\n from tensorflow.keras.applications.resnet50 import preprocess_input\n from tensorflow_serving.apis import predict_pb2\n from tensorflow_serving.apis import prediction_service_pb2_grpc\n from tensorflow.keras.applications.resnet50 import decode_predictions\n\n tf.keras.backend.set_image_data_format('channels_last')\n\n if __name__ == '__main__':\n channel = grpc.insecure_channel('localhost:8500')\n stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n img_file = tf.keras.utils.get_file(\n \"./kitten_small.jpg\",\n \"https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/kitten_small.jpg\")\n img = image.load_img(img_file, target_size=(224, 224))\n img_array = preprocess_input(image.img_to_array(img)[None, ...])\n request = predict_pb2.PredictRequest()\n request.model_spec.name = 'resnet50_inf1'\n request.inputs['input_1'].CopyFrom(\n tf.make_tensor_proto(img_array, shape=img_array.shape))\n result = stub.Predict(request)\n prediction = tf.make_ndarray(result.outputs['output_1'])\n print(decode_predictions(prediction))\n" }, { "path": "src/examples/tensorflow/yolo_v3_demo/yolo_v3.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# [Broken] Evaluate YOLO v3 on Inferentia\\n\",\n \"## Note: this tutorial runs on tensorflow-neuron 1.x only\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction\\n\",\n \"This tutorial walks through compiling and evaluating YOLO v3 model on Inferentia using the AWS Neuron SDK.\\n\",\n \"\\n\",\n \"\\n\",\n \"In this tutorial we provide two main sections:\\n\",\n \"\\n\",\n \"1. Download Dataset and Generate Pretrained SavedModel\\n\",\n \"\\n\",\n \"2. Compile the YOLO v3 model.\\n\",\n \"\\n\",\n \"3. Deploy the same compiled model.\\n\",\n \"\\n\",\n \"Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [Tensorflow Installation Guide](../../../../frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install.html#install-neuron-tensorflow). You can select the Kernel from the “Kernel -> Change Kernel” option on the top of this Jupyter notebook page.\\n\",\n \"\\n\",\n \"Instructions of how to setup Neuron Tensorflow environment and run the tutorial as a Jupyter notebook are available in the Tutorial main page [Tensorflow-YOLO_v3 Tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/tensorflow/tensorflow-neuron/tutorials/yolo_v3_demo/yolo_v3_demo.html)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Prerequisites\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"This demo requires the following pip packages:\\n\",\n \"\\n\",\n \"`pillow matplotlib pycocotools`\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"%pip install tensorflow_neuron==1.15.5.2.8.9.0 neuron_cc==1.13.5.0 requests pillow matplotlib pycocotools==2.0.1 numpy==1.18.2 torch~=1.5.0 --force \\\\\\n\",\n \" --extra-index-url=https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 1: Download Dataset and Generate Pretrained SavedModel\\n\",\n \"### Download COCO 2017 validation dataset\\n\",\n \"\\n\",\n \"We start by downloading the COCO validation dataset, which we will use to validate our model. The COCO 2017 dataset is widely used for object-detection, segmentation and image captioning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"!curl -LO http://images.cocodataset.org/zips/val2017.zip\\n\",\n \"!curl -LO http://images.cocodataset.org/annotations/annotations_trainval2017.zip\\n\",\n \"!unzip -q val2017.zip\\n\",\n \"!unzip annotations_trainval2017.zip\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!ls\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"## Generate YOLO v3 tensorflow SavedModel (pretrained on COCO 2017 dataset)\\n\",\n \"\\n\",\n \"Script yolo_v3_coco_saved_model.py will generate a tensorflow SavedModel using pretrained weights from https://github.com/YunYang1994/tensorflow-yolov3/releases/download/v1.0/yolov3_coco.tar.gz.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%run yolo_v3_coco_saved_model.py ./yolo_v3_coco_saved_model\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"This tensorflow SavedModel can be loaded as a tensorflow predictor. When a JPEG format image is provided as input, the output result of the tensorflow predictor contains information for drawing bounding boxes and classification results.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"import tensorflow as tf\\n\",\n \"from PIL import Image\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"import matplotlib.patches as patches\\n\",\n \"\\n\",\n \"# launch predictor and run inference on an arbitrary image in the validation dataset\\n\",\n \"yolo_pred_cpu = tf.contrib.predictor.from_saved_model('./yolo_v3_coco_saved_model')\\n\",\n \"image_path = './val2017/000000581781.jpg'\\n\",\n \"with open(image_path, 'rb') as f:\\n\",\n \" feeds = {'image': [f.read()]}\\n\",\n \"results = yolo_pred_cpu(feeds)\\n\",\n \"\\n\",\n \"# load annotations to decode classification result\\n\",\n \"with open('./annotations/instances_val2017.json') as f:\\n\",\n \" annotate_json = json.load(f)\\n\",\n \"label_info = {idx+1: cat['name'] for idx, cat in enumerate(annotate_json['categories'])}\\n\",\n \"\\n\",\n \"# draw picture and bounding boxes\\n\",\n \"fig, ax = plt.subplots(figsize=(10, 10))\\n\",\n \"ax.imshow(Image.open(image_path).convert('RGB'))\\n\",\n \"wanted = results['scores'][0] > 0.1\\n\",\n \"for xyxy, label_no_bg in zip(results['boxes'][0][wanted], results['classes'][0][wanted]):\\n\",\n \" xywh = xyxy[0], xyxy[1], xyxy[2] - xyxy[0], xyxy[3] - xyxy[1]\\n\",\n \" rect = patches.Rectangle((xywh[0], xywh[1]), xywh[2], xywh[3], linewidth=1, edgecolor='g', facecolor='none')\\n\",\n \" ax.add_patch(rect)\\n\",\n \" rx, ry = rect.get_xy()\\n\",\n \" rx = rx + rect.get_width() / 2.0\\n\",\n \" ax.annotate(label_info[label_no_bg + 1], (rx, ry), color='w', backgroundcolor='g', fontsize=10,\\n\",\n \" ha='center', va='center', bbox=dict(boxstyle='square,pad=0.01', fc='g', ec='none', alpha=0.5))\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 2: Compile the Pretrained SavedModel for Neuron\\n\",\n \"\\n\",\n \"We make use of the Python compilation API `tfn.saved_model.compile` that is available in `tensorflow-neuron<2`. For the purpose of reducing Neuron runtime overhead, it is necessary to make use of arguments `no_fuse_ops` and `minimum_segment_size`.\\n\",\n \"Compiled model is saved in ./yolo_v3_coco_saved_model_neuron.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import shutil\\n\",\n \"import tensorflow as tf\\n\",\n \"import tensorflow.neuron as tfn\\n\",\n \"\\n\",\n \"\\n\",\n \"def no_fuse_condition(op):\\n\",\n \" return op.name.startswith('Preprocessor') or op.name.startswith('Postprocessor')\\n\",\n \"\\n\",\n \"with tf.Session(graph=tf.Graph()) as sess:\\n\",\n \" tf.saved_model.loader.load(sess, ['serve'], './yolo_v3_coco_saved_model')\\n\",\n \" no_fuse_ops = [op.name for op in sess.graph.get_operations() if no_fuse_condition(op)]\\n\",\n \"shutil.rmtree('./yolo_v3_coco_saved_model_neuron', ignore_errors=True)\\n\",\n \"result = tfn.saved_model.compile(\\n\",\n \" './yolo_v3_coco_saved_model', './yolo_v3_coco_saved_model_neuron',\\n\",\n \" # to enforce trivial compilable subgraphs to run on CPU\\n\",\n \" no_fuse_ops=no_fuse_ops,\\n\",\n \" minimum_segment_size=100,\\n\",\n \" batch_size=2,\\n\",\n \" dynamic_batch_size=True,\\n\",\n \")\\n\",\n \"print(result)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Deploy the model on Inferentia\\n\",\n \"## Part 3:Evaluate Model Quality after Compilation\\n\",\n \"\\n\",\n \"### Define evaluation functions\\n\",\n \"We first define some handy helper functions for running evaluation on the COCO 2017 dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import json\\n\",\n \"import time\\n\",\n \"import numpy as np\\n\",\n \"import tensorflow as tf\\n\",\n \"from pycocotools.coco import COCO\\n\",\n \"from pycocotools.cocoeval import COCOeval\\n\",\n \"\\n\",\n \"\\n\",\n \"def cocoapi_eval(jsonfile,\\n\",\n \" style,\\n\",\n \" coco_gt=None,\\n\",\n \" anno_file=None,\\n\",\n \" max_dets=(100, 300, 1000)):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Args:\\n\",\n \" jsonfile: Evaluation json file, eg: bbox.json, mask.json.\\n\",\n \" style: COCOeval style, can be `bbox` , `segm` and `proposal`.\\n\",\n \" coco_gt: Whether to load COCOAPI through anno_file,\\n\",\n \" eg: coco_gt = COCO(anno_file)\\n\",\n \" anno_file: COCO annotations file.\\n\",\n \" max_dets: COCO evaluation maxDets.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" assert coco_gt is not None or anno_file is not None\\n\",\n \"\\n\",\n \" if coco_gt is None:\\n\",\n \" coco_gt = COCO(anno_file)\\n\",\n \" print(\\\"Start evaluate...\\\")\\n\",\n \" coco_dt = coco_gt.loadRes(jsonfile)\\n\",\n \" if style == 'proposal':\\n\",\n \" coco_eval = COCOeval(coco_gt, coco_dt, 'bbox')\\n\",\n \" coco_eval.params.useCats = 0\\n\",\n \" coco_eval.params.maxDets = list(max_dets)\\n\",\n \" else:\\n\",\n \" coco_eval = COCOeval(coco_gt, coco_dt, style)\\n\",\n \" coco_eval.evaluate()\\n\",\n \" coco_eval.accumulate()\\n\",\n \" coco_eval.summarize()\\n\",\n \" return coco_eval.stats\\n\",\n \"\\n\",\n \"\\n\",\n \"def bbox_eval(anno_file, bbox_list):\\n\",\n \" coco_gt = COCO(anno_file)\\n\",\n \"\\n\",\n \" outfile = 'bbox_detections.json'\\n\",\n \" print('Generating json file...')\\n\",\n \" with open(outfile, 'w') as f:\\n\",\n \" json.dump(bbox_list, f)\\n\",\n \"\\n\",\n \" map_stats = cocoapi_eval(outfile, 'bbox', coco_gt=coco_gt)\\n\",\n \" return map_stats\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_image_as_bytes(images, eval_pre_path):\\n\",\n \" batch_im_id_list = []\\n\",\n \" batch_im_name_list = []\\n\",\n \" batch_img_bytes_list = []\\n\",\n \" n = len(images)\\n\",\n \" batch_im_id = []\\n\",\n \" batch_im_name = []\\n\",\n \" batch_img_bytes = []\\n\",\n \" for i, im in enumerate(images):\\n\",\n \" im_id = im['id']\\n\",\n \" file_name = im['file_name']\\n\",\n \" if i % eval_batch_size == 0 and i != 0:\\n\",\n \" batch_im_id_list.append(batch_im_id)\\n\",\n \" batch_im_name_list.append(batch_im_name)\\n\",\n \" batch_img_bytes_list.append(batch_img_bytes)\\n\",\n \" batch_im_id = []\\n\",\n \" batch_im_name = []\\n\",\n \" batch_img_bytes = []\\n\",\n \" batch_im_id.append(im_id)\\n\",\n \" batch_im_name.append(file_name)\\n\",\n \"\\n\",\n \" with open(os.path.join(eval_pre_path, file_name), 'rb') as f:\\n\",\n \" batch_img_bytes.append(f.read())\\n\",\n \" return batch_im_id_list, batch_im_name_list, batch_img_bytes_list\\n\",\n \"\\n\",\n \"\\n\",\n \"def analyze_bbox(results, batch_im_id, _clsid2catid):\\n\",\n \" bbox_list = []\\n\",\n \" k = 0\\n\",\n \" for boxes, scores, classes in zip(results['boxes'], results['scores'], results['classes']):\\n\",\n \" if boxes is not None:\\n\",\n \" im_id = batch_im_id[k]\\n\",\n \" n = len(boxes)\\n\",\n \" for p in range(n):\\n\",\n \" clsid = classes[p]\\n\",\n \" score = scores[p]\\n\",\n \" xmin, ymin, xmax, ymax = boxes[p]\\n\",\n \" catid = (_clsid2catid[int(clsid)])\\n\",\n \" w = xmax - xmin + 1\\n\",\n \" h = ymax - ymin + 1\\n\",\n \"\\n\",\n \" bbox = [xmin, ymin, w, h]\\n\",\n \" # Round to the nearest 10th to avoid huge file sizes, as COCO suggests\\n\",\n \" bbox = [round(float(x) * 10) / 10 for x in bbox]\\n\",\n \" bbox_res = {\\n\",\n \" 'image_id': im_id,\\n\",\n \" 'category_id': catid,\\n\",\n \" 'bbox': bbox,\\n\",\n \" 'score': float(score),\\n\",\n \" }\\n\",\n \" bbox_list.append(bbox_res)\\n\",\n \" k += 1\\n\",\n \" return bbox_list\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Here is the actual evaluation loop. To fully utilize all four cores on one Inferentia, the optimal setup is to run multi-threaded inference using a `ThreadPoolExecutor`. The following cell is a multi-threaded adaptation of the evaluation routine at https://github.com/miemie2013/Keras-YOLOv4/blob/910c4c6f7265f5828fceed0f784496a0b46516bf/tools/cocotools.py#L97.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from concurrent import futures\\n\",\n \"\\n\",\n \"def evaluate(yolo_predictor, images, eval_pre_path, anno_file, eval_batch_size, _clsid2catid):\\n\",\n \" batch_im_id_list, batch_im_name_list, batch_img_bytes_list = get_image_as_bytes(images, eval_pre_path)\\n\",\n \"\\n\",\n \" # warm up\\n\",\n \" yolo_predictor({'image': np.array(batch_img_bytes_list[0], dtype=object)})\\n\",\n \"\\n\",\n \" with futures.ThreadPoolExecutor(4) as exe:\\n\",\n \" fut_im_list = []\\n\",\n \" fut_list = []\\n\",\n \" start_time = time.time()\\n\",\n \" for batch_im_id, batch_im_name, batch_img_bytes in zip(batch_im_id_list, batch_im_name_list, batch_img_bytes_list):\\n\",\n \" if len(batch_img_bytes) != eval_batch_size:\\n\",\n \" continue\\n\",\n \" fut = exe.submit(yolo_predictor, {'image': np.array(batch_img_bytes, dtype=object)})\\n\",\n \" fut_im_list.append((batch_im_id, batch_im_name))\\n\",\n \" fut_list.append(fut)\\n\",\n \" bbox_list = []\\n\",\n \" count = 0\\n\",\n \" for (batch_im_id, batch_im_name), fut in zip(fut_im_list, fut_list):\\n\",\n \" results = fut.result()\\n\",\n \" bbox_list.extend(analyze_bbox(results, batch_im_id, _clsid2catid))\\n\",\n \" for _ in batch_im_id:\\n\",\n \" count += 1\\n\",\n \" if count % 100 == 0:\\n\",\n \" print('Test iter {}'.format(count))\\n\",\n \" print('==================== Performance Measurement ====================')\\n\",\n \" print('Finished inference on {} images in {} seconds'.format(len(images), time.time() - start_time))\\n\",\n \" print('=================================================================')\\n\",\n \" # start evaluation\\n\",\n \" box_ap_stats = bbox_eval(anno_file, bbox_list)\\n\",\n \" return box_ap_stats\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Evaluate mean average precision (mAP) score\\n\",\n \"Here is the code to calculate mAP scores of the YOLO v3 model. The expected mAP score is around 0.328 if we use the pretrained weights.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"yolo_pred = tf.contrib.predictor.from_saved_model('./yolo_v3_coco_saved_model_neuron')\\n\",\n \"\\n\",\n \"val_coco_root = './val2017'\\n\",\n \"val_annotate = './annotations/instances_val2017.json'\\n\",\n \"clsid2catid = {0: 1, 1: 2, 2: 3, 3: 4, 4: 5, 5: 6, 6: 7, 7: 8, 8: 9, 9: 10, 10: 11, 11: 13, 12: 14, 13: 15, 14: 16,\\n\",\n \" 15: 17, 16: 18, 17: 19, 18: 20, 19: 21, 20: 22, 21: 23, 22: 24, 23: 25, 24: 27, 25: 28, 26: 31,\\n\",\n \" 27: 32, 28: 33, 29: 34, 30: 35, 31: 36, 32: 37, 33: 38, 34: 39, 35: 40, 36: 41, 37: 42, 38: 43,\\n\",\n \" 39: 44, 40: 46, 41: 47, 42: 48, 43: 49, 44: 50, 45: 51, 46: 52, 47: 53, 48: 54, 49: 55, 50: 56,\\n\",\n \" 51: 57, 52: 58, 53: 59, 54: 60, 55: 61, 56: 62, 57: 63, 58: 64, 59: 65, 60: 67, 61: 70, 62: 72,\\n\",\n \" 63: 73, 64: 74, 65: 75, 66: 76, 67: 77, 68: 78, 69: 79, 70: 80, 71: 81, 72: 82, 73: 84, 74: 85,\\n\",\n \" 75: 86, 76: 87, 77: 88, 78: 89, 79: 90}\\n\",\n \"eval_batch_size = 8\\n\",\n \"with open(val_annotate, 'r', encoding='utf-8') as f2:\\n\",\n \" for line in f2:\\n\",\n \" line = line.strip()\\n\",\n \" dataset = json.loads(line)\\n\",\n \" images = dataset['images']\\n\",\n \"box_ap = evaluate(yolo_pred, images, val_coco_root, val_annotate, eval_batch_size, clsid2catid)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.9 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.9\"\n },\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/tensorflow/yolo_v3_demo/yolo_v3_coco_saved_model.py", "content": "import argparse\nimport os\nimport urllib.request\nimport tempfile\nimport shutil\nfrom functools import partial\nimport numpy as np\nimport tensorflow as tf\n\n\nSTRIDES = [8, 16, 32]\nANCHORS = np.array([1.25,1.625, 2.0,3.75, 4.125,2.875, 1.875,3.8125, 3.875,2.8125, 3.6875,7.4375, 3.625,2.8125, 4.875,6.1875, 11.65625,10.1875]).astype(np.float32).reshape([3, 3, 2])\nANCHOR_PER_SCALE = 3\nBOX_SCORE_THRESH = 0.3\nUPSAMPLE_METHOD = \"resize\"\nNUM_CLASSES = 80\n\n\nclass YOLOV3(object):\n \"\"\"Implement tensoflow yolov3 here\"\"\"\n def __init__(self, input_data, input_size, trainable):\n\n self.trainable = trainable\n self.num_class = NUM_CLASSES\n self.strides = STRIDES\n self.anchors = ANCHORS\n self.anchor_per_scale = ANCHOR_PER_SCALE\n self.box_score_thresh = BOX_SCORE_THRESH\n self.upsample_method = UPSAMPLE_METHOD\n\n input_data, decoded_shape = preprocessor(input_data, [input_size, input_size])\n self.conv_lbbox, self.conv_mbbox, self.conv_sbbox = self.__build_nework(input_data)\n\n def decode_boxes(bboxes_and_decoded_shape):\n conv_lbbox, conv_mbbox, conv_sbbox, decoded_shape = bboxes_and_decoded_shape\n conv_lbbox = tf.cast(conv_lbbox, tf.float32)\n conv_mbbox = tf.cast(conv_mbbox, tf.float32)\n conv_sbbox = tf.cast(conv_sbbox, tf.float32)\n conv_lbbox = conv_lbbox[tf.newaxis, ...]\n conv_mbbox = conv_mbbox[tf.newaxis, ...]\n conv_sbbox = conv_sbbox[tf.newaxis, ...]\n decoded_shape = decoded_shape[tf.newaxis, ...]\n with tf.variable_scope('pred_sbbox'):\n pred_sbbox_coors, pred_sbbox_class_scores = self.decode(conv_sbbox, self.anchors[0], self.strides[0], decoded_shape, input_size)\n\n with tf.variable_scope('pred_mbbox'):\n pred_mbbox_coors, pred_mbbox_class_scores = self.decode(conv_mbbox, self.anchors[1], self.strides[1], decoded_shape, input_size)\n\n with tf.variable_scope('pred_lbbox'):\n pred_lbbox_coors, pred_lbbox_class_scores = self.decode(conv_lbbox, self.anchors[2], self.strides[2], decoded_shape, input_size)\n\n with tf.variable_scope('pred_bbox_filter'):\n pred_bbox_coors = tf.concat([pred_sbbox_coors, pred_mbbox_coors, pred_lbbox_coors], axis=1)\n pred_bbox_class_scores = tf.concat([pred_sbbox_class_scores, pred_mbbox_class_scores, pred_lbbox_class_scores], axis=1)\n nms_top_k = 100\n nms_thresh= 0.45\n coors, scores, classes, valid_detections = tf.image.combined_non_max_suppression(\n pred_bbox_coors,\n pred_bbox_class_scores,\n max_output_size_per_class=nms_top_k,\n max_total_size=nms_top_k,\n iou_threshold=nms_thresh,\n score_threshold=self.box_score_thresh,\n pad_per_class=False,\n clip_boxes=False,\n name='CombinedNonMaxSuppression',\n )\n scores = scores[..., tf.newaxis]\n classes = classes[..., tf.newaxis]\n return coors[0], scores[0], classes[0]\n\n with tf.name_scope('Postprocessor'):\n coors, scores, classes = tf.map_fn(\n decode_boxes, [self.conv_lbbox, self.conv_mbbox, self.conv_sbbox, decoded_shape],\n dtype=(tf.float32, tf.float32, tf.float32), back_prop=False, parallel_iterations=16)\n\n with tf.variable_scope('pred_bbox'):\n self.pred_bbox_boxes = tf.identity(coors, name='boxes')\n self.pred_bbox_scores = tf.identity(scores[..., 0], name='scores')\n self.pred_bbox_classes = tf.identity(classes[..., 0], name='classes')\n\n def __build_nework(self, input_data):\n route_1, route_2, input_data = darknet53(input_data, self.trainable)\n\n input_data = convolutional(input_data, (1, 1, 1024, 512), self.trainable, 'conv52')\n input_data = convolutional(input_data, (3, 3, 512, 1024), self.trainable, 'conv53')\n input_data = convolutional(input_data, (1, 1, 1024, 512), self.trainable, 'conv54')\n input_data = convolutional(input_data, (3, 3, 512, 1024), self.trainable, 'conv55')\n input_data = convolutional(input_data, (1, 1, 1024, 512), self.trainable, 'conv56')\n\n conv_lobj_branch = convolutional(input_data, (3, 3, 512, 1024), self.trainable, name='conv_lobj_branch')\n conv_lbbox = convolutional(conv_lobj_branch, (1, 1, 1024, 3*(self.num_class + 5)),\n trainable=self.trainable, name='conv_lbbox', activate=False, bn=False)\n\n input_data = convolutional(input_data, (1, 1, 512, 256), self.trainable, 'conv57')\n input_data = upsample(input_data, name='upsample0', method=self.upsample_method)\n\n with tf.variable_scope('route_1'):\n input_data = tf.concat([input_data, route_2], axis=-1)\n\n input_data = convolutional(input_data, (1, 1, 768, 256), self.trainable, 'conv58')\n input_data = convolutional(input_data, (3, 3, 256, 512), self.trainable, 'conv59')\n input_data = convolutional(input_data, (1, 1, 512, 256), self.trainable, 'conv60')\n input_data = convolutional(input_data, (3, 3, 256, 512), self.trainable, 'conv61')\n input_data = convolutional(input_data, (1, 1, 512, 256), self.trainable, 'conv62')\n\n conv_mobj_branch = convolutional(input_data, (3, 3, 256, 512), self.trainable, name='conv_mobj_branch' )\n conv_mbbox = convolutional(conv_mobj_branch, (1, 1, 512, 3*(self.num_class + 5)),\n trainable=self.trainable, name='conv_mbbox', activate=False, bn=False)\n\n input_data = convolutional(input_data, (1, 1, 256, 128), self.trainable, 'conv63')\n input_data = upsample(input_data, name='upsample1', method=self.upsample_method)\n\n with tf.variable_scope('route_2'):\n input_data = tf.concat([input_data, route_1], axis=-1)\n\n input_data = convolutional(input_data, (1, 1, 384, 128), self.trainable, 'conv64')\n input_data = convolutional(input_data, (3, 3, 128, 256), self.trainable, 'conv65')\n input_data = convolutional(input_data, (1, 1, 256, 128), self.trainable, 'conv66')\n input_data = convolutional(input_data, (3, 3, 128, 256), self.trainable, 'conv67')\n input_data = convolutional(input_data, (1, 1, 256, 128), self.trainable, 'conv68')\n\n conv_sobj_branch = convolutional(input_data, (3, 3, 128, 256), self.trainable, name='conv_sobj_branch')\n conv_sbbox = convolutional(conv_sobj_branch, (1, 1, 256, 3*(self.num_class + 5)),\n trainable=self.trainable, name='conv_sbbox', activate=False, bn=False)\n\n return conv_lbbox, conv_mbbox, conv_sbbox\n\n def decode(self, conv_output, anchors, stride, decoded_shape, input_size):\n conv_output = tf.cast(conv_output, tf.float32)\n \"\"\"\n return tensor of shape [batch_size, output_size, output_size, anchor_per_scale, 5 + num_classes]\n contains (x, y, w, h, score, probability)\n \"\"\"\n\n conv_shape = tf.shape(conv_output)\n batch_size = conv_shape[0]\n output_size = conv_shape[1]\n anchor_per_scale = len(anchors)\n\n conv_output = tf.reshape(conv_output, (batch_size, output_size, output_size, anchor_per_scale, 5 + self.num_class))\n\n conv_raw_dxdy = conv_output[:, :, :, :, 0:2]\n conv_raw_dwdh = conv_output[:, :, :, :, 2:4]\n conv_raw_conf = conv_output[:, :, :, :, 4:5]\n conv_raw_prob = conv_output[:, :, :, :, 5: ]\n\n y = tf.tile(tf.range(output_size, dtype=tf.int32)[:, tf.newaxis], [1, output_size])\n x = tf.tile(tf.range(output_size, dtype=tf.int32)[tf.newaxis, :], [output_size, 1])\n\n xy_grid = tf.concat([x[:, :, tf.newaxis], y[:, :, tf.newaxis]], axis=-1)\n xy_grid = tf.tile(xy_grid[tf.newaxis, :, :, tf.newaxis, :], [batch_size, 1, 1, anchor_per_scale, 1])\n xy_grid = tf.cast(xy_grid, tf.float32)\n\n pred_xy = (tf.sigmoid(conv_raw_dxdy) + xy_grid) * stride\n pred_wh = (tf.exp(conv_raw_dwdh) * anchors) * stride\n pred_xywh = tf.concat([pred_xy, pred_wh], axis=-1)\n\n pred_conf = tf.sigmoid(conv_raw_conf)\n pred_prob = tf.sigmoid(conv_raw_prob)\n\n pred_xywh = tf.reshape(pred_xywh, (-1, output_size*output_size*3, pred_xywh.shape[-1]))\n pred_conf = tf.reshape(pred_conf, (-1, output_size*output_size*3))\n pred_prob = tf.reshape(pred_prob, (-1, output_size*output_size*3, pred_prob.shape[-1]))\n\n return tf_postprocess_boxes(pred_xywh, pred_conf, pred_prob, decoded_shape, input_size, self.box_score_thresh)\n\n\ndef darknet53(input_data, trainable):\n\n with tf.variable_scope('darknet'):\n\n input_data = convolutional(input_data, filters_shape=(3, 3, 3, 32), trainable=trainable, name='conv0')\n input_data = convolutional(input_data, filters_shape=(3, 3, 32, 64), trainable=trainable, name='conv1', downsample=True)\n\n for i in range(1):\n input_data = residual_block(input_data, 64, 32, 64, trainable=trainable, name='residual%d' %(i+0))\n\n input_data = convolutional(input_data, filters_shape=(3, 3, 64, 128), trainable=trainable, name='conv4', downsample=True)\n\n for i in range(2):\n input_data = residual_block(input_data, 128, 64, 128, trainable=trainable, name='residual%d' %(i+1))\n\n input_data = convolutional(input_data, filters_shape=(3, 3, 128, 256), trainable=trainable, name='conv9', downsample=True)\n\n for i in range(8):\n input_data = residual_block(input_data, 256, 128, 256, trainable=trainable, name='residual%d' %(i+3))\n\n route_1 = input_data\n input_data = convolutional(input_data, filters_shape=(3, 3, 256, 512), trainable=trainable, name='conv26', downsample=True)\n\n for i in range(8):\n input_data = residual_block(input_data, 512, 256, 512, trainable=trainable, name='residual%d' %(i+11))\n\n route_2 = input_data\n input_data = convolutional(input_data, filters_shape=(3, 3, 512, 1024), trainable=trainable, name='conv43', downsample=True)\n\n for i in range(4):\n input_data = residual_block(input_data, 1024, 512, 1024, trainable=trainable, name='residual%d' %(i+19))\n\n return route_1, route_2, input_data\n\n\ndef convolutional(input_data, filters_shape, trainable, name, downsample=False, activate=True, bn=True):\n\n with tf.variable_scope(name):\n if downsample:\n pad_h, pad_w = (filters_shape[0] - 2) // 2 + 1, (filters_shape[1] - 2) // 2 + 1\n paddings = tf.constant([[0, 0], [pad_h, pad_h], [pad_w, pad_w], [0, 0]])\n input_data = tf.pad(input_data, paddings, 'CONSTANT')\n strides = (1, 2, 2, 1)\n padding = 'VALID'\n else:\n strides = (1, 1, 1, 1)\n padding = \"SAME\"\n\n weight = tf.get_variable(name='weight', dtype=tf.float32, trainable=True,\n shape=filters_shape, initializer=tf.random_normal_initializer(stddev=0.01))\n weight = tf.cast(weight, tf.float16)\n conv = tf.nn.conv2d(input=input_data, filter=weight, strides=strides, padding=padding)\n\n if bn:\n conv = tf.layers.batch_normalization(conv, beta_initializer=tf.zeros_initializer(),\n gamma_initializer=tf.ones_initializer(),\n moving_mean_initializer=tf.zeros_initializer(),\n moving_variance_initializer=tf.ones_initializer(), training=trainable,\n fused=False)\n else:\n bias = tf.get_variable(name='bias', shape=filters_shape[-1], trainable=True,\n dtype=tf.float32, initializer=tf.constant_initializer(0.0))\n bias = tf.cast(bias, tf.float16)\n conv = tf.nn.bias_add(conv, bias)\n\n if activate == True: conv = tf.nn.leaky_relu(conv, alpha=0.1)\n\n return conv\n\n\ndef residual_block(input_data, input_channel, filter_num1, filter_num2, trainable, name):\n short_cut = input_data\n with tf.variable_scope(name):\n input_data = convolutional(input_data, filters_shape=(1, 1, input_channel, filter_num1),\n trainable=trainable, name='conv1')\n input_data = convolutional(input_data, filters_shape=(3, 3, filter_num1, filter_num2),\n trainable=trainable, name='conv2')\n residual_output = input_data + short_cut\n return residual_output\n\n\ndef upsample(input_data, name, method=\"deconv\"):\n assert method in [\"resize\", \"deconv\"]\n\n if method == \"resize\":\n with tf.variable_scope(name):\n input_shape = tf.shape(input_data)\n output = tf.image.resize_nearest_neighbor(input_data, (input_shape[1] * 2, input_shape[2] * 2))\n\n if method == \"deconv\":\n # replace resize_nearest_neighbor with conv2d_transpose To support TensorRT optimization\n numm_filter = input_data.shape.as_list()[-1]\n output = tf.layers.conv2d_transpose(input_data, numm_filter, kernel_size=2, padding='same',\n strides=(2,2), kernel_initializer=tf.random_normal_initializer())\n\n return output\n\n\ndef decode_jpeg_resize(input_tensor, image_size):\n tensor = tf.image.decode_png(input_tensor, channels=3)\n shape = tf.shape(tensor)\n tensor = tf.cast(tensor, tf.float32)\n tensor = tf.image.resize_image_with_pad(tensor, image_size[0], image_size[1])\n tensor /= 255.0\n return tf.cast(tensor, tf.float16), shape\n\n\ndef preprocessor(input_tensor, image_size):\n with tf.name_scope('Preprocessor'):\n batch_tensor, batch_shape = tf.map_fn(\n partial(decode_jpeg_resize, image_size=image_size), input_tensor,\n dtype=(tf.float16, tf.int32), back_prop=False, parallel_iterations=16)\n return batch_tensor, batch_shape\n\n\ndef tf_postprocess_boxes(pred_xywh, pred_conf, pred_prob, org_img_shape, input_size, score_threshold):\n batch_size = tf.shape(pred_xywh)[0]\n\n # # (1) (x, y, w, h) --> (xmin, ymin, xmax, ymax)\n pred_coor = tf.concat([pred_xywh[:, :, :2] - pred_xywh[:, :, 2:] * 0.5,\n pred_xywh[:, :, :2] + pred_xywh[:, :, 2:] * 0.5], axis=-1)\n # # (2) (xmin, ymin, xmax, ymax) -> (xmin_org, ymin_org, xmax_org, ymax_org)\n org_wh = org_img_shape[:, tf.newaxis, 1::-1]\n org_whwh = tf.concat([org_wh, org_wh], axis=-1)\n org_whwh = tf.cast(org_whwh, tf.float32)\n input_size = np.float32(input_size)\n resize_ratio = input_size / tf.reduce_max(org_whwh, axis=-1)\n dwhwh = (input_size - resize_ratio * org_whwh) / 2\n pred_coor = (pred_coor - dwhwh) / resize_ratio\n\n # # (5) discard some boxes with low scores\n scores = pred_conf * tf.reduce_max(pred_prob, axis=-1)\n score_mask = scores > score_threshold\n coors = pred_coor[score_mask]\n pred_conf = pred_conf[score_mask]\n pred_conf = tf.reshape(pred_conf, [batch_size, -1, 1])\n pred_prob = pred_prob[score_mask]\n pred_prob = tf.reshape(pred_prob, [batch_size, -1, pred_prob.shape[-1]])\n class_scores = pred_conf * pred_prob\n coors = tf.reshape(coors, [batch_size, -1, 1, coors.shape[-1]])\n class_scores = tf.reshape(class_scores, [batch_size, -1, class_scores.shape[-1]])\n return coors, class_scores\n\n\ndef convert_weights(org_weights_path, cur_weights_path, input_size):\n org_weights_mess = []\n with tf.Session(graph=tf.Graph()) as sess:\n load = tf.train.import_meta_graph(org_weights_path + '.meta')\n load.restore(sess, org_weights_path)\n for var in tf.global_variables():\n var_name = var.op.name\n var_name_mess = str(var_name).split('/')\n var_shape = var.shape\n org_weights_mess.append([var_name, var_shape])\n print(\"=> \" + str(var_name).ljust(50), var_shape)\n print()\n\n cur_weights_mess = []\n with tf.Session(graph=tf.Graph()) as sess:\n with tf.name_scope('input'):\n input_data = tf.placeholder(dtype=tf.string, shape=(None,), name='input_data')\n training = tf.placeholder(dtype=tf.bool, name='trainable')\n model = YOLOV3(input_data, input_size, training)\n for var in tf.global_variables():\n var_name = var.op.name\n var_name_mess = str(var_name).split('/')\n var_shape = var.shape\n print(var_name_mess[0])\n cur_weights_mess.append([var_name, var_shape])\n print(\"=> \" + str(var_name).ljust(50), var_shape)\n\n org_weights_num = len(org_weights_mess)\n cur_weights_num = len(cur_weights_mess)\n if cur_weights_num != org_weights_num:\n raise RuntimeError\n\n print('=> Number of weights that will rename:\\t%d' % cur_weights_num)\n cur_to_org_dict = {}\n for index in range(org_weights_num):\n org_name, org_shape = org_weights_mess[index]\n cur_name, cur_shape = cur_weights_mess[index]\n if cur_shape != org_shape:\n print(org_weights_mess[index])\n print(cur_weights_mess[index])\n raise RuntimeError\n cur_to_org_dict[cur_name] = org_name\n print(\"=> \" + str(cur_name).ljust(50) + ' : ' + org_name)\n\n with tf.name_scope('load_save'):\n name_to_var_dict = {var.op.name: var for var in tf.global_variables()}\n restore_dict = {cur_to_org_dict[cur_name]: name_to_var_dict[cur_name] for cur_name in cur_to_org_dict}\n load = tf.train.Saver(restore_dict)\n save = tf.train.Saver(tf.global_variables())\n for var in tf.global_variables():\n print(\"=> \" + var.op.name)\n\n sess.run(tf.global_variables_initializer())\n print('=> Restoring weights from:\\t %s' % org_weights_path)\n load.restore(sess, org_weights_path)\n save.save(sess, cur_weights_path)\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('model_dir')\n args = parser.parse_args()\n if os.path.exists(args.model_dir):\n raise OSError('Directory {} already exists; please specify a different path for the tensorflow SavedModel'.format(args.model_dir))\n with tempfile.TemporaryDirectory() as workdir:\n ckpt_file = os.path.join(workdir, './yolov3_coco_demo.ckpt')\n input_size = 416\n if not os.path.isfile(ckpt_file + '.meta'):\n yolov3_coco_tar_gz = os.path.join(workdir, './yolov3_coco.tar.gz')\n url = 'https://github.com/YunYang1994/tensorflow-yolov3/releases/download/v1.0/yolov3_coco.tar.gz'\n print('Downloading from {}'.format(url))\n urllib.request.urlretrieve(url, yolov3_coco_tar_gz)\n shutil.unpack_archive(yolov3_coco_tar_gz, extract_dir=workdir)\n convert_weights(os.path.join(workdir, './yolov3_coco.ckpt'), ckpt_file, input_size)\n\n input_tensor_name = 'input/input_data:0'\n output_names = ['boxes', 'scores', 'classes']\n output_tensor_names = ['pred_bbox/boxes:0', 'pred_bbox/scores:0', 'pred_bbox/classes:0']\n with tf.Session(graph=tf.Graph()) as sess:\n with tf.name_scope('input'):\n input_data = tf.placeholder(dtype=tf.string, shape=[None], name='input_data')\n model = YOLOV3(input_data, input_size, trainable=False)\n print(model.conv_sbbox, model.conv_mbbox, model.conv_lbbox)\n saver = tf.train.Saver()\n saver.restore(sess, ckpt_file)\n input_tensor = sess.graph.get_tensor_by_name(input_tensor_name)\n inputs = {'image': input_tensor}\n outputs = {name: sess.graph.get_tensor_by_name(tensor_name) for name, tensor_name in zip(output_names, output_tensor_names)}\n tf.saved_model.simple_save(sess, args.model_dir, inputs, outputs)\n print('tensorflow YOLO v3 SavedModel generated at {}'.format(args.model_dir))\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/examples/tensorflow/yolo_v4_demo/README.md", "content": "
\n
\n\nPlease view our documentation at **[https://awsdocs-neuron.readthedocs-hosted.com/](https://awsdocs-neuron.readthedocs-hosted.com/)** \n\n" }, { "path": "src/examples/tensorflow/yolo_v4_demo/evaluate.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Evaluate YOLO v4 on Inferentia\\n\",\n \"## Note: this tutorial runs on tensorflow-neuron 1.x only\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Introduction\\n\",\n \"This tutorial walks through compiling and evaluating YOLO v4 model on Inferentia using the AWS Neuron SDK 09/2020 release. We recommend running this tutorial on an EC2 `inf1.2xlarge` instance which contains one Inferentia and 8 vCPU cores, as well as 16 GB of memory.Verify that this Jupyter notebook is running the Python kernel environment that was set up according to the [Tensorflow Installation Guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/tensorflow/tensorflow-neuron/setup/tensorflow-install.html#install-neuron-tensorflow) You can select the Kernel from the “Kernel -> Change Kernel” option on the top of this Jupyter notebook page.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Prerequisites\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"This demo requires the following pip packages:\\n\",\n \"\\n\",\n \"`neuron-cc tensorflow-neuron<2 requests pillow matplotlib pycocotools torch`\\n\",\n \"\\n\",\n \"and debian/rpm package `aws-neuron-runtime`.\\n\",\n \"\\n\",\n \"On DLAMI, `aws-neuron-runtime` is already pre-installed.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install tensorflow_neuron==1.15.5.2.8.9.0 neuron_cc==1.13.5.0 requests pillow matplotlib pycocotools==2.0.1 numpy==1.18.2 torch~=1.5.0 --force \\\\\\n\",\n \" --extra-index-url=https://pip.repos.neuron.amazonaws.com\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 1: Download Dataset and Generate Pretrained SavedModel\\n\",\n \"### Download COCO 2017 validation dataset\\n\",\n \"We start by downloading the COCO validation dataset, which we will use to validate our model. The COCO 2017 dataset is widely used for object-detection, segmentation and image captioning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!curl -LO http://images.cocodataset.org/zips/val2017.zip\\n\",\n \"!curl -LO http://images.cocodataset.org/annotations/annotations_trainval2017.zip\\n\",\n \"!unzip -q val2017.zip\\n\",\n \"!unzip annotations_trainval2017.zip\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!ls\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Check required package versions\\n\",\n \"Here are the minimum required versions of AWS Neuron packages. We run a check.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pkg_resources\\n\",\n \"from distutils.version import LooseVersion\\n\",\n \"\\n\",\n \"assert LooseVersion(pkg_resources.get_distribution('neuron-cc').version) > LooseVersion('1.0.20000')\\n\",\n \"assert LooseVersion(pkg_resources.get_distribution('tensorflow-neuron').version) > LooseVersion('1.15.3.1.0.2000')\\n\",\n \"print('passed package version checks')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Generate YOLO v4 tensorflow SavedModel (pretrained on COCO 2017 dataset)\\n\",\n \"Script `yolo_v4_coco_saved_model.py` will generate a tensorflow SavedModel using pretrained weights from https://github.com/Tianxiaomo/pytorch-YOLOv4.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!python3 yolo_v4_coco_saved_model.py\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"This tensorflow SavedModel can be loaded as a tensorflow predictor. When a JPEG format image is provided as input, the output result of the tensorflow predictor contains information for drawing bounding boxes and classification results.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"import tensorflow as tf\\n\",\n \"from PIL import Image\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"import matplotlib.patches as patches\\n\",\n \"\\n\",\n \"# launch predictor and run inference on an arbitrary image in the validation dataset\\n\",\n \"yolo_pred_cpu = tf.contrib.predictor.from_saved_model('./yolo_v4_coco_saved_model')\\n\",\n \"image_path = './val2017/000000581781.jpg'\\n\",\n \"with open(image_path, 'rb') as f:\\n\",\n \" feeds = {'image': [f.read()]}\\n\",\n \"results = yolo_pred_cpu(feeds)\\n\",\n \"\\n\",\n \"# load annotations to decode classification result\\n\",\n \"with open('./annotations/instances_val2017.json') as f:\\n\",\n \" annotate_json = json.load(f)\\n\",\n \"label_info = {idx+1: cat['name'] for idx, cat in enumerate(annotate_json['categories'])}\\n\",\n \"\\n\",\n \"# draw picture and bounding boxes\\n\",\n \"fig, ax = plt.subplots(figsize=(10, 10))\\n\",\n \"ax.imshow(Image.open(image_path).convert('RGB'))\\n\",\n \"wanted = results['scores'][0] > 0.1\\n\",\n \"for xyxy, label_no_bg in zip(results['boxes'][0][wanted], results['classes'][0][wanted]):\\n\",\n \" xywh = xyxy[0], xyxy[1], xyxy[2] - xyxy[0], xyxy[3] - xyxy[1]\\n\",\n \" rect = patches.Rectangle((xywh[0], xywh[1]), xywh[2], xywh[3], linewidth=1, edgecolor='g', facecolor='none')\\n\",\n \" ax.add_patch(rect)\\n\",\n \" rx, ry = rect.get_xy()\\n\",\n \" rx = rx + rect.get_width() / 2.0\\n\",\n \" ax.annotate(label_info[label_no_bg + 1], (rx, ry), color='w', backgroundcolor='g', fontsize=10,\\n\",\n \" ha='center', va='center', bbox=dict(boxstyle='square,pad=0.01', fc='g', ec='none', alpha=0.5))\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 2: Compile the Pretrained SavedModel for Inferentia\\n\",\n \"We make use of the Python compilation API `tfn.saved_model.compile` that is avaiable in `tensorflow-neuron<2`. For the purpose of reducing Neuron runtime overhead, it is necessary to make use of arguments `no_fuse_ops` and `minimum_segment_size`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import shutil\\n\",\n \"import tensorflow as tf\\n\",\n \"import tensorflow.neuron as tfn\\n\",\n \"\\n\",\n \"\\n\",\n \"def no_fuse_condition(op):\\n\",\n \" return any(op.name.startswith(pat) for pat in ['reshape', 'lambda_1/Cast', 'lambda_2/Cast', 'lambda_3/Cast'])\\n\",\n \"\\n\",\n \"with tf.Session(graph=tf.Graph()) as sess:\\n\",\n \" tf.saved_model.loader.load(sess, ['serve'], './yolo_v4_coco_saved_model')\\n\",\n \" no_fuse_ops = [op.name for op in sess.graph.get_operations() if no_fuse_condition(op)]\\n\",\n \"shutil.rmtree('./yolo_v4_coco_saved_model_neuron', ignore_errors=True)\\n\",\n \"result = tfn.saved_model.compile(\\n\",\n \" './yolo_v4_coco_saved_model', './yolo_v4_coco_saved_model_neuron',\\n\",\n \" # we partition the graph before casting from float16 to float32, to help reduce the output tensor size by 1/2\\n\",\n \" no_fuse_ops=no_fuse_ops,\\n\",\n \" # to enforce trivial compilable subgraphs to run on CPU\\n\",\n \" minimum_segment_size=100,\\n\",\n \" batch_size=1,\\n\",\n \" dynamic_batch_size=True,\\n\",\n \")\\n\",\n \"print(result)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Part 3: Evaluate Model Quality after Compilation\\n\",\n \"### Define evaluation functions\\n\",\n \"We first define some handy helper functions for running evaluation on the COCO 2017 dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import json\\n\",\n \"import time\\n\",\n \"import numpy as np\\n\",\n \"import tensorflow as tf\\n\",\n \"from pycocotools.coco import COCO\\n\",\n \"from pycocotools.cocoeval import COCOeval\\n\",\n \"\\n\",\n \"\\n\",\n \"def cocoapi_eval(jsonfile,\\n\",\n \" style,\\n\",\n \" coco_gt=None,\\n\",\n \" anno_file=None,\\n\",\n \" max_dets=(100, 300, 1000)):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Args:\\n\",\n \" jsonfile: Evaluation json file, eg: bbox.json, mask.json.\\n\",\n \" style: COCOeval style, can be `bbox` , `segm` and `proposal`.\\n\",\n \" coco_gt: Whether to load COCOAPI through anno_file,\\n\",\n \" eg: coco_gt = COCO(anno_file)\\n\",\n \" anno_file: COCO annotations file.\\n\",\n \" max_dets: COCO evaluation maxDets.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" assert coco_gt is not None or anno_file is not None\\n\",\n \"\\n\",\n \" if coco_gt is None:\\n\",\n \" coco_gt = COCO(anno_file)\\n\",\n \" print(\\\"Start evaluate...\\\")\\n\",\n \" coco_dt = coco_gt.loadRes(jsonfile)\\n\",\n \" if style == 'proposal':\\n\",\n \" coco_eval = COCOeval(coco_gt, coco_dt, 'bbox')\\n\",\n \" coco_eval.params.useCats = 0\\n\",\n \" coco_eval.params.maxDets = list(max_dets)\\n\",\n \" else:\\n\",\n \" coco_eval = COCOeval(coco_gt, coco_dt, style)\\n\",\n \" coco_eval.evaluate()\\n\",\n \" coco_eval.accumulate()\\n\",\n \" coco_eval.summarize()\\n\",\n \" return coco_eval.stats\\n\",\n \"\\n\",\n \"\\n\",\n \"def bbox_eval(anno_file, bbox_list):\\n\",\n \" coco_gt = COCO(anno_file)\\n\",\n \"\\n\",\n \" outfile = 'bbox_detections.json'\\n\",\n \" print('Generating json file...')\\n\",\n \" with open(outfile, 'w') as f:\\n\",\n \" json.dump(bbox_list, f)\\n\",\n \"\\n\",\n \" map_stats = cocoapi_eval(outfile, 'bbox', coco_gt=coco_gt)\\n\",\n \" return map_stats\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_image_as_bytes(images, eval_pre_path):\\n\",\n \" batch_im_id_list = []\\n\",\n \" batch_im_name_list = []\\n\",\n \" batch_img_bytes_list = []\\n\",\n \" n = len(images)\\n\",\n \" batch_im_id = []\\n\",\n \" batch_im_name = []\\n\",\n \" batch_img_bytes = []\\n\",\n \" for i, im in enumerate(images):\\n\",\n \" im_id = im['id']\\n\",\n \" file_name = im['file_name']\\n\",\n \" if i % eval_batch_size == 0 and i != 0:\\n\",\n \" batch_im_id_list.append(batch_im_id)\\n\",\n \" batch_im_name_list.append(batch_im_name)\\n\",\n \" batch_img_bytes_list.append(batch_img_bytes)\\n\",\n \" batch_im_id = []\\n\",\n \" batch_im_name = []\\n\",\n \" batch_img_bytes = []\\n\",\n \" batch_im_id.append(im_id)\\n\",\n \" batch_im_name.append(file_name)\\n\",\n \"\\n\",\n \" with open(os.path.join(eval_pre_path, file_name), 'rb') as f:\\n\",\n \" batch_img_bytes.append(f.read())\\n\",\n \" return batch_im_id_list, batch_im_name_list, batch_img_bytes_list\\n\",\n \"\\n\",\n \"\\n\",\n \"def analyze_bbox(results, batch_im_id, _clsid2catid):\\n\",\n \" bbox_list = []\\n\",\n \" k = 0\\n\",\n \" for boxes, scores, classes in zip(results['boxes'], results['scores'], results['classes']):\\n\",\n \" if boxes is not None:\\n\",\n \" im_id = batch_im_id[k]\\n\",\n \" n = len(boxes)\\n\",\n \" for p in range(n):\\n\",\n \" clsid = classes[p]\\n\",\n \" score = scores[p]\\n\",\n \" xmin, ymin, xmax, ymax = boxes[p]\\n\",\n \" catid = (_clsid2catid[int(clsid)])\\n\",\n \" w = xmax - xmin + 1\\n\",\n \" h = ymax - ymin + 1\\n\",\n \"\\n\",\n \" bbox = [xmin, ymin, w, h]\\n\",\n \" # Round to the nearest 10th to avoid huge file sizes, as COCO suggests\\n\",\n \" bbox = [round(float(x) * 10) / 10 for x in bbox]\\n\",\n \" bbox_res = {\\n\",\n \" 'image_id': im_id,\\n\",\n \" 'category_id': catid,\\n\",\n \" 'bbox': bbox,\\n\",\n \" 'score': float(score),\\n\",\n \" }\\n\",\n \" bbox_list.append(bbox_res)\\n\",\n \" k += 1\\n\",\n \" return bbox_list\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Here is the actual evaluation loop. To fully utilize all four cores on one Inferentia, the optimal setup is to run multi-threaded inference using a `ThreadPoolExecutor`. The following cell is a multi-threaded adaptation of the evaluation routine at https://github.com/miemie2013/Keras-YOLOv4/blob/910c4c6f7265f5828fceed0f784496a0b46516bf/tools/cocotools.py#L97.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from concurrent import futures\\n\",\n \"\\n\",\n \"NUM_THREADS = 4\\n\",\n \"\\n\",\n \"def evaluate(yolo_predictor, images, eval_pre_path, anno_file, eval_batch_size, _clsid2catid):\\n\",\n \" batch_im_id_list, batch_im_name_list, batch_img_bytes_list = get_image_as_bytes(images, eval_pre_path)\\n\",\n \"\\n\",\n \" # warm up\\n\",\n \" yolo_predictor({'image': np.array(batch_img_bytes_list[0], dtype=object)})\\n\",\n \" \\n\",\n \" def yolo_predictor_timer(yolo_pred, image):\\n\",\n \" begin = time.time()\\n\",\n \" result = yolo_pred(image)\\n\",\n \" delta = time.time() - begin\\n\",\n \" return result, delta\\n\",\n \"\\n\",\n \" latency = []\\n\",\n \" with futures.ThreadPoolExecutor(NUM_THREADS) as exe:\\n\",\n \" fut_im_list = []\\n\",\n \" fut_list = []\\n\",\n \"\\n\",\n \" start_time = time.time()\\n\",\n \" for batch_im_id, batch_im_name, batch_img_bytes in zip(batch_im_id_list, batch_im_name_list, batch_img_bytes_list):\\n\",\n \" if len(batch_img_bytes) != eval_batch_size:\\n\",\n \" continue\\n\",\n \" fut = exe.submit(yolo_predictor_timer, yolo_predictor, {'image': np.array(batch_img_bytes, dtype=object)})\\n\",\n \" fut_im_list.append((batch_im_id, batch_im_name))\\n\",\n \" fut_list.append(fut)\\n\",\n \" bbox_list = []\\n\",\n \" sum_time = 0.0\\n\",\n \" count = 0\\n\",\n \" for (batch_im_id, batch_im_name), fut in zip(fut_im_list, fut_list):\\n\",\n \" results, times = fut.result()\\n\",\n \" # Adjust latency since we are in batch\\n\",\n \" latency.append(times / eval_batch_size)\\n\",\n \" sum_time += times\\n\",\n \" bbox_list.extend(analyze_bbox(results, batch_im_id, _clsid2catid))\\n\",\n \" for _ in batch_im_id:\\n\",\n \" count += 1\\n\",\n \" if count % 1000 == 0:\\n\",\n \" print('Test iter {}'.format(count))\\n\",\n \"\\n\",\n \" throughput = len(images) / (sum_time / NUM_THREADS)\\n\",\n \"\\n\",\n \" \\n\",\n \" print('Average Images Per Second:', throughput)\\n\",\n \" print(\\\"Latency P50: {:.1f} ms\\\".format(np.percentile(latency, 50)*1000.0))\\n\",\n \" print(\\\"Latency P90: {:.1f} ms\\\".format(np.percentile(latency, 90)*1000.0))\\n\",\n \" print(\\\"Latency P95: {:.1f} ms\\\".format(np.percentile(latency, 95)*1000.0))\\n\",\n \" print(\\\"Latency P99: {:.1f} ms\\\".format(np.percentile(latency, 99)*1000.0))\\n\",\n \"\\n\",\n \" # start evaluation\\n\",\n \" box_ap_stats = bbox_eval(anno_file, bbox_list)\\n\",\n \" return box_ap_stats\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Evaluate mean average precision (mAP) score\\n\",\n \"Here is the code to calculate mAP scores of the YOLO v4 model. The expected mAP score is around 0.487 if we use the pretrained weights.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"yolo_pred = tf.contrib.predictor.from_saved_model('./yolo_v4_coco_saved_model_neuron')\\n\",\n \"\\n\",\n \"val_coco_root = './val2017'\\n\",\n \"val_annotate = './annotations/instances_val2017.json'\\n\",\n \"clsid2catid = {0: 1, 1: 2, 2: 3, 3: 4, 4: 5, 5: 6, 6: 7, 7: 8, 8: 9, 9: 10, 10: 11, 11: 13, 12: 14, 13: 15, 14: 16,\\n\",\n \" 15: 17, 16: 18, 17: 19, 18: 20, 19: 21, 20: 22, 21: 23, 22: 24, 23: 25, 24: 27, 25: 28, 26: 31,\\n\",\n \" 27: 32, 28: 33, 29: 34, 30: 35, 31: 36, 32: 37, 33: 38, 34: 39, 35: 40, 36: 41, 37: 42, 38: 43,\\n\",\n \" 39: 44, 40: 46, 41: 47, 42: 48, 43: 49, 44: 50, 45: 51, 46: 52, 47: 53, 48: 54, 49: 55, 50: 56,\\n\",\n \" 51: 57, 52: 58, 53: 59, 54: 60, 55: 61, 56: 62, 57: 63, 58: 64, 59: 65, 60: 67, 61: 70, 62: 72,\\n\",\n \" 63: 73, 64: 74, 65: 75, 66: 76, 67: 77, 68: 78, 69: 79, 70: 80, 71: 81, 72: 82, 73: 84, 74: 85,\\n\",\n \" 75: 86, 76: 87, 77: 88, 78: 89, 79: 90}\\n\",\n \"eval_batch_size = 8\\n\",\n \"with open(val_annotate, 'r', encoding='utf-8') as f2:\\n\",\n \" for line in f2:\\n\",\n \" line = line.strip()\\n\",\n \" dataset = json.loads(line)\\n\",\n \" images = dataset['images']\\n\",\n \"box_ap = evaluate(yolo_pred, images, val_coco_root, val_annotate, eval_batch_size, clsid2catid)\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Environment (conda_aws_neuron_tensorflow_p36)\",\n \"language\": \"python\",\n \"name\": \"conda_aws_neuron_tensorflow_p36\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n" }, { "path": "src/examples/tensorflow/yolo_v4_demo/yolo_v4_coco_saved_model.py", "content": "import os\nimport io\nfrom functools import partial\nimport requests\nimport numpy as np\nimport torch\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\n\n\ndef rename_weights(checkpoint):\n name_mapping = {\n 'down1.conv1.conv.0.weight': 'models.0.conv1.weight',\n 'down1.conv1.conv.1.weight': 'models.0.bn1.weight',\n 'down1.conv1.conv.1.bias': 'models.0.bn1.bias',\n 'down1.conv1.conv.1.running_mean': 'models.0.bn1.running_mean',\n 'down1.conv1.conv.1.running_var': 'models.0.bn1.running_var',\n 'down1.conv1.conv.1.num_batches_tracked': 'models.0.bn1.num_batches_tracked',\n 'down1.conv2.conv.0.weight': 'models.1.conv2.weight',\n 'down1.conv2.conv.1.weight': 'models.1.bn2.weight',\n 'down1.conv2.conv.1.bias': 'models.1.bn2.bias',\n 'down1.conv2.conv.1.running_mean': 'models.1.bn2.running_mean',\n 'down1.conv2.conv.1.running_var': 'models.1.bn2.running_var',\n 'down1.conv2.conv.1.num_batches_tracked': 'models.1.bn2.num_batches_tracked',\n 'down1.conv3.conv.0.weight': 'models.2.conv3.weight',\n 'down1.conv3.conv.1.weight': 'models.2.bn3.weight',\n 'down1.conv3.conv.1.bias': 'models.2.bn3.bias',\n 'down1.conv3.conv.1.running_mean': 'models.2.bn3.running_mean',\n 'down1.conv3.conv.1.running_var': 'models.2.bn3.running_var',\n 'down1.conv3.conv.1.num_batches_tracked': 'models.2.bn3.num_batches_tracked',\n 'down1.conv4.conv.0.weight': 'models.4.conv4.weight',\n 'down1.conv4.conv.1.weight': 'models.4.bn4.weight',\n 'down1.conv4.conv.1.bias': 'models.4.bn4.bias',\n 'down1.conv4.conv.1.running_mean': 'models.4.bn4.running_mean',\n 'down1.conv4.conv.1.running_var': 'models.4.bn4.running_var',\n 'down1.conv4.conv.1.num_batches_tracked': 'models.4.bn4.num_batches_tracked',\n 'down1.conv5.conv.0.weight': 'models.5.conv5.weight',\n 'down1.conv5.conv.1.weight': 'models.5.bn5.weight',\n 'down1.conv5.conv.1.bias': 'models.5.bn5.bias',\n 'down1.conv5.conv.1.running_mean': 'models.5.bn5.running_mean',\n 'down1.conv5.conv.1.running_var': 'models.5.bn5.running_var',\n 'down1.conv5.conv.1.num_batches_tracked': 'models.5.bn5.num_batches_tracked',\n 'down1.conv6.conv.0.weight': 'models.6.conv6.weight',\n 'down1.conv6.conv.1.weight': 'models.6.bn6.weight',\n 'down1.conv6.conv.1.bias': 'models.6.bn6.bias',\n 'down1.conv6.conv.1.running_mean': 'models.6.bn6.running_mean',\n 'down1.conv6.conv.1.running_var': 'models.6.bn6.running_var',\n 'down1.conv6.conv.1.num_batches_tracked': 'models.6.bn6.num_batches_tracked',\n 'down1.conv7.conv.0.weight': 'models.8.conv7.weight',\n 'down1.conv7.conv.1.weight': 'models.8.bn7.weight',\n 'down1.conv7.conv.1.bias': 'models.8.bn7.bias',\n 'down1.conv7.conv.1.running_mean': 'models.8.bn7.running_mean',\n 'down1.conv7.conv.1.running_var': 'models.8.bn7.running_var',\n 'down1.conv7.conv.1.num_batches_tracked': 'models.8.bn7.num_batches_tracked',\n 'down1.conv8.conv.0.weight': 'models.10.conv8.weight',\n 'down1.conv8.conv.1.weight': 'models.10.bn8.weight',\n 'down1.conv8.conv.1.bias': 'models.10.bn8.bias',\n 'down1.conv8.conv.1.running_mean': 'models.10.bn8.running_mean',\n 'down1.conv8.conv.1.running_var': 'models.10.bn8.running_var',\n 'down1.conv8.conv.1.num_batches_tracked': 'models.10.bn8.num_batches_tracked',\n 'down2.conv1.conv.0.weight': 'models.11.conv9.weight',\n 'down2.conv1.conv.1.weight': 'models.11.bn9.weight',\n 'down2.conv1.conv.1.bias': 'models.11.bn9.bias',\n 'down2.conv1.conv.1.running_mean': 'models.11.bn9.running_mean',\n 'down2.conv1.conv.1.running_var': 'models.11.bn9.running_var',\n 'down2.conv1.conv.1.num_batches_tracked': 'models.11.bn9.num_batches_tracked',\n 'down2.conv2.conv.0.weight': 'models.12.conv10.weight',\n 'down2.conv2.conv.1.weight': 'models.12.bn10.weight',\n 'down2.conv2.conv.1.bias': 'models.12.bn10.bias',\n 'down2.conv2.conv.1.running_mean': 'models.12.bn10.running_mean',\n 'down2.conv2.conv.1.running_var': 'models.12.bn10.running_var',\n 'down2.conv2.conv.1.num_batches_tracked': 'models.12.bn10.num_batches_tracked',\n 'down2.conv3.conv.0.weight': 'models.14.conv11.weight',\n 'down2.conv3.conv.1.weight': 'models.14.bn11.weight',\n 'down2.conv3.conv.1.bias': 'models.14.bn11.bias',\n 'down2.conv3.conv.1.running_mean': 'models.14.bn11.running_mean',\n 'down2.conv3.conv.1.running_var': 'models.14.bn11.running_var',\n 'down2.conv3.conv.1.num_batches_tracked': 'models.14.bn11.num_batches_tracked',\n 'down2.resblock.module_list.0.0.conv.0.weight': 'models.15.conv12.weight',\n 'down2.resblock.module_list.0.0.conv.1.weight': 'models.15.bn12.weight',\n 'down2.resblock.module_list.0.0.conv.1.bias': 'models.15.bn12.bias',\n 'down2.resblock.module_list.0.0.conv.1.running_mean': 'models.15.bn12.running_mean',\n 'down2.resblock.module_list.0.0.conv.1.running_var': 'models.15.bn12.running_var',\n 'down2.resblock.module_list.0.0.conv.1.num_batches_tracked': 'models.15.bn12.num_batches_tracked',\n 'down2.resblock.module_list.0.1.conv.0.weight': 'models.16.conv13.weight',\n 'down2.resblock.module_list.0.1.conv.1.weight': 'models.16.bn13.weight',\n 'down2.resblock.module_list.0.1.conv.1.bias': 'models.16.bn13.bias',\n 'down2.resblock.module_list.0.1.conv.1.running_mean': 'models.16.bn13.running_mean',\n 'down2.resblock.module_list.0.1.conv.1.running_var': 'models.16.bn13.running_var',\n 'down2.resblock.module_list.0.1.conv.1.num_batches_tracked': 'models.16.bn13.num_batches_tracked',\n 'down2.resblock.module_list.1.0.conv.0.weight': 'models.18.conv14.weight',\n 'down2.resblock.module_list.1.0.conv.1.weight': 'models.18.bn14.weight',\n 'down2.resblock.module_list.1.0.conv.1.bias': 'models.18.bn14.bias',\n 'down2.resblock.module_list.1.0.conv.1.running_mean': 'models.18.bn14.running_mean',\n 'down2.resblock.module_list.1.0.conv.1.running_var': 'models.18.bn14.running_var',\n 'down2.resblock.module_list.1.0.conv.1.num_batches_tracked': 'models.18.bn14.num_batches_tracked',\n 'down2.resblock.module_list.1.1.conv.0.weight': 'models.19.conv15.weight',\n 'down2.resblock.module_list.1.1.conv.1.weight': 'models.19.bn15.weight',\n 'down2.resblock.module_list.1.1.conv.1.bias': 'models.19.bn15.bias',\n 'down2.resblock.module_list.1.1.conv.1.running_mean': 'models.19.bn15.running_mean',\n 'down2.resblock.module_list.1.1.conv.1.running_var': 'models.19.bn15.running_var',\n 'down2.resblock.module_list.1.1.conv.1.num_batches_tracked': 'models.19.bn15.num_batches_tracked',\n 'down2.conv4.conv.0.weight': 'models.21.conv16.weight',\n 'down2.conv4.conv.1.weight': 'models.21.bn16.weight',\n 'down2.conv4.conv.1.bias': 'models.21.bn16.bias',\n 'down2.conv4.conv.1.running_mean': 'models.21.bn16.running_mean',\n 'down2.conv4.conv.1.running_var': 'models.21.bn16.running_var',\n 'down2.conv4.conv.1.num_batches_tracked': 'models.21.bn16.num_batches_tracked',\n 'down2.conv5.conv.0.weight': 'models.23.conv17.weight',\n 'down2.conv5.conv.1.weight': 'models.23.bn17.weight',\n 'down2.conv5.conv.1.bias': 'models.23.bn17.bias',\n 'down2.conv5.conv.1.running_mean': 'models.23.bn17.running_mean',\n 'down2.conv5.conv.1.running_var': 'models.23.bn17.running_var',\n 'down2.conv5.conv.1.num_batches_tracked': 'models.23.bn17.num_batches_tracked',\n 'down3.conv1.conv.0.weight': 'models.24.conv18.weight',\n 'down3.conv1.conv.1.weight': 'models.24.bn18.weight',\n 'down3.conv1.conv.1.bias': 'models.24.bn18.bias',\n 'down3.conv1.conv.1.running_mean': 'models.24.bn18.running_mean',\n 'down3.conv1.conv.1.running_var': 'models.24.bn18.running_var',\n 'down3.conv1.conv.1.num_batches_tracked': 'models.24.bn18.num_batches_tracked',\n 'down3.conv2.conv.0.weight': 'models.25.conv19.weight',\n 'down3.conv2.conv.1.weight': 'models.25.bn19.weight',\n 'down3.conv2.conv.1.bias': 'models.25.bn19.bias',\n 'down3.conv2.conv.1.running_mean': 'models.25.bn19.running_mean',\n 'down3.conv2.conv.1.running_var': 'models.25.bn19.running_var',\n 'down3.conv2.conv.1.num_batches_tracked': 'models.25.bn19.num_batches_tracked',\n 'down3.conv3.conv.0.weight': 'models.27.conv20.weight',\n 'down3.conv3.conv.1.weight': 'models.27.bn20.weight',\n 'down3.conv3.conv.1.bias': 'models.27.bn20.bias',\n 'down3.conv3.conv.1.running_mean': 'models.27.bn20.running_mean',\n 'down3.conv3.conv.1.running_var': 'models.27.bn20.running_var',\n 'down3.conv3.conv.1.num_batches_tracked': 'models.27.bn20.num_batches_tracked',\n 'down3.resblock.module_list.0.0.conv.0.weight': 'models.28.conv21.weight',\n 'down3.resblock.module_list.0.0.conv.1.weight': 'models.28.bn21.weight',\n 'down3.resblock.module_list.0.0.conv.1.bias': 'models.28.bn21.bias',\n 'down3.resblock.module_list.0.0.conv.1.running_mean': 'models.28.bn21.running_mean',\n 'down3.resblock.module_list.0.0.conv.1.running_var': 'models.28.bn21.running_var',\n 'down3.resblock.module_list.0.0.conv.1.num_batches_tracked': 'models.28.bn21.num_batches_tracked',\n 'down3.resblock.module_list.0.1.conv.0.weight': 'models.29.conv22.weight',\n 'down3.resblock.module_list.0.1.conv.1.weight': 'models.29.bn22.weight',\n 'down3.resblock.module_list.0.1.conv.1.bias': 'models.29.bn22.bias',\n 'down3.resblock.module_list.0.1.conv.1.running_mean': 'models.29.bn22.running_mean',\n 'down3.resblock.module_list.0.1.conv.1.running_var': 'models.29.bn22.running_var',\n 'down3.resblock.module_list.0.1.conv.1.num_batches_tracked': 'models.29.bn22.num_batches_tracked',\n 'down3.resblock.module_list.1.0.conv.0.weight': 'models.31.conv23.weight',\n 'down3.resblock.module_list.1.0.conv.1.weight': 'models.31.bn23.weight',\n 'down3.resblock.module_list.1.0.conv.1.bias': 'models.31.bn23.bias',\n 'down3.resblock.module_list.1.0.conv.1.running_mean': 'models.31.bn23.running_mean',\n 'down3.resblock.module_list.1.0.conv.1.running_var': 'models.31.bn23.running_var',\n 'down3.resblock.module_list.1.0.conv.1.num_batches_tracked': 'models.31.bn23.num_batches_tracked',\n 'down3.resblock.module_list.1.1.conv.0.weight': 'models.32.conv24.weight',\n 'down3.resblock.module_list.1.1.conv.1.weight': 'models.32.bn24.weight',\n 'down3.resblock.module_list.1.1.conv.1.bias': 'models.32.bn24.bias',\n 'down3.resblock.module_list.1.1.conv.1.running_mean': 'models.32.bn24.running_mean',\n 'down3.resblock.module_list.1.1.conv.1.running_var': 'models.32.bn24.running_var',\n 'down3.resblock.module_list.1.1.conv.1.num_batches_tracked': 'models.32.bn24.num_batches_tracked',\n 'down3.resblock.module_list.2.0.conv.0.weight': 'models.34.conv25.weight',\n 'down3.resblock.module_list.2.0.conv.1.weight': 'models.34.bn25.weight',\n 'down3.resblock.module_list.2.0.conv.1.bias': 'models.34.bn25.bias',\n 'down3.resblock.module_list.2.0.conv.1.running_mean': 'models.34.bn25.running_mean',\n 'down3.resblock.module_list.2.0.conv.1.running_var': 'models.34.bn25.running_var',\n 'down3.resblock.module_list.2.0.conv.1.num_batches_tracked': 'models.34.bn25.num_batches_tracked',\n 'down3.resblock.module_list.2.1.conv.0.weight': 'models.35.conv26.weight',\n 'down3.resblock.module_list.2.1.conv.1.weight': 'models.35.bn26.weight',\n 'down3.resblock.module_list.2.1.conv.1.bias': 'models.35.bn26.bias',\n 'down3.resblock.module_list.2.1.conv.1.running_mean': 'models.35.bn26.running_mean',\n 'down3.resblock.module_list.2.1.conv.1.running_var': 'models.35.bn26.running_var',\n 'down3.resblock.module_list.2.1.conv.1.num_batches_tracked': 'models.35.bn26.num_batches_tracked',\n 'down3.resblock.module_list.3.0.conv.0.weight': 'models.37.conv27.weight',\n 'down3.resblock.module_list.3.0.conv.1.weight': 'models.37.bn27.weight',\n 'down3.resblock.module_list.3.0.conv.1.bias': 'models.37.bn27.bias',\n 'down3.resblock.module_list.3.0.conv.1.running_mean': 'models.37.bn27.running_mean',\n 'down3.resblock.module_list.3.0.conv.1.running_var': 'models.37.bn27.running_var',\n 'down3.resblock.module_list.3.0.conv.1.num_batches_tracked': 'models.37.bn27.num_batches_tracked',\n 'down3.resblock.module_list.3.1.conv.0.weight': 'models.38.conv28.weight',\n 'down3.resblock.module_list.3.1.conv.1.weight': 'models.38.bn28.weight',\n 'down3.resblock.module_list.3.1.conv.1.bias': 'models.38.bn28.bias',\n 'down3.resblock.module_list.3.1.conv.1.running_mean': 'models.38.bn28.running_mean',\n 'down3.resblock.module_list.3.1.conv.1.running_var': 'models.38.bn28.running_var',\n 'down3.resblock.module_list.3.1.conv.1.num_batches_tracked': 'models.38.bn28.num_batches_tracked',\n 'down3.resblock.module_list.4.0.conv.0.weight': 'models.40.conv29.weight',\n 'down3.resblock.module_list.4.0.conv.1.weight': 'models.40.bn29.weight',\n 'down3.resblock.module_list.4.0.conv.1.bias': 'models.40.bn29.bias',\n 'down3.resblock.module_list.4.0.conv.1.running_mean': 'models.40.bn29.running_mean',\n 'down3.resblock.module_list.4.0.conv.1.running_var': 'models.40.bn29.running_var',\n 'down3.resblock.module_list.4.0.conv.1.num_batches_tracked': 'models.40.bn29.num_batches_tracked',\n 'down3.resblock.module_list.4.1.conv.0.weight': 'models.41.conv30.weight',\n 'down3.resblock.module_list.4.1.conv.1.weight': 'models.41.bn30.weight',\n 'down3.resblock.module_list.4.1.conv.1.bias': 'models.41.bn30.bias',\n 'down3.resblock.module_list.4.1.conv.1.running_mean': 'models.41.bn30.running_mean',\n 'down3.resblock.module_list.4.1.conv.1.running_var': 'models.41.bn30.running_var',\n 'down3.resblock.module_list.4.1.conv.1.num_batches_tracked': 'models.41.bn30.num_batches_tracked',\n 'down3.resblock.module_list.5.0.conv.0.weight': 'models.43.conv31.weight',\n 'down3.resblock.module_list.5.0.conv.1.weight': 'models.43.bn31.weight',\n 'down3.resblock.module_list.5.0.conv.1.bias': 'models.43.bn31.bias',\n 'down3.resblock.module_list.5.0.conv.1.running_mean': 'models.43.bn31.running_mean',\n 'down3.resblock.module_list.5.0.conv.1.running_var': 'models.43.bn31.running_var',\n 'down3.resblock.module_list.5.0.conv.1.num_batches_tracked': 'models.43.bn31.num_batches_tracked',\n 'down3.resblock.module_list.5.1.conv.0.weight': 'models.44.conv32.weight',\n 'down3.resblock.module_list.5.1.conv.1.weight': 'models.44.bn32.weight',\n 'down3.resblock.module_list.5.1.conv.1.bias': 'models.44.bn32.bias',\n 'down3.resblock.module_list.5.1.conv.1.running_mean': 'models.44.bn32.running_mean',\n 'down3.resblock.module_list.5.1.conv.1.running_var': 'models.44.bn32.running_var',\n 'down3.resblock.module_list.5.1.conv.1.num_batches_tracked': 'models.44.bn32.num_batches_tracked',\n 'down3.resblock.module_list.6.0.conv.0.weight': 'models.46.conv33.weight',\n 'down3.resblock.module_list.6.0.conv.1.weight': 'models.46.bn33.weight',\n 'down3.resblock.module_list.6.0.conv.1.bias': 'models.46.bn33.bias',\n 'down3.resblock.module_list.6.0.conv.1.running_mean': 'models.46.bn33.running_mean',\n 'down3.resblock.module_list.6.0.conv.1.running_var': 'models.46.bn33.running_var',\n 'down3.resblock.module_list.6.0.conv.1.num_batches_tracked': 'models.46.bn33.num_batches_tracked',\n 'down3.resblock.module_list.6.1.conv.0.weight': 'models.47.conv34.weight',\n 'down3.resblock.module_list.6.1.conv.1.weight': 'models.47.bn34.weight',\n 'down3.resblock.module_list.6.1.conv.1.bias': 'models.47.bn34.bias',\n 'down3.resblock.module_list.6.1.conv.1.running_mean': 'models.47.bn34.running_mean',\n 'down3.resblock.module_list.6.1.conv.1.running_var': 'models.47.bn34.running_var',\n 'down3.resblock.module_list.6.1.conv.1.num_batches_tracked': 'models.47.bn34.num_batches_tracked',\n 'down3.resblock.module_list.7.0.conv.0.weight': 'models.49.conv35.weight',\n 'down3.resblock.module_list.7.0.conv.1.weight': 'models.49.bn35.weight',\n 'down3.resblock.module_list.7.0.conv.1.bias': 'models.49.bn35.bias',\n 'down3.resblock.module_list.7.0.conv.1.running_mean': 'models.49.bn35.running_mean',\n 'down3.resblock.module_list.7.0.conv.1.running_var': 'models.49.bn35.running_var',\n 'down3.resblock.module_list.7.0.conv.1.num_batches_tracked': 'models.49.bn35.num_batches_tracked',\n 'down3.resblock.module_list.7.1.conv.0.weight': 'models.50.conv36.weight',\n 'down3.resblock.module_list.7.1.conv.1.weight': 'models.50.bn36.weight',\n 'down3.resblock.module_list.7.1.conv.1.bias': 'models.50.bn36.bias',\n 'down3.resblock.module_list.7.1.conv.1.running_mean': 'models.50.bn36.running_mean',\n 'down3.resblock.module_list.7.1.conv.1.running_var': 'models.50.bn36.running_var',\n 'down3.resblock.module_list.7.1.conv.1.num_batches_tracked': 'models.50.bn36.num_batches_tracked',\n 'down3.conv4.conv.0.weight': 'models.52.conv37.weight',\n 'down3.conv4.conv.1.weight': 'models.52.bn37.weight',\n 'down3.conv4.conv.1.bias': 'models.52.bn37.bias',\n 'down3.conv4.conv.1.running_mean': 'models.52.bn37.running_mean',\n 'down3.conv4.conv.1.running_var': 'models.52.bn37.running_var',\n 'down3.conv4.conv.1.num_batches_tracked': 'models.52.bn37.num_batches_tracked',\n 'down3.conv5.conv.0.weight': 'models.54.conv38.weight',\n 'down3.conv5.conv.1.weight': 'models.54.bn38.weight',\n 'down3.conv5.conv.1.bias': 'models.54.bn38.bias',\n 'down3.conv5.conv.1.running_mean': 'models.54.bn38.running_mean',\n 'down3.conv5.conv.1.running_var': 'models.54.bn38.running_var',\n 'down3.conv5.conv.1.num_batches_tracked': 'models.54.bn38.num_batches_tracked',\n 'down4.conv1.conv.0.weight': 'models.55.conv39.weight',\n 'down4.conv1.conv.1.weight': 'models.55.bn39.weight',\n 'down4.conv1.conv.1.bias': 'models.55.bn39.bias',\n 'down4.conv1.conv.1.running_mean': 'models.55.bn39.running_mean',\n 'down4.conv1.conv.1.running_var': 'models.55.bn39.running_var',\n 'down4.conv1.conv.1.num_batches_tracked': 'models.55.bn39.num_batches_tracked',\n 'down4.conv2.conv.0.weight': 'models.56.conv40.weight',\n 'down4.conv2.conv.1.weight': 'models.56.bn40.weight',\n 'down4.conv2.conv.1.bias': 'models.56.bn40.bias',\n 'down4.conv2.conv.1.running_mean': 'models.56.bn40.running_mean',\n 'down4.conv2.conv.1.running_var': 'models.56.bn40.running_var',\n 'down4.conv2.conv.1.num_batches_tracked': 'models.56.bn40.num_batches_tracked',\n 'down4.conv3.conv.0.weight': 'models.58.conv41.weight',\n 'down4.conv3.conv.1.weight': 'models.58.bn41.weight',\n 'down4.conv3.conv.1.bias': 'models.58.bn41.bias',\n 'down4.conv3.conv.1.running_mean': 'models.58.bn41.running_mean',\n 'down4.conv3.conv.1.running_var': 'models.58.bn41.running_var',\n 'down4.conv3.conv.1.num_batches_tracked': 'models.58.bn41.num_batches_tracked',\n 'down4.resblock.module_list.0.0.conv.0.weight': 'models.59.conv42.weight',\n 'down4.resblock.module_list.0.0.conv.1.weight': 'models.59.bn42.weight',\n 'down4.resblock.module_list.0.0.conv.1.bias': 'models.59.bn42.bias',\n 'down4.resblock.module_list.0.0.conv.1.running_mean': 'models.59.bn42.running_mean',\n 'down4.resblock.module_list.0.0.conv.1.running_var': 'models.59.bn42.running_var',\n 'down4.resblock.module_list.0.0.conv.1.num_batches_tracked': 'models.59.bn42.num_batches_tracked',\n 'down4.resblock.module_list.0.1.conv.0.weight': 'models.60.conv43.weight',\n 'down4.resblock.module_list.0.1.conv.1.weight': 'models.60.bn43.weight',\n 'down4.resblock.module_list.0.1.conv.1.bias': 'models.60.bn43.bias',\n 'down4.resblock.module_list.0.1.conv.1.running_mean': 'models.60.bn43.running_mean',\n 'down4.resblock.module_list.0.1.conv.1.running_var': 'models.60.bn43.running_var',\n 'down4.resblock.module_list.0.1.conv.1.num_batches_tracked': 'models.60.bn43.num_batches_tracked',\n 'down4.resblock.module_list.1.0.conv.0.weight': 'models.62.conv44.weight',\n 'down4.resblock.module_list.1.0.conv.1.weight': 'models.62.bn44.weight',\n 'down4.resblock.module_list.1.0.conv.1.bias': 'models.62.bn44.bias',\n 'down4.resblock.module_list.1.0.conv.1.running_mean': 'models.62.bn44.running_mean',\n 'down4.resblock.module_list.1.0.conv.1.running_var': 'models.62.bn44.running_var',\n 'down4.resblock.module_list.1.0.conv.1.num_batches_tracked': 'models.62.bn44.num_batches_tracked',\n 'down4.resblock.module_list.1.1.conv.0.weight': 'models.63.conv45.weight',\n 'down4.resblock.module_list.1.1.conv.1.weight': 'models.63.bn45.weight',\n 'down4.resblock.module_list.1.1.conv.1.bias': 'models.63.bn45.bias',\n 'down4.resblock.module_list.1.1.conv.1.running_mean': 'models.63.bn45.running_mean',\n 'down4.resblock.module_list.1.1.conv.1.running_var': 'models.63.bn45.running_var',\n 'down4.resblock.module_list.1.1.conv.1.num_batches_tracked': 'models.63.bn45.num_batches_tracked',\n 'down4.resblock.module_list.2.0.conv.0.weight': 'models.65.conv46.weight',\n 'down4.resblock.module_list.2.0.conv.1.weight': 'models.65.bn46.weight',\n 'down4.resblock.module_list.2.0.conv.1.bias': 'models.65.bn46.bias',\n 'down4.resblock.module_list.2.0.conv.1.running_mean': 'models.65.bn46.running_mean',\n 'down4.resblock.module_list.2.0.conv.1.running_var': 'models.65.bn46.running_var',\n 'down4.resblock.module_list.2.0.conv.1.num_batches_tracked': 'models.65.bn46.num_batches_tracked',\n 'down4.resblock.module_list.2.1.conv.0.weight': 'models.66.conv47.weight',\n 'down4.resblock.module_list.2.1.conv.1.weight': 'models.66.bn47.weight',\n 'down4.resblock.module_list.2.1.conv.1.bias': 'models.66.bn47.bias',\n 'down4.resblock.module_list.2.1.conv.1.running_mean': 'models.66.bn47.running_mean',\n 'down4.resblock.module_list.2.1.conv.1.running_var': 'models.66.bn47.running_var',\n 'down4.resblock.module_list.2.1.conv.1.num_batches_tracked': 'models.66.bn47.num_batches_tracked',\n 'down4.resblock.module_list.3.0.conv.0.weight': 'models.68.conv48.weight',\n 'down4.resblock.module_list.3.0.conv.1.weight': 'models.68.bn48.weight',\n 'down4.resblock.module_list.3.0.conv.1.bias': 'models.68.bn48.bias',\n 'down4.resblock.module_list.3.0.conv.1.running_mean': 'models.68.bn48.running_mean',\n 'down4.resblock.module_list.3.0.conv.1.running_var': 'models.68.bn48.running_var',\n 'down4.resblock.module_list.3.0.conv.1.num_batches_tracked': 'models.68.bn48.num_batches_tracked',\n 'down4.resblock.module_list.3.1.conv.0.weight': 'models.69.conv49.weight',\n 'down4.resblock.module_list.3.1.conv.1.weight': 'models.69.bn49.weight',\n 'down4.resblock.module_list.3.1.conv.1.bias': 'models.69.bn49.bias',\n 'down4.resblock.module_list.3.1.conv.1.running_mean': 'models.69.bn49.running_mean',\n 'down4.resblock.module_list.3.1.conv.1.running_var': 'models.69.bn49.running_var',\n 'down4.resblock.module_list.3.1.conv.1.num_batches_tracked': 'models.69.bn49.num_batches_tracked',\n 'down4.resblock.module_list.4.0.conv.0.weight': 'models.71.conv50.weight',\n 'down4.resblock.module_list.4.0.conv.1.weight': 'models.71.bn50.weight',\n 'down4.resblock.module_list.4.0.conv.1.bias': 'models.71.bn50.bias',\n 'down4.resblock.module_list.4.0.conv.1.running_mean': 'models.71.bn50.running_mean',\n 'down4.resblock.module_list.4.0.conv.1.running_var': 'models.71.bn50.running_var',\n 'down4.resblock.module_list.4.0.conv.1.num_batches_tracked': 'models.71.bn50.num_batches_tracked',\n 'down4.resblock.module_list.4.1.conv.0.weight': 'models.72.conv51.weight',\n 'down4.resblock.module_list.4.1.conv.1.weight': 'models.72.bn51.weight',\n 'down4.resblock.module_list.4.1.conv.1.bias': 'models.72.bn51.bias',\n 'down4.resblock.module_list.4.1.conv.1.running_mean': 'models.72.bn51.running_mean',\n 'down4.resblock.module_list.4.1.conv.1.running_var': 'models.72.bn51.running_var',\n 'down4.resblock.module_list.4.1.conv.1.num_batches_tracked': 'models.72.bn51.num_batches_tracked',\n 'down4.resblock.module_list.5.0.conv.0.weight': 'models.74.conv52.weight',\n 'down4.resblock.module_list.5.0.conv.1.weight': 'models.74.bn52.weight',\n 'down4.resblock.module_list.5.0.conv.1.bias': 'models.74.bn52.bias',\n 'down4.resblock.module_list.5.0.conv.1.running_mean': 'models.74.bn52.running_mean',\n 'down4.resblock.module_list.5.0.conv.1.running_var': 'models.74.bn52.running_var',\n 'down4.resblock.module_list.5.0.conv.1.num_batches_tracked': 'models.74.bn52.num_batches_tracked',\n 'down4.resblock.module_list.5.1.conv.0.weight': 'models.75.conv53.weight',\n 'down4.resblock.module_list.5.1.conv.1.weight': 'models.75.bn53.weight',\n 'down4.resblock.module_list.5.1.conv.1.bias': 'models.75.bn53.bias',\n 'down4.resblock.module_list.5.1.conv.1.running_mean': 'models.75.bn53.running_mean',\n 'down4.resblock.module_list.5.1.conv.1.running_var': 'models.75.bn53.running_var',\n 'down4.resblock.module_list.5.1.conv.1.num_batches_tracked': 'models.75.bn53.num_batches_tracked',\n 'down4.resblock.module_list.6.0.conv.0.weight': 'models.77.conv54.weight',\n 'down4.resblock.module_list.6.0.conv.1.weight': 'models.77.bn54.weight',\n 'down4.resblock.module_list.6.0.conv.1.bias': 'models.77.bn54.bias',\n 'down4.resblock.module_list.6.0.conv.1.running_mean': 'models.77.bn54.running_mean',\n 'down4.resblock.module_list.6.0.conv.1.running_var': 'models.77.bn54.running_var',\n 'down4.resblock.module_list.6.0.conv.1.num_batches_tracked': 'models.77.bn54.num_batches_tracked',\n 'down4.resblock.module_list.6.1.conv.0.weight': 'models.78.conv55.weight',\n 'down4.resblock.module_list.6.1.conv.1.weight': 'models.78.bn55.weight',\n 'down4.resblock.module_list.6.1.conv.1.bias': 'models.78.bn55.bias',\n 'down4.resblock.module_list.6.1.conv.1.running_mean': 'models.78.bn55.running_mean',\n 'down4.resblock.module_list.6.1.conv.1.running_var': 'models.78.bn55.running_var',\n 'down4.resblock.module_list.6.1.conv.1.num_batches_tracked': 'models.78.bn55.num_batches_tracked',\n 'down4.resblock.module_list.7.0.conv.0.weight': 'models.80.conv56.weight',\n 'down4.resblock.module_list.7.0.conv.1.weight': 'models.80.bn56.weight',\n 'down4.resblock.module_list.7.0.conv.1.bias': 'models.80.bn56.bias',\n 'down4.resblock.module_list.7.0.conv.1.running_mean': 'models.80.bn56.running_mean',\n 'down4.resblock.module_list.7.0.conv.1.running_var': 'models.80.bn56.running_var',\n 'down4.resblock.module_list.7.0.conv.1.num_batches_tracked': 'models.80.bn56.num_batches_tracked',\n 'down4.resblock.module_list.7.1.conv.0.weight': 'models.81.conv57.weight',\n 'down4.resblock.module_list.7.1.conv.1.weight': 'models.81.bn57.weight',\n 'down4.resblock.module_list.7.1.conv.1.bias': 'models.81.bn57.bias',\n 'down4.resblock.module_list.7.1.conv.1.running_mean': 'models.81.bn57.running_mean',\n 'down4.resblock.module_list.7.1.conv.1.running_var': 'models.81.bn57.running_var',\n 'down4.resblock.module_list.7.1.conv.1.num_batches_tracked': 'models.81.bn57.num_batches_tracked',\n 'down4.conv4.conv.0.weight': 'models.83.conv58.weight',\n 'down4.conv4.conv.1.weight': 'models.83.bn58.weight',\n 'down4.conv4.conv.1.bias': 'models.83.bn58.bias',\n 'down4.conv4.conv.1.running_mean': 'models.83.bn58.running_mean',\n 'down4.conv4.conv.1.running_var': 'models.83.bn58.running_var',\n 'down4.conv4.conv.1.num_batches_tracked': 'models.83.bn58.num_batches_tracked',\n 'down4.conv5.conv.0.weight': 'models.85.conv59.weight',\n 'down4.conv5.conv.1.weight': 'models.85.bn59.weight',\n 'down4.conv5.conv.1.bias': 'models.85.bn59.bias',\n 'down4.conv5.conv.1.running_mean': 'models.85.bn59.running_mean',\n 'down4.conv5.conv.1.running_var': 'models.85.bn59.running_var',\n 'down4.conv5.conv.1.num_batches_tracked': 'models.85.bn59.num_batches_tracked',\n 'down5.conv1.conv.0.weight': 'models.86.conv60.weight',\n 'down5.conv1.conv.1.weight': 'models.86.bn60.weight',\n 'down5.conv1.conv.1.bias': 'models.86.bn60.bias',\n 'down5.conv1.conv.1.running_mean': 'models.86.bn60.running_mean',\n 'down5.conv1.conv.1.running_var': 'models.86.bn60.running_var',\n 'down5.conv1.conv.1.num_batches_tracked': 'models.86.bn60.num_batches_tracked',\n 'down5.conv2.conv.0.weight': 'models.87.conv61.weight',\n 'down5.conv2.conv.1.weight': 'models.87.bn61.weight',\n 'down5.conv2.conv.1.bias': 'models.87.bn61.bias',\n 'down5.conv2.conv.1.running_mean': 'models.87.bn61.running_mean',\n 'down5.conv2.conv.1.running_var': 'models.87.bn61.running_var',\n 'down5.conv2.conv.1.num_batches_tracked': 'models.87.bn61.num_batches_tracked',\n 'down5.conv3.conv.0.weight': 'models.89.conv62.weight',\n 'down5.conv3.conv.1.weight': 'models.89.bn62.weight',\n 'down5.conv3.conv.1.bias': 'models.89.bn62.bias',\n 'down5.conv3.conv.1.running_mean': 'models.89.bn62.running_mean',\n 'down5.conv3.conv.1.running_var': 'models.89.bn62.running_var',\n 'down5.conv3.conv.1.num_batches_tracked': 'models.89.bn62.num_batches_tracked',\n 'down5.resblock.module_list.0.0.conv.0.weight': 'models.90.conv63.weight',\n 'down5.resblock.module_list.0.0.conv.1.weight': 'models.90.bn63.weight',\n 'down5.resblock.module_list.0.0.conv.1.bias': 'models.90.bn63.bias',\n 'down5.resblock.module_list.0.0.conv.1.running_mean': 'models.90.bn63.running_mean',\n 'down5.resblock.module_list.0.0.conv.1.running_var': 'models.90.bn63.running_var',\n 'down5.resblock.module_list.0.0.conv.1.num_batches_tracked': 'models.90.bn63.num_batches_tracked',\n 'down5.resblock.module_list.0.1.conv.0.weight': 'models.91.conv64.weight',\n 'down5.resblock.module_list.0.1.conv.1.weight': 'models.91.bn64.weight',\n 'down5.resblock.module_list.0.1.conv.1.bias': 'models.91.bn64.bias',\n 'down5.resblock.module_list.0.1.conv.1.running_mean': 'models.91.bn64.running_mean',\n 'down5.resblock.module_list.0.1.conv.1.running_var': 'models.91.bn64.running_var',\n 'down5.resblock.module_list.0.1.conv.1.num_batches_tracked': 'models.91.bn64.num_batches_tracked',\n 'down5.resblock.module_list.1.0.conv.0.weight': 'models.93.conv65.weight',\n 'down5.resblock.module_list.1.0.conv.1.weight': 'models.93.bn65.weight',\n 'down5.resblock.module_list.1.0.conv.1.bias': 'models.93.bn65.bias',\n 'down5.resblock.module_list.1.0.conv.1.running_mean': 'models.93.bn65.running_mean',\n 'down5.resblock.module_list.1.0.conv.1.running_var': 'models.93.bn65.running_var',\n 'down5.resblock.module_list.1.0.conv.1.num_batches_tracked': 'models.93.bn65.num_batches_tracked',\n 'down5.resblock.module_list.1.1.conv.0.weight': 'models.94.conv66.weight',\n 'down5.resblock.module_list.1.1.conv.1.weight': 'models.94.bn66.weight',\n 'down5.resblock.module_list.1.1.conv.1.bias': 'models.94.bn66.bias',\n 'down5.resblock.module_list.1.1.conv.1.running_mean': 'models.94.bn66.running_mean',\n 'down5.resblock.module_list.1.1.conv.1.running_var': 'models.94.bn66.running_var',\n 'down5.resblock.module_list.1.1.conv.1.num_batches_tracked': 'models.94.bn66.num_batches_tracked',\n 'down5.resblock.module_list.2.0.conv.0.weight': 'models.96.conv67.weight',\n 'down5.resblock.module_list.2.0.conv.1.weight': 'models.96.bn67.weight',\n 'down5.resblock.module_list.2.0.conv.1.bias': 'models.96.bn67.bias',\n 'down5.resblock.module_list.2.0.conv.1.running_mean': 'models.96.bn67.running_mean',\n 'down5.resblock.module_list.2.0.conv.1.running_var': 'models.96.bn67.running_var',\n 'down5.resblock.module_list.2.0.conv.1.num_batches_tracked': 'models.96.bn67.num_batches_tracked',\n 'down5.resblock.module_list.2.1.conv.0.weight': 'models.97.conv68.weight',\n 'down5.resblock.module_list.2.1.conv.1.weight': 'models.97.bn68.weight',\n 'down5.resblock.module_list.2.1.conv.1.bias': 'models.97.bn68.bias',\n 'down5.resblock.module_list.2.1.conv.1.running_mean': 'models.97.bn68.running_mean',\n 'down5.resblock.module_list.2.1.conv.1.running_var': 'models.97.bn68.running_var',\n 'down5.resblock.module_list.2.1.conv.1.num_batches_tracked': 'models.97.bn68.num_batches_tracked',\n 'down5.resblock.module_list.3.0.conv.0.weight': 'models.99.conv69.weight',\n 'down5.resblock.module_list.3.0.conv.1.weight': 'models.99.bn69.weight',\n 'down5.resblock.module_list.3.0.conv.1.bias': 'models.99.bn69.bias',\n 'down5.resblock.module_list.3.0.conv.1.running_mean': 'models.99.bn69.running_mean',\n 'down5.resblock.module_list.3.0.conv.1.running_var': 'models.99.bn69.running_var',\n 'down5.resblock.module_list.3.0.conv.1.num_batches_tracked': 'models.99.bn69.num_batches_tracked',\n 'down5.resblock.module_list.3.1.conv.0.weight': 'models.100.conv70.weight',\n 'down5.resblock.module_list.3.1.conv.1.weight': 'models.100.bn70.weight',\n 'down5.resblock.module_list.3.1.conv.1.bias': 'models.100.bn70.bias',\n 'down5.resblock.module_list.3.1.conv.1.running_mean': 'models.100.bn70.running_mean',\n 'down5.resblock.module_list.3.1.conv.1.running_var': 'models.100.bn70.running_var',\n 'down5.resblock.module_list.3.1.conv.1.num_batches_tracked': 'models.100.bn70.num_batches_tracked',\n 'down5.conv4.conv.0.weight': 'models.102.conv71.weight',\n 'down5.conv4.conv.1.weight': 'models.102.bn71.weight',\n 'down5.conv4.conv.1.bias': 'models.102.bn71.bias',\n 'down5.conv4.conv.1.running_mean': 'models.102.bn71.running_mean',\n 'down5.conv4.conv.1.running_var': 'models.102.bn71.running_var',\n 'down5.conv4.conv.1.num_batches_tracked': 'models.102.bn71.num_batches_tracked',\n 'down5.conv5.conv.0.weight': 'models.104.conv72.weight',\n 'down5.conv5.conv.1.weight': 'models.104.bn72.weight',\n 'down5.conv5.conv.1.bias': 'models.104.bn72.bias',\n 'down5.conv5.conv.1.running_mean': 'models.104.bn72.running_mean',\n 'down5.conv5.conv.1.running_var': 'models.104.bn72.running_var',\n 'down5.conv5.conv.1.num_batches_tracked': 'models.104.bn72.num_batches_tracked',\n 'neek.conv1.conv.0.weight': 'models.105.conv73.weight',\n 'neek.conv1.conv.1.weight': 'models.105.bn73.weight',\n 'neek.conv1.conv.1.bias': 'models.105.bn73.bias',\n 'neek.conv1.conv.1.running_mean': 'models.105.bn73.running_mean',\n 'neek.conv1.conv.1.running_var': 'models.105.bn73.running_var',\n 'neek.conv1.conv.1.num_batches_tracked': 'models.105.bn73.num_batches_tracked',\n 'neek.conv2.conv.0.weight': 'models.106.conv74.weight',\n 'neek.conv2.conv.1.weight': 'models.106.bn74.weight',\n 'neek.conv2.conv.1.bias': 'models.106.bn74.bias',\n 'neek.conv2.conv.1.running_mean': 'models.106.bn74.running_mean',\n 'neek.conv2.conv.1.running_var': 'models.106.bn74.running_var',\n 'neek.conv2.conv.1.num_batches_tracked': 'models.106.bn74.num_batches_tracked',\n 'neek.conv3.conv.0.weight': 'models.107.conv75.weight',\n 'neek.conv3.conv.1.weight': 'models.107.bn75.weight',\n 'neek.conv3.conv.1.bias': 'models.107.bn75.bias',\n 'neek.conv3.conv.1.running_mean': 'models.107.bn75.running_mean',\n 'neek.conv3.conv.1.running_var': 'models.107.bn75.running_var',\n 'neek.conv3.conv.1.num_batches_tracked': 'models.107.bn75.num_batches_tracked',\n 'neek.conv4.conv.0.weight': 'models.114.conv76.weight',\n 'neek.conv4.conv.1.weight': 'models.114.bn76.weight',\n 'neek.conv4.conv.1.bias': 'models.114.bn76.bias',\n 'neek.conv4.conv.1.running_mean': 'models.114.bn76.running_mean',\n 'neek.conv4.conv.1.running_var': 'models.114.bn76.running_var',\n 'neek.conv4.conv.1.num_batches_tracked': 'models.114.bn76.num_batches_tracked',\n 'neek.conv5.conv.0.weight': 'models.115.conv77.weight',\n 'neek.conv5.conv.1.weight': 'models.115.bn77.weight',\n 'neek.conv5.conv.1.bias': 'models.115.bn77.bias',\n 'neek.conv5.conv.1.running_mean': 'models.115.bn77.running_mean',\n 'neek.conv5.conv.1.running_var': 'models.115.bn77.running_var',\n 'neek.conv5.conv.1.num_batches_tracked': 'models.115.bn77.num_batches_tracked',\n 'neek.conv6.conv.0.weight': 'models.116.conv78.weight',\n 'neek.conv6.conv.1.weight': 'models.116.bn78.weight',\n 'neek.conv6.conv.1.bias': 'models.116.bn78.bias',\n 'neek.conv6.conv.1.running_mean': 'models.116.bn78.running_mean',\n 'neek.conv6.conv.1.running_var': 'models.116.bn78.running_var',\n 'neek.conv6.conv.1.num_batches_tracked': 'models.116.bn78.num_batches_tracked',\n 'neek.conv7.conv.0.weight': 'models.117.conv79.weight',\n 'neek.conv7.conv.1.weight': 'models.117.bn79.weight',\n 'neek.conv7.conv.1.bias': 'models.117.bn79.bias',\n 'neek.conv7.conv.1.running_mean': 'models.117.bn79.running_mean',\n 'neek.conv7.conv.1.running_var': 'models.117.bn79.running_var',\n 'neek.conv7.conv.1.num_batches_tracked': 'models.117.bn79.num_batches_tracked',\n 'neek.conv8.conv.0.weight': 'models.120.conv80.weight',\n 'neek.conv8.conv.1.weight': 'models.120.bn80.weight',\n 'neek.conv8.conv.1.bias': 'models.120.bn80.bias',\n 'neek.conv8.conv.1.running_mean': 'models.120.bn80.running_mean',\n 'neek.conv8.conv.1.running_var': 'models.120.bn80.running_var',\n 'neek.conv8.conv.1.num_batches_tracked': 'models.120.bn80.num_batches_tracked',\n 'neek.conv9.conv.0.weight': 'models.122.conv81.weight',\n 'neek.conv9.conv.1.weight': 'models.122.bn81.weight',\n 'neek.conv9.conv.1.bias': 'models.122.bn81.bias',\n 'neek.conv9.conv.1.running_mean': 'models.122.bn81.running_mean',\n 'neek.conv9.conv.1.running_var': 'models.122.bn81.running_var',\n 'neek.conv9.conv.1.num_batches_tracked': 'models.122.bn81.num_batches_tracked',\n 'neek.conv10.conv.0.weight': 'models.123.conv82.weight',\n 'neek.conv10.conv.1.weight': 'models.123.bn82.weight',\n 'neek.conv10.conv.1.bias': 'models.123.bn82.bias',\n 'neek.conv10.conv.1.running_mean': 'models.123.bn82.running_mean',\n 'neek.conv10.conv.1.running_var': 'models.123.bn82.running_var',\n 'neek.conv10.conv.1.num_batches_tracked': 'models.123.bn82.num_batches_tracked',\n 'neek.conv11.conv.0.weight': 'models.124.conv83.weight',\n 'neek.conv11.conv.1.weight': 'models.124.bn83.weight',\n 'neek.conv11.conv.1.bias': 'models.124.bn83.bias',\n 'neek.conv11.conv.1.running_mean': 'models.124.bn83.running_mean',\n 'neek.conv11.conv.1.running_var': 'models.124.bn83.running_var',\n 'neek.conv11.conv.1.num_batches_tracked': 'models.124.bn83.num_batches_tracked',\n 'neek.conv12.conv.0.weight': 'models.125.conv84.weight',\n 'neek.conv12.conv.1.weight': 'models.125.bn84.weight',\n 'neek.conv12.conv.1.bias': 'models.125.bn84.bias',\n 'neek.conv12.conv.1.running_mean': 'models.125.bn84.running_mean',\n 'neek.conv12.conv.1.running_var': 'models.125.bn84.running_var',\n 'neek.conv12.conv.1.num_batches_tracked': 'models.125.bn84.num_batches_tracked',\n 'neek.conv13.conv.0.weight': 'models.126.conv85.weight',\n 'neek.conv13.conv.1.weight': 'models.126.bn85.weight',\n 'neek.conv13.conv.1.bias': 'models.126.bn85.bias',\n 'neek.conv13.conv.1.running_mean': 'models.126.bn85.running_mean',\n 'neek.conv13.conv.1.running_var': 'models.126.bn85.running_var',\n 'neek.conv13.conv.1.num_batches_tracked': 'models.126.bn85.num_batches_tracked',\n 'neek.conv14.conv.0.weight': 'models.127.conv86.weight',\n 'neek.conv14.conv.1.weight': 'models.127.bn86.weight',\n 'neek.conv14.conv.1.bias': 'models.127.bn86.bias',\n 'neek.conv14.conv.1.running_mean': 'models.127.bn86.running_mean',\n 'neek.conv14.conv.1.running_var': 'models.127.bn86.running_var',\n 'neek.conv14.conv.1.num_batches_tracked': 'models.127.bn86.num_batches_tracked',\n 'neek.conv15.conv.0.weight': 'models.130.conv87.weight',\n 'neek.conv15.conv.1.weight': 'models.130.bn87.weight',\n 'neek.conv15.conv.1.bias': 'models.130.bn87.bias',\n 'neek.conv15.conv.1.running_mean': 'models.130.bn87.running_mean',\n 'neek.conv15.conv.1.running_var': 'models.130.bn87.running_var',\n 'neek.conv15.conv.1.num_batches_tracked': 'models.130.bn87.num_batches_tracked',\n 'neek.conv16.conv.0.weight': 'models.132.conv88.weight',\n 'neek.conv16.conv.1.weight': 'models.132.bn88.weight',\n 'neek.conv16.conv.1.bias': 'models.132.bn88.bias',\n 'neek.conv16.conv.1.running_mean': 'models.132.bn88.running_mean',\n 'neek.conv16.conv.1.running_var': 'models.132.bn88.running_var',\n 'neek.conv16.conv.1.num_batches_tracked': 'models.132.bn88.num_batches_tracked',\n 'neek.conv17.conv.0.weight': 'models.133.conv89.weight',\n 'neek.conv17.conv.1.weight': 'models.133.bn89.weight',\n 'neek.conv17.conv.1.bias': 'models.133.bn89.bias',\n 'neek.conv17.conv.1.running_mean': 'models.133.bn89.running_mean',\n 'neek.conv17.conv.1.running_var': 'models.133.bn89.running_var',\n 'neek.conv17.conv.1.num_batches_tracked': 'models.133.bn89.num_batches_tracked',\n 'neek.conv18.conv.0.weight': 'models.134.conv90.weight',\n 'neek.conv18.conv.1.weight': 'models.134.bn90.weight',\n 'neek.conv18.conv.1.bias': 'models.134.bn90.bias',\n 'neek.conv18.conv.1.running_mean': 'models.134.bn90.running_mean',\n 'neek.conv18.conv.1.running_var': 'models.134.bn90.running_var',\n 'neek.conv18.conv.1.num_batches_tracked': 'models.134.bn90.num_batches_tracked',\n 'neek.conv19.conv.0.weight': 'models.135.conv91.weight',\n 'neek.conv19.conv.1.weight': 'models.135.bn91.weight',\n 'neek.conv19.conv.1.bias': 'models.135.bn91.bias',\n 'neek.conv19.conv.1.running_mean': 'models.135.bn91.running_mean',\n 'neek.conv19.conv.1.running_var': 'models.135.bn91.running_var',\n 'neek.conv19.conv.1.num_batches_tracked': 'models.135.bn91.num_batches_tracked',\n 'neek.conv20.conv.0.weight': 'models.136.conv92.weight',\n 'neek.conv20.conv.1.weight': 'models.136.bn92.weight',\n 'neek.conv20.conv.1.bias': 'models.136.bn92.bias',\n 'neek.conv20.conv.1.running_mean': 'models.136.bn92.running_mean',\n 'neek.conv20.conv.1.running_var': 'models.136.bn92.running_var',\n 'neek.conv20.conv.1.num_batches_tracked': 'models.136.bn92.num_batches_tracked',\n 'head.conv1.conv.0.weight': 'models.137.conv93.weight',\n 'head.conv1.conv.1.weight': 'models.137.bn93.weight',\n 'head.conv1.conv.1.bias': 'models.137.bn93.bias',\n 'head.conv1.conv.1.running_mean': 'models.137.bn93.running_mean',\n 'head.conv1.conv.1.running_var': 'models.137.bn93.running_var',\n 'head.conv1.conv.1.num_batches_tracked': 'models.137.bn93.num_batches_tracked',\n 'head.conv2.conv.0.weight': 'models.138.conv94.weight',\n 'head.conv2.conv.0.bias': 'models.138.conv94.bias',\n 'head.conv3.conv.0.weight': 'models.141.conv95.weight',\n 'head.conv3.conv.1.weight': 'models.141.bn95.weight',\n 'head.conv3.conv.1.bias': 'models.141.bn95.bias',\n 'head.conv3.conv.1.running_mean': 'models.141.bn95.running_mean',\n 'head.conv3.conv.1.running_var': 'models.141.bn95.running_var',\n 'head.conv3.conv.1.num_batches_tracked': 'models.141.bn95.num_batches_tracked',\n 'head.conv4.conv.0.weight': 'models.143.conv96.weight',\n 'head.conv4.conv.1.weight': 'models.143.bn96.weight',\n 'head.conv4.conv.1.bias': 'models.143.bn96.bias',\n 'head.conv4.conv.1.running_mean': 'models.143.bn96.running_mean',\n 'head.conv4.conv.1.running_var': 'models.143.bn96.running_var',\n 'head.conv4.conv.1.num_batches_tracked': 'models.143.bn96.num_batches_tracked',\n 'head.conv5.conv.0.weight': 'models.144.conv97.weight',\n 'head.conv5.conv.1.weight': 'models.144.bn97.weight',\n 'head.conv5.conv.1.bias': 'models.144.bn97.bias',\n 'head.conv5.conv.1.running_mean': 'models.144.bn97.running_mean',\n 'head.conv5.conv.1.running_var': 'models.144.bn97.running_var',\n 'head.conv5.conv.1.num_batches_tracked': 'models.144.bn97.num_batches_tracked',\n 'head.conv6.conv.0.weight': 'models.145.conv98.weight',\n 'head.conv6.conv.1.weight': 'models.145.bn98.weight',\n 'head.conv6.conv.1.bias': 'models.145.bn98.bias',\n 'head.conv6.conv.1.running_mean': 'models.145.bn98.running_mean',\n 'head.conv6.conv.1.running_var': 'models.145.bn98.running_var',\n 'head.conv6.conv.1.num_batches_tracked': 'models.145.bn98.num_batches_tracked',\n 'head.conv7.conv.0.weight': 'models.146.conv99.weight',\n 'head.conv7.conv.1.weight': 'models.146.bn99.weight',\n 'head.conv7.conv.1.bias': 'models.146.bn99.bias',\n 'head.conv7.conv.1.running_mean': 'models.146.bn99.running_mean',\n 'head.conv7.conv.1.running_var': 'models.146.bn99.running_var',\n 'head.conv7.conv.1.num_batches_tracked': 'models.146.bn99.num_batches_tracked',\n 'head.conv8.conv.0.weight': 'models.147.conv100.weight',\n 'head.conv8.conv.1.weight': 'models.147.bn100.weight',\n 'head.conv8.conv.1.bias': 'models.147.bn100.bias',\n 'head.conv8.conv.1.running_mean': 'models.147.bn100.running_mean',\n 'head.conv8.conv.1.running_var': 'models.147.bn100.running_var',\n 'head.conv8.conv.1.num_batches_tracked': 'models.147.bn100.num_batches_tracked',\n 'head.conv9.conv.0.weight': 'models.148.conv101.weight',\n 'head.conv9.conv.1.weight': 'models.148.bn101.weight',\n 'head.conv9.conv.1.bias': 'models.148.bn101.bias',\n 'head.conv9.conv.1.running_mean': 'models.148.bn101.running_mean',\n 'head.conv9.conv.1.running_var': 'models.148.bn101.running_var',\n 'head.conv9.conv.1.num_batches_tracked': 'models.148.bn101.num_batches_tracked',\n 'head.conv10.conv.0.weight': 'models.149.conv102.weight',\n 'head.conv10.conv.0.bias': 'models.149.conv102.bias',\n 'head.conv11.conv.0.weight': 'models.152.conv103.weight',\n 'head.conv11.conv.1.weight': 'models.152.bn103.weight',\n 'head.conv11.conv.1.bias': 'models.152.bn103.bias',\n 'head.conv11.conv.1.running_mean': 'models.152.bn103.running_mean',\n 'head.conv11.conv.1.running_var': 'models.152.bn103.running_var',\n 'head.conv11.conv.1.num_batches_tracked': 'models.152.bn103.num_batches_tracked',\n 'head.conv12.conv.0.weight': 'models.154.conv104.weight',\n 'head.conv12.conv.1.weight': 'models.154.bn104.weight',\n 'head.conv12.conv.1.bias': 'models.154.bn104.bias',\n 'head.conv12.conv.1.running_mean': 'models.154.bn104.running_mean',\n 'head.conv12.conv.1.running_var': 'models.154.bn104.running_var',\n 'head.conv12.conv.1.num_batches_tracked': 'models.154.bn104.num_batches_tracked',\n 'head.conv13.conv.0.weight': 'models.155.conv105.weight',\n 'head.conv13.conv.1.weight': 'models.155.bn105.weight',\n 'head.conv13.conv.1.bias': 'models.155.bn105.bias',\n 'head.conv13.conv.1.running_mean': 'models.155.bn105.running_mean',\n 'head.conv13.conv.1.running_var': 'models.155.bn105.running_var',\n 'head.conv13.conv.1.num_batches_tracked': 'models.155.bn105.num_batches_tracked',\n 'head.conv14.conv.0.weight': 'models.156.conv106.weight',\n 'head.conv14.conv.1.weight': 'models.156.bn106.weight',\n 'head.conv14.conv.1.bias': 'models.156.bn106.bias',\n 'head.conv14.conv.1.running_mean': 'models.156.bn106.running_mean',\n 'head.conv14.conv.1.running_var': 'models.156.bn106.running_var',\n 'head.conv14.conv.1.num_batches_tracked': 'models.156.bn106.num_batches_tracked',\n 'head.conv15.conv.0.weight': 'models.157.conv107.weight',\n 'head.conv15.conv.1.weight': 'models.157.bn107.weight',\n 'head.conv15.conv.1.bias': 'models.157.bn107.bias',\n 'head.conv15.conv.1.running_mean': 'models.157.bn107.running_mean',\n 'head.conv15.conv.1.running_var': 'models.157.bn107.running_var',\n 'head.conv15.conv.1.num_batches_tracked': 'models.157.bn107.num_batches_tracked',\n 'head.conv16.conv.0.weight': 'models.158.conv108.weight',\n 'head.conv16.conv.1.weight': 'models.158.bn108.weight',\n 'head.conv16.conv.1.bias': 'models.158.bn108.bias',\n 'head.conv16.conv.1.running_mean': 'models.158.bn108.running_mean',\n 'head.conv16.conv.1.running_var': 'models.158.bn108.running_var',\n 'head.conv16.conv.1.num_batches_tracked': 'models.158.bn108.num_batches_tracked',\n 'head.conv17.conv.0.weight': 'models.159.conv109.weight',\n 'head.conv17.conv.1.weight': 'models.159.bn109.weight',\n 'head.conv17.conv.1.bias': 'models.159.bn109.bias',\n 'head.conv17.conv.1.running_mean': 'models.159.bn109.running_mean',\n 'head.conv17.conv.1.running_var': 'models.159.bn109.running_var',\n 'head.conv17.conv.1.num_batches_tracked': 'models.159.bn109.num_batches_tracked',\n 'head.conv18.conv.0.weight': 'models.160.conv110.weight',\n 'head.conv18.conv.0.bias': 'models.160.conv110.bias',\n }\n pth_weights = torch.load(checkpoint)\n pt_weights = type(pth_weights)()\n for name, new_name in name_mapping.items():\n pt_weights[new_name] = pth_weights[name]\n return pt_weights\n\n\ndef convert_pt_checkpoint_to_keras_h5(state_dict):\n print('============================================================')\n\n def copy1(conv, bn, idx):\n keyword1 = 'conv%d.weight' % idx\n keyword2 = 'bn%d.weight' % idx\n keyword3 = 'bn%d.bias' % idx\n keyword4 = 'bn%d.running_mean' % idx\n keyword5 = 'bn%d.running_var' % idx\n for key in state_dict:\n value = state_dict[key].numpy()\n if keyword1 in key:\n w = value\n elif keyword2 in key:\n y = value\n elif keyword3 in key:\n b = value\n elif keyword4 in key:\n m = value\n elif keyword5 in key:\n v = value\n w = w.transpose(2, 3, 1, 0)\n conv.set_weights([w])\n bn.set_weights([y, b, m, v])\n\n def copy2(conv, idx):\n keyword1 = 'conv%d.weight' % idx\n keyword2 = 'conv%d.bias' % idx\n for key in state_dict:\n value = state_dict[key].numpy()\n if keyword1 in key:\n w = value\n elif keyword2 in key:\n b = value\n w = w.transpose(2, 3, 1, 0)\n conv.set_weights([w, b])\n\n num_classes = 80\n num_anchors = 3\n\n with tf.Session(graph=tf.Graph()):\n inputs = layers.Input(shape=[], dtype='string')\n model_body = YOLOv4(inputs, num_classes, num_anchors)\n model_body.summary()\n layer_name_to_idx = {layer.name: idx for idx, layer in enumerate(model_body.layers)}\n\n print('\\nCopying...')\n i1 = layer_name_to_idx['conv2d']\n i2 = layer_name_to_idx['batch_normalization']\n copy1(model_body.layers[i1], model_body.layers[i2], 1)\n for i in range(2, 94, 1):\n i1 = layer_name_to_idx['conv2d_%d' % (i - 1)]\n i2 = layer_name_to_idx['batch_normalization_%d' % (i - 1)]\n copy1(model_body.layers[i1], model_body.layers[i2], i)\n for i in range(95, 102, 1):\n i1 = layer_name_to_idx['conv2d_%d' % (i - 1)]\n i2 = layer_name_to_idx['batch_normalization_%d' % (i - 2,)]\n copy1(model_body.layers[i1], model_body.layers[i2], i)\n for i in range(103, 110, 1):\n i1 = layer_name_to_idx['conv2d_%d' % (i - 1)]\n i2 = layer_name_to_idx['batch_normalization_%d' % (i - 3,)]\n copy1(model_body.layers[i1], model_body.layers[i2], i)\n\n i1 = layer_name_to_idx['conv2d_93']\n copy2(model_body.layers[i1], 94)\n i1 = layer_name_to_idx['conv2d_101']\n copy2(model_body.layers[i1], 102)\n i1 = layer_name_to_idx['conv2d_109']\n copy2(model_body.layers[i1], 110)\n\n weights = model_body.get_weights()\n print('\\nDone.')\n return weights\n\n\nclass Mish(layers.Layer):\n\n def __init__(self):\n super(Mish, self).__init__()\n\n def compute_output_shape(self, input_shape):\n return input_shape\n\n def call(self, x):\n return x * tf.tanh(tf.math.softplus(x))\n\n\ndef conv2d_unit(x, filters, kernels, strides=1, padding='valid', bn=1, act='mish'):\n use_bias = (bn != 1)\n x = layers.Conv2D(filters, kernels,\n padding=padding,\n strides=strides,\n use_bias=use_bias,\n activation='linear',\n kernel_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.01))(x)\n if bn:\n x = layers.BatchNormalization(fused=False)(x)\n if act == 'leaky':\n x = keras.layers.LeakyReLU(alpha=0.1)(x)\n elif act == 'mish':\n x = Mish()(x)\n return x\n\n\ndef residual_block(inputs, filters_1, filters_2):\n x = conv2d_unit(inputs, filters_1, 1, strides=1, padding='valid')\n x = conv2d_unit(x, filters_2, 3, strides=1, padding='same')\n x = layers.add([inputs, x])\n return x\n\n\ndef stack_residual_block(inputs, filters_1, filters_2, n):\n x = residual_block(inputs, filters_1, filters_2)\n for i in range(n - 1):\n x = residual_block(x, filters_1, filters_2)\n return x\n\n\ndef spp(x):\n x_1 = x\n x_2 = layers.MaxPooling2D(pool_size=5, strides=1, padding='same')(x)\n x_3 = layers.MaxPooling2D(pool_size=9, strides=1, padding='same')(x)\n x_4 = layers.MaxPooling2D(pool_size=13, strides=1, padding='same')(x)\n out = layers.Concatenate()([x_4, x_3, x_2, x_1])\n return out\n\n\ndef YOLOv4(inputs, num_classes, num_anchors, input_shape=(608, 608), initial_filters=32,\n fast=False, anchors=None, conf_thresh=0.05, nms_thresh=0.45, keep_top_k=100, nms_top_k=100):\n i32 = initial_filters\n i64 = i32 * 2\n i128 = i32 * 4\n i256 = i32 * 8\n i512 = i32 * 16\n i1024 = i32 * 32\n\n x, image_shape = layers.Lambda(lambda t: preprocessor(t, input_shape))(inputs)\n\n # cspdarknet53\n x = conv2d_unit(x, i32, 3, strides=1, padding='same')\n\n # ============================= s2 =============================\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(x)\n x = conv2d_unit(x, i64, 3, strides=2)\n s2 = conv2d_unit(x, i64, 1, strides=1)\n x = conv2d_unit(x, i64, 1, strides=1)\n x = stack_residual_block(x, i32, i64, n=1)\n x = conv2d_unit(x, i64, 1, strides=1)\n x = layers.Concatenate()([x, s2])\n s2 = conv2d_unit(x, i64, 1, strides=1)\n\n # ============================= s4 =============================\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(s2)\n x = conv2d_unit(x, i128, 3, strides=2)\n s4 = conv2d_unit(x, i64, 1, strides=1)\n x = conv2d_unit(x, i64, 1, strides=1)\n x = stack_residual_block(x, i64, i64, n=2)\n x = conv2d_unit(x, i64, 1, strides=1)\n x = layers.Concatenate()([x, s4])\n s4 = conv2d_unit(x, i128, 1, strides=1)\n\n # ============================= s8 =============================\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(s4)\n x = conv2d_unit(x, i256, 3, strides=2)\n s8 = conv2d_unit(x, i128, 1, strides=1)\n x = conv2d_unit(x, i128, 1, strides=1)\n x = stack_residual_block(x, i128, i128, n=8)\n x = conv2d_unit(x, i128, 1, strides=1)\n x = layers.Concatenate()([x, s8])\n s8 = conv2d_unit(x, i256, 1, strides=1)\n\n # ============================= s16 =============================\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(s8)\n x = conv2d_unit(x, i512, 3, strides=2)\n s16 = conv2d_unit(x, i256, 1, strides=1)\n x = conv2d_unit(x, i256, 1, strides=1)\n x = stack_residual_block(x, i256, i256, n=8)\n x = conv2d_unit(x, i256, 1, strides=1)\n x = layers.Concatenate()([x, s16])\n s16 = conv2d_unit(x, i512, 1, strides=1)\n\n # ============================= s32 =============================\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(s16)\n x = conv2d_unit(x, i1024, 3, strides=2)\n s32 = conv2d_unit(x, i512, 1, strides=1)\n x = conv2d_unit(x, i512, 1, strides=1)\n x = stack_residual_block(x, i512, i512, n=4)\n x = conv2d_unit(x, i512, 1, strides=1)\n x = layers.Concatenate()([x, s32])\n s32 = conv2d_unit(x, i1024, 1, strides=1)\n\n # fpn\n x = conv2d_unit(s32, i512, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i1024, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i512, 1, strides=1, act='leaky')\n x = spp(x)\n\n x = conv2d_unit(x, i512, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i1024, 3, strides=1, padding='same', act='leaky')\n fpn_s32 = conv2d_unit(x, i512, 1, strides=1, act='leaky')\n\n # pan01\n x = conv2d_unit(fpn_s32, i256, 1, strides=1, act='leaky')\n x = layers.UpSampling2D(2)(x)\n s16 = conv2d_unit(s16, i256, 1, strides=1, act='leaky')\n x = layers.Concatenate()([s16, x])\n x = conv2d_unit(x, i256, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i512, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i256, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i512, 3, strides=1, padding='same', act='leaky')\n fpn_s16 = conv2d_unit(x, i256, 1, strides=1, act='leaky')\n\n # pan02\n x = conv2d_unit(fpn_s16, i128, 1, strides=1, act='leaky')\n x = layers.UpSampling2D(2)(x)\n s8 = conv2d_unit(s8, i128, 1, strides=1, act='leaky')\n x = layers.Concatenate()([s8, x])\n x = conv2d_unit(x, i128, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i256, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i128, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i256, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i128, 1, strides=1, act='leaky')\n\n # output_s, doesn't need concat()\n output_s = conv2d_unit(x, i256, 3, strides=1, padding='same', act='leaky')\n output_s = conv2d_unit(output_s, num_anchors * (num_classes + 5), 1, strides=1, bn=0, act=None)\n\n # output_m, need concat()\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(x)\n x = conv2d_unit(x, i256, 3, strides=2, act='leaky')\n x = layers.Concatenate()([x, fpn_s16])\n x = conv2d_unit(x, i256, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i512, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i256, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i512, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i256, 1, strides=1, act='leaky')\n output_m = conv2d_unit(x, i512, 3, strides=1, padding='same', act='leaky')\n output_m = conv2d_unit(output_m, num_anchors * (num_classes + 5), 1, strides=1, bn=0, act=None)\n\n # output_l, need concat()\n x = layers.ZeroPadding2D(padding=((1, 0), (1, 0)))(x)\n x = conv2d_unit(x, i512, 3, strides=2, act='leaky')\n x = layers.Concatenate()([x, fpn_s32])\n x = conv2d_unit(x, i512, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i1024, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i512, 1, strides=1, act='leaky')\n x = conv2d_unit(x, i1024, 3, strides=1, padding='same', act='leaky')\n x = conv2d_unit(x, i512, 1, strides=1, act='leaky')\n output_l = conv2d_unit(x, i1024, 3, strides=1, padding='same', act='leaky')\n output_l = conv2d_unit(output_l, num_anchors * (num_classes + 5), 1, strides=1, bn=0, act=None)\n\n def cast_float32(tensor):\n return tf.cast(tensor, tf.float32)\n\n output_l = layers.Lambda(cast_float32)(output_l)\n output_m = layers.Lambda(cast_float32)(output_m)\n output_s = layers.Lambda(cast_float32)(output_s)\n\n # originally reshape in multi_thread_post\n output_lr = layers.Reshape((1, input_shape[0] // 32, input_shape[1] // 32, 3, 5 + num_classes))(output_l)\n output_mr = layers.Reshape((1, input_shape[0] // 16, input_shape[1] // 16, 3, 5 + num_classes))(output_m)\n output_sr = layers.Reshape((1, input_shape[0] // 8, input_shape[1] // 8, 3, 5 + num_classes))(output_s)\n\n # originally _yolo_out\n masks = [[6, 7, 8], [3, 4, 5], [0, 1, 2]]\n anchors = [[12, 16], [19, 36], [40, 28], [36, 75], [76, 55],\n [72, 146], [142, 110], [192, 243], [459, 401]]\n\n def batch_process_feats(out, anchors, mask):\n grid_h, grid_w, num_boxes = map(int, out.shape[2:5])\n\n anchors = [anchors[i] for i in mask]\n anchors_tensor = np.array(anchors).reshape(1, 1, len(anchors), 2)\n\n # Reshape to batch, height, width, num_anchors, box_params.\n box_xy = tf.sigmoid(out[..., :2])\n box_wh = tf.exp(out[..., 2:4])\n box_wh = box_wh * anchors_tensor\n\n box_confidence = tf.sigmoid(out[..., 4])\n box_confidence = tf.expand_dims(box_confidence, axis=-1)\n box_class_probs = tf.sigmoid(out[..., 5:])\n\n col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)\n row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)\n\n col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)\n row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)\n grid = np.concatenate((col, row), axis=-1).astype(np.float32)\n\n box_xy += grid\n box_xy /= (grid_w, grid_h)\n box_wh /= input_shape\n box_xy -= (box_wh / 2.) # normalized xywh\n boxes = tf.concat((box_xy, box_xy + box_wh), axis=-1)\n\n box_scores = box_confidence * box_class_probs\n num_boxes = np.prod(boxes.shape[1:-1])\n boxes = tf.reshape(boxes, [-1, num_boxes, boxes.shape[-1]])\n box_scores = tf.reshape(box_scores, [-1, num_boxes, box_scores.shape[-1]])\n return boxes, box_scores\n\n def filter_boxes(outputs):\n boxes_l, boxes_m, boxes_s, box_scores_l, box_scores_m, box_scores_s, image_shape = outputs\n boxes_l, box_scores_l = filter_boxes_one_size(boxes_l, box_scores_l)\n boxes_m, box_scores_m = filter_boxes_one_size(boxes_m, box_scores_m)\n boxes_s, box_scores_s = filter_boxes_one_size(boxes_s, box_scores_s)\n boxes = tf.concat([boxes_l, boxes_m, boxes_s], axis=0)\n box_scores = tf.concat([box_scores_l, box_scores_m, box_scores_s], axis=0)\n image_shape_wh = image_shape[1::-1]\n image_shape_whwh = tf.concat([image_shape_wh, image_shape_wh], axis=-1)\n image_shape_whwh = tf.cast(image_shape_whwh, tf.float32)\n boxes *= image_shape_whwh\n boxes = tf.expand_dims(boxes, 0)\n box_scores = tf.expand_dims(box_scores, 0)\n boxes = tf.expand_dims(boxes, 2)\n nms_boxes, nms_scores, nms_classes, valid_detections = tf.image.combined_non_max_suppression(\n boxes,\n box_scores,\n max_output_size_per_class=nms_top_k,\n max_total_size=nms_top_k,\n iou_threshold=nms_thresh,\n score_threshold=conf_thresh,\n pad_per_class=False,\n clip_boxes=False,\n name='CombinedNonMaxSuppression',\n )\n return nms_boxes[0], nms_scores[0], nms_classes[0]\n\n def filter_boxes_one_size(boxes, box_scores):\n box_class_scores = tf.reduce_max(box_scores, axis=-1)\n keep = box_class_scores > conf_thresh\n boxes = boxes[keep]\n box_scores = box_scores[keep]\n return boxes, box_scores\n\n def batch_yolo_out(outputs):\n with tf.name_scope('yolo_out'):\n b_output_lr, b_output_mr, b_output_sr, b_image_shape = outputs\n with tf.name_scope('process_feats'):\n b_boxes_l, b_box_scores_l = batch_process_feats(b_output_lr, anchors, masks[0])\n with tf.name_scope('process_feats'):\n b_boxes_m, b_box_scores_m = batch_process_feats(b_output_mr, anchors, masks[1])\n with tf.name_scope('process_feats'):\n b_boxes_s, b_box_scores_s = batch_process_feats(b_output_sr, anchors, masks[2])\n with tf.name_scope('filter_boxes'):\n b_nms_boxes, b_nms_scores, b_nms_classes = tf.map_fn(\n filter_boxes, [b_boxes_l, b_boxes_m, b_boxes_s, b_box_scores_l, b_box_scores_m, b_box_scores_s, b_image_shape],\n dtype=(tf.float32, tf.float32, tf.float32), back_prop=False, parallel_iterations=16)\n return b_nms_boxes, b_nms_scores, b_nms_classes\n\n boxes_scores_classes = layers.Lambda(batch_yolo_out)([output_lr, output_mr, output_sr, image_shape])\n\n model_body = keras.models.Model(inputs=inputs, outputs=boxes_scores_classes)\n return model_body\n\n\ndef decode_jpeg_resize(input_tensor, image_size):\n tensor = tf.image.decode_png(input_tensor, channels=3)\n shape = tf.shape(tensor)\n tensor = tf.cast(tensor, tf.float32)\n tensor = tf.image.resize(tensor, image_size)\n tensor /= 255.0\n return tf.cast(tensor, tf.float16), shape\n\n\ndef preprocessor(input_tensor, image_size):\n with tf.name_scope('Preprocessor'):\n tensor = tf.map_fn(\n partial(decode_jpeg_resize, image_size=image_size), input_tensor,\n dtype=(tf.float16, tf.int32), back_prop=False, parallel_iterations=16)\n return tensor\n\n\ndef main():\n os.system('aws s3 cp s3://neuron-s3/training_checkpoints/pytorch/yolov4/yolov4.pth . --no-sign-request')\n torch_weights = rename_weights('./yolov4.pth')\n keras_weights = convert_pt_checkpoint_to_keras_h5(torch_weights)\n keras.backend.set_learning_phase(0)\n num_anchors = 3\n num_classes = 80\n input_shape = (608, 608)\n conf_thresh = 0.001\n nms_thresh = 0.45\n inputs = layers.Input(shape=[], dtype='string')\n yolo = YOLOv4(inputs, num_classes, num_anchors, input_shape, conf_thresh=conf_thresh, nms_thresh=nms_thresh)\n yolo.set_weights(keras_weights)\n sess = keras.backend.get_session()\n inputs = {'image': yolo.inputs[0]}\n output_names = ['boxes', 'scores', 'classes']\n outputs = {name: ts for name, ts in zip(output_names, yolo.outputs)}\n tf.saved_model.simple_save(sess, './yolo_v4_coco_saved_model', inputs, outputs)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/helperscripts/installationScripts/python_instructions.txt", "content": "\n# AL2 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# U20 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# AL2 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx Upgrade(1.13)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Upgrade(1.13)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx Upgrade(1.12)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.12.0 --neuron-version=2.6.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Upgrade(1.12)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.12.0 --neuron-version=2.6.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx Upgrade(1.11)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --neuron-version=2.4.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Upgrade(1.11)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.11.0 --neuron-version=2.4.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2 tensorflow Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.10.1.1.0.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1.1.0.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 tensorflow Neuronx upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1.1.0.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1.1.0.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 EFA Installation\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=efa --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U20 EFA Installation\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=efa --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2 PyTorch DLAMI\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=all --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework\n\n# U20 PyTorch DLAMI\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=all --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework\n\n# AL2 tensorflow Neuronx upgrade(2.10)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade(2.10)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 tensorflow Neuronx upgrade(2.9)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade(2.9)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 tensorflow Neuronx upgrade(2.8)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade(2.8)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 tensorflow Neuronx Install(2.10)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx Install(2.10)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 tensorflow Neuronx Install(2.8)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.8 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx Install(2.8)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.8 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 tensorflow Neuronx Install(2.7)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.7 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 tensorflow Neuronx Install(2.7)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.7 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# AL2 Tensorflow DLAMI\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=all --framework=tensorflow --framework-version=2.10 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework\n\n# U20 Tensorflow DLAMI\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=all --framework=tensorflow --framework-version=2.10 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework\n\n# AL2 PyTorch Neuron DLAMI\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=all --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=inf1 --ami=dlami-framework\n\n# U20 PyTorch Neuron DLAMI\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=all --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=inf1 --ami=dlami-framework\n\n# U22 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# U22 Tensorflow Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# U22 Pytorch Neuron Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n# U22 Tensorflow Neuron Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx DLAMI Upgrade(1.13)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework\n\n# U20 Pytorch Neuronx DLAMI Upgrade(1.13)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework\n\n# AL2 tensorflow Neuronx upgrade DLAMI(2.10)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework --category=compiler_framework\n\n# AL2 tensorflow Neuronx upgrade DLAMI(2.9)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework --category=compiler_framework\n\n# AL2 tensorflow Neuronx upgrade DLAMI(2.8)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade DLAMI(2.10)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade DLAMI(2.9)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.9.3 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework --category=compiler_framework\n\n# U20 tensorflow Neuronx upgrade(2.8)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=tensorflow --framework-version=2.8.4 --neuron-version=2.10.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework --category=compiler_framework\n\n# U20 Pytorch Neuronx 2.0 Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx 2.0 Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U22 Pytorch Neuronx 2.0 Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx Upgrade(2.0)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Upgrade(2.0)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# U22 Pytorch Neuronx Upgrade(2.0)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2 Pytorch Neuronx DLAMI Upgrade(2.0)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2 --instance=trn1 --ami=dlami-framework\n\n# U20 Pytorch Neuronx DLAMI Upgrade(2.0)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework\n\n# AL2023 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# AL2023 tensorflow Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.10.1.1.0.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# Al2023 Pytorch Neuronx 2.0 Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# AL2023 tensorflow Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=tensorflow --framework-version=2.10.1.1.0.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami --category=compiler_framework\n\n# U20 Pytorch Neuronx 2.1 Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2023 Pytorch Neuronx 2.1 Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 2.5 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 Pytorch Neuronx Upgrade(2.1)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Upgrade(2.1)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# U22 2.5.1 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 Pytorch Neuronx DLAMI Upgrade(2.1)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=dlami-framework\n\n# U20 Pytorch Neuronx DLAMI Upgrade(2.1)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=dlami-framework\n\n# U22 Neuron DLAMI - Torch-Neuronx-1.13.1\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=dlami-neuron\n\n# U22 Neuron DLAMI - Torch-Neuronx- 2.1.1\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=dlami-neuron\n\n# U22 Neuron DLAMI - Tensorflow-Neuronx- 2.10.1\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=dlami-neuron\n\n# U22 Neuron DLAMI - Transofrmers-Neuronx\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=transformers-neuronx --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=dlami-neuron\n\n# U22 Neuron DLAMI - Torch-Neuron-1.13.1\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=dlami-neuron\n\n# U22 Neuron DLAMI - Tensorflow-Neuron- 2.10.1\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=tensorflow --framework-version=2.10.1 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=inf1 --ami=dlami-neuron\n\n# Rocky Linux 9 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=rockylinux9 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# AL2023 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=1.13.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# U22 2.1 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 2.1 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U20 2.1 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# U20 Pytorch Neuronx Upgrade(2.1)\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.1.2 --file=src/helperscripts/n2-manifest.json --os=ubuntu20 --instance=trn1 --ami=non-dlami\n\n# AL2023 2.5.1 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.5.1 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# AL2023 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# U22 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# AL2 EFA Installation\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=efa --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 EFA Installation\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=efa --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# U22 2.6.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.6.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 2.6.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.6.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 2.6.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.6.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 2.7.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.7.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 2.7.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.7.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 2.7.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.7.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 2.7.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.7.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 2.8.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 2.8.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# AL2023 Latest Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=amazonlinux2023 --instance=trn1 --ami=non-dlami\n\n# U22 2.8.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# U22 2.9.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# U22 Latest Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu22 --instance=trn1 --ami=non-dlami\n\n# U24 EFA Installation\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=efa --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu24 --instance=trn1 --ami=non-dlami\n\n# U24 2.9.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu24 --instance=trn1 --ami=non-dlami\n\n# U24 2.9.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu24 --instance=trn1 --ami=non-dlami\n\n# U24 Driver and Tools\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --framework=pytorch --framework-version=2.9.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu24 --instance=trn1 --ami=non-dlami --category=driver_runtime_tools\n\n# U24 2.8.0 Pytorch Neuronx Upgrade\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=update --category=compiler_framework --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu24 --instance=trn1 --ami=non-dlami\n\n# U24 2.8.0 Pytorch Neuronx Install\n.. program-output:: python3 src/helperscripts/n2-helper.py --install-type=install --category=compiler_framework --framework=pytorch --framework-version=2.8.0 --file=src/helperscripts/n2-manifest.json --os=ubuntu24 --instance=trn1 --ami=non-dlami\n\n\n\n" }, { "path": "src/helperscripts/n2-helper.py", "content": "import json\nimport argparse\nfrom packaging.version import Version, parse\nimport pandas as pd\nfrom pandas import json_normalize\n\n\nclass manifest:\n def __init__(self, manifest_file):\n\n self.manifest_file = manifest_file\n self.df_packages = pd.DataFrame()\n\n def parse_manifest(self):\n\n with open(self.manifest_file, 'r') as f:\n manifest = json.load(f)\n\n # repos\n self.df_repos = json_normalize(manifest['repos_n2'])\n\n # latest release\n self.df_latest_release = json_normalize(manifest['latest_release'])\n\n # os properties\n self.df_os_properties = json_normalize(manifest['os_properties'])\n\n # ami properties\n self.df_ami_properties = json_normalize(manifest['ami_properties'])\n\n # dlami properties\n self.df_dlami_properties = json_normalize(manifest['dlami_properties'])\n\n # major version properties\n self.df_major_version_properties = json_normalize(manifest['major_version_properties'])\n\n # package properties\n self.df_package_properties = json_normalize(manifest['package_properties'])\n\n # neuron releases\n for release in manifest['neuron_releases']:\n df_release = json_normalize(release['packages'])\n df_release['neuron_version'] = release['neuron_version']\n self.df_packages = pd.concat([self.df_packages, df_release])\n\n # merge release packages\n self.df_release_packages = self.df_packages.merge(self.df_package_properties, how='left', on='name')\n self.df_release_packages['supported_instances'] = self.df_release_packages['supported_instances'].tolist()\n\n def merge_release_packages(self):\n\n self.df_release_packages = self.df_packages.merge(self.df_package_properties, how='left', on='name')\n\n def extract_major_minor_version(self, version):\n\n return str(version.major) + '.' + str(version.minor)\n\n def get_pip_packages_supporting_python_versions(self, args):\n '''\n Get supported python version by packages (compiler and framework)\n e.g., {\"3.6\",\"3.7\",\"3.8\"}\n '''\n\n if args.neuron_version == None:\n neuron_version = self.get_latest_neuron_version_per_instance(args.instance)\n else:\n neuron_version = args.neuron_version\n\n df_instance = self.df_release_packages[\n (self.df_release_packages['supported_instances'].map(lambda x: args.instance in x)) & (\n self.df_release_packages['neuron_version'] == neuron_version)]\n\n # Compiler supporting Python versions\n compiler_python_versions = \\\n df_instance.loc[df_instance['component'] == 'Compiler']['supported_python_versions'].values[0]\n\n # Specific framework version supporting Python versions\n df_framework = df_instance.loc[df_instance['category'] == args.framework].copy()\n df_framework['version'] = df_framework['version'].map(lambda x: Version(x))\n df_framework['major_minor_version'] = df_framework['version'].map(lambda x: str(x.major) + '.' + str(x.minor))\n\n framework_python_versions = df_framework.loc[\n df_framework['major_minor_version'] == self.extract_major_minor_version(Version(args.framework_version))][\n 'supported_python_versions'].values[0]\n return list(set(compiler_python_versions) & set(framework_python_versions))\n\n def get_major_version(self, package_name, instance):\n return self.df_major_version_properties.loc[(self.df_major_version_properties['name'] == package_name)][\n args.instance].values[0]\n\n def generate_script(self, args):\n '''\n It generates:\n (1) str_preamble\n (2) str_driver\n (3) str_runtime\n (4) str_tools\n (5) str_python\n (6) str_compiler\n (7) str_framework\n '''\n\n str_preamble = ''\n\n # Install and enable EPEL (required only for rocky linux 9 currently)\n str_preamble += self.install_and_enable_epel(args)\n\n # Configure Neuron repository\n str_preamble += self.config_neuron_repository(args)\n\n # Update OS packages\n str_preamble += self.update_os_packages(args)\n\n # Install OS headers\n str_preamble += self.install_os_headers(args)\n\n # Install git\n str_preamble += self.install_git(args)\n\n # Install Neuron driver\n str_driver = self.install_neuron_driver(args)\n\n # Install Neuron runtime\n str_runtime = self.install_neuron_runtime(args)\n\n # Install EFA driver\n str_efa = self.install_efa_driver(args)\n\n # Install Neuron Tools\n str_tools = self.install_neuron_system_tools(args)\n\n # Add PATH\n if args.mode != 'compile' or args.ami != 'dlami-framework':\n str_tools += '\\n# Add PATH\\n'\n str_tools += 'export PATH=/opt/aws/neuron/bin:$PATH\\n'\n\n # Install Python virtual environment\n str_python = self.set_python_venv(args)\n\n # Activate Pythohn venv\n str_python += self.activate_python_venv(args)\n\n # install jupyter notebook\n str_python += self.jupyter_notebook(args)\n\n # Set pip repository\n str_python += self.set_pip_repository()\n\n # Install wget, awscli\n str_python += self.install_aux(args)\n\n # install extra dependencies\n str_deps = self.install_extra_dependencies(args)\n\n # Install Neuron compiler\n str_compiler = self.install_neuron_compiler(args)\n\n # Install Neuron framework\n str_framework = self.install_neuron_framework(args)\n\n # install neuron compiler and framework\n str_compiler_framework = self.install_neuron_compiler_and_framework(args)\n if args.ami == 'dlami-framework':\n # dlami instructions\n str_dlami = self.install_dlami(args)\n return str_dlami\n elif args.ami == 'dlami-neuron':\n str_dlami = self.install_neuron_dlami(args)\n return str_dlami\n elif args.category == 'all':\n if args.instance == 'trn1':\n str_runtime += str_efa\n return str_preamble + str_driver + str_runtime + str_tools + str_deps + str_python + str_compiler_framework\n elif args.category == 'driver_runtime_tools':\n return str_preamble + str_driver + str_runtime + str_tools\n elif args.category == 'compiler_framework':\n return str_deps + str_python + str_compiler_framework\n elif args.category == 'driver':\n return str_preamble + str_driver\n elif args.category == 'runtime':\n return str_runtime\n elif args.category == 'tools':\n return str_tools\n elif args.category == 'compiler':\n if args.instance != 'inf1':\n return str_python + str_compiler\n else:\n return str_python\n elif args.category == 'framework':\n return str_framework\n elif args.category == 'efa':\n return str_efa\n\n def install_dlami(self, args):\n latest_release_for_instance = \\\n self.df_latest_release.loc[self.df_latest_release['instance'] == args.instance]['version'].values[0]\n latest_release_for_dlami = self.df_dlami_properties[\n (self.df_dlami_properties['framework'] == args.framework) & (\n self.df_dlami_properties['supported_instances'].map(lambda x: args.instance in x))][\n 'neuron_released_version'].values[0]\n\n if (latest_release_for_instance == latest_release_for_dlami):\n return self.activate_python_venv(args)\n else:\n args.install_type = 'update'\n str_dlami = self.activate_python_venv(args)\n str_dlami += self.jupyter_notebook(args)\n str_dlami += self.set_pip_repository()\n str_dlami += self.install_neuron_compiler_and_framework(args)\n return str_dlami\n\n\n def install_neuron_dlami(self, args):\n str_dlami = \"\"\n if ((args.instance == 'trn1' or args.instance == 'inf2') and args.category == \"transformers-neuronx\"):\n str_dlami = '\\n# Activate Python venv for Transformers-NeuronX \\n'\n str_dlami += \"source /opt/aws_neuronx_venv_transformers_neuronx/bin/activate\"\n elif ((args.instance == 'trn1' or args.instance == 'inf2') and args.framework == \"pytorch\" and args.framework_version == \"1.13.1\"):\n str_dlami = '\\n# Activate Python venv for Pytorch 1.13 \\n'\n str_dlami += \"source /opt/aws_neuronx_venv_pytorch_1_13/bin/activate\"\n elif ((args.instance == 'trn1' or args.instance == 'inf2') and args.framework == \"pytorch\" and args.framework_version == \"2.1\"):\n str_dlami = '\\n# Activate Python venv for Pytorch 2.1 \\n'\n str_dlami += \"source /opt/aws_neuronx_venv_pytorch_2_1/bin/activate\"\n elif ((args.instance == 'trn1' or args.instance == 'inf2') and args.framework == \"tensorflow\" and args.framework_version == \"2.10.1\"):\n str_dlami = '\\n# Activate Python venv for Tensorflow 2.10 \\n'\n str_dlami += \"source /opt/aws_neuronx_venv_tensorflow_2_10/bin/activate\"\n elif (args.instance == 'inf1' and args.framework == \"tensorflow\" and args.framework_version == \"2.10.1\"):\n str_dlami = '\\n# Activate Python venv for Tensorflow 2.10 \\n'\n str_dlami += \"source /opt/aws_neuron_venv_tensorflow_2_10_inf1/bin/activate\"\n elif (args.instance == 'inf1' and args.framework == \"pytorch\" and args.framework_version == \"1.13.1\"):\n str_dlami = '\\n# Activate Python venv for Pytorch 1.13 \\n'\n str_dlami += \"source /opt/aws_neuron_venv_pytorch_1_13_inf1/bin/activate\"\n return str_dlami\n\n def jupyter_notebook(self, args):\n os_default_python_version = \\\n self.df_os_properties.loc[self.df_os_properties['os'] == args.os]['default_python_version'].values[0]\n packages_supporting_python_versions = self.get_pip_packages_supporting_python_versions(args)\n\n if os_default_python_version in packages_supporting_python_versions:\n target_python_version = os_default_python_version\n else:\n target_python_version = max(packages_supporting_python_versions)\n\n framework_name = self.get_package_names(category=args.framework, instance=args.instance)[0]\n\n str_jupiter = '\\n# Install Jupyter notebook kernel\\n'\n str_jupiter += 'pip install ipykernel ' + '\\n'\n str_jupiter += 'python' + target_python_version + ' -m ipykernel install --user --name '\n str_jupiter += 'aws_neuron_venv_' + args.framework\n if args.instance == 'inf1':\n str_jupiter += '_inf1'\n str_jupiter += ' --display-name \"Python (' + framework_name + ')\"' + '\\n'\n str_jupiter += 'pip install jupyter notebook' + '\\n'\n str_jupiter += 'pip install environment_kernels' + '\\n'\n return str_jupiter\n\n def install_and_enable_epel(self, args):\n str = ''\n if args.mode != 'compile':\n if args.install_type == 'install':\n if args.os == 'rockylinux9':\n str += '\\n# Install and enable EPEL\\n'\n str += 'sudo dnf config-manager --set-enabled crb\\n'\n str += 'sudo dnf install epel-release -y\\n'\n return str\n\n def config_neuron_repository(self, args):\n \"\"\"\n Reads OS type from the arguments and generates scripts for configuration of Neuron repository\n \"\"\"\n str = ''\n if args.mode != 'compile':\n # Neuron repository needs when mode is 'develop' or 'deploy'\n if args.install_type == 'install':\n str += '\\n# Configure Linux for Neuron repository updates' + '\\n'\n if args.os == 'ubuntu18' or args.os == 'ubuntu20' or args.os == 'ubuntu22' or args.os == 'ubuntu24':\n str += '. /etc/os-release' + '\\n'\n str += 'sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null <\", \"package_categories\": [\"driver\",\"runtime\",\"tools\",\"compiler\"]}\n ],\n \"dlami_properties\": [\n {\"framework\":\"pytorch\", \"dlami\": \"1.13\", \"neuron_released_version\": \"2.17.0\", \"supported_instances\":[\"trn1\",\"inf2\",\"inf1\"]},\n {\"framework\":\"tensorflow\", \"dlami\": \"2.10\", \"neuron_released_version\": \"2.17.0\", \"supported_instances\":[\"trn1\",\"inf2\"]}\n ],\n \"major_version_properties\": [\n {\"name\":\"neuronx-cc\",\"inf1\":\"\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-k8-plugin\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-oci-hooks\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"tensorflow-neuronx\",\"inf1\":\"\",\"trn1\":\"1\",\"inf2\":\"1\"},\n {\"name\":\"torch-neuronx\",\"inf1\":\"\",\"trn1\":\"1\",\"inf2\":\"1\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-dkms\",\"inf1\":\"2.21\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-collectives\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-runtime-lib\",\"inf1\":\"\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"aws-neuronx-tools\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"},\n {\"name\":\"tensorflow-model-server-neuronx\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\"},\n {\"name\":\"neuronperf\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\"},\n {\"name\":\"tensorboard-plugin-neuronx\",\"inf1\":\"2\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\"},\n {\"name\":\"nki\",\"trn1\":\"2\",\"inf2\":\"2\",\"trn2\":\"2\",\"trn3\":\"2\"}\n ],\n \"package_properties\": [\n {\"name\":\"aws-neuronx-runtime-discovery\", \"component\":\"General\",\"category\":\"general\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"aws_neuron_sdk_release_version\", \"component\":\"Github\",\"category\":\"github\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"libneuronxla\",\"component\":\"Framework\",\"category\":\"general\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"neuron-cc\",\"component\":\"Compiler\",\"category\":\"compiler\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"neuronx-cc\",\"component\":\"Compiler\",\"category\":\"compiler\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"neuronx-cc-stubs\",\"component\":\"Compiler\",\"category\":\"compiler\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"aws-neuronx-k8-plugin\",\"component\":\"Kubernetes Plugin\",\"category\":\"container\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"component\":\"Kubernetes Scheduler\",\"category\":\"container\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"aws-neuronx-oci-hooks\",\"component\":\"OCI Hooks\",\"category\":\"container\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"mxnet-neuron\",\"component\":\"MXNet\",\"category\":\"mxnet\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"tensorflow-neuron\",\"component\":\"TensorFlow\",\"category\":\"tensorflow\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"tensorflow\",\"component\":\"TensorFlow\",\"category\":\"tensorflow\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"tensorflow-neuronx\",\"component\":\"TensorFlow\",\"category\":\"tensorflow\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"torch-neuron\",\"component\":\"PyTorch\",\"category\":\"pytorch\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"torch-neuronx\",\"component\":\"PyTorch\",\"category\":\"pytorch\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"transformers-neuronx\",\"component\":\"Transformers Neuron\",\"category\":\"transformers-neuronx\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"mxnet_neuron\",\"component\":\"MXNet\",\"category\":\"mxnet\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"mx_neuron\",\"component\":\"MXNet\",\"category\":\"mxnet\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"aws-neuronx-dkms\",\"component\":\"Driver\",\"category\":\"driver\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"aws-neuronx-collectives\",\"component\":\"Collective Communication Library\",\"category\":\"runtime\",\"package_type\":\"os\",\"use_cases\":[\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"efa-installer\",\"component\":\"EFA\",\"category\":\"efa\",\"package_type\":\"na\",\"use_cases\":[\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"aws-neuronx-runtime-lib\",\"component\":\"Runtime Library\",\"category\":\"runtime\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"aws-neuron-tools\",\"component\":\"System Tools\",\"category\":\"system-tools\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"true\"},\n {\"name\":\"aws-neuronx-tools\",\"component\":\"System Tools\",\"category\":\"system-tools\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"tensorflow-model-server-neuron\",\"component\":\"TensorFlow Model Server\",\"category\":\"model-server\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"tensorflow-model-server-neuronx\",\"component\":\"TensorFlow Model Server\",\"category\":\"model-server\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"neuronperf\",\"component\":\"Perf Tools\",\"category\":\"helper-tools\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"tensorboard-plugin-neuron\",\"component\":\"TensorBoard\",\"category\":\"profiling-tools\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"true\"},\n {\"name\":\"tensorboard-plugin-neuronx\",\"component\":\"TensorBoard\",\"category\":\"profiling-tools\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"},\n {\"name\":\"libnrt.so\",\"component\":\"Runtime Library\",\"category\":\"libnrt\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"torch_xla\",\"component\":\"PyTorch\",\"category\":\"helper-lib\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"component\":\"CustomOps Tools\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"component\":\"CustomOps\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"aws-neuronx-oci-hook\",\"component\":\"OCI\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"dmlc_nnvm\",\"component\":\"Compiler\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"neuronx_hwm\",\"component\":\"Compiler\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"dmlc_topi\",\"component\":\"Compiler\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"dmlc_tvm\",\"component\":\"Compiler\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"inferentia_hwm\",\"component\":\"Compiler\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"neuronx_distributed\",\"component\":\"Neuron Distributed\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"neuronx_distributed_training\",\"component\":\"Neuron Distributed Training\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"false\"},\n {\"name\":\"neuronx_distributed_inference\",\"component\":\"Neuron Distributed Inference\",\"category\":\"na\",\"package_type\":\"os\",\"use_cases\":[\"inference\"],\"pin_major\":\"false\"},\n {\"name\":\"jax_neuronx\",\"component\":\"Jax\",\"category\":\"jax\",\"package_type\":\"pip\",\"use_cases\":[\"inference\"],\"pin_major\":\"true\"},\n {\"name\":\"nki\",\"component\":\"NKI\",\"category\":\"nki\",\"package_type\":\"pip\",\"use_cases\":[\"inference\",\"training\"],\"pin_major\":\"true\"}\n ],\n \"neuron_releases\": [\n {\"neuron_version\":\"2.29.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.31.24.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.27.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.21.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"14.09.x\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.29.147.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn3\"] ,\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.29.147.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.15.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.31.24.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.29.18.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.7.0.1.0.8181\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.16408.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.24.5133.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.24.5133.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.18.27753\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.9.17334\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"nki\",\"version\":\"0.3.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.918.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.9.0.2.13.24727\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\",\"trn3\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"efa-installer\",\"version\":\"1.47\",\"supported_instances\":[\"trn1\",\"trn2\",\"trn3\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.28.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.30.59.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.26.10.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.20.7.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.20.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.29.71.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.29.71.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.14.102.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.30.51.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.28.23.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.7.0.1.0.7584\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.15515.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.23.6484.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.23.6484.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.17.26814\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.7.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.8.16251\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"nki\",\"version\":\"0.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.918.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.9.0.2.12.22436\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.12.22436\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.8.0.2.12.22436\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"efa-installer\",\"version\":\"1.47\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.28.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.30.59.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.26.5.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.20.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.20.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.29.71.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.29.71.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.14.102.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.30.51.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.28.23.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.7.0.1.0.7584\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.15515.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.23.6484.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.23.6484.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.17.26814\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.7.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.8.16251\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"nki\",\"version\":\"0.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.918.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.9.0.2.12.22436\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.12.22436\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.8.0.2.12.22436\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"efa-installer\",\"version\":\"1.47\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.27.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.29.41.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.25.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.19.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.19.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.29.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.29.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.13.52.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.29.40.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.27.33.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.7.0.1.0.7377\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.14584.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.22.12471.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.22.12471.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.16.25997\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.7.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.7.15063\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"nki\",\"version\":\"0.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.918.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.9.0.2.11.19912\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.11.19912\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.8.0.2.11.19912\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.27.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.29.41.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.25.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.19.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.19.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.29.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.29.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.13.52.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.29.40.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.27.33.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.7.0.1.0.7377\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.14584.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.22.12471.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.22.12471.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.16.25997\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.7.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.7.14366\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"nki\",\"version\":\"0.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.918.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.9.0.2.11.19912\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.11.19912\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.8.0.2.11.19912\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\",\"3.12\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.26.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.28.27.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.24.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.18.0.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.18.0.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.28.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.28.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.12.36.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.28.23.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.26.14.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.6.2.1.0.6446\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.12677.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.21.33363.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.21.33363.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.15.22404\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.6.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.6.10598\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.837.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.6.0.2.10.16998\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.10.16998\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.8.0.2.10.16998\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.1315\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.26.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.28.27.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.24.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.18.0.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.18.0.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.28.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.28.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.12.36.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.28.23.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.26.14.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.6.2.1.0.6446\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.12677.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.21.18209.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.21.18209.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.15.22404\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.6.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.6.10598\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.837.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.6.0.2.10.13553\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.10.13553\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.8.0.2.10.13553\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.1315\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.25.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.27.34.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.27.34.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.23.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.17.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.17.0.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.27.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.27.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.11.42.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.27.23.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.25.145.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.6.1.1.0.3499\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.8201.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.20.9961.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.20.9961.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.14.18461\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.5.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.5.9230\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.813.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.6.0.2.9.9357\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.9.9357\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.1216\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.24.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.26.43.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.26.43.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.22.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.16.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.16.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.26.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.26.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.10.56.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.26.42.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.24.54.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.6.0.1.0.1296\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.4410.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.19.8089.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.19.8089.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.13.14393\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.4.1\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.4.7422\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.760.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.5.1.2.8.6734\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.6.0.2.8.6734\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.8.6734\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.6\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.985\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.24.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.26.43.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.26.43.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.22.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.16.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.16.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.26.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.26.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.10.56.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.26.42.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.24.54.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.6.0.1.0.1296\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.4410.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.19.8089.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.19.8089.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.13.14393\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.4.0\",\"supported_instances\":[\"trn1\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.4.7422\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.760.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.5.1.2.8.6734\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.6.0.2.8.6734\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.7.0.2.8.6734\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.6\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.985\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.23.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.25.65.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.21.37.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.15.12.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.15.1.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.25.24.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.25.24.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.9.88.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.25.57.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.23.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.5.3.1.0.719\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.3493.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.18.121.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.18.121.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.12.12111\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.3.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.3.5591\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.0.670.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.5.1.2.7.5413\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.6.0.2.7.5413\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.6\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.798\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.22.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.24.59.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.24.59.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.20.28.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.14.12.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.14.6.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.24.23.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.24.23.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.7.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.24.53.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.24.53.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.22.61.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.1.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.2.1630.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.3396\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.17.194.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.17.194.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.11.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.2.0\",\"supported_instances\":[\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.117.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.5.1.2.6.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.6\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.470\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.21.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.23.135.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.35.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.19.64.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.13.16.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.13.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.23.45.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.23.45.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.6.36.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.23.112.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.20.204.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.1.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.1.714.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.3396\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.16.372.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.16.372.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.10.1\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.1.1\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.1.1\",\"supported_instances\":[\"inf2\",\"trn2\",\"trn1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.52.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]}, \n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.17.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.5.1.2.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurong\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.6\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.380\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.21.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.23.133.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.35.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.19.64.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.13.16.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.13.2.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.23.30.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.23.30.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.6.36.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.23.110.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.20.204.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.1.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.1.681.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.3388\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.16.345.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.16.345.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.10.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.1.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_inference\",\"version\":\"0.1.0\",\"supported_instances\":[\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.52.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]}, \n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.17.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.5.1.2.4.0\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurong\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.6\",\"supported_instances\":[\"trn1\",\"inf2\",\"trn2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.13.322\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n \n {\"neuron_version\":\"2.20.2\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.22.33.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.35.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.18.20.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.12.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.12.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.22.20.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.22.20.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.5.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.22.19.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.19.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.1.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.5347.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.3278\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.15.143.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.15.143.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.0.1\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.63.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]}, \n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.11.13.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.3.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurong\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.5\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.12.313\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.20.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.22.26.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.35.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.18.12.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.12.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.12.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.22.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.22.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.5.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.22.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.19.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.1.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.4986.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.2978\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.15.141.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.15.141.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.0.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.63.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.12.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]}, \n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.3.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurong\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.4\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.12.313\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.20.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.22.26.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.35.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.18.12.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.12.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.12.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.22.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.22.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.5.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.22.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.19.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"jax_neuronx\",\"version\":\"0.1.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.9\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.4115.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.2978\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.24.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.15.128.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx-cc-stubs\",\"version\":\"2.15.128.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"neuronx_distributed_training\",\"version\":\"1.0.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.63.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.12.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.12.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.12.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.12.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.12.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.12.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]}, \n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurong\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.4\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.12.313\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\",\"3.11\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.19.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.21.46.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.17.17.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.11.4.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.11.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.21.14.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.21.14.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.4.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.21.41.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.2335\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.1795\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.23.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.14.227.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.63.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.11.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.11.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.11.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.15.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronf\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.11.351\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.19.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.21.46.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.17.17.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.11.4.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.11.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.21.14.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.21.14.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.4.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.21.41.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.2335\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.1795\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.147.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.23.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.93.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.14.213.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.63.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.11.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.11.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.11.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.11.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.10.12.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.15.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronf\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.11.351\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.18.2\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.20.22.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.16.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.9.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.9.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.20.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.20.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.3.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.20.22.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.17.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.965\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.971\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.50.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.22.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.55.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.13.72.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurone\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.10.0.360\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.18.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.20.22.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.16.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.9.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.9.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.20.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.20.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.3.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.20.22.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.17.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.965\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.971\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.50.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.22.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.55.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.13.68.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurone\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.10.0.360\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.18.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.20.22.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.16.7.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.9.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.9.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.20.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.20.13.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.3.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.20.22.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.17.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.17.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.965\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.971\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.50.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.22.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.55.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.13.66.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.19.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.19.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.74.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.2.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurone\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.10.0.21\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.17.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.20.11.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.15.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.9.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.9.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.19.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.19.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.45.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.20.11.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.17.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.16.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.755\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.809\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.40.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.21.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.15.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.12.68.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.6.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.12.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.13.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.0.0.2.0.1b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.1.2.0.1b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurond\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.9.474\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.16.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.19.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.15.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.9.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.9.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.19.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.19.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.45.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.19.5.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.16.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.16.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.498\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.669\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.40.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.21.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.15.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.12.68.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.6.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.12.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.13.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.0.0.2.0.1b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.1.2.0.0b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurond\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.9.474\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.16.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.19.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.15.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.9.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.9.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.19.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.19.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.45.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.19.5.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.16.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.18.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.16.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"2.0.498\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.669\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.40.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.21.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.15.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.12.54.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.6.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.12.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.6.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.8.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.17.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.13.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.0.0.2.0.1b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.1.1.2.0.0b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurond\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.1.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.9.474\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.15.2\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.18.19.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.14.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.8.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.8.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.27.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.18.15.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.15.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"1.0.680\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.570\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.25.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.20.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.11.0.35\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.5.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.11.0.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.43.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.12.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.0.0.2.0.1b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronc\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.0.0+torchneuron0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.8.268\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.18.15\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.15.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.18.19.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.14.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.8.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.8.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.27.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.18.15.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.15.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"1.0.680\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.570\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.25.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.20.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.11.0.34\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.5.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.11.0.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.43.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.12.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.0.0.2.0.1b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronc\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.0.0+torchneuron0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.8.268\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.18.15\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.15.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.18.18.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.14.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.8.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.8.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.18.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.27.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.18.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.15.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.18.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"1.0.663\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.538\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.25.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.20.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.11.0.34\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.5.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.11.0.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.43.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.12.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"2.0.0.2.0.0b0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronc\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"2.0.0+torchneuron0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.8.268\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.18.14\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.14.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.17.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.7.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.7.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.17.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.14.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.476\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.10.0.5\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronb\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.10.0.35\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.17.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.17.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.22.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.11.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.7.84\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.17.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.17.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.17.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.15.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.4.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.14.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.17.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.7.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.7.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.17.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.14.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.476\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.10.0.5\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuronb\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.10.0.34\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.19.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.17.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.17.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.22.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.11.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.7.84\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.17.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.17.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.17.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.15.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.4.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.13.2\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.16.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.12.18.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.6.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.6.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.16.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.440\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.9.0.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurona\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.9.0.40\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.18.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.16.18.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.16.18.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.25.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.10.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.6.106\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.15.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.13.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.16.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.12.11.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.6.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.6.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.16.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.425\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.9.0.2\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurona\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.9.0.40\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.18.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.16.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.16.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.21.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.10.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.6.106\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.15.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.13.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.16.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.12.11.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.6.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.6.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.16.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.425\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.9.0.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneurona\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.9.0.16\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.18.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.16.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.16.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.21.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf.\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.9.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.10.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.6.106\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.15.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.12.2\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.15.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.11.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.5.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.5.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.15.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.413\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.8.0.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuron8\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.8.0.25\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.17.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.15.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.15.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.9.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.5.58\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.14.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]}, \n {\"neuron_version\":\"2.12.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.15.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.11.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.5.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.5.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.15.14.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.413\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.8.0.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuron8\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.8.0.25\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.17.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.15.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.15.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.5.58\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.14.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.12.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.15.13.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.11.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.5.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.5.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.15.11.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.12.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.391\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.8.0.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7,\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuron8\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.8.0.25\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.17.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.15.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.15.6.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.16.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.9.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.9.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.39.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.5.58\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.14.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.2.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.11.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.14.9.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.10.11.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.4.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.4.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.14.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.11.10.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.326\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.7.0.3\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1+torchneuron7\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.7.0.40\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.16.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.14.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.14.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.8.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.8.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.8.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.8.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.8.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.8.9.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.37.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.7.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.7.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.7.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.7.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.7.10.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.9.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.8.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.4.60\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.16.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.16.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.16.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.14.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"islpy\",\"version\":\"2021.1+aws2021.x.169.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_distributed\",\"version\":\"0.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.10.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.13.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.9.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop-lib\",\"version\":\"0.3.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.3.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.13.6.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.10.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.207\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.6.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.6.0.19\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.15.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.13.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hook\",\"version\":\"2.2.0.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.8.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.8.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.8.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.8.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.8.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.7.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.4.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.9.3.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.8.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.8.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.8.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.8.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.8.1.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.26.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.39.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.4.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.1.1.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\",\"3.10\"]},\n {\"name\":\"transformers-neuronx\",\"version\":\"0.3.32\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.8.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_nnvm\",\"version\":\"1.15.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_topi\",\"version\":\"1.15.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"dmlc_tvm\",\"version\":\"1.15.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"inferentia_hwm\",\"version\":\"1.14.1\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"islpy\",\"version\":\"2021.1\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.9.1\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.35.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.8.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop\",\"version\":\"0.2.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.23.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.9.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.205\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.5.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.5.0.28\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.14.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.12.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.12.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.97.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.7.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.7.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.7.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.7.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.7.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.7.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.7.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.7.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.7.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.7.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.25.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.6.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.6.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.6.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.6.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.6.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.37.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.2.127.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.0.1.6.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.16.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.9.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.12.27.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.8.4.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop\",\"version\":\"0.2.3.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.2.1.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.12.16.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.9.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.173\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.5.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.5.0.28\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.14.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.12.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.12.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.97.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.7.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.7.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.7.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.7.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.7.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.2.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.7.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.7.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.7.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.7.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.7.3.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.25.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.13.1.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.37.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.2.127.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.0.1.6.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.7.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.12.16.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.8.0\", \"packages\": [\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.11.47.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.7.33.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-customop\",\"version\":\"0.1.23.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-gpsimd-tools\",\"version\":\"0.1.7.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-discovery\",\"version\":\"2.9\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.11.43.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.8.2.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"libneuronxla\",\"version\":\"0.5.144\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx_hwm\",\"version\":\"2.4.0.1\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch_xla\",\"version\":\"1.13.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.4.0.21\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"neuron-cc\",\"version\":\"1.13.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.81.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.4.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.4.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.9.3.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.10.1.2.6.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.10.1.1.0.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.6.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.4.2.6.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.4.2.6.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.9.3.2.6.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.10.1.2.6.5.0\",\"supported_instances\":[\"inf1\",\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.19.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuron\",\"version\":\"2.4.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.11.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.2.43.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.0.1.5.0\",\"supported_instances\":[\"trn1\",\"inf2\"],\"supported_python_versions\":[\"3.7\",\"3.8\",\"3.9\"]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.6.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.10.30.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.7.0\", \"packages\": [\n {\"name\":\"neuronx-cc\",\"version\":\"2.4.0.21\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.60.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.2.1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.5.4.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.6.3.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.15.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-gpsimd-customop\",\"version\":\"0.1.23.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronx-gpsimd-tools\",\"version\":\"0.1.7.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.13.0.1.4.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-xla\",\"version\":\"1.13.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.7.15.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.11.47.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.11.43.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.7.2.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.6.0\", \"packages\": [\n {\"name\":\"neuronx-cc\",\"version\":\"2.3.0.4\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.14.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.2.1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.5.4.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.6.3.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuronx\",\"version\":\"2.5.3.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.12.0.1.4.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"torch-xla\",\"version\":\"1.12.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.6.33.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.10.37.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.10.30.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.6.1.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.5.0\", \"packages\": [\n {\"name\":\"neuron-cc\",\"version\":\"1.13.5.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.2.0.73\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.1.12.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.14.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.5.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.5.3.2.5.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.6.5.2.5.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.3.2.5.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.2.2.5.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.2.1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"1.15.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.5.4.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.6.3.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.7.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuronx\",\"version\":\"2.8.0.2.5.6.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuron\",\"version\":\"2.4.6.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.12.1.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.7.1.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.8.1.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.5.8.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.11.0.1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.11.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.2.43.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.6.33.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.10.34.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.10.27.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.5.19.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.6.1.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.10.27.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.4.0\", \"packages\": [\n {\"name\":\"neuron-cc\",\"version\":\"1.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.2.0.73\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.1.2.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.1.2.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.1.2.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.5.3.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.6.3.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.1.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.0.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.2.1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"1.15.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.5.4.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.6.3.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.7.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.8.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuron\",\"version\":\"2.4.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.7.1.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.8.1.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.11.0.1.2.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.2.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.6.5.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.10.17.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.10.15.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.5.16.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.2.51.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]},\n {\"neuron_version\":\"2.3.0\", \"packages\": [\n {\"name\":\"neuron-cc\",\"version\":\"1.11.7.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"neuronx-cc\",\"version\":\"2.1.0.76\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"aws-neuronx-k8-plugin\",\"version\":\"2.0.1.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-k8-scheduler\",\"version\":\"2.0.1.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-oci-hooks\",\"version\":\"2.0.1.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"1.15.5.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.5.3.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.6.3.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.7.1.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuron\",\"version\":\"2.8.0.2.3.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-neuronx\",\"version\":\"2.8.2.1.1.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"1.15.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.5.4.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.6.3.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.7.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorflow-model-server-neuron\",\"version\":\"2.8.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"tensorboard-plugin-neuron\",\"version\":\"2.4.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"torch-neuron\",\"version\":\"1.7.1.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.8.1.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.9.1.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.10.2.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuron\",\"version\":\"1.11.0.2.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"torch-neuronx\",\"version\":\"1.11.0.1.1.1\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[\"3.7\",\"3.8\"]},\n {\"name\":\"mxnet_neuron\",\"version\":\"1.5.1.1.10.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"mx_neuron\",\"version\":\"1.8.0.2.2.2.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[\"3.7\"]},\n {\"name\":\"aws-neuronx-dkms\",\"version\":\"2.5.41.0\",\"supported_instances\":[\"inf1\",\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-collectives\",\"version\":\"2.9.86.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"efa-installer\",\"version\":\"na\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-runtime-lib\",\"version\":\"2.9.64.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuron-tools\",\"version\":\"2.1.4.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"aws-neuronx-tools\",\"version\":\"2.4.14.0\",\"supported_instances\":[\"trn1\"],\"supported_python_versions\":[]},\n {\"name\":\"neuronperf\",\"version\":\"1.3.0.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]},\n {\"name\":\"libnrt.so\",\"version\":\"2.2.51.0\",\"supported_instances\":[\"inf1\"],\"supported_python_versions\":[]}\n ]}\n ]\n}\n" }, { "path": "src/helperscripts/neuron-releases-manifest.json", "content": "{\n \"repos\": {\n \"whl\": \"https://pip.repos.neuron.amazonaws.com/\",\n \"rpm\": \"https://yum.repos.neuron.amazonaws.com/\",\n \"deb\": \"https://apt.repos.neuron.amazonaws.com/\"\n },\n \"manifest_date\": \"2022-12-12\",\n \"manifest_version\": \"1.0.1\",\n \"dlami_conda_env\": {\n \"tensorflow\": {\n \"1.15.5\": [\n \"aws_neuron_tensorflow_p36\",\n \"aws_neuron_tensorflow_p36\"\n ],\n \"2.1.4\": [\n \"None\",\n \"None\"\n ],\n \"2.2.3\": [\n \"None\",\n \"None\"\n ],\n \"2.3.4\": [\n \"None\",\n \"None\"\n ],\n \"2.4.3\": [\n \"None\",\n \"None\"\n ],\n \"2.5.1\": [\n \"None\",\n \"None\"\n ],\n \"2.5.2\": [\n \"None\",\n \"None\"\n ],\n \"2.5.3\": [\n \"None\",\n \"None\"\n ],\n \"2.6.3\": [\n \"None\",\n \"None\"\n ],\n \"2.6.5\": [\n \"None\",\n \"None\"\n ],\n \"2.7.1\": [\n \"None\",\n \"None\"\n ],\n \"2.7.3\": [\n \"None\",\n \"None\"\n ],\n \"2.8.0\": [\n \"None\",\n \"None\"\n ],\n \"2.8.2\": [\n \"None\",\n \"None\"\n ]\n },\n \"pytorch\": {\n \"1.5.1\": [\n \"None\",\n \"aws_neuron_pytorch_p36\"\n ],\n \"1.6.0\": [\n \"None\",\n \"aws_neuron_pytorch_p36\"\n ],\n \"1.7.1\": [\n \"None\",\n \"aws_neuron_pytorch_p36\"\n ],\n \"1.8.1\": [\n \"aws_neuron_pytorch_p36\",\n \"aws_neuron_pytorch_p36\"\n ],\n \"1.9.1\": [\n \"None\",\n \"aws_neuron_pytorch_p36\"\n ],\n \"1.10.1\": [\n \"None\",\n \"aws_neuron_pytorch_p36\"\n ],\n \"1.10.2\": [\n \"None\",\n \"None\"\n ],\n \"1.11.0\": [\n \"None\",\n \"None\"\n ]\n },\n \"mxnet\": {\n \"1.5.1\": [\n \"aws_neuron_mxnet_p36\",\n \"aws_neuron_mxnet_p36\"\n ],\n \"1.8.0\": [\n \"None\",\n \"aws_neuron_mxnet_p36\"\n ]\n }\n },\n \"latest_version_of_maintained_packages\": {\n \"runtime-server\": {\n \"framework\": false,\n \"package-name\": \"aws-neuron-runtime\",\n \"package-version\": \"1.6.24.0\",\n \"neuron-version\": \"1.15.2\"\n },\n \"mxnet-1.5.1\": {\n \"framework\": true,\n \"package-name\": \"mxnet_neuron\",\n \"package-version\": \"1.5.1.1.6.5.1\",\n \"neuron-version\": \"1.16.0\"\n }\n },\n \"fal_supported_runtime\": {\n \"tensorflow\": {\n \"1.15.5\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"1.15.5.1.6.10.0\"\n ],\n \"libnrt\": [\n \"1.15.5.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.1.4\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"2.1.4.1.6.10.0\"\n ],\n \"libnrt\": [\n \"2.1.4.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.2.3\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"2.2.3.1.6.10.0\"\n ],\n \"libnrt\": [\n \"2.2.3.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.3.3\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.3.4\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.3.4.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.4.2\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.4.3\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.4.3.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.5.0\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.5.1\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.5.2\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.5.3\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.6.3\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.6.5\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.7.1\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.7.3\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.8.0\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"2.8.2\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"2.5.1.2.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n }\n },\n \"pytorch\": {\n \"1.5.1\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"1.5.1.1.5.21.0\"\n ],\n \"libnrt\": [\n \"1.5.1.1.5.21.1\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.7.1\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"1.7.1.1.5.21.0\"\n ],\n \"libnrt\": [\n \"1.7.1.1.5.21.1\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.8.1\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"1.8.1.1.5.21.0\"\n ],\n \"libnrt\": [\n \"1.8.1.1.5.21.1\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.9.1\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"1.9.1.0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.10.1\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"1.9.1.0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.10.2\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"1.9.1.0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.11.0\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"1.9.1.0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.12.1\": {\n \"neuron-rtd\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"1.9.1.0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ]\n }\n },\n \"mxnet\": {\n \"1.5.1\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"99.99.99.99.99.99.99\"\n ],\n \"libnrt\": [\n \"99.99.99.99.99.99.99\",\n \"99.99.99.99.99.99.99\"\n ]\n },\n \"1.8.0\": {\n \"neuron-rtd\": [\n \"0.0.0.0\",\n \"1.8.0.1.3.4.0\"\n ],\n \"libnrt\": [\n \"1.8.0.1.3.4.1\",\n \"99.99.99.99.99.99.99\"\n ]\n }\n }\n },\n \"latest_release\": {\n \"inf1\": {\n \"version\": \"2.8.0\"\n }\n },\n \"neuron_versions\": {\n \"2.6.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.6.33.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.10.27.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.13.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.12.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.3.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.2.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuronx\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.11.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.43.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"2.7.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.7.15.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.10.27.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.7.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.13.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.12.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.3.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.2.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuronx\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.11.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.43.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"2.8.0\": {\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.7.33.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.10.30.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.8.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.13.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.12.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.6.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.4.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.4.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.9.3.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.10.1.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuronx\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.6.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.4.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.4.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.9.3.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.10.1.2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.11.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.43.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n },\n \"instance_support\": [\n \"inf1\"\n ],\n \"python_ver\": [\n \"3.7\"\n ]\n },\n \"2.5.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.6.33.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.10.27.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.5.19.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.13.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.5.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.12.1.2.5.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.5.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.3.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.2.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuronx\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.5.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.11.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.43.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"2.4.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.51.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.5.16.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.11.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"2.3.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.5.41.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.51.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuronx-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.11.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.19.2\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.3.26.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.51.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.9.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.9.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.11.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.19.1\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.3.11.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.51.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.9.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.9.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.11.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.19.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.3.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.51.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.9.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.9.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.11.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.2.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.11.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.1.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.3.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.8.0.2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.10.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.18.0\": {\n \"python_ver\": [\n \"3.7\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.14.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.51.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.8.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.8.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.790.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.10.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.1.2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.3.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.6.3.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.7.1.2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.4.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.6.3.2.2.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.7.0.2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.9.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.17.2\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.31.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.623.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.9.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.1.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.1.2.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.2.2.1.14.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.1.14.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.3.2.1.14.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.8.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.1.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.17.1\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.31.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.623.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.9.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.1.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.1.2.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.1.13.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.2.2.1.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.1.13.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.3.2.1.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.8.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.1.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.17.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.31.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.623.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.9.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.1.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.10.1.2.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.1.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.2.2.1.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.1.6.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.3.2.1.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.8.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.1.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.16.3\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.18.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.494.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.0.85.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.0.536.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.0.536.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.0.536.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.0.536.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.1.2.0.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.0.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.2.2.0.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.7.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.0.290.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.16.2\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.18.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.327.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.0.85.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.0.468.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.0.468.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.0.468.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.0.468.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.1.2.0.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.2.2.0.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.7.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.0.276.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.16.1\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.18.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.327.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.0.85.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.0.392.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.0.392.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.0.392.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.0.392.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.1.2.0.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.0.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.2.2.0.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.7.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.0.276.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.16.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"libnrt\": {\n \"framework\": false,\n \"packages\": {\n \"libnrt\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.15.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"lib\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.277.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.7.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"neuronperf\": {\n \"framework\": false,\n \"packages\": {\n \"neuronperf\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.0.85.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.2.0.318.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.2.0.318.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.2.0.318.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.9.1.2.0.318.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.4.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.3.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.1.2.0.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.3.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.4.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.3.2.0.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.2.2.0.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.7.0.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.2.0.271.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.15.2\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.24.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.22.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.22.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.21.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.25.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.6.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.5.21.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.5.21.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.1.5.21.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.3.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.2.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.0.1.6.10.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.2.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.0.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.1.1.6.10.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.1.1.6.10.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.6.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.3.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.15.1\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.24.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.22.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.22.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.21.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.25.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.6.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.5.21.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.5.21.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.1.5.21.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.3.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.2.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.0.1.6.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.2.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.0.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.1.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.1.1.6.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.6.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.3.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.15.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.450.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.19.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.17.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.17.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.16.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.20.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.6.13.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.5.21.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.5.21.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.1.5.21.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.1.4.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.2.3.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.3.3.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.4.2.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"2.5.0.1.6.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.1.4.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.2.2.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.3.0.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.4.1.1.6.8.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n },\n \"2.5.1.1.6.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.6.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.3.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.14.2\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.386.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.10.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.5.12.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.5.12.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.1.5.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.5.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.5.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.6.1.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.14.1\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.7.4.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.5.12.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.5.12.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.1.5.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.5.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.5.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.6.1.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.3.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.14.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.6.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.4.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.4.1.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.4.1.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.8.1.1.4.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.1.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.4.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.5.1.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.2.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.13.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.17.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.5.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.3.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.3.5.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.3.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.3.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-plugin-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"2.0.29.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.3.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.4.4.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n },\n \"mx_neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.8.0.1.1.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [\n \"wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\",\n \"pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl\"\n ],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.12.3\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.2.11.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.2.24.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.2.24.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.2.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.3.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.12.2\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.12.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.2.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.2.16.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.2.16.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.2.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.3.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.12.1\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.9.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.2.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.2.15.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.2.15.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.2.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.6.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.8.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.3.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.12.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.4.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.2.3.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.2.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.5.1.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.2.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.3.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.11.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.2.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.3.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.1.7.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n },\n \"1.7.1.1.1.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.4.1.1.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.1.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.1.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.2.1.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n },\n \"1.10.0\": {\n \"python_ver\": [\n \"3.6\"\n ],\n \"instance_support\": [\n \"inf1\"\n ],\n \"arch\": [\n \"x86_64\"\n ],\n \"components\": {\n \"driver\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-dkms\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.3.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\",\n \"src\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-server\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.5.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-plugin\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-plugin\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"k8-scheduler\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-k8-scheduler\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"runtime-base\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-runtime-base\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.0.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-rtd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"tools\": {\n \"framework\": false,\n \"packages\": {\n \"aws-neuron-tools\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.2.7.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"neuron-monitor\",\n \"neuron-cli\",\n \"neuron-top\",\n \"neuron-htop\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"compiler\": {\n \"framework\": false,\n \"packages\": {\n \"neuron-cc\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.0.24045.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": \"whl\"\n }\n }\n }\n }\n },\n \"pytorch\": {\n \"framework\": true,\n \"packages\": {\n \"torch-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.0.1978.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"torch-neuron\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow\": {\n \"framework\": true,\n \"packages\": {\n \"tensorflow-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.15.4.1.0.2168.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorboard\": {\n \"framework\": false,\n \"packages\": {\n \"tensorboard-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.0.615.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n },\n \"tensorflow-model-server\": {\n \"framework\": false,\n \"packages\": {\n \"tensorflow-model-server-neuron\": {\n \"install_on_compute_instance\": false,\n \"versions\": {\n \"1.15.0.1.0.2168.0\": {\n \"main_version\": true,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"deb\",\n \"rpm\"\n ]\n }\n }\n }\n }\n },\n \"mxnet\": {\n \"framework\": true,\n \"packages\": {\n \"mxnet-neuron\": {\n \"install_on_compute_instance\": true,\n \"versions\": {\n \"1.5.1.1.1.88.0\": {\n \"main_version\": false,\n \"pre_install_cmds\": [],\n \"post_install_cmds\": [],\n \"format\": [\n \"bin\"\n ],\n \"content\": [\n \"tbd\"\n ],\n \"package_type\": [\n \"whl\"\n ]\n }\n }\n }\n }\n }\n }\n }\n }\n}" }, { "path": "src/helperscripts/neuron-setup-example.py", "content": "from neuronsetuphelper import neuron_setup_helper\n\n\nnr_setup=neuron_setup_helper(manifest_file='default',neuron_version='latest')\n\nsetup_cmd = nr_setup.instructions(framework='tensorflow',action='Install',os='ubuntu',ami='non-dlami',mode='develop',framework_version='latest')\nprint (setup_cmd)\n" }, { "path": "src/helperscripts/neuronsetuphelper.py", "content": "import json\nimport argparse\nfrom packaging.version import Version, parse\n\n\n\n########################################\n# neuron_setup_helper\n########################################\n\nclass neuron_release_info:\n def __init__(self):\n\n self.release_frameworks_all = {}\n self.release_frameworks_main = {}\n self.release_packages_all ={}\n self.release_package_main={}\n self.release_frameworks_list=[]\n self.release_components_list = []\n self.release_tf_package_to_model_server_package={}\n self.release_os_install_list =[]\n self.python_ver=\"\"\n\n\n\n# release_frameworks_all\n# Desc: Dictionary - all framewors included in the release\n# example: 'pytorch-1.5.1': {'framework': 'pytorch', 'package': 'torch-neuron', 'version': '1.5.1.1.5.3.0', 'main': False, 'framework_version': '1.5.1', 'package_name': 'torch-neuron-1.5.1.1.5.3.0', 'pre_install_cmds': [], 'post_install_cmds': []}\n# release_frameworks_all = {}\n\n# release_frameworks_main\n# Desc: Dictionary - the main frameworks in each rlease (single version of the same framework)\n# example: 'mxnet': {'framework': 'mxnet-1.8.0', 'package': 'mx_neuron', 'version': '1.8.0.1.3.0.0', 'framework_version': '1.5.1', 'full_package_name': 'mx_neuron-1.8.0.1.3.0.0', 'pre_install_cmds': ['wget https://aws-mx-pypi.s3-us-west-2.amazonaws.com/1.8.0/aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl', 'pip install aws_mx_cu110-1.8.0-py2.py3-none-manylinux2014_x86_64.whl'], 'post_install_cmds': []}\n# release_frameworks_main = {}\n\n# release_packages_all\n# Desc: Dictionary - all packages included in the release\n# example: 'aws-neuron-dkms-1.5.0.0': {'component': 'driver', 'package': 'aws-neuron-dkms', 'version': '1.5.0.0', 'main': True, 'pre_install_cmds': [], 'post_install_cmds': []}\n# release_packages_all ={}\n\n# release_package_main\n# Desc: Dictionary - only single package from each component\n# example: 'driver': {'package': 'aws-neuron-dkms', 'version': '1.5.0.0', 'full_package_name': 'aws-neuron-dkms-1.5.0.0', 'pre_install_cmds': [], 'post_install_cmds': []}\n# release_package_main={}\n\n\n# list of all framewoks included in the specific neuron release\n# release_frameworks_list=[]\n\n# list of all neuron components included in the specific neuron release\n# release_components_list = []\n\n# dictionary to correlate tf version with model server version\n# release_tf_package_to_model_server_package = {}\n\n\n# list of all Neuron versions included in the manifest\nneuron_ver_list = []\n\n\n# release_os_install_list =[]\n\ndlami_conda_env= {}\n\n\n\n\npackage_formal_name= {\n \"compiler\":\"Neuron Compiler\",\n \"tensorflow\":\"Neuron TensorFlow\",\n \"pytorch\":\"Neuron PyTorch\",\n \"mxnet\":\"Neuron MXNet\",\n \"runtime-server\":\"Neuron Runtime server\",\n \"libnrt\":\"Neuron Runtime library\",\n \"runtime-base\":\"Neuron Runtime base\",\n \"driver\":\"Neuron Driver\",\n \"tools\":\"Neuron Tools\",\n \"tensorboard\":\"Neuron TensorBoard\",\n \"tensorflow-model-server\":\"Neuron TensorFlow model server\"\n }\n\n\n\n\n########################################\n# parse_arguments\n########################################\n\ndef cli_parse_arguments():\n __name__='neuron-install-helper.py'\n parser = argparse.ArgumentParser(prog=__name__\n ,usage='\\npython3 %(prog)s --list {neuron_versions,packages,components,frameworks} [--neuron-version=X.Y.Z] [--file FILE] \\n'\n +'python3 %(prog)s --install {pytorch,tensorflow,mxnet} [--neuron-version=X.Y.Z] [--framework-version=FRAMEWORK-X.Y.Z] [options]\\n'\n +'python3 %(prog)s --install {driver,runtime,tools} [--neuron-version=X.Y.Z] [options]\\n'\n +'python3 %(prog)s --update {pytorch,tensorflow,mxnet} [--framework-version=framework-X.Y.Z] [options]\\n'\n +'python3 %(prog)s --update {driver,runtime,tools} [options]\\n'\n +'options= [--file FILE] [--ami {dlami,non-dlami}] [--os {ubuntu,amazonlinux}]\\n'\n ,description='Installer helper for Neuron SDK')\n\n group = parser.add_mutually_exclusive_group(required=True)\n parser.add_argument(\"--neuron-version\",metavar='X.Y.Z')\n group.add_argument(\"--list\",choices=['neuron_versions','packages','components','frameworks'])\n group.add_argument(\"--install\",choices=['pytorch','tensorflow','mxnet'])\n group.add_argument(\"--update\",choices=['pytorch','tensorflow','mxnet'])\n parser.add_argument(\"--mode\",choices=['develop','compile','deploy'],default='develop')\n parser.add_argument(\"--framework-version\",metavar='framework-X.Y.Z')\n parser.add_argument(\"--os\",choices=['ubuntu','amazonlinux'],default='ubuntu',help='default=ubuntu')\n parser.add_argument(\"--ami\",choices=['dlami','non-dlami'],default='non-dlami',help='default=non-dlami')\n parser.add_argument(\"--file\",default='neuron-releases-manifest.json',help='default=neuron-releases-manifest.json')\n\n return parser.parse_args()\n\n\n\n\ndef enumerate_release_manifest(nr_setup, in_neuron_version):\n\n ########################################\n # Enumerate the Json file\n ########################################\n\n if nr_setup.file==None:\n nr_setup.file='neuron-releases-manifest.json'\n\n try:\n read_file = open(nr_setup.file, \"r\")\n except:\n print(__name__,\": error:\",\"Can't open \" + nr_setup.file + \" \")\n exit(-1)\n\n neuron_releases = json.load (read_file)\n\n latest_neuron_version = neuron_releases[\"latest_release\"][\"inf1\"][\"version\"]\n\n nr_setup.dlami_conda_env = neuron_releases[\"dlami_conda_env\"]\n\n nr_setup.fal_supported_runtime = neuron_releases[\"fal_supported_runtime\"]\n\n if (in_neuron_version == None) | (in_neuron_version == 'latest'):\n neuron_version=latest_neuron_version\n else:\n neuron_version = in_neuron_version\n\n\n\n for n_ver in neuron_releases[\"neuron_versions\"]:\n neuron_ver_list.append(n_ver)\n\n\n\n for neuron_release_ver in neuron_releases[\"neuron_versions\"]:\n m_release=neuron_releases[\"neuron_versions\"][neuron_release_ver][\"components\"]\n n_info=neuron_release_info()\n n_info.python_ver= neuron_releases[\"neuron_versions\"][neuron_release_ver][\"python_ver\"][0]\n\n for component_name in m_release:\n if m_release[component_name][\"framework\"]==False:\n n_info.release_components_list.append(component_name)\n m_packages=m_release[component_name][\"packages\"]\n for package_name in m_packages:\n for package_ver in m_packages[package_name][\"versions\"]:\n m_package_ver=m_packages[package_name][\"versions\"][package_ver]\n\n full_package_name=package_name+'-'+package_ver\n\n n_info.release_packages_all[full_package_name]= {\"component\":component_name,\"package\":package_name,\"version\":package_ver,\"main\":m_package_ver[\"main_version\"],\"pre_install_cmds\":m_package_ver[\"pre_install_cmds\"],\"post_install_cmds\":m_package_ver[\"post_install_cmds\"],\"package_type\":m_package_ver[\"package_type\"]}\n\n if m_package_ver[\"main_version\"]:\n n_info.release_package_main[component_name]={\"package\":package_name,\"version\":package_ver,\"full_package_name\":full_package_name,\"pre_install_cmds\":m_package_ver[\"pre_install_cmds\"],\"post_install_cmds\":m_package_ver[\"post_install_cmds\"],\"package_type\":m_package_ver[\"package_type\"]}\n\n if m_release[component_name][\"framework\"]:\n ver_digits = package_ver.rsplit('.')\n fw_ver=ver_digits[0]+'.'+ver_digits[1]+'.'+ver_digits[2]\n fw_name_ver=component_name+'-'+fw_ver\n\n if m_release[component_name][\"framework\"]:\n n_info.release_components_list.append(fw_name_ver)\n n_info.release_frameworks_list.append(fw_name_ver)\n\n if m_package_ver[\"main_version\"]:\n n_info.release_frameworks_main[component_name]={\"framework\":fw_name_ver,\"package\":package_name,\"version\":package_ver,\"framework_version\":fw_ver,\"package_name\":full_package_name,\"full_package_name\":full_package_name,\"pre_install_cmds\":m_package_ver[\"pre_install_cmds\"],\"post_install_cmds\":m_package_ver[\"post_install_cmds\"],\"package_type\":m_package_ver[\"package_type\"]}\n\n\n n_info.release_frameworks_all[fw_name_ver]={\"framework\":component_name,\"package\":package_name,\"version\":package_ver,\"main\":m_package_ver[\"main_version\"],\"framework_version\":fw_ver,\"package_name\":full_package_name,\"pre_install_cmds\":m_package_ver[\"pre_install_cmds\"],\"post_install_cmds\":m_package_ver[\"post_install_cmds\"],\"package_type\":m_package_ver[\"package_type\"]}\n\n if 'driver' in n_info.release_components_list:\n n_info.release_os_install_list.append('driver')\n if 'runtime-server' in n_info.release_components_list:\n n_info.release_os_install_list.append('runtime-server')\n if 'tools' in n_info.release_components_list:\n n_info.release_os_install_list.append('tools')\n if 'tensorflow-model-server' in n_info.release_components_list:\n n_info.release_os_install_list.append('tensorflow-model-server')\n\n # correlate TF and TF model server versions\n for pkg in n_info.release_packages_all.keys():\n if n_info.release_packages_all[pkg]['component'] == 'tensorflow':\n package_ver=n_info.release_packages_all[pkg]['version']\n ver_digits = package_ver.rsplit('.')\n tf_small_ver=ver_digits[0]+'.'+ver_digits[1]\n for pkg2 in n_info.release_packages_all.keys():\n if n_info.release_packages_all[pkg2]['component'] == 'tensorflow-model-server':\n package_ver=n_info.release_packages_all[pkg2]['version']\n ver_digits = package_ver.rsplit('.')\n tf_model_server_small_ver=ver_digits[0]+'.'+ver_digits[1]\n if tf_model_server_small_ver==tf_small_ver:\n n_info.release_tf_package_to_model_server_package[pkg]=pkg2\n break\n\n nr_setup.releases_info[neuron_release_ver]=n_info\n\n\n try:\n m_release=neuron_releases[\"neuron_versions\"][neuron_version][\"components\"]\n except:\n print(__name__,\": error: \",\"Version \" + neuron_version + \" is not a Neuron version or it is not supported\")\n exit(-1)\n\n\n\n\n return (neuron_version,latest_neuron_version)\n\n\n\n\n\n################\n# Sanity Checks\n################\ndef cli_validate(update,neuron_version,framework_version,is_latest_neuron,ami):\n # --update_cmd Sanity check\n # When choosing update, it always updating to latest , should not provide neuron_version\n if (update!=None) & (is_latest_neuron == False):\n print (__name__,\": error: \",\"--update always update to latest Neuron versions, can't specify Neuron version\")\n exit(-1)\n\n #if neuron_version != None:\n # if ami == 'dlami':\n # print (__name__,\": error: \",\"--neuron_version should not be specified together with --ami=dlami\")\n # exit(-1)\n\n if (framework_version != None):\n if (framework_version not in nr_setup.releases_info[neuron_version].release_frameworks_list):\n print (__name__,\": error: \",\" \" + framework_version + \" is not a supported framework\")\n exit(-1)\n\n########################################\n# version to tuple\n########################################\n\ndef versiontuple(v):\n filled = []\n for point in v.split(\".\"):\n filled.append(point.zfill(8))\n return tuple(filled)\n\n\n########################################\n# --list command\n########################################\ndef cli_list_cmd(nr_setup, neuron_version, list):\n\n\n str =''\n\n if (list == 'neuron_versions'):\n str += '\\nList of Neuron release versions supported by this helper:\\n' + '\\n'\n for ver in neuron_ver_list:\n str += 'neuron-'+ver + '\\n'\n\n #TODO: add \"[main]\" label to main packages\n if (list == 'packages'):\n str += '\\nList of Neuron packages included in Neuron release version ' + neuron_version + ':\\n' + '\\n'\n for package in nr_setup.releases_info[neuron_version].release_packages_all:\n if len( nr_setup.releases_info[neuron_version].release_packages_all[package]['package_type']):\n #FIXME Runtime library hardcode print\n if (nr_setup.releases_info[neuron_version].release_packages_all[package][\"component\"] == 'libnrt'):\n str += nr_setup.releases_info[neuron_version].release_packages_all[package][\"component\"] +' : \\t' + \\\n \"libnrt.so (version \"+ \\\n nr_setup.releases_info[neuron_version].release_packages_all[package][\"version\"] + \")\" + '\\n'\n else:\n str += nr_setup.releases_info[neuron_version].release_packages_all[package][\"component\"] +' : \\t' + package + '\\n'\n\n if (list == 'components'):\n str += '\\nList of Neuron components included in Neuron release version ' + neuron_version + ':\\n' + '\\n'\n for comp in nr_setup.releases_info[neuron_version].release_components_list:\n str += comp + '\\n'\n\n #TODO: add \"[main]\" label to main frameworks\n if (list == 'frameworks'):\n str += '\\nList of frameworks included in Neuron release version ' + neuron_version + ':\\n' + '\\n'\n for fw in nr_setup.releases_info[neuron_version].release_frameworks_all:\n str += nr_setup.releases_info[neuron_version].release_frameworks_all[fw][\"framework\"] +' : \\t' + fw + '\\n'\n\n return str\n\n\n########################################\n# Print configuration\n########################################\n\ndef hlpr_print_config(nr_setup, neuron_version):\n str = ''\n str += '\\n'\n str += '###########################################################################' + '\\n'\n str += '# ' + nr_setup.action + ' ' + nr_setup.framework + ' '\n if (nr_setup.framework_version != 'latest') & (nr_setup.framework_version != None):\n str += '(' + nr_setup.framework_version + ')' + ' '\n if nr_setup.action == 'Update':\n str += 'from latest Neuron version ' + neuron_version\n else:\n str += 'from Neuron version ' + neuron_version\n\n str += '\\n# '\n\n str += 'On '\n if (nr_setup.os == 'ubuntu'):\n str += 'Ubuntu '\n elif (nr_setup.os == 'amazonlinux'):\n str += 'Amazon Linux '\n\n if (nr_setup.ami == 'dlami'):\n str += 'DLAMI'\n else:\n str += 'AMI'\n\n str += ' for '\n if (nr_setup.mode == 'compile'):\n str += 'compilation on compute instance'\n elif (nr_setup.mode == 'develop'):\n str += 'development on inf1 instance'\n elif (nr_setup.mode == 'deploy'):\n str += 'deployment on inf1 instance'\n str += '\\n'\n str += '###########################################################################' + '\\n'\n str += '\\n'\n\n return str\n\n###################################\n# Build Pip command\n###################################\ndef hlpr_build_pip_command(nr_setup, neuron_version, component,include_compiler,optional):\n\n\n package_dict= nr_setup.releases_info[neuron_version].release_package_main\n\n if (nr_setup.framework_version==None):\n fw_package_dict= nr_setup.releases_info[neuron_version].release_frameworks_main\n fw_comp=component\n else:\n fw_package_dict= nr_setup.releases_info[neuron_version].release_frameworks_all\n fw_comp=nr_setup.framework_version\n\n pip_cmd_prefix=''\n pip_cmd =''\n\n\n if nr_setup.action=='Install':\n pip_cmd_prefix = 'pip install '\n else:\n pip_cmd_prefix = 'pip install --upgrade '\n\n cmd=pip_cmd_prefix\n\n if (component == 'mxnet') | (component == 'pytorch') | (component == 'tensorflow'):\n\n # Framework installation\n if (component == 'mxnet') | (component == 'pytorch'):\n pip_cmd += cmd + fw_package_dict[fw_comp]['package']\n if (nr_setup.is_latest_neuron==False) | (nr_setup.force_versions == True):\n pip_cmd += '=='+fw_package_dict[fw_comp]['version']\n elif (nr_setup.is_latest_neuron==True)&(nr_setup.framework_version!=None):\n pip_cmd += '=='+fw_package_dict[fw_comp]['framework_version']+'.*'\n\n elif (component == 'tensorflow'):\n if ((parse(neuron_version)=parse('2.99.99')):\n os_cmd += '\\n'\n os_cmd += '################################################################################################################\\n'\n os_cmd += '# To install or update to Neuron versions 2.99.99 and newer from previous releases:'+ '\\n'\n if (nr_setup.os=='ubuntu'):\n os_cmd += '# - Unstall aws-neuron-dkms by calling \\`sudo dnf remove aws-neuron-dkms -y\\` -y'+ '\\n'\n elif (nr_setup.os=='amazonlinux'):\n os_cmd += '# - Unstall aws-neuron-dkms by calling \\`sudo apt-get remove aws-neuron-dkms\\` -y'+ '\\n'\n os_cmd += '# - DO NOT skip \\'aws-neuronx-dkms\\' install or upgrade step, you MUST install or upgrade to latest Neuron driver'+ '\\n'\n os_cmd += '################################################################################################################\\n'\n elif (parse(neuron_version)>=parse('1.19.1')):\n os_cmd += '\\n'\n os_cmd += '################################################################################################################\\n'\n os_cmd += '# To install or update to Neuron versions 1.19.1 and newer from previous releases:'+ '\\n'\n os_cmd += '# - DO NOT skip \\'aws-neuron-dkms\\' install or upgrade step, you MUST install or upgrade to latest Neuron driver'+ '\\n'\n os_cmd += '################################################################################################################\\n'\n\n\n # Update header files if driver should be installed or updated\n if (comp=='driver'):\n os_cmd += hlpr_os_headers_update(nr_setup)\n\n\n\n\n if nr_setup.os=='ubuntu':\n os_cmd_prefix = 'sudo apt-get install '\n elif (nr_setup.action=='Install')&(nr_setup.os=='amazonlinux'):\n os_cmd_prefix = 'sudo dnf install '\n elif (nr_setup.action=='Update')&(nr_setup.os=='amazonlinux'):\n os_cmd_prefix = 'sudo dnf update '\n\n if comp in nr_setup.releases_info[neuron_version].release_os_install_list:\n # install only if there is a package associated with the component\n if (len(pkg_dict[key]['package_type']) != 0):\n #os_cmd = build_os_command(cmd=os_cmd_prefix,component=comp,is_latest_release=is_latest_neuron)\n os_cmd += '\\n'\n if (optional==False):\n os_cmd += '# ' + nr_setup.action + ' ' + package_formal_name[comp]\n else:\n os_cmd += '# Optional: ' + nr_setup.action + ' ' + package_formal_name[comp]\n\n if (nr_setup.is_latest_neuron==False)&(nr_setup.os=='ubuntu'):\n os_cmd += '\\n'\n os_cmd += '# If you are downgrading from newer version, please add \\'--allow-downgrades\\' option to \\'sudo apt-get install\\' '\n if (nr_setup.is_latest_neuron==False)&(nr_setup.os=='amazonlinux'):\n os_cmd += '\\n'\n os_cmd += '# If you are downgrading from newer version , please remove existing package using \\'sudo dnf remove\\' before installing the older package'\n os_cmd += '\\n'\n # Amazon Linux DLAMI will not allow updating tensorflow-model-server and aws-neuron-dkms without adding sudo dnf versionlock delete\n if ((comp=='tensorflow-model-server') | (comp=='driver')) & (nr_setup.ami == 'dlami') & (nr_setup.os == 'amazonlinux'):\n os_cmd += 'sudo dnf versionlock delete '\n os_cmd += pkg_dict[key]['package']\n os_cmd += '\\n'\n\n os_cmd += os_cmd_prefix + pkg_dict[key]['package']\n\n # Amazon Linux yum installation packaging versioning is set via hyphen not equals\n version_key = \"=\"\n if (nr_setup.os=='amazonlinux'):\n version_key = \"-\"\n\n if (nr_setup.is_latest_neuron==False) | (nr_setup.force_versions):\n os_cmd += version_key + pkg_dict[key]['version']\n elif (pkg!=None):\n if ( nr_setup.releases_info[neuron_version].release_package_main[comp]['version']!= nr_setup.releases_info[neuron_version].release_packages_all[pkg]['version']):\n os_cmd += version_key + pkg_dict[key]['version']\n\n # Ubuntu DLAMI will not allow updating tensorflow-model-server and aws-neuron-dkms without adding --allow-change-held-packages\n if ((comp=='tensorflow-model-server') | (comp=='driver')) & (nr_setup.ami == 'dlami') & (nr_setup.os == 'ubuntu'):\n os_cmd += ' --allow-change-held-packages'\n\n os_cmd += ' -y'\n os_cmd += '\\n'\n\n # Update header files if driver should be installed or updated\n if (comp=='driver'):\n os_cmd += '\\n'\n os_cmd += '####################################################################################\\n'\n os_cmd += '# Warning: If Linux kernel is updated as a result of OS package update'+ '\\n'\n if (parse(neuron_version)>=parse('2.99.99')):\n os_cmd += '# Neuron driver (aws-neuronx-dkms) should be re-installed after reboot'+ '\\n'\n else:\n os_cmd += '# Neuron driver (aws-neuron-dkms) should be re-installed after reboot'+ '\\n'\n os_cmd += '####################################################################################\\n'\n\n if (comp=='tools'):\n if (parse(neuron_version)>=parse('2.99.99')):\n os_cmd += '\\n'\n os_cmd += '################################################################################################################\\n'\n os_cmd += '# To install or update to Neuron versions 2.99.99 and newer from previous releases:'+ '\\n'\n if (nr_setup.os=='ubuntu'):\n os_cmd += '# - Unstall aws-neuron-tools by calling \\`sudo dnf remove aws-neuron-tools -y\\` -y'+ '\\n'\n elif (nr_setup.os=='amazonlinux'):\n os_cmd += '# - Unstall aws-neuron-tools by calling \\`sudo apt-get remove aws-neuron-tools\\` -y'+ '\\n'\n os_cmd += '################################################################################################################\\n'\n\n return os_cmd\n\n\n########################################\n## installation / Update instructions\n########################################\ndef hlpr_instructions(nr_setup, neuron_version):\n\n cmd_string = ''\n\n setup_mode=nr_setup.mode\n\n\n\n # look for conda environment for this framework version\n for fw_env in nr_setup.dlami_conda_env:\n if fw_env != nr_setup.framework:\n continue\n fw_ver_conda_env=nr_setup.dlami_conda_env[fw_env]\n for conda_env_fw_ver in fw_ver_conda_env:\n if (conda_env_fw_ver == nr_setup.fw_package_dict[nr_setup.fw_comp]['framework_version']):\n nr_setup.conda_env=nr_setup.dlami_conda_env[fw_env][conda_env_fw_ver][0]\n nr_setup.generic_conda_env=nr_setup.dlami_conda_env[fw_env][conda_env_fw_ver][1]\n break\n\n\n # look what runtime works with this framework version\n fal_rtd=False\n fal_libnrt=False\n for fw in nr_setup.fal_supported_runtime:\n if fw != nr_setup.framework:\n continue\n if fw == nr_setup.framework:\n if (nr_setup.framework_version == None):\n fw_ver= nr_setup.releases_info[neuron_version].release_frameworks_main[nr_setup.framework]['framework_version']\n fal_version= nr_setup.releases_info[neuron_version].release_frameworks_main[nr_setup.framework]['version']\n else:\n fw_ver= nr_setup.releases_info[neuron_version].release_frameworks_all[nr_setup.framework_version]['framework_version']\n fal_version= nr_setup.releases_info[neuron_version].release_frameworks_all[nr_setup.framework_version]['version']\n fal_supported_rtd=nr_setup.fal_supported_runtime[fw][fw_ver]['neuron-rtd']\n fal_supported_libnrt=nr_setup.fal_supported_runtime[fw][fw_ver]['libnrt']\n if (parse(fal_version) >= parse(fal_supported_rtd[0])) & \\\n (parse(fal_version) <= parse(fal_supported_rtd[1])):\n fal_rtd=True\n elif (parse(fal_version) >= parse(fal_supported_libnrt[0])) & \\\n (parse(fal_version) <= parse(fal_supported_libnrt[1])):\n fal_libnrt=True\n\n if nr_setup.conda_env == \"None\":\n dlami_ev_exists=False\n else:\n dlami_ev_exists=True\n\n #cmd_string += hlpr_print_config(nr_setup, neuron_version)\n\n if (nr_setup.framework_version==None):\n fw_package_dict= nr_setup.releases_info[neuron_version].release_frameworks_main\n fw_comp=nr_setup.framework\n else:\n fw_package_dict= nr_setup.releases_info[neuron_version].release_frameworks_all\n fw_comp=nr_setup.framework_version\n\n\n\n if (nr_setup.framework !=None): #if install or update\n # If we are not using DLAMI\n if (nr_setup.ami=='non-dlami') | \\\n ((nr_setup.ami=='dlami') & \\\n (\n (nr_setup.action == 'Update') | \\\n (dlami_ev_exists==False) | \\\n (nr_setup.is_latest_neuron==False)) \\\n ):\n\n\n\n if (nr_setup.ami=='dlami') & (dlami_ev_exists==False):\n cmd_string += '\\n'\n cmd_string += '# Note: There is no DLAMI Conda environment for this framework version'+ '\\n'\n cmd_string += '# Framework will be installed/updated inside a Python environment'+ '\\n'\n\n\n if (setup_mode == 'develop') | (setup_mode == 'deploy'):\n if (nr_setup.action =='Install')&(nr_setup.ami!='dlami'):\n # For First install, setup Neuron OS packagaes repo (dnf or apt)\n cmd_string += hlpr_os_packages_first_setup(nr_setup)\n\n # Always update to latest OS packages\n cmd_string += hlpr_os_packages_update(nr_setup)\n\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version, comp='driver',optional=False,pkg=None)\n\n\n #FIXME Temporary check for MXNET 1.5 in maintenance mode\n if (neuron_version == \"1.16.0\") & (nr_setup.framework==\"mxnet\")& \\\n (fw_package_dict[fw_comp]['framework_version']==\"1.5.1\"):\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version=\"1.15.2\", comp='runtime-server',optional=False,pkg=None)\n elif (fal_rtd):\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version, comp='runtime-server',optional=False,pkg=None)\n\n #if mode = develop, install tools\n if (setup_mode == 'develop'):\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version, comp='tools',optional=False,pkg=None)\n if (nr_setup.framework == 'tensorflow'):\n cmd_string += hlpr_build_pip_command(nr_setup, neuron_version, component='tensorboard',include_compiler=False,optional=False)\n\n if (nr_setup.action =='Install'):\n cmd_string += hlpr_os_export_path(nr_setup)\n\n if (nr_setup.ami=='non-dlami') | \\\n ((nr_setup.ami=='dlami')&(nr_setup.generic_conda_env==\"None\")):\n\n if (nr_setup.action =='Install'):\n # For first install , install python venv and activate a venv\n cmd_string += hlpr_pip_install_create_python_venv(nr_setup, neuron_version)\n elif (nr_setup.action =='Update'):\n # For nect times, activate the venv used for initial install\n cmd_string += hlpr_pip_activate_python_venv(nr_setup, neuron_version)\n elif (nr_setup.ami=='dlami'):\n cmd_string += hlpr_framework_dlami_activate(nr_setup)\n\n # Setup Neuron pip packages\n cmd_string += hlpr_pip_repos_setup()\n\n # Now install framework\n if (setup_mode == 'deploy'):\n # do not install compiler when deploying\n cmd_string += hlpr_framework_compiler_setup(nr_setup, neuron_version, include_compiler=False)\n else:\n # install compiler when mode = developer or mode = compile\n cmd_string += hlpr_framework_compiler_setup(nr_setup, neuron_version, include_compiler=True)\n\n\n #if mode = deploy, install model server\n if (setup_mode != 'compile'):\n if (nr_setup.framework == 'tensorflow'):\n if (nr_setup.framework_version==None):\n tf_package= nr_setup.releases_info[neuron_version].release_frameworks_main[nr_setup.framework]['package_name']\n else:\n tf_package= nr_setup.releases_info[neuron_version].release_frameworks_all[nr_setup.framework_version]['package_name']\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version, comp='tensorflow-model-server',optional=True,pkg= nr_setup.releases_info[neuron_version].release_tf_package_to_model_server_package[tf_package])\n\n\n # if running DLAMI\n elif (nr_setup.ami=='dlami'):\n if (nr_setup.action =='Install'):\n\n cmd_string += '\\n'\n cmd_string += '# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions '+ '\\n'\n cmd_string += '# To update to latest Neuron version, follow \"Update to latest release\" instruction on Neuron documentation'+ '\\n'\n\n # WARNING: Exception\n # Starting Neuron 1.16.0 , new kernel is needed to work with Runtime 2.x (library mode)\n if (parse(neuron_version)>=parse('1.16.0')):\n if (setup_mode == 'develop') | (setup_mode == 'deploy'):\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version, comp='driver',optional=False,pkg=None)\n\n #FIXME Temporary check for MXNET 1.5 in maintenance mode\n if (neuron_version == \"1.16.0\") & (nr_setup.framework==\"mxnet\")& \\\n (fw_package_dict[fw_comp]['framework_version']==\"1.5.1\"):\n cmd_string += hlpr_os_comp_setup_cmd(nr_setup, neuron_version=\"1.15.2\", comp='runtime-server',optional=False,pkg=None)\n\n cmd_string += '\\n'\n cmd_string += hlpr_framework_dlami_activate(nr_setup)\n\n\n\n return cmd_string\n\n\n\n\n\n########################################\n# neuron_setup_helper\n########################################\n\nclass neuron_setup_helper:\n def __init__(self, manifest_file,neuron_version):\n\n # All Neuron releases\n self.releases_info = {}\n\n if (manifest_file== None) | (manifest_file== 'default') :\n self.file = 'neuron-releases-manifest.json'\n else:\n self.file = manifest_file\n\n ver_tuple = enumerate_release_manifest(nr_setup=self,in_neuron_version=neuron_version)\n self.neuron_version = ver_tuple[0]\n self.latest_neuron_version = ver_tuple[1]\n\n self.conda_env=\"\"\n self.python_ver=\"\"\n self.generic_conda_env=\"\"\n\n if self.neuron_version == self.latest_neuron_version:\n self.is_latest_neuron=True\n else:\n self.is_latest_neuron=False\n\n if (self.is_latest_neuron) & (neuron_version !=None) & (neuron_version !='latest'):\n # User explicitly specified the version, although it is the latest version\n # in this case the instructions will include the exact versions of the packages\n self.force_versions=True\n else:\n self.force_versions=False\n\n\n def instructions(self,framework,action,framework_version,os,ami,mode):\n\n self.framework=framework\n self.action=action\n self.mode=mode\n self.os=os\n self.ami=ami\n if (framework_version=='latest'):\n self.framework_version=None\n else:\n self.framework_version=framework_version\n setup_cmd = \"\"\n\n if (self.framework_version==None):\n self.fw_package_dict= self.releases_info[self.neuron_version].release_frameworks_main\n self.fw_comp=self.framework\n else:\n self.fw_package_dict= self.releases_info[self.neuron_version].release_frameworks_all\n self.fw_comp=self.framework_version\n\n setup_cmd=hlpr_instructions(self,self.neuron_version)\n\n return setup_cmd\n\nif __name__ == '__main__':\n setup_cmd =''\n args = cli_parse_arguments()\n nr_setup=neuron_setup_helper(manifest_file=args.file,neuron_version=args.neuron_version)\n\n cli_validate(update=args.update,neuron_version=nr_setup.neuron_version,framework_version=args.framework_version,is_latest_neuron=nr_setup.is_latest_neuron,ami=args.ami)\n if (args.list):\n setup_cmd += cli_list_cmd(nr_setup=nr_setup,neuron_version=nr_setup.neuron_version, list=args.list)\n else:\n if (args.install != None)|(args.update !=None):\n if args.install:\n framework=args.install\n action = 'Install'\n elif args.update:\n framework=args.update\n action = 'Update'\n else:\n action = None\n framework=None\n\n setup_cmd += nr_setup.instructions(framework=framework,action=action,framework_version=args.framework_version,os=args.os,ami=args.ami,mode=args.mode)\n print (setup_cmd)\n\n\n\n\n\n\n" }, { "path": "src/helperscripts/release-manifest-def.py", "content": "\nneuron_releases={\n \"repos\":{\n \"whl\"=\"_url\", # url of the wheel repo\n \"rpm\"=\"_url\", # url of the rpm repo (yum)\n \"deb\"=\"_url\", # url of the debian repo (apt)\n }\n \"manifest_date\": \"_date\",\n \"manifest_version\":\"_ver\" # Will increment when format change\n \"latest_release\":{\n \"_instance\":{ # can be \"inf1\", \"trn1\", etc.. \n \"version\":\"_ver\" # latest neuron release that support the _instance \n }\n }\n \"neuron_versions\"={ # all neuron release versions supported by this manifest\n \"_neuron_version\":{ # Neuron release version entry e.g. \"1.14.0\"\n \"python_ver\": [\"_ver\"] # list of python versions supported by this neuron release, e.g. \"3.6\"\n \"instance_support\": [\"_instance\"] # list of instances supported by this neuron release\n \"arch\":[\"_arch\"] # list of architectures supported by this neuron release (e.g. x86)\n \"components\":{ # all components included in this neuron release \n # (e.g. compiler, driver , pytorch ...)\n \"_component_name\":{ # component entry (e.g. driver, compiler)\n \"framework\":_boolean # is this component a framework ? \n # needed since there is a differces in versioning and content etc .. \n \"packages\":{ # all packages of this component that included in this release \n # e.g. mxnet support mx_neuron and mxnet-neuron\n \"_package_name\":{ # package entry (e.g. mx_neuron)\n \"install_on_compute_instance\":_booolean # can this package installed on compute instance?\n \"versions\":{ # all versions of the specific package\n # e.g. torch-neuron may include multiple versions\n \"_ver\":{ # package version entry (e.g. 1.4.1.0)\n \"pre_install_cmds\":[\"_cmd\"] # a list of commands to call before installing\n # the package, e.g. when a plugin need to install the\n # framework first , as in mx_neuron\n \"post_install_cmds\":[\"_cmd\"] # a list of commands to call after installaing the package\n \"format\":[\"_format\"] # package format (e.g. bin or src)\n \"content\":[\"_content\"] # package content \n # (e.g. tools include neuron-top, neuron monitor etc .. )\n \"package_type\":[\"_type\"] # list of package type supported ( e.g. whl, rpm, deb)\n }\n } \n }\n }\n }\n }\n }\n },\n \"softwarelifecycle\":{ # Status of neuron software releases (supported, maintained, deprecated)\n # Releases that are not under \"supported\" or \"maintained\" should be \"supported\"\n \"maintained\":{ # Releases that are being maintained, no active development, bug fixes can be provided\n # releases can be Neuron release, component (e.g. runtime), or a framework (e.g. pytorch-1.5.x)\n \"neuron_versions\":{ # Neuron versions that are under maintanance status\n \"from\":\"_ver\" # from neuron release version\n \"to\":\"_ver\" # to neuron release version\n },\n \"components\":{ # Components that are under maintanance status\n \"_component_name\":{ # packages in that component\n \"_package_name\":{ # package entry\n \"from\":\"_ver\" # from version\n \"to\":\"_ver\" # to version\n }\n }\n\n },\n \"frameworks\":{ # Frameworks that are under maintanance status\n \"pytorch\":{ # Pytorch versions that are under maintanance status\n \"from\":\"_ver\" # from version\n \"to\":\"_ver\" # to version\n },\n \"tensorflow\":{ # Pytorch versions that are under maintanance status\n \"from\":\"_ver\" # from version\n \"to\":\"_ver\" # to version\n },\n \"mxnewt\":{ # MXNet versions that are under maintanance status\n \"from\":\"_ver\" # from version\n \"to\":\"_ver\" # to version\n }\n }\n },\n \"deprecated\":{ # Releases that are deprecated, no bug fixes\n # format similar to \"maintained\" section\n },\n },\n \"compatability\": { # compatability section\n \"_component_name\": { # component entry\n \"_package_name\": { # package entry\n \"_ver_to__ver\": { # compatability entry\n \"from\": \"_ver\", # from version\n \"to\": \"_ver\", # to version\n \"instance_support\": [ # instance compatability\n \"_instance\"\n ],\n \"arch\": [ # arch compatability\n \"_arch\"\n ],\n \"components\": { # components compatability section \n \"_component_name\": { # component entry\n \"_package_name\": { # package entry\n \"from\": \"_ver\", # from version\n \"to\": \"_ver\" # to version\n }\n }\n }\n }\n }\n }\n }\n}\n" }, { "path": "src/k8/bert_service.yml", "content": "---\nkind: Service\napiVersion: v1\nmetadata:\n name: inf-k8s-test\n labels:\n app: inf-k8s-test\nspec:\n ports:\n - name: http-tf-serving\n port: 8500\n targetPort: 8500\n - name: grpc-tf-serving\n port: 9000\n targetPort: 9000\n selector:\n app: inf-k8s-test\n role: master\n type: ClusterIP\n---\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: inf-k8s-test\n labels:\n app: inf-k8s-test\n role: master\nspec:\n replicas: 1 # Number of desired replicas. Increase to desired number.\n selector:\n matchLabels:\n app: inf-k8s-test\n role: master\n template:\n metadata:\n labels:\n app: inf-k8s-test\n role: master\n spec:\n volumes:\n - name: sock\n emptyDir: {}\n containers:\n - name: inf-k8s-test\n image: tf-serving-ctr\n imagePullPolicy: IfNotPresent\n command: [\"/bin/sh\",\"-c\"]\n\n # Pull model from s3, then start tensorflow_model_server_neuron with the model.\n args:\n - \"aws s3 sync s3:///bert /tmp/bert && \\\n tensorflow_model_server_neuron --port=9000 --rest_api_port=8500 --model_name=bert_mrpc_hc_gelus_b4_l24_0926_02 --model_base_path=/tmp//bert/\"\n\n # Open grpc and rest API ports\n ports:\n - containerPort: 8500\n - containerPort: 9000\n\n # Arbitrary resource requirements\n resources:\n limits:\n cpu: 4\n memory: 4Gi\n aws.amazon.com/neuron: 1 # desired number of Inferentia devices.\n requests:\n cpu: \"1\"\n memory: 1Gi\n aws.amazon.com/neuron: 1 # desired number of Inferentia devices.\n" }, { "path": "src/k8/k8s-neuron-device-plugin-rbac.yml", "content": "# rbac.yaml\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: neuron-device-plugin\nrules:\n- apiGroups:\n - \"\"\n resources:\n - nodes\n verbs:\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - events\n verbs:\n - create\n - patch\n- apiGroups:\n - \"\"\n resources:\n - pods\n verbs:\n - update\n - patch\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - nodes/status\n verbs:\n - patch\n - update\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: neuron-device-plugin\n namespace: kube-system\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: neuron-device-plugin\n namespace: kube-system\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: neuron-device-plugin\nsubjects:\n- kind: ServiceAccount\n name: neuron-device-plugin\n namespace: kube-system\n" }, { "path": "src/k8/k8s-neuron-device-plugin.yml", "content": "# https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: neuron-device-plugin-daemonset\n namespace: kube-system\nspec:\n selector:\n matchLabels:\n name: neuron-device-plugin-ds\n updateStrategy:\n type: RollingUpdate\n template:\n metadata:\n # Uncomment the annotation below if k8s version is 1.13 or lower\n # annotations:\n # scheduler.alpha.kubernetes.io/critical-pod: \"\"\n labels:\n name: neuron-device-plugin-ds\n spec:\n serviceAccount: neuron-device-plugin\n tolerations:\n - key: CriticalAddonsOnly\n operator: Exists\n - key: aws.amazon.com/neuron\n operator: Exists\n effect: NoSchedule\n # Mark this pod as a critical add-on; when enabled, the critical add-on\n # scheduler reserves resources for critical add-on pods so that they can\n # be rescheduled after a failure.\n # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/\n priorityClassName: \"system-node-critical\"\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n # Uncomment following matchExpressions if using k8s 1.16 or lower\n #- matchExpressions:\n # - key: \"beta.kubernetes.io/instance-type\"\n # operator: In\n # values:\n # - inf1.xlarge\n # - inf1.2xlarge\n # - inf1.6xlarge\n # - inf1.24xlarge\n # - inf2.xlarge\n # - inf2.8xlarge\n # - inf2.24xlarge\n # - inf2.48xlarge\n # - trn1.2xlarge\n # - trn1.32xlarge\n # - trn1n.32xlarge\n - matchExpressions:\n - key: \"node.kubernetes.io/instance-type\"\n operator: In\n values:\n - inf1.xlarge\n - inf1.2xlarge\n - inf1.6xlarge\n - inf1.24xlarge\n - inf2.xlarge\n - inf2.8xlarge\n - inf2.24xlarge\n - inf2.48xlarge\n - trn1.2xlarge\n - trn1.32xlarge\n - trn1n.32xlarge\n containers:\n # Find all neuron-device-plugin images at https://gallery.ecr.aws/neuron/neuron-device-plugin\n - image: public.ecr.aws/neuron/neuron-device-plugin:2.22.4.0\n imagePullPolicy: Always\n name: neuron-device-plugin\n env:\n - name: KUBECONFIG\n value: /etc/kubernetes/kubelet.conf\n - name: NODE_NAME\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n securityContext:\n allowPrivilegeEscalation: false\n capabilities:\n drop: [\"ALL\"]\n volumeMounts:\n - name: device-plugin\n mountPath: /var/lib/kubelet/device-plugins\n - name: infa-map\n mountPath: /run\n volumes:\n - name: device-plugin\n hostPath:\n path: /var/lib/kubelet/device-plugins\n - name: infa-map\n hostPath:\n path: /run\n\n\n\n" }, { "path": "src/k8/k8s-neuron-monitor-daemonset.yml", "content": "apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: neuron-monitor\n namespace: neuron-monitor\n labels:\n app: neuron-monitor\n version: v1\nspec:\n selector:\n matchLabels:\n app: neuron-monitor\n template:\n metadata:\n labels:\n app: neuron-monitor\n version: v1\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/os\n operator: In\n values:\n - linux\n - key: node.kubernetes.io/instance-type\n operator: In\n values:\n - trn1.2xlarge\n - trn1.32xlarge\n - trn1n.32xlarge\n - inf1.xlarge\n - inf1.2xlarge\n - inf1.6xlarge\n - inf2.xlarge\n - inf2.8xlarge\n - inf2.24xlarge\n - inf2.48xlarge\n containers:\n - name: neuron-monitor\n image: public.ecr.aws/neuron/neuron-monitor:1.3.0\n ports:\n - containerPort: 8000\n command:\n - \"/opt/bin/entrypoint.sh\"\n args: \n - \"--port\"\n - \"8000\"\n - \"--neuron-monitor-config\"\n - \"/opt/aws/neuron/bin/neuron-monitor.conf\"\n resources:\n limits:\n cpu: 500m\n memory: 256Mi\n requests:\n cpu: 256m\n memory: 128Mi\n env:\n - name: GOMEMLIMIT\n value: 160MiB\n securityContext:\n privileged: true\n" }, { "path": "src/k8/k8s-neuron-scheduler-configmap.yml", "content": "apiVersion: v1\ndata:\n policy.cfg: |\n {\n \"kind\": \"Policy\",\n \"apiVersion\": \"v1\",\n \"extenders\": [\n {\n \"urlPrefix\": \"http://127.0.0.1:32700\",\n \"filterVerb\": \"filter\",\n \"bindVerb\": \"bind\",\n \"enableHttps\": false,\n \"nodeCacheCapable\": true,\n \"managedResources\": [\n {\n \"name\": \"aws.amazon.com/neuron\",\n \"ignoredByScheduler\": false\n },\n {\n \"name\": \"aws.amazon.com/neurondevice\",\n \"ignoredByScheduler\": false\n },\n {\n \"name\": \"aws.amazon.com/neuroncore\",\n \"ignoredByScheduler\": false\n }\n ],\n \"ignorable\": false\n }\n ]\n }\nkind: ConfigMap\nmetadata:\n name: scheduler-policy\n namespace: kube-system\n" }, { "path": "src/k8/k8s-neuron-scheduler-eks.yml", "content": "# rbac.yaml\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: k8s-neuron-scheduler\nrules:\n- apiGroups:\n - \"\"\n resources:\n - nodes\n verbs:\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - nodes/status\n verbs:\n - update\n - patch\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - events\n verbs:\n - create\n - patch\n- apiGroups:\n - \"\"\n resources:\n - pods\n verbs:\n - update\n - patch\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - bindings\n - pods/binding\n verbs:\n - create\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: k8s-neuron-scheduler\nsubjects:\n- kind: ServiceAccount\n name: k8s-neuron-scheduler\n namespace: kube-system\n\n# deployment yaml\n---\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\nspec:\n replicas: 1\n strategy:\n type: Recreate\n selector:\n matchLabels:\n app: neuron-scheduler\n component: k8s-neuron-scheduler\n template:\n metadata:\n labels:\n app: neuron-scheduler\n component: k8s-neuron-scheduler\n annotations:\n scheduler.alpha.kubernetes.io/critical-pod: ''\n spec:\n serviceAccount: k8s-neuron-scheduler\n schedulerName: my-scheduler\n containers:\n - name: neuron-scheduler-exp\n # Find all neuron-scheduler images at https://gallery.ecr.aws/neuron/neuron-scheduler\n image: public.ecr.aws/neuron/neuron-scheduler:2.22.4.0\n env:\n - name: PORT\n value: \"12345\"\n\n# service.yaml \n---\napiVersion: v1\nkind: Service\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\n labels:\n app: neuron-scheduler\n component: k8s-neuron-scheduler\nspec:\n ports:\n - port: 12345\n name: http\n targetPort: 12345\n selector:\n # select app=ingress-nginx pods\n app: neuron-scheduler\n component: k8s-neuron-scheduler \n" }, { "path": "src/k8/k8s-neuron-scheduler.yml", "content": "# rbac.yaml\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: k8s-neuron-scheduler\nrules:\n- apiGroups:\n - \"\"\n resources:\n - nodes\n verbs:\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - events\n verbs:\n - create\n - patch\n- apiGroups:\n - \"\"\n resources:\n - pods\n verbs:\n - update\n - patch\n - get\n - list\n - watch\n- apiGroups:\n - \"\"\n resources:\n - bindings\n - pods/binding\n verbs:\n - create\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: k8s-neuron-scheduler\nsubjects:\n- kind: ServiceAccount\n name: k8s-neuron-scheduler\n namespace: kube-system\n\n# deployment yaml\n---\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\nspec:\n replicas: 1\n strategy:\n type: Recreate\n selector:\n matchLabels:\n app: neuron-scheduler\n component: k8s-neuron-scheduler\n template:\n metadata:\n labels:\n app: neuron-scheduler\n component: k8s-neuron-scheduler\n annotations:\n scheduler.alpha.kubernetes.io/critical-pod: ''\n spec:\n hostNetwork: true\n tolerations:\n - effect: NoSchedule\n operator: Exists\n key: node-role.kubernetes.io/master\n - effect: NoSchedule\n operator: Exists\n key: node.cloudprovider.kubernetes.io/uninitialized\n nodeSelector:\n node-role.kubernetes.io/master: \"\"\n serviceAccount: k8s-neuron-scheduler\n containers:\n - name: neuron-scheduler\n # Find all neuron-scheduler images at https://gallery.ecr.aws/neuron/neuron-scheduler\n image: public.ecr.aws/neuron/neuron-scheduler:2.22.4.0\n env:\n - name: PORT\n value: \"12345\"\n\n# service.yaml \n---\napiVersion: v1\nkind: Service\nmetadata:\n name: k8s-neuron-scheduler\n namespace: kube-system\n labels:\n app: neuron-scheduler\n component: k8s-neuron-scheduler\nspec:\n type: NodePort\n ports:\n - port: 12345\n name: http\n targetPort: 12345\n nodePort: 32700\n selector:\n # select app=ingress-nginx pods\n app: neuron-scheduler\n component: k8s-neuron-scheduler \n" }, { "path": "src/k8/k8s-ultraserver-init-script.sh", "content": "#!/bin/bash\n\nMPI_HOST_FILE=/etc/mpi/hostfile\n\nNEURON_ULTRASERVER_MODE_UNSET=0\nNEURON_ULTRASERVER_MODE_X4=1\nNEURON_ULTRASERVER_MODE_X2H=2\nNEURON_ULTRASERVER_MODE_X2V=3\nNEURON_ULTRASERVER_MODE_X1=4\n\nULTRASERVER_INIT_DIR=/root/ultraserver_init\nSORTED_NODES_FILE=$ULTRASERVER_INIT_DIR/sorted_nodes.txt\nFQDN_MODE_FILE=$ULTRASERVER_INIT_DIR/fqdn_mode.txt\nENV_VARS_FILE=$ULTRASERVER_INIT_DIR/us_env_vars.txt\nNEW_HOST_FILE=$ULTRASERVER_INIT_DIR/new_hostfile\n\nexport NEURON_ULTRASERVER_SERVER_ID_DEFAULT_VALUE=\"0000000000000000\"\nexport NEURON_ULTRASERVER_NODE_ID_DEFAULT_VALUE=-1\n\nexport NEURON_GLOBAL_TOPOID0_HOST=\"\"\n\nexport NUM_WORKERS=0\n\ncat /dev/null > $SORTED_NODES_FILE\ncat /dev/null > $FQDN_MODE_FILE\ncat /dev/null > $ENV_VARS_FILE\ncat /dev/null > $NEW_HOST_FILE\n\nsave_sorted_node_list() {\n # Gather ultraserver information from each worker node\n mpirun --allow-run-as-root \\\n --mca orte_keep_fqdn_hostnames 1 \\\n -host $ip_list \\\n -x NEURON_ULTRASERVER_SERVER_ID_DEFAULT_VALUE \\\n -x NEURON_ULTRASERVER_NODE_ID_DEFAULT_VALUE \\\n -x NEURON_ULTRASERVER_NODE_CONFIG \\\n sh -c '\n if [ -f \"/sys/class/neuron_device/server_id_${NEURON_ULTRASERVER_NODE_CONFIG}\" ]; then\n NEURON_ULTRASERVER_SERVER_ID=$(cat /sys/class/neuron_device/server_id_${NEURON_ULTRASERVER_NODE_CONFIG})\n else\n NEURON_ULTRASERVER_SERVER_ID=$NEURON_ULTRASERVER_SERVER_ID_DEFAULT_VALUE\n fi\n\n if [ -f \"/sys/class/neuron_device/node_id_${NEURON_ULTRASERVER_NODE_CONFIG}\" ]; then\n NEURON_ULTRASERVER_NODE_ID=$(cat /sys/class/neuron_device/node_id_${NEURON_ULTRASERVER_NODE_CONFIG})\n else\n NEURON_ULTRASERVER_NODE_ID=$NEURON_ULTRASERVER_NODE_ID_DEFAULT_VALUE\n fi\n\n FQDN=$(hostname --fqdn)\n echo $NEURON_ULTRASERVER_SERVER_ID:$NEURON_ULTRASERVER_NODE_ID:$FQDN\n ' | sort -t':' -k1,1 -k2,2 -k3,3 > $SORTED_NODES_FILE\n\n # Set the topology ids for each worker node\n local i=0\n while IFS= read -r line; do\n echo \"${i}:${line}\"\n ((i++))\n done < $SORTED_NODES_FILE > temp && mv temp $SORTED_NODES_FILE\n NEURON_GLOBAL_TOPOID0_HOST=$(head -n1 $SORTED_NODES_FILE | cut -d: -f4)\n}\n\nvalidate_node_config() {\n while read -r server_id; do\n # Server id and node id are only valid for node configs > 1\n if [ $NEURON_ULTRASERVER_NODE_CONFIG -ne 1 ]; then\n # Validate server id exists\n if [ \"$server_id\" = \"$NEURON_ULTRASERVER_SERVER_ID_DEFAULT_VALUE\" ]; then\n echo \"$NEURON_ULTRASERVER_NODE_CONFIG-node config is not supported\"\n exit 1\n fi\n\n # Validate there is the correct amount of nodes that share the same server id\n count=$(grep \"$server_id\" \"$SORTED_NODES_FILE\" | wc -l)\n if [ $count -ne $NEURON_ULTRASERVER_NODE_CONFIG ]; then\n echo \"Error: Incorrect number of nodes with server id $server_id, need $NEURON_ULTRASERVER_NODE_CONFIG nodes but saw $count\"\n exit 1\n fi\n\n # Validate all the node ids are unique\n node_ids_count=$(grep \"$server_id\" \"$SORTED_NODES_FILE\" | cut -d':' -f3 | sort | uniq | wc -l)\n if [ $node_ids_count -ne $NEURON_ULTRASERVER_NODE_CONFIG ]; then\n echo \"Error: Found $node_ids_count unique node IDs, expected $NEURON_ULTRASERVER_NODE_CONFIG\"\n exit 1\n fi\n fi\n\n while IFS=':' read -r tid sid nid fqdn; do\n # Validate mode is valid for each node\n modes=\"${fqdn_modes_map[$fqdn]}\"\n if [ $NEURON_ULTRASERVER_NODE_CONFIG -eq 4 ]; then\n if echo \"$modes\" | grep -q \"\\b$NEURON_ULTRASERVER_MODE_X4\\b\"; then\n mode=$NEURON_ULTRASERVER_MODE_X4\n else\n echo \"Error: Node $fqdn does not support 4-node config\"\n exit 1\n fi\n elif [ $NEURON_ULTRASERVER_NODE_CONFIG -eq 2 ]; then\n if echo \"$modes\" | grep -q \"\\b$NEURON_ULTRASERVER_MODE_X2V\\b\"; then\n mode=$NEURON_ULTRASERVER_MODE_X2V\n elif echo \"$modes\" | grep -q \"\\b$NEURON_ULTRASERVER_MODE_X2H\\b\"; then\n mode=$NEURON_ULTRASERVER_MODE_X2H\n else\n echo \"Error: Node $fqdn does not support 2-node config\"\n exit 1\n fi\n else\n mode=$NEURON_ULTRASERVER_MODE_X1\n fi\n\n # Save each worker node's environments variables to a file\n echo \"${tid}:${mode}:${sid}:${nid}:${fqdn}\" >> \"$ENV_VARS_FILE\"\n done < <(grep \"$server_id\" \"$SORTED_NODES_FILE\")\n done < <(cut -d':' -f2 \"$SORTED_NODES_FILE\" | sort | uniq)\n}\n\nreorder_hostfile() {\n # Check if files exist\n if [ ! -f \"$MPI_HOST_FILE\" ] || [ ! -f \"$SORTED_NODES_FILE\" ]; then\n echo \"Error: One or both input files do not exist\"\n exit 1\n fi\n\n # Extract FQDNs from SORTED_NODES_FILE and reorder entries\n while IFS=: read -r _ _ _ fqdn; do\n # Remove .cluster.local suffix\n clean_fqdn=${fqdn%.cluster.local}\n\n # Find the matching line in original file\n while read -r line; do\n if [[ \"$line\" == \"$clean_fqdn\"* ]]; then\n echo \"$line\" >> \"$NEW_HOST_FILE\"\n break\n fi\n done < \"$MPI_HOST_FILE\"\n done < \"$SORTED_NODES_FILE\"\n}\n\n# Validate node config\nif [ -z \"${NEURON_ULTRASERVER_NODE_CONFIG}\" ]; then\n NEURON_ULTRASERVER_NODE_CONFIG=4\nfi\nif [ $NEURON_ULTRASERVER_NODE_CONFIG -ne 1 ] && [ $NEURON_ULTRASERVER_NODE_CONFIG -ne 2 ] && [ $NEURON_ULTRASERVER_NODE_CONFIG -ne 4 ]; then\n echo \"Error: Invalid ultraserver node config: $NEURON_ULTRASERVER_NODE_CONFIG. Must be 1, 2, or 4.\"\n exit 1\nfi\necho \"Using $NEURON_ULTRASERVER_NODE_CONFIG-node config\"\n\necho -e \"\\nCurrent hostfile:\"\ncat $MPI_HOST_FILE\n\n# Read the file, extract the first column, resolve IPs, and build the comma-separated string\nip_list=\"\"\nwhile read line; do\n ip=$(getent hosts \"$line\" | awk '{print $1}')\n if [ -z \"$ip\" ]; then\n echo \"error: Unable to resolve IP address for host: $line\"\n exit 1\n fi\n if [ -z \"$ip_list\" ]; then\n ip_list=\"$ip\"\n else\n ip_list=\"${ip_list},${ip}\"\n fi\ndone < <(cut -d' ' -f1 $MPI_HOST_FILE)\necho \"Worker pod IPs:\" \"$ip_list\"\n\n# Count unique IPs from ip_list and store in NUM_WORKERS\nNUM_WORKERS=$(echo \"$ip_list\" | tr -cd ',' | wc -c)\nNUM_WORKERS=$((NUM_WORKERS + 1))\necho \"Number of worker nodes: $NUM_WORKERS\"\n\n# Validate that the number of workers is a multiple of the node config\nif [ $((NUM_WORKERS % NEURON_ULTRASERVER_NODE_CONFIG)) -ne 0 ]; then\n echo \"Error: Invalid number of worker nodes for $NEURON_ULTRASERVER_NODE_CONFIG-node config: $NUM_WORKERS.\"\n exit 1\nfi\n\n# Create a map of workers to their possible ultraserver modes\nmpirun --allow-run-as-root \\\n --mca orte_keep_fqdn_hostnames 1 \\\n -host $ip_list \\\n sh -c '\n FQDN=$(hostname --fqdn)\n NEURON_ULTRASERVER_MODE=$(cat /sys/class/neuron_device/ultraserver_mode)\n echo $FQDN:$NEURON_ULTRASERVER_MODE\n ' | sort -t':' -k1 > $FQDN_MODE_FILE\ndeclare -A fqdn_modes_map\nwhile IFS=':' read -r fqdn mode; do\n fqdn_modes_map[\"$fqdn\"]=\"$mode\"\ndone < $FQDN_MODE_FILE\n(echo \"FQDN:Modes\" && cat $FQDN_MODE_FILE) | tr ':' ' '\n\n# Validate worker nodes\necho -e \"\\nSorted nodes:\"\nsave_sorted_node_list\n(echo \"TOPO_ID:SERVER_ID:NODE_ID:FQDN\" && cat $SORTED_NODES_FILE) | tr ':' ' '\necho -e \"\\nNEURON_GLOBAL_TOPOID0 node will be: $NEURON_GLOBAL_TOPOID0_HOST\"\nvalidate_node_config\n\n# Update hostlist\necho -e \"\\nUpdated hostfile:\"\nreorder_hostfile\ncat $NEW_HOST_FILE\n\n# Write environment variables to each worker node\nfor line in `cat $ENV_VARS_FILE`; do\n IFS=':' read -r topo_id mode server_id node_id fqdn <<< \"$line\"\n export mode server_id node_id fqdn topo_id\n mpirun --allow-run-as-root \\\n --mca orte_keep_fqdn_hostnames 1 \\\n -host $fqdn \\\n -x topo_id \\\n -x NEURON_GLOBAL_TOPOID0_HOST \\\n -x mode \\\n -x server_id \\\n -x node_id \\\n sh -c '\n sed -i \"/^NEURON_GLOBAL_TOPOID=/d\" /etc/environment\n sed -i \"/^NEURON_GLOBAL_TOPOID0_HOST=/d\" /etc/environment\n sed -i \"/^NEURON_RT_ULTRASERVER_MODE=/d\" /etc/environment\n sed -i \"/^NEURON_RT_ULTRASERVER_SERVER_ID=/d\" /etc/environment\n sed -i \"/^NEURON_RT_ULTRASERVER_NODE_ID=/d\" /etc/environment\n\n echo \"NEURON_GLOBAL_TOPOID=$topo_id\" >> /etc/environment\n echo \"NEURON_GLOBAL_TOPOID0_HOST=$NEURON_GLOBAL_TOPOID0_HOST\" >> /etc/environment\n echo \"NEURON_RT_ULTRASERVER_MODE=$mode\" >> /etc/environment\n echo \"NEURON_RT_ULTRASERVER_SERVER_ID=$server_id\" >> /etc/environment\n echo \"NEURON_RT_ULTRASERVER_NODE_ID=$node_id\" >> /etc/environment\n\n echo \"Node $(hostname --fqdn): Variables set and persisted\"\n echo \"NEURON_GLOBAL_TOPOID=$topo_id\"\n echo \"NEURON_GLOBAL_TOPOID0_HOST=$NEURON_GLOBAL_TOPOID0_HOST\"\n echo \"NEURON_RT_ULTRASERVER_MODE=$mode\"\n echo \"NEURON_RT_ULTRASERVER_SERVER_ID=$server_id\"\n echo \"NEURON_RT_ULTRASERVER_NODE_ID=$node_id\"\n '\ndone\n" }, { "path": "src/k8/my-scheduler.yml", "content": "apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: my-scheduler\n namespace: kube-system\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: my-scheduler-as-kube-scheduler\nsubjects:\n- kind: ServiceAccount\n name: my-scheduler\n namespace: kube-system\nroleRef:\n kind: ClusterRole\n name: system:kube-scheduler\n apiGroup: rbac.authorization.k8s.io\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: my-scheduler-as-volume-scheduler\nsubjects:\n- kind: ServiceAccount\n name: my-scheduler\n namespace: kube-system\nroleRef:\n kind: ClusterRole\n name: system:volume-scheduler\n apiGroup: rbac.authorization.k8s.io\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: my-scheduler\nrules:\n- apiGroups:\n - \"\"\n resources:\n - configmaps\n verbs:\n - get\n - list\n - watch\n- apiGroups:\n - coordination.k8s.io\n resources:\n - leases\n verbs:\n - create\n - get\n - list\n - update\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: my-scheduler\n namespace: kube-system\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: my-scheduler\nsubjects:\n- kind: ServiceAccount\n name: my-scheduler\n namespace: kube-system\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: my-scheduler-config\n namespace: kube-system\ndata:\n my-scheduler-config.yaml: |\n apiVersion: kubescheduler.config.k8s.io/v1\n kind: KubeSchedulerConfiguration\n profiles:\n - schedulerName: my-scheduler\n extenders:\n - urlPrefix: 'http://k8s-neuron-scheduler.kube-system.svc.cluster.local:12345'\n filterVerb: filter\n bindVerb: bind\n enableHTTPS: false\n nodeCacheCapable: true\n managedResources:\n - name: 'aws.amazon.com/neuron'\n ignoredByScheduler: false\n - name: 'aws.amazon.com/neuroncore'\n ignoredByScheduler: false\n - name: 'aws.amazon.com/neurondevice'\n ignoredByScheduler: false\n ignorable: false\n leaderElection:\n leaderElect: true\n resourceNamespace: kube-system \n resourceName: my-scheduler\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n labels:\n component: scheduler\n tier: control-plane\n name: my-scheduler\n namespace: kube-system\nspec:\n selector:\n matchLabels:\n component: scheduler\n tier: control-plane\n replicas: 1\n template:\n metadata:\n labels:\n component: scheduler\n tier: control-plane\n version: second\n spec:\n serviceAccountName: my-scheduler\n containers:\n - args:\n - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml\n - --leader-elect=true\n - --v=2\n command:\n - /usr/local/bin/kube-scheduler\n image: public.ecr.aws/eks-distro/kubernetes/kube-scheduler:v1.28.5-eks-1-28-latest\n # or use below for your version of k8s\n # image: registry.k8s.io/kube-scheduler:\n livenessProbe:\n httpGet:\n path: /healthz\n port: 10259\n scheme: HTTPS\n initialDelaySeconds: 15\n name: kube-second-scheduler\n readinessProbe:\n httpGet:\n path: /healthz\n port: 10259\n scheme: HTTPS\n resources:\n requests:\n cpu: '0.1'\n securityContext:\n privileged: false\n volumeMounts:\n - name: config-volume\n mountPath: /etc/kubernetes/my-scheduler\n hostNetwork: false\n hostPID: false\n volumes:\n - name: config-volume\n configMap:\n name: my-scheduler-config\n\n" }, { "path": "src/k8/neuron-problem-detector/k8s-neuron-problem-detector-and-recovery-config.yml", "content": "apiVersion: v1\ndata:\n kernel-monitor.json: |\n {\n \"plugin\": \"kmsg\",\n \"logPath\": \"/dev/kmsg\",\n \"lookback\": \"5m\",\n \"bufferSize\": 10,\n \"source\": \"kernel-monitor\",\n \"conditions\": [\n {\n \"type\": \"NeuronHealth\",\n \"reason\": \"NeuronHasNoError\",\n \"message\": \"Neuron has no error\"\n }\n ],\n \"rules\": [\n {\n \"type\": \"permanent\",\n \"condition\": \"NeuronHealth\",\n \"reason\": \"NeuronHasError_SRAM_UNCORRECTABLE_ERROR\",\n \"pattern\": \".* NEURON_HW_ERR=SRAM_UNCORRECTABLE_ERROR .*\"\n },\n {\n \"type\": \"permanent\",\n \"condition\": \"NeuronHealth\",\n \"reason\": \"NeuronHasError_NC_UNCORRECTABLE_ERROR\",\n \"pattern\": \".* NEURON_HW_ERR=NC_UNCORRECTABLE_ERROR .*\"\n },\n {\n \"type\": \"permanent\",\n \"condition\": \"NeuronHealth\",\n \"reason\": \"NeuronHasError_HBM_UNCORRECTABLE_ERROR\",\n \"pattern\": \".* NEURON_HW_ERR=HBM_UNCORRECTABLE_ERROR .*\"\n },\n {\n \"type\": \"permanent\",\n \"condition\": \"NeuronHealth\",\n \"reason\": \"NeuronHasError_DMA_ERROR\",\n \"pattern\": \".* NEURON_HW_ERR=DMA_ERROR .*\"\n }\n ]\n }\nkind: ConfigMap\nmetadata:\n name: node-problem-detector-config\n namespace: neuron-healthcheck-system\n\n" }, { "path": "src/k8/neuron-problem-detector/k8s-neuron-problem-detector-and-recovery-rbac.yml", "content": "apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: node-problem-detector\n namespace: neuron-healthcheck-system\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: npd-binding\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: system:node-problem-detector\nsubjects:\n - kind: ServiceAccount\n name: node-problem-detector\n namespace: neuron-healthcheck-system\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n labels:\n kubernetes.io/bootstrapping: rbac-defaults\n name: system:node-problem-detector\nrules:\n- apiGroups:\n - \"\"\n resources:\n - nodes\n verbs:\n - get\n- apiGroups:\n - \"\"\n resources:\n - nodes/status\n verbs:\n - patch\n- apiGroups:\n - \"\"\n - events.k8s.io\n resources:\n - events\n verbs:\n - create\n - patch\n - update\n" }, { "path": "src/k8/neuron-problem-detector/k8s-neuron-problem-detector-and-recovery.yml", "content": "apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: node-problem-detector\n namespace: neuron-healthcheck-system\n labels:\n app: node-problem-detector\nspec:\n selector:\n matchLabels:\n app: node-problem-detector\n template:\n metadata:\n labels:\n app: node-problem-detector\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n - key: \"node.kubernetes.io/instance-type\"\n operator: In\n values:\n - inf1.xlarge\n - inf1.2xlarge\n - inf1.6xlarge\n - inf1.24xlarge\n - inf2.xlarge\n - inf2.8xlarge\n - inf2.24xlarge\n - inf2.48xlarge\n - trn1.2xlarge\n - trn1.32xlarge\n - trn1n.32xlarge\n\n containers:\n - name: node-problem-detector\n command:\n - /node-problem-detector\n - --logtostderr\n - --config.system-log-monitor=/config/kernel-monitor.json\n image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.19\n ports:\n - containerPort: 20257\n resources:\n limits:\n cpu: 10m\n memory: 80Mi\n requests:\n cpu: 10m\n memory: 80Mi\n imagePullPolicy: Always\n securityContext:\n privileged: true\n env:\n - name: NODE_NAME\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n volumeMounts:\n - name: log\n mountPath: /var/log\n readOnly: true\n - name: kmsg\n mountPath: /dev/kmsg\n readOnly: true\n # Make sure node problem detector is in the same timezone\n # with the host.\n - name: localtime\n mountPath: /etc/localtime\n readOnly: true\n - name: config\n mountPath: /config\n readOnly: true\n - name: node-recovery\n command:\n - /bin/sh\n - -c\n - \"sleep 60 && /scripts/check-health.py\"\n image: public.ecr.aws/neuron/neuron-node-recovery:1.3.0\n resources:\n limits:\n cpu: 10m\n memory: 150Mi\n requests:\n cpu: 10m\n memory: 150Mi\n imagePullPolicy: Always\n env:\n - name: NODE_NAME\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n - name: ENABLE_RECOVERY\n value: \"false\"\n serviceAccountName: node-problem-detector\n volumes:\n - name: log\n # Config `log` to your system log directory\n hostPath:\n path: /var/log/\n - name: kmsg\n hostPath:\n path: /dev/kmsg\n - name: localtime\n hostPath:\n path: /etc/localtime\n - name: config\n configMap:\n name: node-problem-detector-config\n defaultMode: 0555\n items:\n - key: kernel-monitor.json\n path: kernel-monitor.json\n tolerations:\n - effect: NoSchedule\n operator: Exists\n - effect: NoExecute\n operator: Exists\n" }, { "path": "src/libnrt/README.md", "content": "# NeuronX Runtime Header Files\n\n## Overview\n\nThe NeuronX Runtime Library provides C APIs for initializing the Neuron hardware,\nloading models and input data, executing iterations on loaded models, and\nretrieving output data.\n\nThis library is provided to customers via a shared object (libnrt.so) that is installed\nthrough the `aws-neuronx-runtime-lib` package. This directory exposes the header files\nthat customers can use to write custom applications utilizing the NeuronX Runtime Library.\n\n## File Location\n\nThese header files will be installed to the user's system under `/opt/aws/neuron/include`\nwhen installing the `aws-neuronx-runtime-lib` package and the `libnrt.so` library is \ninstalled under the `/opt/aws/neuron/lib` directory.\n\n## Experimental Headers\n\nThe following files contain experimental function declarations and are subject to change in \nfuture releases.\n\n- nrt_async.h\n- nrt_async_sendrecv.h\n- nrt_experimental.h\n\n## Documentation\n\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-api-guide.html\n" }, { "path": "src/libnrt/include/ndl/ndl.h", "content": "/*\n * Copyright 2020-2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \n#include \n#include \n#include \n\n#include \"neuron_driver_shared.h\"\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\ntypedef enum NQ_DEV_TYPE {\n NQ_DEV_TYPE_NEURON_CORE = 0,\n NQ_DEV_TYPE_TOPSP,\n NQ_DEV_TYPE_MAX,\n} ndl_nq_dev_t;\n\n#define NEURON_MAX_DEVICES MAX_NEURON_DEVICE_COUNT\n#define NEURON_DEVICE_PREFIX \"/dev/neuron\"\n#define NEURON_DRIVER_LIBRARY_MAJOR 1\n#define NEURON_DRIVER_LIB_MINOR 0\n#define MAX_HBM_PER_DEVICE 4\n\n#define DRIVER_VERSION_MAX_SIZE 32\ntypedef struct ndl_version_info {\n uint16_t driver_major_version; // Major version of the driver\n uint16_t driver_minor_version; // Minor version of the driver\n char driver_full_version[DRIVER_VERSION_MAX_SIZE];\n uint16_t library_major_version; // Major version of the library\n uint16_t library_minor_version; // Minor version of the library\n} ndl_version_info_t;\n\n/** Get version info.\n *\n * @param[out] version - Buffer to store the version information.\n *\n * @return 0 on success.\n * -1 on failed to read driver version.\n */\nint ndl_get_version(ndl_version_info_t *version);\n\n/** Gets the range of compatible version\n *\n * @param min_compatible_version_min [out] - Lowest supported version\n * @param max_compatible_version_max [out] - Highest supported version\n *\n * @return 0 on success.\n *\n */\nint ndl_get_compatible_version(uint32_t *min_compatible_version, uint32_t *max_compatible_version);\n\ntypedef struct ndl_device_init_param {\n bool initialize_device; // if set to true, device is initialized as part of open()\n int num_dram_regions; // splits device DRAMs into given number of regions.\n bool map_hbm; // if set to true, HBM will be mapped during device open\n} ndl_device_init_param_t;\n\n\n#define NDL_COPY_BUF_SIZE (2ull * 1024 * 1024)\ntypedef struct ndl_copy_buf {\n uint64_t mem_handle;\n void *mmap_va;\n pthread_mutex_t lock;\n} ndl_copy_buf_t;\n\n// Maximum neuron devices supported on a system.\n#define MAX_NEURON_DEVICE_COUNT 64\n\n// Maximum neuron cores per device\n#define MAX_NC_PER_DEVICE 8\n\ntypedef struct ndl_device {\n uint8_t device_index; // Device Index\n uint8_t device_type; // Device Type (V1, V2..)\n uint16_t device_revision; // Revision id of board\n uint8_t connected_device_count; // Number of devices connected to this device\n uint8_t connected_devices[MAX_NEURON_DEVICE_COUNT]; // Array of devices(IDs) connected to this device\n uint64_t csr_base[2]; // BAR0/BAR2 base\n uint64_t csr_size[2]; // BAR0/BAR2 size\n ndl_copy_buf_t cpy_bufs[MAX_NC_PER_DEVICE]; // MMAP buffers for efficiently copying data in/out of the device\n void *hbm_va[MAX_HBM_PER_DEVICE]; // HBM virtual addresses\n size_t hbm_size; // HBM sizes\n uint32_t hbm_va_cnt; // Number of active HBM regions\n uint32_t shift_hbm_size; // Cached number of bits to shift\n uint64_t hbm_offset[MAX_HBM_PER_DEVICE]; // HBM offsets\n uint8_t context[]; // Library reserved fields\n} ndl_device_t;\n\ntypedef struct ndl_device_nc {\n ndl_device_t *device;\n uint32_t nc_id;\n} ndl_device_nc_t;\n\ntypedef struct ndl_device_context {\n int nd_fd;\n} ndl_device_context_t;\n\ntypedef struct ndl_mem_info {\n ndl_device_t *device;\n __u64 driver_handle;\n uint64_t pa;\n uint64_t mmap_offset;\n uint64_t size;\n uint32_t align;\n void *mmap_va;\n uint32_t host_memory;\n int nc_id;\n} ndl_mem_info_t;\n\ntypedef struct ndl_notification_context {\n union {\n uint8_t nc_id; // neuron core index\n uint8_t nq_dev_id; // notification device index\n };\n ndl_nq_dev_t nq_dev_type; // notification device type\n uint8_t nq_type; // type of the notification queue\n uint8_t engine_index; // engine index\n uint32_t size; // size of the NQ in bytes\n int fd; // file descriptor of /dev/ndX/ncY/nqZ\n uint64_t offset; //mmap offset in the nd\n uint64_t mem_handle;\n void *va; // mmapped address\n\n ndl_mem_info_t *mem_info; // NQ memory info\n} ndl_notification_context_t;\n\n/**\n * Called by app the first time when it accesses the device.\n *\n * @param[in] device_index - device index that is to be opened\n * @param[in] num_tdram_regions - number of tdram regions\n * @param[out] device - device specific information\n *\n * @return 0 on success.\n * -1 on failure\n */\nint ndl_open_device(int device_index, ndl_device_init_param_t *params, ndl_device_t **device);\n\n/**\n * Called by app when it is done. After this, device cannot be accessed\n *\n * @param[in] device - Device to close.\n *\n * @return 0 on success.\n * -1 on failure\n */\nint ndl_close_device(ndl_device_t *device);\n\n/**\n * Get all the device index\n *\n * @param[out] device_indexes - Buffer to store device indexes.\n * @param[in] device_indexes_size - Size of the buffer in dwords.\n *\n * @return Number of devices found.\n */\nint ndl_available_devices(int *device_indexes, int device_indexes_size);\n\n/** Read from one or more registers.\n *\n * @param device[in] - Device handle.\n * @param bar[in] - BAR to read.\n * @param addresses[in] - Array of register addresses.\n * @param count[in] - Number of registers in the array.\n * @param buffer[out] - Buffer to store read data.\n *\n * @return 0 on success.\n */\nint ndl_bar_read(ndl_device_t *device, uint8_t bar, uint64_t *addresses, uint32_t count, uint32_t *buffer);\n\n/** Write to one or more registers.\n *\n * @param device[in] - Device handle.\n * @param bar[in] - BAR to write.\n * @param addresses[in] - Array of register addresses.\n * @param count[in] - Number of registers in the array.\n * @param data[in] - Data to write.\n *\n * @return 0 on success.\n */\nint ndl_bar_write(ndl_device_t *device, uint8_t bar, uint64_t *addresses, uint32_t count, uint32_t *data);\n\n/** Read hw counters from one or more addresses\n *\n * @param device[in] - Device handle.\n * @param addresses[in] - Array of register addresses.\n * @param count[in] - Number of registers in the array.\n * @param buffer[out] - Buffer to store read data.\n *\n * @return 0 on success.\n */\nint ndl_read_hw_counters(ndl_device_t *device, uint64_t *addresses, uint32_t count, uint32_t *data);\n\n/**\n * Retrieves the cached HBM virtual address for the specified device.\n *\n * @param device[in] - Device handle.\n * @param hbm_idx[in] - HBM index.\n * @param va[out] - Resulting virtual address.\n * @param size[out] - Size of the HBM\n * \n * @return 0 on success, -EINVAL on failure, and -ENOENT when there are no more entries to be found.\n */\nint ndl_get_hbm_va(ndl_device_t *device, int hbm_idx, void **va, size_t *size);\n\n/** Allocates memory.\n *\n * @param device[in] - Device to be associated with the allocation.\n * @param size[in] - Number of bytes to allocate.\n * @param host_memory[in] - If true allocate from host memory instead of using device memory.\n * @param dram_channel[in] - DRAM channel to use in the device memory.\n * @param dram_region[in] - DRAM region to use in the device memory.\n * @param nc_id[in] - NC ID to use in the device\n * @param mem_alloc_type[in]- Type of memory allocation \n * @param mem_handle[out] - Allocated memory handle would be stored here.\n *\n * @return 0 on success.\n */\nint ndl_memory_alloc(ndl_device_t *device, size_t size, uint64_t align, uint32_t host_memory, uint32_t dram_channel, uint32_t dram_region,\n uint32_t nc_id, uint32_t mem_alloc_type, uint64_t *mem_handle);\n\n/** Given a mem handle gets it PA - HACK to be removed\n * @param mem_handle[in] - Memory handle\n * @parama pa[out] - Physical address of handle\n *\n * @return the PA\n */\nint ndl_memory_get_pa(uint64_t mem_handle, uint64_t *pa);\n\n/** Map given m memory handle into virtual address space.\n *\n * @param mem_handle[in] - Handle to map.\n * @param va[out] - Resulting virtual address.\n *\n * @return 0 on success\n */\nint ndl_memory_map(uint64_t mem_handle, void **va);\n\n/** Unmap given memory handle from virtual address space.\n *\n * @param mem_handle[in] - Handle to unmap.\n *\n * @return 0 on success\n */\nint ndl_memory_unmap(uint64_t mem_handle);\n\n/** Frees already allocated memory.\n *\n * @param mem_handle[in] - Memory handle to be freed.\n *\n * @return 0 on success.\n */\nint ndl_memory_free(uint64_t mem_handle);\n\n/** Copy data from buffer to mem_handle.\n *\n * @param mem_handle[in] - Handle on which data needs to be copied in.\n * @param buffer - Buffer from which data needs to be copied.\n * @param offset - Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n *\n * @return 0 on success.\n */\nint ndl_memory_buf_copyin(uint64_t mem_handle, void *buffer, uint64_t offset, size_t size);\n\n/** Copy data from mem_handle to buffer.\n *\n * @param mem_handle[in] - Handle from which data needs to be copied out.\n * @param buffer - Buffer to which data needs to be copied.\n * @param offset - Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n *\n * @return 0 on success.\n */\nint ndl_memory_buf_copyout(uint64_t mem_handle, void *buffer, uint64_t offset, size_t size);\n\n/** Copy data from buffer to mem_handle (zero copy, buffer is pinned and used directly).\n *\n * @param mem_handle[in] - Handle on which data needs to be copied in.\n * @param buffer - Buffer from which data needs to be copied.\n * @param offset - Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n *\n * @return 0 on success.\n */\nint ndl_memory_buf_zerocopyin(uint64_t mem_handle, void *buffer, uint64_t offset, size_t size, int qid, uint32_t bar4_wr_threshold);\n\n/** Copy data from mem_handle to buffer (zero copy, buffer is pinned and used directly).\n *\n * @param mem_handle[in] - Handle from which data needs to be copied out.\n * @param buffer - Buffer to which data needs to be copied.\n * @param offset - Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n * @param qid - H2T queue to use. NEURON_DMA_H2T_DEFAULT_QID is default\n *\n * @return 0 on success.\n */\nint ndl_memory_buf_zerocopyout(uint64_t mem_handle, void *buffer, uint64_t offset, size_t size, int qid);\n\n/** Batch transfer data between host buffers and device memory.\n *\n * @param mem_handle[in] - Device memory handle\n * @param ops[in] - Array of batch operations\n * @param num_ops[in] - Number of operations in batch\n * @param direction[in] - Transfer direction (0=write to device, 1=read from device)\n * @param qid[in] - H2T queue to use (-1 for default)\n *\n * @return 0 on success.\n */\nint ndl_memory_buf_batch_copy(neuron_memcpy_batch_t *batches, uint64_t num_batches, uint32_t direction, int qid);\n\n/** Copy data from buffer to addr in engine.\n *\n * @param device[in] - Device information.\n * @param nc_id [in] - Neuron core id.\n * @param dst [in] - Address on which data needs to be copied in.\n * @param buffer - Buffer from which data needs to be copied.\n * @param offset - Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n * @param qid - H2T queue to use. NEURON_DMA_H2T_DEFAULT_QID is default\n *\n * @return 0 on success.\n */\nint ndl_program_engine(ndl_device_t *device, uint32_t nc_id, uint64_t dst, void *buffer, uint64_t offset, size_t size);\n\n/** Memset the given memhandle with passed byte value\n *\n * @param src_mem_handle[in]- Handle which needs to be filled with byte value\n * @param offset - Src Offset in the mem handle.\n * @param value - Byte value to fill the memory with\n * @param size - Size in bytes to be copied.\n *\n * @return 0 on success.\n */\nint ndl_memset(const uint64_t addr, uint64_t offset, const int value, const size_t size);\n\n/** Copy data between mem_handles.\n *\n * @param src_mem_handle[in]- Handle from which data needs to be copied out.\n * @param dst_mem_handle[in]- Handle from which data needs to be copied to.\n * @param src_offset - Src Offset in the mem handle.\n * @param dst_offset - Dest Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n *\n * @return 0 on success.\n */\nint ndl_memory_copy(uint64_t src_mem_handle, uint64_t dst_mem_handle, uint64_t src_offset, uint64_t dst_offset,\n size_t size);\n\n\n/** Copy data between mem_handles asynchronously.\n *\n * @param src_mem_handle[in] - Handle from which data needs to be copied out.\n * @param dst_mem_handle[in] - Handle from which data needs to be copied to.\n * @param src_offset - Src Offset in the mem handle.\n * @param dst_offset - Dest Offset in the mem handle.\n * @param size - Size in bytes to be copied.\n * @param prefetch_addr - Host destination address associate with copy out operation to prefetch\n * @param wait_handle [in/out] - wait_handle [in] is for prev request, [out] is handle for this request\n *\n * @return 0 on success.\n */\nint ndl_memory_copy_as(uint64_t src_mem_handle, uint64_t dst_mem_handle, uint64_t src_offset, uint64_t dst_offset, \n size_t size, uint64_t prefetch_addr, int * wait_handle);\n\n\n/** Copy data between mem_handles.\n *\n * @param mem_handle[in] - Handle from which data for this tran (either src or dst)\n * @param wait_handle - wait_handle for an async dma\n *\n * @return 0 on success.\n */\nint ndl_memory_copy_as_wait(uint64_t mem_handle, int wait_handle);\n\n/** Set the dma engine state\n *\n * @param device_index[in] - Device index.\n * @param eng_id[in] - Eng ID that is initialized.\n * @param state[in] - State that is set UDMA_NORMAL/UDMA_DISABLE etc\n *\n * @return 0 on success.\n */\nint ndl_dma_eng_set_state(int device_index, uint32_t eng_id, uint32_t state);\n\n/** Get the dma engine state\n *\n * @param device_index[in] - Device index.\n * @param eng_id[in] - Engine index which status needs to be collected.\n * @param state[out] - Buffer to store engine state.\n *\n * @return 0 on success.\n */\nint ndl_dma_eng_get_state(int device_index, uint32_t eng_id, struct neuron_dma_eng_state *state);\n\n/** Get DMA queue state\n *\n * @param device_index[in] - Device index.\n * @param eng_id [in] - DMA engine index.\n * @param qid [in] - DMA queue index.\n * @param tx [out] - Tx queue state.\n * @param rx [out] - Rx queue state.\n *\n * @return 0 on success.\n */\nint ndl_dma_queue_get_state(int device_index, uint8_t eng_id, uint8_t qid, struct neuron_dma_queue_state *tx, struct neuron_dma_queue_state *rx);\n\n/** Copy DMA descriptors to userspace.\n *\n * This API needs root privilege.\n *\n * @param device_index[in] - Device index.\n * @param eng_id [in] - DMA engine index.\n * @param qid [in] - DMA queue index.\n * @param type [in] - Type of the queue.\n * @param index [in] - Start descriptor index.\n * @param count [in] - Number of descriptor needs to be copied.\n * @param buffer [out] - Buffer to store the descriptors.\n *\n * @return 0 on success.\n */\nint ndl_dma_descriptor_copyout(int device_index, uint8_t eng_id, uint8_t qid, enum neuron_dma_queue_type type, uint32_t start_index, uint32_t count, void *buffer);\n\n/** Initialize the dma queue for a given engine\n *\n * @param device_index[in] - Device index\n * @param eng_id[in] - Engine for which the queue is initialized\n * @param qid[in] - Queue id that needs to be initialized\n * @param tx_desc_count[in] - number of tx desc's need to be allocated\n * @param rx_desc_count[in] - number of rx desc's need to be allocated\n * @param tx_handle[in] - TX mem handle\n * @param rx_handle[in] - RX mem handle\n * @param rxc_handle[in] - Completion mem handle\n *\n * @return 0 on success.\n */\nint ndl_dma_queue_init(int device_index, uint32_t eng_id, uint32_t qid, uint32_t tx_desc_count, uint32_t rx_desc_count,\n uint64_t tx_handle, uint64_t rx_handle, uint64_t rxc_handle, uint32_t axi_port);\n\nstruct ndl_queue_init {\n __u32 eng_id; // [in] DMA engine index\n __u32 qid; // [in] Queue index in the DMA engine\n __u32 tx_desc_count; // [in] number of tx desc's need to be allocated\n __u32 rx_desc_count; // [in] number of rx desc's need to be allocated\n __u64 tx_handle; // [in] mem handle for the tx ring\n __u64 rx_handle; // [in] mem handle for the rx ring\n __u64 rxc_handle; // [in] mem handle for the rxc ring\n __u32 axi_port; // [in] axi port\n};\n\n#define MAX_NDL_QUEUE_INIT_BATCH 256\nstruct ndl_queue_init_batch {\n __u32 count;\n struct ndl_queue_init entries[MAX_NDL_QUEUE_INIT_BATCH];\n};\n\n/** Initialize a batch of dma queues\n *\n * @param device_index[in] - Device index\n * @param batch[in] - Batch of dma queue initialization requests\n *\n * @return 0 on success.\n */\nint ndl_dma_queue_init_batch(int device_idx, struct ndl_queue_init_batch *batch);\n\n/** Release the dma queue for a given engine - only used in tests\n *\n * @param device_index[in] - Device index\n * @param eng_id[in] - Engine for which the queue is initialized\n * @param qid[in] - Queue id that needs to be initialized\n *\n * @return 0 on success.\n */\nint ndl_dma_queue_release(int device_index, uint32_t eng_id, uint32_t qid);\n\n/** Starts DMA by copying the given number of descriptors or prefetch s2m\n *\n * @param device_index[in] - Device index\n * @param eng_id[in] - Engine for which the queue is initialized\n * @param qid[in] - Queue id that needs to be initialized\n * @param tx_desc_count[in] - number of tx desc's need to be copied, could be 0 if called for s2m prefetch\n * @param rx_desc_count[in] - number of rx desc's need to be copied\n *\n * @return 0 on success.\n */\nint ndl_dma_queue_copy_start(int device_index, uint32_t eng_id, uint32_t qid, uint32_t tx_desc_count, uint32_t rx_desc_count);\n\n/** Acks the completed desc count for the eng/queue - only used in tests\n *\n * @param device_index[in] - Device index\n * @param eng_id[in] - Engine for which the queue is initialized\n * @param qid[in] - Queue id that needs to be initialized\n * @param count[in] - Number of desc's to ack\n *\n * @return 0 on success.\n */\nint ndl_dma_ack_completed_desc(int device_index, uint32_t eng_id, uint32_t qid, uint32_t count);\n\n/** Copy data from buffer to mem_handle. Buffer has dma desc\n *\n * @param mem_handle[in] - Handle on which data needs to be copied in.\n * @param buffer[in] - Buffer from which data needs to be copied. Buffer has dma desc\n * @param offset[in] - Offset in the mem handle.\n * @param num_descs[in] - Number of descriptors to copy\n * @param queue_type[in] - From which queue copy descriptors.\n *\n * @return 0 on success.\n */\nint ndl_dma_copy_descriptors(uint64_t mem_handle, void *buffer, uint64_t offset, uint32_t num_descs, enum neuron_dma_queue_type queue_type);\n\n/** Reset given NCs within a device.\n *\n * @param device_index[in] - Device to reset.\n * @param nc_map[in] - NCs to reset (-1 to reset entire device)\n * @param request_id[out] - ID for this reset request\n *\n * @return 0 on success.\n */\nint ndl_reset_ncs(int device_index, int nc_map, uint32_t *request_id);\n\n/** Register the callback to NRT to warn/nudge users when hitting soft incompatibility\n *\n * @param callback - the call back function\n * @return int - 0 on success, otherwise on failure\n */\nint ndl_register_soft_incompat_callback(void (*callback)(const char *));\n\n/** Waits for readiness of given NCs within a device.\n *\n * @param device_index[in] - Device index.\n * @param request_id[in] - ID for the reset request to wait on\n * @param result[out] - Buffer to store the result.\n * If the device is ready then this would be set to 1.\n *\n * @return 0 on success.\n *\n */\nint ndl_ready_ncs(int device_index, uint32_t request_id, uint8_t *result);\n\n/** Get info on all the apps that are currently using the device, caller needs to free returned info (*info)\n *\n * @param device_index[in] - Device index.\n * @param info[out] - Pointer to a pointer which will hold app data, needs to be deallocated by caller\n * @param size[out] - Number of entries in neuron_app_info\n *\n * @return 0 - on success\n */\nint ndl_get_all_apps_info(ndl_device_t *device, struct neuron_app_info **info, size_t *count, uint16_t apps_info_flags);\n\n/** Increment a semaphore in Neuron Core.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron Core index\n * @param semaphore_index[in] - Semaphore which needs to be incremented.\n * @param value[in] - Value to decrement.\n *\n * @return 0 on success\n */\nint ndl_nc_semaphore_increment(ndl_device_t *device, int nc_index, uint32_t semaphore_index, uint32_t value);\n\n/** Decrement a semaphore in Neuron Core.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron Core index\n * @param semaphore_index[in] - Semaphore which needs to be decremented.\n * @param value[in] - Value to increment.\n *\n * @return 0 on success\n */\nint ndl_nc_semaphore_decrement(ndl_device_t *device, int nc_index, uint32_t semaphore_index, uint32_t value);\n\n/** Get semaphore value in Neuron Core.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron Core index\n * @param semaphore_index[in] - Semaphore index.\n * @param value[out] - Buffer where read value would be stored.\n *\n * @return 0 on success\n */\nint ndl_nc_semaphore_read(ndl_device_t *device, int nc_index, uint32_t semaphore_index, uint32_t *value);\n\n\n/** Write given value into the semaphore in Neuron Core.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron Core index\n * @param semaphore_index[in] - Semaphore index.\n * @param value[in] - Value to write.\n *\n * @return 0 on success\n */\nint ndl_nc_semaphore_write(ndl_device_t *device, int nc_index, uint32_t semaphore_index, uint32_t value);\n\n\n/** Get event value in Neuron Core.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron Core index\n * @param semaphore_index[in] - Semaphore index.\n * @param value[out] - Buffer where read value would be stored.\n *\n * @return 0 on success\n */\nint ndl_nc_event_get(ndl_device_t *device, int nc_index, uint32_t event_index, uint32_t *value);\n\n\n/** Set a event in Neuron Core.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron Core index\n * @param event_index[in] - Event index.\n * @param value[in] - Value to write.\n *\n * @return 0 on success\n */\nint ndl_nc_event_set(ndl_device_t *device, int nc_index, uint32_t event_index, uint32_t value);\n\n\n/** Configure notification queue\n *\n * Neuron device has multiple of neuron cores and TOP_SPs. If nq_dev_type is\n * NQ_DEV_TYPE_NEURON_CORE, nq_dev_index conveys neuron core index. In case of\n * NQ_DEV_TYPE_NEURON_TOPSP, nq_dev_index means TOP_SP index.\n *\n * @param device[in] - Device\n * @param nq_dev_id[in] - Notification device index\n * @param nq_dev_type[in] - Notification device type\n * @param nq_type[in] - Notification queue type\n * @param engine_index[in] - Engine index\n * @param size[in] - Size in bytes\n * @param on_host_memory[in] - If true, NQ is created on host memory\n * @param dram_channel - If NQ is created on device, DRAM channel to use\n * @param dram_region - If NQ is created on device, DRAM region to use\n * @param force_alloc_mem - If true, force allocate new memory (and delete already allocated memory, if any)\n * @param context[out] - Resulting NQ context.\n *\n * @return 0 on success.\n */\nint ndl_notification_init(ndl_device_t *device, int nq_dev_id, ndl_nq_dev_t nq_dev_type, uint8_t nq_type, uint8_t engine_index,\n uint32_t size, bool on_host_memory, uint32_t dram_channel, uint32_t dram_region,\n uint64_t *notification_context);\n\n/** Configure notification queue with option to force re-allocate/re-size\n *\n * Neuron device has multiple of neuron cores and TOP_SPs. If nq_dev_type is\n * NQ_DEV_TYPE_NEURON_CORE, nq_dev_index conveys neuron core index. In case of\n * NQ_DEV_TYPE_NEURON_TOPSP, nq_dev_index means TOP_SP index.\n *\n * @param device[in] - Device\n * @param nq_dev_id[in] - Notification device index\n * @param nq_dev_type[in] - Notification device type\n * @param nq_type[in] - Notification queue type\n * @param engine_index[in] - Engine index\n * @param size[in] - Size in bytes\n * @param on_host_memory[in] - If true, NQ is created on host memory\n * @param dram_channel - If NQ is created on device, DRAM channel to use\n * @param dram_region - If NQ is created on device, DRAM region to use\n * @param force_alloc_mem - If true, force allocate new memory (and delete already allocated memory, if any)\n * @param context[out] - Resulting NQ context.\n *\n * @return 0 on success.\n */\nint ndl_notification_init_with_realloc(ndl_device_t *device, int nq_dev_id, ndl_nq_dev_t nq_dev_type, uint8_t nq_type, uint8_t engine_index,\n uint32_t size, bool on_host_memory, uint32_t dram_channel, uint32_t dram_region, bool force_alloc_mem,\n uint64_t *notification_context);\n\n/** Returns mem_handle associated with the NQ\n *\n * @param notification_context[in] - Notification context\n * @param mem_handle[out] - Notification's memory handle would be stored here.\n *\n * @return 0 on success, 1 on failure\n */\nint ndl_notification_get_mem_handle(uint64_t notification_context, uint64_t *mem_handle);\n\n/** Returns size associated with the NQ\n *\n * @param notification_context[in] - Notification context\n * @param size[out] - Notification's size would be stored here.\n *\n * @return 0 on success, 1 on failure\n */\n\nint ndl_notification_get_size(uint64_t notification_context, uint32_t *size);\n\n/** Maps NQ to virtual address.\n *\n * @param notification_context[in] - Notification context.\n * @param va [out] - Virtual address where the mapping is done.\n * @return 0 on success\n */\nint ndl_notification_map(uint64_t notification_context, void **va);\n\n/** Stops and destroys already configured notification queue.\n *\n * @param notification_context[in] - Notification context.\n *\n * @return 0 on success.\n */\nint ndl_notification_destroy(uint64_t notification_context);\n\n/** Makes neuron ds available for use and returns a valid pointer in **data and a valid size in *size\n *\n * @param device[in] - Device\n * @param pid[in] - PID for this NDS (if 0 it allocates a new one)\n * @param data[out] - Will contain a valid pointer to the datastore\n * @param size[out] - Will contain a valid size for the datastore\n *\n * @return 0 on success.\n */\nint ndl_nds_open(ndl_device_t *device, int32_t pid, void **data, size_t *size);\n\n/** Decreases ref count for the given pid\n *\n * @param device - Device\n * @param pid - PID owning the datastore\n * @param data - Pointer to datastore raw data (returned by ndl_nds_open)\n * @param size - Size of datastore (returned by ndl_nds_open)\n *\n * @return 0 on success.\n */\nint ndl_nds_close(ndl_device_t *device, int32_t pid, void *data, size_t size);\n\n/** Enter inference critical section.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron core index\n * @param uuid[in] - UUID of the model expected to be loaded\n *\n * This function would fail if the UUID is different or PID\n * which loaded the UUID is different.\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_crwl_reader_enter(ndl_device_t *device, int nc_index, struct neuron_uuid uuid);\n\n/** Exit inference critical section.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron core index\n * @param uuid[in] - UUID of the model expected to be loaded\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_crwl_reader_exit(ndl_device_t *device, int nc_index, struct neuron_uuid uuid);\n\n/** Enter model load critical section.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron core index\n * @param uuid[in] - UUID of the model to be loaded\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_crwl_writer_enter(ndl_device_t *device, int nc_index, struct neuron_uuid uuid);\n\n/** Exit model load critical section and enter inference critical section.\n *\n * @param device[in] - Device\n * @param nc_index[in] - Neuron core index\n * @param uuid[in] - UUID of the loaded model\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_crwl_writer_downgrade(ndl_device_t *device, int nc_index, struct neuron_uuid uuid);\n\n/** Find given number of free NCs and mark them as used.\n *\n * @param nc_count[in] - Number of free neuron cores needed.\n * @param start_nc[in] - From where to start the free core search.\n * @param end_nc[in] - Last NC where to stop the free core search.\n * @param max_nc_available[out] - Maximum number of free cores available.\n * @param bitmap[out] - Bitmap of marked neuron core indexes.\n * @param size[in] - size of the bitmap in bytes\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_crwl_nc_range_mark(uint32_t nc_count, uint32_t start_nc, uint32_t end_nc,\n uint32_t *max_nc_available, uint64_t *bitmap, size_t size);\n\n/** Unmark neuron cores as free.\n *\n * @param bitmap[in] - Bitmap of marked neuron core indexes.\n * @param size[in] - size of the bitmap in bytes\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_crwl_nc_range_unmark(uint64_t *bitmap, size_t size);\n\n/** Gets the info for the copy buffer for copying data to/from device\n *\n * To dma data in and out of the device, app needs a host dram buffer allocated\n * by the driver. Allocating this every-time is expensive especially if we want\n * a bigger copy size. To avoid this performance penalty, applications can use\n * this preallocated buffer.\n *\n * @param device[in] - Device\n * @param nc_id[in] - nc id the copy buffer is from\n * @param cpy_buf[out] - Pointer to copy buffer\n *\n * @return 0 on success\n */\nint ndl_get_copy_buf(ndl_device_t *device, uint32_t nc_id, ndl_copy_buf_t **cpy_buf);\n\n/** Set the neuron core init state\n * Initially the state is set to started and then app intializes the neuron core. Then\n * it sets the state to completed. If any other app tries to set the state to started when it\n * is already started then this routine will block until the init is done or timeout\n *\n * @param device[in] - Device\n * @param state[in] - State that will be state\n * @param new_state[out] - State after the set is done\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_nc_init_set_state(ndl_device_t *device, uint32_t nc_id, uint32_t state, uint32_t *new_state);\n\n/** Gets the state of model start. If this is the first model that will be loaded in the nc.\n *\n * @param device[in] - Device\n * @param nc_id[in] - nc id\n * @param started_count[out] - number of times model started in that nc\n *\n * @return 0 on success, -1 on failure.\n */\nint ndl_nc_model_started_count(ndl_device_t *device, uint32_t nc_id, uint64_t *started_count);\n\n/** Gets the architecture & revision of the board\n *\n * @param architecture[out] - Architecture of the board\n * @param revision[out] - Revision of the board\n *\n * @return 0 on success\n */\nint ndl_get_board_info(uint32_t *architecture, uint32_t *revision);\n\n/** Gets BDF for a device - only for devices opened by the calling process - DEPRECATED don't use\n *\n * @param bus_num[out] - Bus number for this device\n * @param pci_slot[out] - PCI slot for this device\n * @param dev_func[out] - Device function for this device\n *\n * @return 0 on success\n */\nint ndl_get_device_bdf(int device_index, uint32_t *bus_num, uint8_t *pci_slot, uint8_t *dev_func);\n\n/**\n * @brief Get the anonymous file-descriptor of dma-buf associated with\n * a Neuron device memory region if it was registered for EFA peer direct\n *\n * @param addr[in] - Device buffer virtual address\n * @param size[in] - Device buffer size (in bytes)\n * @param fd[out] - dma-buf fd\n *\n * @return 0 on success\n */\nint ndl_get_dmabuf_fd(uint64_t addr, uint64_t size, int* fd);\n\n/** Gets BDF for a device\n *\n * @param device_index[in] - Neuron device index\n * @param domain[out] - PCIe domain for the device\n * @param bus_num[out] - Bus number for the device\n * @param pci_slot[out] - PCI slot for the device\n * @param dev_func[out] - Device function for the device\n *\n * @return 0 on success\n */\nint ndl_get_device_bdf_ext(int device_index, uint32_t *domain, uint32_t *bus_num, uint8_t *pci_slot, uint8_t *dev_func);\n\n/** retrieve offset/size where to mmap around a physical address\n * \n * @param device[in] - Neuron device\n * @param pa[in] - physical address in device mem to retrieve mc mmap info for\n * @param mmap_offset[out] - mmap offset\n * @param mem_handle[out] - The handle for the given physical address.\n * Set to 0 when using backwards compatible interface with old drivers.\n * @param size[out] - size\n *\n */\nint ndl_mem_get_mc_mmap_info(ndl_device_t *device, uint64_t pa, uint64_t *mmap_offset, uint64_t *size, uint64_t *mem_handle);\n\n/** mmap a bar region into user address\n *\n * @param device[in] - Neuron device\n * @param block[in] - block type containing the resource\n * @param block_id[in] - id of the block if is more than one block\n * @param resource[in] - resource the caller wants to mmap\n * @param va[out] - virual address of the resource\n * @param size[out] - size of the resource\n *\n */\nint ndl_mmap_bar_region( ndl_device_t *device, enum neuron_dm_block_type block, uint32_t block_id, enum neuron_dm_resource_type resource, \n void ** va, uint64_t * size);\n\n/** Close all cached FDs\n *\n */\nvoid ndl_device_cached_fd_close_all(void);\n\n/** Log an error message to kernel messages/serial console\n * \n * @param str[in] - The error message\n * @param size[in] - The size of the error message including null terminator\n * @param action[in] - Additional action to perform\n * \n * @return On success: 0\n * On failure: -1 and:\n * * errno == EFAULT when size is too large\n * * errno == EBADMSG when str is not null terminated\n */\nint ndl_printk(char *str, uint32_t size, uint32_t action);\n\n/** get the host device id for an open device (for containers)\n * \n * @param device[in] - Neuron device\n * @param host_device_id[out] - host device id \n * \n */\nint ndl_get_host_device_id(ndl_device_t *device, uint32_t *host_device_id);\n\n/** return device id to routing id mapping table along with number of entries in the table\n *\n * @param count[in/out] - [in] size of map in entries. [out] # entries returned\n * @param host_did_to_rid_map[out] - map of host device id to routing ids \n *\n */\nint ndl_get_host_device_id_to_rid_map(uint32_t *count, uint32_t *host_did_to_rid_map);\n\nint ndl_dump_device_allocation_info(ndl_device_t *device, uint32_t hbm_index, struct neuron_ioctl_mem_chunk_info *data, uint32_t *num_entries);\n\n/** ask the driver to dump neuron core process info \n *\n * @param nc_id[in] - [in] neuron core to dump process info for\n * @param filter_log_owner[in] - [in] only dump log entries for the owner pid of the neuron core\n * @param log_dump_limit[in] - [in] max number of log entries to dump\n *\n */\nint ndl_dump_nc_pid_info(uint32_t nc_id, bool filter_log_owner, uint32_t log_dump_limit);\n\n/** write a value to entire HBM accessible to Neuron (so excludes firmware carveout)\n *\n * @param hbm_index - HBM to write to\n * @param init_val - value to write\n */\nint ndl_hbm_scrub_start(ndl_device_t *device, uint32_t nc_id, uint32_t hbm_index, uint32_t axi_port, uint32_t init_val);\nint ndl_hbm_scrub_wait(ndl_device_t *device, uint32_t nc_id, uint32_t hbm_index);\n\n/** Gets the tpb mapping.\n *\n * @param map[out] - Location to store the mapping information\n * @param max_num_entries[in] - Maximum number of entries we can store in `map`\n * @param mapping_version[in] - Flavor of mapping to get from the driver\n *\n * @return 0 on success\n */\nint ndl_get_logical_to_physical_nc_map(struct neuron_ioctl_nc_map *map, uint32_t max_num_entries, enum neuron_ioctl_nc_mapping_type mapping_version);\n\n/** return pod information\n *\n * @param pod_type[out] - type of pod\n * @param pod_sz[out] - size of the pod\n *\n */\nint ndl_pod_info(uint32_t * pod_type, uint32_t * pod_sz);\n\n/** return pod election state\n *\n * @param state[out] - election state\n *\n */\nint ndl_pod_election_state(uint32_t * state);\n\n/** return pod mapping information.\n *\n * @param node_id[out] - node id of the pod node. -1 if the node is not part of a configured pod\n *\n */\nint ndl_pod_mapping_info(int * node_id);\n\n\n/** return pod status\n *\n * @param pod_id[out] - pod id. Only valid it the pod is configured as a pod\n * @param state[out] - state of the pod election \n * @param pod_type[out] - type of pod \n * @param pod_sz[out] - size of the pod. 0 if the node is not part of a pod\n * @param node_id[out] - node id of the pod node. -1 if the node is not part of a configured pod\n * @param mode[out] - current operating mode\n * @param modes_supported[out] - supported operating modes\n *\n */\nint ndl_pod_status(uint8_t *pod_id, uint32_t *state, uint32_t *pod_type, uint32_t *pod_sz, int *node_id, \n enum neuron_ultraserver_mode *mode, uint32_t *modes_supported);\n\n\n/** control pod election state\n *\n * @param ctrl[in] - control request. (enum neuron_pod_ctrl_req)\n * @param mode[in] - requested operating mode\n * @param timeout[in] - timeout for control operation\n * @param state[out] - state of the pod election \n *\n */\nint ndl_pod_ctrl(uint32_t ctrl, enum neuron_ultraserver_mode mode, uint32_t timeout, uint32_t *state);\n\nint ndl_alloc_contiguous_scratchpad(ndl_device_t *device, uint64_t size, uint32_t hbm_index, uint32_t nc_id, uint64_t *mem_handle);\n\n/** Similar to ndl_memory_map - only difference is that a contiguous scratchpad var may span multiple contiguous memchunks. So size of memory mapping is different from just the size of the first contiguous memchunk.\n *\n * @param mem_handle[in] - Handle to map.\n * @param va[out] - Resulting virtual address.\n * @param size[in] - Size to map \n *\n * @return 0 on success\n */\n\nint ndl_memory_map_contiguous_scratchpad(uint64_t mem_handle, void **va, uint64_t size);\n\n/** Set performance profile\n *\n * @param device[in] - Device handle.\n * @param profile[in] - Performance profile to set.\n *\n * @return 0 on success.\n */\nint ndl_set_performance_profile(ndl_device_t *device, uint32_t profile);\n\n/** Enable or disable throttling notifications\n *\n * @param device[in] - Device handle.\n * @param enable[in] - true to enable, false to disable.\n *\n * @return 0 on success.\n */\nint ndl_enable_throttling_notifications(ndl_device_t *device, bool enable);\n\nbool ndl_feature_supported(int nd_fd, uint64_t feature);\n\n/** dynamically allocate h2t queues (rings)\n *\n * @param device[in] - Neuron device\n * @param nc_id[in] - neuron core to allocate h2t queues for\n * @param copy_queue_cnt[in] - number of h2t copy queues to allocate\n * @param service_queue_cnt[in] - number of service queues to allocate\n * @param copy_queue_bmap[out] - bitmap of the allocated copy queues\n * @param servic_equeue_bmap[out] - bitmap of the allocated service queues\n * @param copy_default_queue[out] - default h2t copy queue\n *\n */\nint ndl_h2t_dma_queue_alloc(ndl_device_t *device, uint32_t nc_id, uint32_t copy_queue_cnt, uint32_t service_queue_cnt,\n uint32_t *copy_queue_bmap, uint32_t *service_queue_bmap, uint32_t *copy_default_queue);\n\n/** free dynamically allocated h2t queues\n *\n * @param device[in] - Neuron device\n * @param nc_id[in] - [in] neuron core to free queues for\n * @param queue_bmap[in] - [in] bitmap of queues to free\n *\n */\nint ndl_h2t_dma_queue_free(ndl_device_t *device, uint32_t nc_id, uint32_t queue_bmap);\n\n/** control metrics posting behavior\n *\n * @param device[in] - Neuron device\n * @param mode[in] - how to modify posting behavior (enable or disable periodic posting)\n */\nint ndl_metrics_ctrl(ndl_device_t *device, enum neuron_metrics_mode mode);\n\n/** get Neuron device and HBM index pointed by VA\n *\n * @param va[in] - VA of Neuron memory\n * @param device_index[out] - Neuron device\n * @param hbm_index[out] - HBM index\n */\nint ndl_get_va_placement(const void *va, int *device_index, int *hbm_index);\n\n/**\n * arbitrary size bitmap support\n *\n */\n#define NBM_NR_BITS(t) (sizeof(t)*8)\n#define NBM_NR_ENT(nr,t) (((nr)+NBM_NR_BITS(t)-1) / NBM_NR_BITS(t))\n\nstatic inline uint32_t nbitmap_test_bit(uint32_t nr, uint64_t *addr)\n{\n return (addr[nr/NBM_NR_BITS(*addr)] & (1ull << (nr % NBM_NR_BITS(*addr)))) != 0ull;\n}\n\nstatic inline void nbitmap_set_bit(uint32_t nr, uint64_t *addr)\n{\n addr[nr/NBM_NR_BITS(*addr)] |= (1ull << (nr % NBM_NR_BITS(*addr)));\n}\n\nstatic inline uint32_t nbitmap_ffs1(uint32_t nr, uint64_t *addr)\n{\n int i;\n for (i=0; i < NBM_NR_ENT(nr, *addr); i++) {\n uint32_t x = __builtin_ffsl(addr[i]);\n if (x) \n return i * NBM_NR_BITS(*addr) + x;\n }\n return 0;\n}\n\nstatic inline uint32_t nbitmap_popcount(uint32_t nr, uint64_t *addr)\n{\n int i;\n uint32_t cnt = 0;\n for (i=0; i < NBM_NR_ENT(nr, *addr); i++) {\n cnt += __builtin_popcountll(addr[i]);\n }\n return cnt;\n}\nstatic inline void nbitmap_clr_bit(uint32_t nr, uint64_t *addr)\n{\n addr[nr/NBM_NR_BITS(*addr)] &= ~(1ull << (nr % NBM_NR_BITS(*addr)));\n}\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/ndl/neuron_driver_shared.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#ifndef NEURON_DRIVER_SHARED_H\n#define NEURON_DRIVER_SHARED_H\n\n#include \n#include \"neuron_driver_shared_tensor_batch_op.h\"\n\nenum neuron_driver_feature_flag {\n\tNEURON_DRIVER_FEATURE_DMABUF = 1ull << 0, \n\tNEURON_DRIVER_FEATURE_ASYNC_DMA = 1ull << 1, \n\tNEURON_DRIVER_FEATURE_BATCH_DMAQ_INIT = 1ull << 2, \n\tNEURON_DRIVER_FEATURE_BIG_CORE_MAPS = 1ull << 3, \n\tNEURON_DRIVER_FEATURE_MEM_ALLOC_TYPE = 1ull << 4,\n\tNEURON_DRIVER_FEATURE_HBM_SCRUB = 1ull << 5,\n\tNEURON_DRIVER_FEATURE_MEM_ALLOC64 = 1ull << 6,\n\tNEURON_DRIVER_FEATURE_CONTIGUOUS_SCRATCHPAD = 1ull << 7,\n\tNEURON_DRIVER_FEATURE_ZEROCOPY = 1ull << 8,\n};\n\n// FIXME this should be more generic - like node type.\nenum {\n\tNEURON_POD_TYPE_NONE = 0,\n\tNEURON_POD_TYPE_P2P,\n\tNEURON_POD_TYPE_SWITCH\n};\n\nenum {\n\tNEURON_POD_E_STATE_NOT_STARTED= 0,\n\tNEURON_POD_E_STATE_IN_PROGRESS,\n\tNEURON_POD_E_STATE_ULTRASERVER,\n\tNEURON_POD_E_STATE_FAILED,\t\t\t\t// TODO we currently don't discriminate between failed and single node (todo for diagnostic/debug purposes)\n\tNEURON_POD_E_STATE_SINGLE_NODE,\n};\n\nenum neuron_pod_ctrl_req {\n\tNEURON_NPE_POD_CTRL_REQ_POD = 0, \t\t // request pod state to pod (on-demand election request)\n\tNEURON_NPE_POD_CTRL_REQ_SINGLE_NODE = 1, // request pod state to single node\n\tNEURON_NPE_POD_CTRL_REQ_KILL = 2,\t\t // request to kill the election\n\tNEURON_NPE_POD_CTRL_SET_MODE = 3,\t\t // request to ultraserver mode\n};\n\nenum neuron_ultraserver_mode {\n\tNEURON_ULTRASERVER_MODE_UNSET = 0, \t // no configuration set\n\tNEURON_ULTRASERVER_MODE_X4 = 1, \t\t // 4 node US configuration\n\tNEURON_ULTRASERVER_MODE_X2H = 2, \t\t // 2 node US configuration using horizontal links \n\tNEURON_ULTRASERVER_MODE_X2V = 3, \t\t // 2 node US configuration using vertical links \n\tNEURON_ULTRASERVER_MODE_X1 = 4, \t\t // 1 node US configuration (standalone)\n};\n\nenum neuron_metrics_mode {\n NEURON_METRICS_MODE_PERIODIC_ENABLE = 0, // enable periodic posting\n NEURON_METRICS_MODE_PERIODIC_DISABLE = 1, // disable periodic posting\n};\n\n#define NEURON_NC_MAP_DEVICE (0xffffffff)\n\nenum neuron_dma_queue_type {\n\tNEURON_DMA_QUEUE_TYPE_TX = 0, // transmit queue\n\tNEURON_DMA_QUEUE_TYPE_RX, // receive queue\n\tNEURON_DMA_QUEUE_TYPE_COMPLETION, // completion queue\n};\n\nenum neuron_cinit_state {\n\tNEURON_CINIT_STATE_STARTED = 1, // Core Init is initiated\n\tNEURON_CINIT_STATE_COMPLETED, // Core Init is completed successfully\n\tNEURON_CINIT_STATE_INVALID // Core Init is not valid\n};\n\n\nstruct neuron_dma_eng_state {\n\t__u32 revision_id; // revision id\n\t__u32 max_queues; // maximum queues supported\n\t__u32 num_queues; // number of queues configured\n\t__u32 tx_state; // Tx statue\n\t__u32 rx_state; // Rx state\n};\n\nstruct neuron_dma_queue_state {\n\t__u32 hw_status; // hardware status\n\t__u32 sw_status; // software status\n\t__u64 base_addr; // base address of the queue\n\t__u32 length; // size of the queue\n\t__u32 head_pointer; // hardware pointer index\n\t__u32 tail_pointer; // software pointer index\n\t__u64 completion_base_addr; // completion queue base address\n\t__u32 completion_head; // completion head\n};\n\nenum neuron_dma_h2t_ctx_handle_type {\n\tNEURON_DMA_H2T_CTX_HANDLE_NONE = -1, // no handle - used as prev handle to start an async dma\n\tNEURON_DMA_H2T_CTX_HANDLE_SYNC = 0, // handle for doing synchronous DMA\n\tNEURON_DMA_H2T_CTX_HANDLE_ASYNC1 = 1, // first of two async handles\n\tNEURON_DMA_H2T_CTX_HANDLE_ASYNC2 = 2, // second of two async handles\n\tNEURON_DMA_H2T_CTX_HANDLE_CNT = 3 // number of dma \n};\n\n/*\n * H2T DMA Default Queue id\n */\n#define NEURON_DMA_H2T_DEFAULT_QID (-1)\n\n/*\n * NOTE: In runtime version 5, this enum was passed in as a bool instead -\n * true if top_sp and false if NC. Match the enum values to the bool to\n * maintain compatibility with older runtime. Do not change these values\n * until the min compatibility version is updated to >=6.\n */\nenum NQ_DEVICE_TYPE {\n NQ_DEVICE_TYPE_NEURON_CORE = 0,\n NQ_DEVICE_TYPE_TOPSP,\n NQ_DEVICE_TYPE_MAX\n};\n\nenum NQ_TYPE {\n\tNQ_TYPE_TRACE = 0, /**< Implicit notifications generated during execution. */\n\tNQ_TYPE_NOTIFY, /**< Explicit notifications generated by NOTIFY instruction */\n\tNQ_TYPE_EVENT, /**< Notifications triggered by event set/clear operations. */\n\tNQ_TYPE_ERROR, /**< Notifications triggered by an error condition. */\n\tNQ_TYPE_TRACE_DMA, /**< Implicit notifications generated by DMA transfers.*/\n\tNQ_TYPE_THROTTLE, /**< Notifications triggered by throttling activity. */\n\tNQ_TYPE_MAX\n};\n\n/**\n * memory mapping enums for selecting what bar0 resources to map.\n * Bar0 mmapping is restricted to a limited set of regions.\n * Resources are selected by block type, block id and resource within the block.\n * TPB 1 State buffer, for example - where type is TPB, block id is 1 and \n * resource is state buffer.\n * NEURON_DM_RESOURCE_ALL resource mapping is restricted to read only.\n *\n */\nenum neuron_dm_block_type {\n NEURON_DM_BLOCK_INVALID = -1, // invalid - tag last entry in the table\n NEURON_DM_BLOCK_TPB = 0,\n NEURON_DM_BLOCK_TOPSP = 1,\n NEURON_DM_BLOCK_HBM = 2\n};\n\nenum neuron_dm_resource_type {\n NEURON_DM_RESOURCE_SEMAPHORE = 0, // resource to mmap is semaphore region\n NEURON_DM_RESOURCE_ALL = 1, // resource to mmap is the entire block (read only). Only available for TOPSP\n NEURON_DM_RESOURCE_SBUF = 2, // resource to mmap is state buffer\n NEURON_DM_RESOURCE_DMEM = 3\t // resource to mmap is device memory\n};\n\nstruct neuron_uuid {\n\t__u8 value[32];\n};\n\n#define NEURON_MAX_PROCESS_PER_DEVICE 16 // 2 per core (arbitrary but needs to small number for fast lookup)\n\n#define APP_INFO_PID_NC_LOCK_INFO\t(1)\n#define APP_INFO_PID_MEM_USAGE\t\t(1 << 1)\n#define APP_INFO_ALL\t\t\t(0xF)\n\n#define APP_INFO_MAX_MODELS_PER_DEVICE\t(4)\n#define NDS_INVALID_ID (-1)\n\nstruct neuron_app_info {\n\t__s32 pid;\t\t\t\t\t\t\t// PID of this app\n\t__u8 nc_lock_map;\t\t\t\t\t\t// NCs which are locked by it (one bit set for each locked NC)\n\tstruct neuron_uuid uuid_data[APP_INFO_MAX_MODELS_PER_DEVICE];\t// UUIDs running for this app for each neuroncore\n\tsize_t host_mem_size;\t\t\t\t\t\t// Amount of host memory used by this PID\n\tsize_t device_mem_size;\t\t\t\t\t\t// Amount of device memory used by this PID\n};\n\ntypedef union nmetric_version {\n\tstruct {\n\t\t__u64 build_num : 32;\n\t\t__u64 minor_ver : 8;\n\t\t__u64 major_ver : 8;\n\t\t__u64 reserved : 16;\n\t};\n\t__u64 all;\n} nmetric_version_t;\n\nstruct neuron_ioctl_mem_chunk_info {\n\t__u64 pa;\n\t__u64 size;\n\t__u32 mem_type;\n};\n\n// Max number of entries this version of the driver\n// will ever give back to the user\n#define NEURON_NC_MAP_MAX_ENTRIES 128\nenum neuron_ioctl_nc_mapping_type {\n NEURON_IOCTL_NC_MAPPING_TYPE_V0 = 0, // seng swap mapping\n};\nstruct neuron_ioctl_nc_map_entry {\n __u32 device_id;\n __u32 device_nc_idx;\n};\nstruct neuron_ioctl_nc_map {\n __u32 num_entries;\n struct neuron_ioctl_nc_map_entry mappings[];\n};\n\n/* A batch of copy operations */\ntypedef struct neuron_memcpy_batch {\n\t__u64 mem_handle; // [in] Source or Destination memory handle from/to data needs to be copied.\n\t__u64 mem_handle_offset; // [in] Memory offset of the memory handle\n\tconst nrt_tensor_batch_op_t *ops_ptr; // [in] Pointer to array of operations\n\t__u32 num_ops; // [in] Number of neuron_memcpy_op operations.\n\t__u16 bar4_wr_threshold; // [in] Threshold below which we will use bar4 direct write vs. DMA. Subject to driver limits.\n\t__u16 flags; // [in] TBD.\n\tvoid *context; // [in] TBD. opaque context pointer passed back in completion queue\n} neuron_memcpy_batch_t;\n\n/*\n * Memory allocation categories for sysfs counters\n*/\ntypedef enum {\n\tNEURON_MEMALLOC_TYPE_UNKNOWN_HOST, // only for old runtimes, do not use elsewhere\n\tNEURON_MEMALLOC_TYPE_CODE_HOST,\n\tNEURON_MEMALLOC_TYPE_TENSORS_HOST,\n\tNEURON_MEMALLOC_TYPE_CONSTANTS_HOST,\n\tNEURON_MEMALLOC_TYPE_MISC_HOST,\n\tNEURON_MEMALLOC_TYPE_NCDEV_HOST,\n\tNEURON_MEMALLOC_TYPE_NOTIFICATION_HOST,\n\n\tNEURON_MEMALLOC_TYPE_UNKNOWN_DEVICE, // only for old runtimes, do not use elsewhere\n\tNEURON_MEMALLOC_TYPE_CODE_DEVICE,\n\tNEURON_MEMALLOC_TYPE_TENSORS_DEVICE,\n\tNEURON_MEMALLOC_TYPE_CONSTANTS_DEVICE,\n\tNEURON_MEMALLOC_TYPE_SCRATCHPAD_DEVICE,\n\tNEURON_MEMALLOC_TYPE_MISC_DEVICE,\n\tNEURON_MEMALLOC_TYPE_NCDEV_DEVICE,\n\tNEURON_MEMALLOC_TYPE_COLLECTIVES_DEVICE,\n\tNEURON_MEMALLOC_TYPE_SCRATCHPAD_NONSHARED_DEVICE,\n\tNEURON_MEMALLOC_TYPE_NOTIFICATION_DEVICE,\n\n\tNEURON_MEMALLOC_TYPE_DMA_RINGS_HOST,\n\tNEURON_MEMALLOC_TYPE_DMA_RINGS_DEVICE,\n\n\tNEURON_MEMALLOC_TYPE_CONTIGUOUS_SCRATCHPAD_DEVICE, // uses same sysfs counter as NEURON_MEMALLOC_TYPE_SCRATCHPAD_DEVICE\n\n\tNEURON_MEMALLOC_TYPE_MAX\n} mem_alloc_category_t;\n\n/*\n * NDS stats\n * Note: \n * \tTo add a new counter type inside the enum, \n * \t\t1. you need to manually decrease NDS_ND_COUNTER_RESERVED or NDS_NC_COUNTER_RESERVED by 1\n * \t\t2. you need to update NDS_ND_COUNTER_COUNT or NDS_NC_COUNTER_COUNT\n * \tTo prevent compatability issues, you need to always append the new counter type to the end of the enum\n */\n#define NDS_ND_COUNTER_RESERVED 18\n\n// Device counter types\nenum {\n\tNDS_ND_COUNTER_RUNTIME_VERSION,\n\tNDS_ND_COUNTER_FRAMEWORK_VERSION,\n\tNDS_ND_COUNTER_FAL_VERSION,\n\tNDS_ND_COUNTER_FEATURE_BITMAP,\n\tNDS_ND_COUNTER_MIN_NEFF_VERSION,\n\tNDS_ND_COUNTER_MAX_NEFF_VERSION,\n\n\t// memory usage counters\n\tNDS_ND_COUNTER_MEM_USAGE_CODE_HOST,\n\tNDS_ND_COUNTER_MEM_USAGE_TENSORS_HOST,\n\tNDS_ND_COUNTER_MEM_USAGE_CONSTANTS_HOST,\n\tNDS_ND_COUNTER_MEM_USAGE_SCRATCHPAD_HOST,\n\tNDS_ND_COUNTER_MEM_USAGE_MISC_HOST,\n\n\tNDS_ND_COUNTER_DYNAMIC_SYSFS_METRIC_BITMAP,\n\n\tNDS_ND_COUNTER_DEVICE_CLUSTER_ID,\n\n\tNDS_ND_COUNTER_COUNT = NDS_ND_COUNTER_DEVICE_CLUSTER_ID + NDS_ND_COUNTER_RESERVED + 1\n};\n\n#define NDS_NC_COUNTER_RESERVED 0\n\n// Neuroncore counter types\nenum {\n\tNDS_NC_COUNTER_TIME_IN_USE = 0,\n\n\tNDS_NC_COUNTER_INFER_COMPLETED,\n\tNDS_NC_COUNTER_INFER_COMPLETED_WITH_ERR,\n\tNDS_NC_COUNTER_INFER_COMPLETED_WITH_NUM_ERR,\n\tNDS_NC_COUNTER_INFER_TIMED_OUT,\n\tNDS_NC_COUNTER_INFER_INCORRECT_INPUT,\n\tNDS_NC_COUNTER_INFER_FAILED_TO_QUEUE,\n\n\t// these must be in this specifc order\n\t// runtime assumes these are offset by\n\t// error code\n\tNDS_NC_COUNTER_ERR_GENERIC,\n\tNDS_NC_COUNTER_ERR_NUMERICAL,\n\tNDS_NC_COUNTER_ERR_MODEL,\n\tNDS_NC_COUNTER_ERR_TRANSIENT,\n\tNDS_NC_COUNTER_ERR_HW,\n\tNDS_NC_COUNTER_ERR_RT,\n\n\tNDS_NC_COUNTER_LATENCY_DEVICE,\n\tNDS_NC_COUNTER_LATENCY_TOTAL,\n\tNDS_NC_COUNTER_NC_TIME,\n\n\t// these are new counters\n\t// these shall be placed at the\n\t// end so there offsets are always\n\t// greater than old counters\n\t// This will ensure\n\t// new runtime + old driver will\n\t// write to reserved setions and not\n\t// break anything\n\tNDS_NC_COUNTER_GENERIC_FAIL,\n\tNDS_NC_COUNTER_ERR_RESOURCE,\n\tNDS_NC_COUNTER_ERR_RESOURCE_NC,\n\tNDS_NC_COUNTER_ERR_INVALID,\n\tNDS_NC_COUNTER_ERR_UNSUPPORTED_NEFF_VERSION,\n\n\tNDS_NC_COUNTER_CC_TIME,\n\n\tNDS_NC_COUNTER_MEM_USAGE_CODE_DEVICE,\n\tNDS_NC_COUNTER_MEM_USAGE_TENSORS_DEVICE,\n\tNDS_NC_COUNTER_MEM_USAGE_CONSTANTS_DEVICE,\n\tNDS_NC_COUNTER_MEM_USAGE_SCRATCHPAD_DEVICE,\n\tNDS_NC_COUNTER_MEM_USAGE_MISC_DEVICE,\n\n\tNDS_NC_COUNTER_MODEL_LOAD_COUNT,\n\tNDS_NC_COUNTER_INFERENCE_COUNT,\n\n\tNDS_NC_COUNTER_MAC_COUNT,\n\n\tNDS_NC_COUNTER_OOB,\n\n\tNDS_NC_COUNTER_COUNT = NDS_NC_COUNTER_OOB + NDS_NC_COUNTER_RESERVED + 1\n};\n\n#define NDS_MAX_NEURONCORE_COUNT (4)\n#define NDS_EXT_MAX_NEURONCORE_COUNT (12)\n\n// Additional NC storage\n// | NDS_EXT_NC_COUNTER_COUNT | ... | NDS_EXT_NC_COUNTER_COUNT | (x NDS_MAX_NEURONCORE_COUNT) - this will only store the 'overflow' from the original counters\n// | NDS_NC_COUNTER_COUNT + NDS_EXT_NC_COUNTER_COUNT | ... (x NDS_EXT_MAX_NEURONCORE_COUNT) - this will store complete data for additional NCs (up to a max of 16)\n#define NDS_EXT_NC_COUNTER_ADDED_RESERVED 54\n// Index of NC counter extensions start at NDS_NC_COUNTER_COUNT not at 0\nenum {\n\tNDS_EXT_NC_COUNTER_HW_ERR_COLLECTIVES = NDS_NC_COUNTER_COUNT,\n\tNDS_EXT_NC_COUNTER_HW_ERR_HBM_UE,\n\tNDS_EXT_NC_COUNTER_HW_ERR_NC_UE,\n\tNDS_EXT_NC_COUNTER_HW_ERR_DMA_ABORT,\n\tNDS_EXT_NC_COUNTER_ERR_SW_NQ_OVERFLOW,\n\tNDS_EXT_NC_COUNTER_ERR_SW_SEMAPHORE_ERROR,\n\tNDS_EXT_NC_COUNTER_ERR_SW_EVENT_ERROR,\n\tNDS_EXT_NC_COUNTER_ERR_SW_PSUM_COLLISION,\n\tNDS_EXT_NC_COUNTER_ERR_SW_SEQUENCER_FATAL,\n\tNDS_EXT_NC_COUNTER_HW_ERR_REPAIRABLE_HBM_UE,\n\tNDS_EXT_NC_COUNTER_LAST,\n\tNDS_EXT_NC_COUNTER_COUNT = NDS_EXT_NC_COUNTER_LAST - NDS_NC_COUNTER_COUNT + NDS_EXT_NC_COUNTER_ADDED_RESERVED\n};\n\n#define NDS_TOTAL_NC_COUNTER_COUNT (NDS_NC_COUNTER_COUNT + NDS_EXT_NC_COUNTER_COUNT) // 31 original + 64 extended = 95 counters\n\ntypedef struct nds_header {\n\tchar signature[4]; // Fixed signature: 'n', 'd', 's', 0\n\tint version; // Version of the datastore's format\n} nds_header_t;\n\n/* --------------------------------------------\n * NDS shared data offsets\n * --------------------------------------------\n */\n\n#define NDS_HEADER_START (0)\n#define NDS_HEADER_SIZE (sizeof(nds_header_t))\n\n#define NDS_ND_COUNTERS_START (NDS_HEADER_START + NDS_HEADER_SIZE)\n#define NDS_ND_COUNTERS_SIZE (NDS_ND_COUNTER_COUNT * sizeof(uint64_t))\n#define NDS_ND_COUNTERS(base_addr) ((uint64_t *)(base_addr + NDS_ND_COUNTERS_START))\n\n// original NC counter section\n#define NDS_NEURONCORE_COUNTERS_COUNT (NDS_NC_COUNTER_COUNT)\n#define NDS_NEURONCORE_COUNTERS_START (NDS_ND_COUNTERS_START + NDS_ND_COUNTERS_SIZE)\n#define NDS_NEURONCORE_COUNTERS_SIZE (NDS_NEURONCORE_COUNTERS_COUNT * NDS_MAX_NEURONCORE_COUNT * sizeof(uint64_t))\n#define NDS_NEURONCORE_COUNTERS(base_addr, nc_index) ((uint64_t *)(base_addr + NDS_NEURONCORE_COUNTERS_START) + (nc_index * NDS_NEURONCORE_COUNTERS_COUNT))\n\n// additional NC counter section at the end of all existing structures in the datastore (i.e. after NDS_PROCESS_EXT_INFO)\n// NDS_PROCESS_EXT_INFO_START + NDS_PROCESS_EXT_INFO_SIZE = 44588 (hardcoded because it's easier than to move all the structs here and sizeof them)\n#define NDS_EXT_NC_COUNTER_COUNT_OLD (65)\n#define NDS_TOTAL_NC_COUNTER_COUNT_OLD (96)\n\n#define NDS_EXT_NEURONCORE_COUNTERS_SIZE_OLD (NDS_EXT_NC_COUNTER_COUNT_OLD * NDS_MAX_NEURONCORE_COUNT * sizeof(uint64_t))\n#define NDS_EXT_NEURONCORE_NC_DATA_SIZE_OLD\t(NDS_TOTAL_NC_COUNTER_COUNT_OLD * NDS_EXT_MAX_NEURONCORE_COUNT * sizeof(uint64_t))\n#define NDS_EXT_SECTION_SIZE_OLD (NDS_EXT_NEURONCORE_COUNTERS_SIZE_OLD + NDS_EXT_NEURONCORE_NC_DATA_SIZE_OLD)\n#define NDS_EXT_OFFSET_OLD (44588)\n\n#define NDS_EXT_ALIGNMENT (64)\n#define NDS_ALIGN(v) ((v) + (-(v) & (NDS_EXT_ALIGNMENT - 1)))\n#define NDS_EXT_OFFSET (NDS_ALIGN(NDS_EXT_OFFSET_OLD + NDS_EXT_SECTION_SIZE_OLD))\n\n#define NDS_EXT_NEURONCORE_COUNTERS_COUNT (NDS_EXT_NC_COUNTER_COUNT) // number of extended counters\n#define NDS_EXT_NEURONCORE_COUNTERS_START (NDS_EXT_OFFSET)\n#define NDS_EXT_NEURONCORE_COUNTERS_SIZE (NDS_EXT_NC_COUNTER_COUNT * NDS_MAX_NEURONCORE_COUNT * sizeof(uint64_t))\n#define NDS_EXT_NEURONCORE_COUNTERS(base_addr, nc_index) ((uint64_t *)(base_addr + NDS_EXT_NEURONCORE_COUNTERS_START) + (nc_index * NDS_EXT_NC_COUNTER_COUNT))\n\n// additional NC data for extra Neuron Cores (12 extra sets which include all 95 counters + 1 for padding)\n#define NDS_EXT_NEURONCORE_NC_DATA_PADDING (1) // 1 added as padding for 64 byte alignment per NC\n#define NDS_EXT_NEURONCORE_NC_DATA_COUNT (NDS_TOTAL_NC_COUNTER_COUNT + NDS_EXT_NEURONCORE_NC_DATA_PADDING) // full set of counters (base + extended) + padding\n#define NDS_EXT_NEURONCORE_NC_DATA_START (NDS_ALIGN(NDS_EXT_NEURONCORE_COUNTERS_START + NDS_EXT_NEURONCORE_COUNTERS_SIZE))\n#define NDS_EXT_NEURONCORE_NC_DATA_SIZE (NDS_EXT_MAX_NEURONCORE_COUNT * NDS_EXT_NEURONCORE_NC_DATA_COUNT * sizeof(uint64_t))\n\n#define NDS_EXT_NEURONCORE_NC_DATA(base_addr, nc_index) ((uint64_t *)(base_addr + NDS_EXT_NEURONCORE_NC_DATA_START) + (nc_index * NDS_EXT_NEURONCORE_NC_DATA_COUNT))\n\n#endif // NEURON_DRIVER_SHARED_H\n" }, { "path": "src/libnrt/include/ndl/neuron_driver_shared_tensor_batch_op.h", "content": "/*\n * Shared tensor batch operation between runtime and driver.\n */\n\n#ifndef NEURON_DRIVER_SHARED_TENSOR_BATCH_OP_H\n#define NEURON_DRIVER_SHARED_TENSOR_BATCH_OP_H\n\n#ifdef __KERNEL__\n#include \ntypedef __u64 nrt_tensor_batch_offset_t;\ntypedef __u64 nrt_tensor_batch_size_t;\n#else\n#include \ntypedef uint64_t nrt_tensor_batch_offset_t;\ntypedef uint64_t nrt_tensor_batch_size_t;\n#endif\n\ntypedef struct nrt_tensor_batch_op {\n nrt_tensor_batch_offset_t offset;\n nrt_tensor_batch_size_t size;\n void *buffer;\n} nrt_tensor_batch_op_t;\n\n#endif // NEURON_DRIVER_SHARED_TENSOR_BATCH_OP_H\n" }, { "path": "src/libnrt/include/nrt/ndebug_stream.h", "content": "/*\n * Copyright 2025, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n/**\n * Overview:\n * The `ndebug_stream` APIs provide applications a way to consume debug events from the runtime (see\n * `ndebug_stream_event_type_t` for the different event types). These debug events are emitted by the\n * runtime per Logical Neuron Core and can be used by applications to get information on events that\n * occured on the device (ie prints, breakpoints, etc.).\n *\n * Connecting, polling, and consuming:\n * Applications that want to consume debug events will first need to connect to a Logical Neuron Core's debug stream via a call to\n * `nrt_debug_client_connect`. Once a client is connected to a core's debug stream, the runtime will will push debug events emitted\n * by the Logical Neuron Core to the stream for clients to consume. To be notified of emitted debug events, clients can utilize the\n * polling APIs provided by the Linux kernel. The `stream_fd` handle obtained from the `nrt_debug_client_connect` is a typical Linux\n * file descriptor and can be passed into any Linux polling API. It is important to note though, that while the `stream_fd` is pollable,\n * all other non-polling related functionality must go through the provided `nrt_debug_client*` APIs. For example, the stream contents\n * can only be accessed from the `nrt_debug_client_read*` API(s) and any other methods of accessing the stream data leads undefined/undesireable\n * behavior.\n *\n * Closing a Connection:\n * Once a connection is not needed anymore, clients can close the connection using the `nrt_debug_client_connect_close` API.\n *\n * Events:\n * Events consist of a header describing the payload type, and a payload representing the contents of the event. Events can be consumed by\n * clients via the `nrt_debug_client_read*` API(s).\n *\n * Notes:\n * * These APIs do not allow for interprocess communication. Debug events are only pushed to the process that owns the Logical Neuron Core.\n * * These APIs do not provide thread safety for multiple threads accessing the SAME stream (thread safety for different streams is guarenteed).\n * * There can only be one outstanding connection per stream. Any attempts to initialize multiple connectiongs will result in an error.\n * * Events are only emitted AFTER a client connects to a Logical Neuron Core's stream. Any event that would have been emitted before connectioning\n * to the stream is dropped.\n * * Events will be dropped if the number of unconsumed events in a stream exceeds the stream's buffer size. Clients must consume events fast\n * enough to prevent dropped events. Additionally, Clients can configure the stream's buffer size via the `NEURON_RT_DEBUG_STREAM_BUFFER_SIZE`\n * environment variable. The buffer size currently defaults to 64K debug events.\n */\n\n#pragma once\n\n#include \n#include \n#include \n#include \n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\ntypedef enum ndebug_stream_event_type {\n NDEBUG_STREAM_EVENT_TYPE_INVALID = 0,\n NDEBUG_STREAM_EVENT_TYPE_DEBUG_TENSOR_READ = 1,\n} ndebug_stream_event_type_t;\n\ntypedef struct ndebug_stream_event_header {\n uint64_t data_size;\n uint32_t type;\n char reserved[52];\n} ndebug_stream_event_header_t;\n\ntypedef struct ndebug_stream_payload_debug_tensor_read {\n char prefix[512];\n uint32_t logical_nc_id;\n uint32_t pipe;\n char tensor_dtype[16];\n uint64_t tensor_shape[8];\n uint64_t tensor_data_size;\n char reserved0[416];\n char tensor_data[];\n} ndebug_stream_payload_debug_tensor_read_t;\n\n/** Establish a connection to a specified Logical Neuron Core's debug stream.\n *\n * @param logical_nc_idx[in] - Core's debug stream to connect to.\n * @param stream_fd[out] - Connection handle to reference and interact with the stream.\n *\n * @return NRT_SUCCESS on success.\n *\n * @note Only one client can connect to a Logical Neuron Core's stream at any given time.\n * Attempts to connect to a stream with multiple clients will result in a NRT_INVALID\n * return status.\n *\n */\nNRT_STATUS nrt_debug_client_connect(int logical_nc_idx, int *stream_fd);\n\n/** Closes connection created by `nrt_debug_client_connect`\n *\n * @param stream_fd[in] - Connection handle to close.\n *\n */\nvoid nrt_debug_client_connect_close(int stream_fd);\n\n/** Consumes a single event from the stream.\n *\n * @param stream_fd[in] - Stream to consume an event from\n * @param header[out] - Comsuned event's header. See `ndebug_stream_event_header_t`.\n * @param payload[out] - Consumed event's payload. See `ndebug_stream_payload*` and `ndebug_stream_event_type_t`.\n * **IMPORTANT**: it is the user's responsibility to free this payload pointer.\n *\n * @return NRT_SUCCESS on success.\n *\n * @note This function must be called from the same process that owns the Logical Neuron Core. Calling this\n * function from any other process results in undefined behavior.\n *\n */\nNRT_STATUS nrt_debug_client_read_one_event(int stream_fd, ndebug_stream_event_header_t *header, void **payload);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nds/neuron_ds.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \n#include \n#include \n#include \n#include \n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n// Main NDS object handle\ntypedef void *nds_obj_handle_t;\n\n// NDS object types\n#define OBJECT_TYPE_MODEL_NODE_INFO (0)\n#define OBJECT_TYPE_PROCESS_INFO (1)\n#define OBJECT_TYPE_PROCESS_INFO_EXT (2)\n\n// Model-related structs\n#define MODEL_MEM_USAGE_LOCATION_COUNT 2\n\n/*\n * Number of slots for mem_usage_type in Neuron Datastore (also used by tools)\n *\n * In the current version of the neuron datastore's format, there are only 12 slots for storing\n * memory usage type, so we aggregate them using the same logic as for the 'per NC' memory tracker.\n * Monitor always aggregated them even further by adding them together, so we aren't breaking any feature.\n *\n * For usage types definiton, go to \"inc/tdrv/dma_mem_usage_type.h\"\n *\n */\nenum {\n NDS_DMA_MEM_USAGE_SLOT_CODE,\n NDS_DMA_MEM_USAGE_SLOT_TENSORS,\n NDS_DMA_MEM_USAGE_SLOT_CONSTANTS,\n NDS_DMA_MEM_USAGE_SLOT_SCRATCHPAD,\n NDS_DMA_MEM_USAGE_SLOT_MISC,\n NDS_DMA_MEM_USAGE_SLOT_COUNT = 12 // do not change\n};\n\n// Aggregated data for all chunks of the same type/location\ntypedef struct nds_mem_usage_info {\n size_t total_size; // Total size\n uint32_t chunk_count; // Number chunks that make up the total size\n} nds_mem_usage_info_t;\n\n// Loaded model node information\ntypedef struct nds_model_node_info {\n uint32_t model_id; // parent model id\n uint32_t model_node_id; // node id\n char name[256]; // model name\n char uuid[16]; // uuid\n uint8_t nc_index; // nc index\n uint8_t sg_index; // subgraph index\n} nds_model_node_info_t;\n\n// Loaded model node memory usage information\ntypedef struct nds_model_node_mem_usage_info {\n // MODEL_MEM_USAGE_LOCATION_COUNT per each usage type\n nds_mem_usage_info_t model_mem_usage[MODEL_MEM_USAGE_LOCATION_COUNT][NDS_DMA_MEM_USAGE_SLOT_COUNT];\n} nds_model_node_mem_usage_info_t;\n\n// Version information\ntypedef struct nds_version_info {\n uint8_t major;\n uint8_t minor;\n uint32_t build;\n} nds_version_info_t;\n\n// Process information-related struct\ntypedef struct nds_process_info {\n int8_t framework_type;\n char tag[32];\n nds_version_info_t framework_version;\n nds_version_info_t fal_version;\n nds_version_info_t runtime_version;\n} nds_process_info_t;\n\n// Extended process information\ntypedef struct nds_process_info_ext {\n char tag[256];\n} nds_process_info_ext_t;\n\ntypedef struct nds_instance nds_instance_t;\ntypedef struct ndl_device ndl_device_t;\n\n\n// Feature bitmap's bit index information\ntypedef enum feature_bitmap_bit_index {\n BIT_INDEX_TEST_FEATURE = 0,\n BIT_INDEX_MULTICORE_FEATURE = 1,\n\n BIT_INDEX_COUNT = BIT_INDEX_MULTICORE_FEATURE + 1\n} feature_bitmap_bit_index_t;\n\n\n/** Opens NDS for the given pid. If pid == 0, it acquires it for the current PID\n * and it's opened in read-write mode. If pid != 0, it acquires it for the provided PID\n * and it's opened as read-only.\n *\n * @param device[in] - ndl_device used to open this NDS\n * @pid pid[in] - pid for which to open the NDS, if 0 - it's opened as r/w for the current process\n * @inst[out] - address of a pointer which will contain the instance handle\n *\n * @return non zero in case of error\n */\nint nds_open(ndl_device_t *device, pid_t pid, nds_instance_t **inst);\n\n/** Releases the NDS instance and frees the data associated with it (mandatory for readers)\n *\n * @param inst[in] - NDS instance to close\n *\n * @return non zero in case of error, the pointer gets deleted regardless\n */\nint nds_close(nds_instance_t *inst);\n\n/* --------------------------------------------\n * NDS Neuroncore Counters\n * --------------------------------------------\n */\n\n/** Increments a simple per-nc counter\n *\n * @param inst[in] - NDS instance\n * @param pnc_index[in] - Neuroncore index\n * @param counter_index[in] - Counter index\n * @param increment[in] - Amount to increment\n *\n * @return 0 on success.\n */\nint nds_increment_nc_counter(nds_instance_t *inst, int pnc_index, uint32_t counter_index, uint64_t increment);\n\n/** Decrements a simple per-nc counter\n *\n * @param inst[in] - NDS instance\n * @param pnc_index[in] - Neuroncore index\n * @param counter_index[in] - Counter index\n * @param increment[in] - Amount to increment\n *\n * @return 0 on success.\n */\nint nds_decrement_nc_counter(nds_instance_t *inst, int pnc_index, uint32_t counter_index, uint64_t decrement);\n\n/** Gets a simple per-nc counter\n *\n * @param inst[in] - NDS instance\n * @param pnc_index[in] - Neuroncore index\n * @param counter_index[in] - Counter index\n * @param value[out] - Counter value\n *\n * @return 0 on success.\n */\nint nds_get_nc_counter(nds_instance_t *inst, int pnc_index, uint32_t counter_index, uint64_t *value);\n\n/** Sets a simple per-nc counter\n *\n * @param inst[in] - NDS instance\n * @param pnc_index[in] - Neuroncore index\n * @param counter_index[in] - Counter index\n * @param value[in] - Value to set the counter to\n *\n * @return 0 on success.\n */\nint nds_set_nc_counter(nds_instance_t *inst, int pnc_index, uint32_t counter_index, uint64_t *value);\n\n/* --------------------------------------------\n * NDS Neuron Device Counters\n * --------------------------------------------\n */\n\n/** Increments a simple per-nd counter - may overflow\n *\n * @param inst[in] - NDS instance\n * @param counter_index[in] - Counter index\n * @param increment[in] - Amount to increment\n *\n * @return 0 on success.\n */\nint nds_increment_nd_counter(nds_instance_t *inst, uint32_t counter_index, uint64_t increment);\n\n/** Decrements a simple per-nd counter - may overflow\n *\n * @param inst[in] - NDS instance\n * @param counter_index[in] - Counter index\n * @param decrement[in] - Amount to decrement\n *\n * @return 0 on success.\n */\nint nds_decrement_nd_counter(nds_instance_t *inst, uint32_t counter_index, uint64_t decrement);\n\n/** Bitwise inclusive OR operation on counter\n * \n * @param inst[in] - NDS instance\n * @param counter_index[in] - Counter index\n * @param 1ull << bit_index - bit mask on the feature bitmap\n * \n * @return 0 on success.\n */\nint nds_or_nd_counter(nds_instance_t *inst, uint32_t counter_index, uint64_t bit_index);\n\n/** Gets a simple per-nd counter\n *\n * @param inst[in] - NDS instance\n * @param counter_index[in] - Counter index\n * @param value[out] - Counter value\n *\n * @return 0 on success.\n */\nint nds_get_nd_counter(nds_instance_t *inst, uint32_t counter_index, uint64_t *value);\n\n/** Sets a simple per-nd counter\n *\n * @param inst[in] - NDS instance\n * @param counter_index[in] - Counter index\n * @param value[in] - Value to set the counter to\n *\n * @return 0 on success.\n */\nint nds_set_nd_counter(nds_instance_t *inst, uint32_t counter_index, uint64_t *value);\n\n/* --------------------------------------------\n * NDS objects\n * --------------------------------------------\n */\n\n/** Writes an NDS object to the NDS memory\n *\n * @param obj[in] - NDS object handle\n *\n * @return 0 on success.\n */\nint nds_obj_commit(nds_obj_handle_t obj);\n\n/** Creates a new NDS object with the given type\n *\n * @param inst[in] - NDS instance\n * @param type[in] - type of object to create\n *\n * @return handle for newly created object\n */\nnds_obj_handle_t nds_obj_new(nds_instance_t *inst, int type);\n\n/** Deletes a NDS object from NDS (and local memory)\n *\n * @param obj[in] - NDS object handle\n *\n * @return 0 on success.\n */\nint nds_obj_delete(nds_obj_handle_t obj);\n\n/** Casts this NDS object to a mode_node_info_t which can be used for r/w\n *\n * @param obj[in] - NDS object handle\n *\n * @return non-NULL on success.\n */\nnds_model_node_info_t *nds_obj_handle_to_model_node_info(nds_obj_handle_t obj);\n\n/** Casts this NDS object to a nds_model_node_mem_usage_info_t which can be used for r/w\n *\n * @param obj[in] - NDS object handle\n *\n * @return non-NULL on success.\n */\nnds_model_node_mem_usage_info_t *nds_obj_handle_to_model_node_mem_usage(nds_obj_handle_t obj);\n\n/** Reads all model info data and returns it as an array (needs to be deleted by caller)\n *\n * @param inst[in] - NDS instance\n * @param models[out] - Pointer where to write the address of an array of length count containing object handles\n * @param count[out] - Number of models loaded (present in the models array)\n *\n * @return non-NULL on success.\n */\nint nds_read_all_model_nodes(nds_instance_t *inst, nds_obj_handle_t **models, size_t *count);\n\n/** Casts this NDS object to a nds_process_info_t which can be used for r/w\n *\n * @param obj[in] - NDS object handle\n *\n * @return non-NULL on success.\n */\nnds_process_info_t *nds_obj_handle_to_process_info(nds_obj_handle_t obj);\n\n/** Casts this NDS object to a nds_process_info_ext_t which can be used for r/w\n *\n * @param obj[in] - NDS object handle\n *\n * @return non-NULL on success.\n */\nnds_process_info_ext_t *nds_obj_handle_to_process_info_ext(nds_obj_handle_t obj);\n\n/** Reads process info and returns a nds_obj_handle\n *\n * @param inst[in] - NDS instance\n *\n * @return non-NULL on success.\n */\nnds_obj_handle_t nds_read_process_info(nds_instance_t *inst);\n\n/** Reads extended process info and returns a nds_obj_handle\n *\n * @param inst[in] - NDS instance\n *\n * @return non-NULL on success.\n */\nnds_obj_handle_t nds_read_process_info_ext(nds_instance_t *inst);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nec.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \n#include \n#include \n#include \n#include \"nrt/nrt_status.h\"\n#include \n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n#define NEC_MAX_CHANNELS 32 /* matches MAXCHANNELS in NCCL */\n#define NEC_MAX_NR_CHANNEL_CHUNKS 32 /* Channel buffers for reduce operation */\n#define NEC_MAX_FOLD_N 16\n\n/*\n * We can set max communicator to anything here but ultimately we will be\n * limited by how much HW resources (such as TOP_SP semaphores or NX DRAM\n * space etc) get used up as number of communicators go up.\n */\n#define NEC_MAX_COMM_N 12 /* Max supported replica-groups in NEFF */\n\n#define NEC_MAX_NET_BUFFERS (2 * NEC_MAX_COMM_N) /* 2(hier & ring) x (# replica groups) */\n\n/*\n * We can set max communicator to anything here but ultimately we will be\n * limited by how much HW resources (such as TOP_SP semaphores or NX DRAM\n * space etc) get used up as number of communicators go up.\n */\n#define NEC_MAX_COMM_N 12 /* Max supported replica-groups in NEFF */\n\n#define NEC_MAX_NET_BUFFERS (2 * NEC_MAX_COMM_N) /* 2(hier & ring) x (# replica groups) */\n\n/*\n * We can set max communicator to anything here but ultimately we will be\n * limited by how much HW resources (such as TOP_SP semaphores or NX DRAM\n * space etc) get used up as number of communicators go up.\n */\n#define NEC_MAX_COMM_N 12 /* Max supported replica-groups in NEFF */\n\n#define NEC_MAX_NET_BUFFERS (2 * NEC_MAX_COMM_N) /* 2(hier & ring) x (# replica groups) */\n\n#define NEC_CACHE_LINE_SIZE 128\n\n/* Rank ID to denote network connector */\n#define NEC_NET_CONNECTOR_RANK -1\n/* MLA dev ID to denote network connector */\n#define NEC_NET_MLA_DEV -1\n/* MLA dev ID to denote POD connector */\n#define NEC_POD_MLA_DEV -2\n/* Rank ID to denote an unknown connector -> possibly not reachable */\n#define NEC_UNKNOWN_RANK -3\n/* MLA dev ID to denote an unknown connector -> possibly not reachable */\n#define NEC_UNKNOWN_MLA_DEV -3\n\n/* the number of hierarchical cc pipeline stage */\n#define NEC_HIER_CC_PIPELINE_STAGE_N (3)\n\n/* the max number of outgoing requests in the recv/send proxy */\n#define NCCL_NET_NEURON_MAX_REQUESTS 128\n\n/**\n * The maximum number of concurrent cc execution. As NCCL needs this\n * information, define the size in the common header file.\n */\n#define NEC_MAX_STREAM_N 4\n\n/**\n * The different types of ofi communicators that are in the netResources\n * object that is used in the recv/send proxy\n */\ntypedef enum ofi_comm_type {\n NET_SEND_COMM,\n NET_RECV_COMM,\n NET_RECV_LISTEN_COMM,\n LOCAL_RECV_COMM,\n LOCAL_SEND_COMM\n} ofi_comm_type_t;\n\nenum enc_comm_type {\n H_COMM_INTRA_ID = 0,\n H_COMM_INTER_ID = 1,\n H_COMM_MAX_ID\n};\n\n/**\n * Neuron Elastic Collectives (NEC)\n *\n * This is the main component for Neuron Elastic Collectives in Neuron Runtime\n * (NRT). This is to provide collective operations to applications offloaded by\n * the device including collective comm init, receiving (post) operations,\n * building resources for the operation, triggering the operation and polling\n * its completion.\n *\n * +-----------------------+\n * | Collectives App |\n * +-----------------------+\n * | Collectives Library |\n * +-----------------------+\n * | NEC / NRT |\n * +-----------------------+\n * | DEVICE |\n * +-----------------------+\n *\n * TODO: ENC will be renamed to NEC\n */\n\ntypedef enum nec_pod_type {\n NEC_POD_TYPE_NONE,\n NEC_POD_TYPE_P2P,\n NEC_POD_TYPE_SWITCH,\n NEC_POD_TYPE_INVALID\n} nec_pod_type_t;\n\ntypedef enum nec_pod_type {\n NEC_POD_TYPE_NONE,\n NEC_POD_TYPE_P2P,\n NEC_POD_TYPE_SWITCH,\n NEC_POD_TYPE_INVALID\n} nec_pod_type_t;\n\n/* Translated from what KaenaDriver returns */\ntypedef enum nec_pod_type {\n NEC_POD_TYPE_NONE,\n NEC_POD_TYPE_P2P,\n NEC_POD_TYPE_SWITCH,\n NEC_POD_TYPE_INVALID\n} nec_pod_type_t;\n\ntypedef struct enc_comm* nec_comm_t;\ntypedef struct enc_channel* nec_channel_t;\ntypedef uint64_t dma_addr_t;\n\nstruct enc_net_host_memory_index {\n union {\n volatile uint32_t index;\n char pad[NEC_CACHE_LINE_SIZE]; /* Avoid false-sharing */\n };\n};\n\n/**\n * Host memory structure for network transport\n *\n * The proxy-thread progress function first waits for the device to be ready by\n * polling host index on fold 0 until it is (-1). Once (-1) was polled, the\n * proxy-thread resets the host index to 0 and notify the device that the\n * proxy-thread is ready by incrementing the handshake semaphore by 1.\n *\n * On the sender side, the device increase the host index to post a buffer to\n * send to a remote device. The proxy-thread send progress function polls the\n * host index and send posted buffers to the respective remote device. The\n * proxy-thread polls for send requests completions and notifies the device on\n * these completions by increasing the send_complete semaphore by the amount of\n * completed send requests. The device may in response to this notification\n * increase the host index further to post additional buffers to send. The\n * proxy-thread recognize the last entry in the FIFO by the fact it is\n * specially marked (See mark_fifo_end())\n *\n * On the receiver side, the device increase the host index to post receive\n * buffers to be filled with data from a remote device. The proxy-thread recv\n * progress function polls the host index and post the receive buffers to the\n * network plugin. The proxy-thread polls for receive completions and notifies\n * the device on these completions by increasing the recv_complete semaphore by\n * the amount of completed recv requests. The device use this notification to\n * know that data is available for processing on the device memory. The device\n * may also in response to this notification increase the host index further to\n * post additional buffers to post as receive buffers. The proxy-thread\n * recognize the last entry in the FIFO by the fact it is specially marked.\n *\n * For the ring algorithm:\n * The sender's handshake and send_complete semaphores\n * are the send-credit semaphore.\n * The receiver's handshake and recv_complete semaphores are the recv-cnt\n * semaphore.\n *\n * For the mesh algorithm:\n * The handshake semaphore is the local-handshake event semaphore for both\n * sender and receiver.\n * The recevier's recv_complete semaphore is the broadcast event semaphore.\n * The sender's send_complete semaphore is the sync event semaphore.\n */\nstruct enc_net_host_memory {\n union {\n struct {\n struct enc_net_host_memory_index post_recv[NEC_MAX_FOLD_N];\n } recv;\n struct {\n struct enc_net_host_memory_index post_send[NEC_MAX_FOLD_N];\n } send;\n };\n};\n\ntypedef struct enc_host_mem {\n void *mem_handle;\n void *va;\n dma_addr_t pa;\n size_t size;\n} enc_host_mem_t;\n\n\ntypedef struct enc_host_mem_shared {\n enc_host_mem_t mem;\n int refcnt;\n} enc_host_mem_shared_t;\n\n/**\n * Network connector structure containing allocated resources for network transport\n */\nstruct enc_net_connector {\n int fold_n;\n\n enc_host_mem_t net_host_mem; /* Used to signal proxy thread */\n enc_host_mem_shared_t *dynamic_input_host_mem; /* Used to pass info only available during execution */\n\n /* Network transport buffer, allocated only for sender */\n void *devmem_res;\n void *nccl_mhandle;\n\n /* Address and mhandle for event semaphores and pre-registered buffers */\n void *inc_recv_sem_nccl_mhandle;\n uint32_t *inc_recv_sem_values_buffer;\n void *inc_recv_sem_values_buffer_mhandle;\n\n /*\n * NCCL network connector data structure. When one proxy worker is used for\n * the same type (recv or send) network operation, connector information\n * should be included in each transaction.\n */\n void *nccl_connector;\n};\n\ntypedef enum enc_pattern {\n ENC_PATTERN_RING,\n ENC_PATTERN_MESH,\n ENC_PATTERN_INVALID,\n} enc_pattern_t;\n\ntypedef enum enc_net_connectivity {\n ENC_CONNECTIVITY_MESH,\n ENC_CONNECTIVITY_RDH,\n ENC_CONNECTIVITY_DEFAULT\n} enc_net_connectivity_t;\n\nstruct enc_channel {\n /*\n * Application parameters for init\n */\n int id;\n enc_pattern_t pattern;\n\n /* Applicable only in case of remote neighbor */\n struct enc_net_connector *net_recv; /* if receving from rank over the network */\n struct enc_net_connector *net_send; /* if sending to rank over the network */\n\n /*\n * Neuron Runtime context\n */\n void *devmem_res;\n void *two_step_pod_mesh_devmem_res;\n /* Gateway buffer is allocated only when hybrid ring is supported */\n void *devmem_gw_buf_res;\n void *nccl_mhandle;\n\n dma_addr_t gw_recv_buffer;\n dma_addr_t gw_send_buffer;\n\n struct enc_channel_context *ch_ctx;\n struct encd_dma_channel *drv_channel;\n};\n\nstruct enc_peer_info {\n int neuron_dev;\n int rid;\n int tpb_index;\n int pod_node_id;\n};\n\ntypedef enum enc_topology_mode {\n ENC_TOPO_NULL = 0,\n ENC_TOPO_4_DEVS_IN_ROW,\n ENC_TOPO_4_DEVS_IN_COLUMN,\n} enc_topology_mode_t;\n\nstruct enc_comm_info {\n int neuron_dev;\n int rank;\n int rank_n;\n int local_rank_n;\n int local_rack_rank_n;\n\n int node;\n int node_n;\n\n enc_topology_mode_t enc_topo_mode;\n\n /* Pod information received from NCCL */\n bool enable_pod;\n bool use_net; /* Whether network interface is used or not with the communicator */\n int pod;\n int pod_n;\n int pod_node;\n int pod_node_n;\n\n struct enc_peer_info *peers;\n};\n\nstruct enc_ring {\n int prev;\n int next;\n int *user_ranks;\n\n /* used by one_rank_per_device rings only */\n bool duplicate;\n};\n\n/* Kangaring */\n#define NEC_KANGARING_MAX_NUM_RANKS (256)\n#define KANGARING_NUM_SENG_PER_DEV (4)\n#define KANGARING_NUM_TPB_PER_DEV (8)\n#define KANGARING_MAX_SECONDARIES (3)\n\nenum SEngine {\n S0 = 0,\n S1 = 1,\n S2 = 2,\n S3 = 3,\n SENGS_PER_DIE = 2,\n SENGS_PER_MLA = 4\n};\n\nstruct enc_kangaring {\n int vnc; // virtual neuron core size\n int logical_path[NEC_KANGARING_MAX_NUM_RANKS]; // the logical kangaring path: p0 s0 p1 s1 ...\n int prev; // upstream\n int next; // downstream\n int port; // port to go to next\n\n /* In VNC 2 case, this is the only peer. For primary ranks, it refer to their secondary rank;\n * for secondary ranks, this refer to their primary rank.\n * In VNC 1 case, it refers specifically to the peer over rmtv with the same tpb index.\n */\n int peer_rmtv;\n /* In VNC 1 case, we have these 2 additional peers.\n * peer_over_rmtv2 refers to the peer over rmtv with a different tpb index.\n * peer_local refers to the local peer with a different tpb index\n */\n int peer_rmtv2;\n int peer_local;\n\n int next_peer_rmtv; // next's peer over rmtv\n\n bool is_primary; // is self rank on data path?\n bool is_next_pcie; // is next primary reached via pcie or d2d?\n bool duplicate; // is this a duplicate channel?\n bool pattern2; // is pattern 2?\n};\n\ntypedef enum metaring_type {\n RING,\n KANGARING,\n SINGLE_CYCLE_RING,\n RDH,\n INVALID_METARING\n} metaring_type_t;\n\nstruct enc_alg_metaring {\n int channel_n;\n struct enc_channel channels[NEC_MAX_CHANNELS];\n\n struct enc_ring ring_ranks[NEC_MAX_CHANNELS];\n struct enc_kangaring kangaring_ranks[NEC_MAX_CHANNELS];\n metaring_type_t type;\n\n /* Does the group contain only on rank per device? This variable is set to true when NCCL\n * returns device level H-cycles to runtime. In this case, we will parse that device H-cycle\n * and generate ring paths on runtime side. We do this because we need to enforce certain\n * pre-defined patterns in the paths so that we avoid dead locks between concurrent groups.\n */\n bool one_rank_per_device;\n /* Hybrid ring is supported when RG have 4 H-cycles of one_rank_per_device */\n bool is_hybrid_ring;\n bool tokens_exchanged; /* reinitialzed tokens from old metaring config*/\n bool deadlock_free_rank_list;\n\n struct enc_comm *comm; /* Backward reference to ENC comm */\n struct encd_alg_metaring *drv_alg;\n\n /* For use by src/tgt pairs only */\n bool skip_send;\n bool skip_recv;\n};\n\n/*\n * The order of the events matter here, so while adding a new event make sure the event is added\n * to the right section of the list\n * \n * ENC_COMMON_NUM_EVENT_TYPE: contains all common events between RDH-Mesh or A2A-mesh\n * ENC_MESH_NUM_EVENT_TYPE-ENC_COMMON_NUM_EVENT_TYPE: contains events used by mesh\n * ENC_A2A_NUM_EVENT_TYPE-ENC_MESH_NUM_EVENT_TYPE: contains events used by A2A only\n * ENC_RDH_NUM_EVENT_TYPE-ENC_A2A_NUM_EVENT_TYPE: contains events used by RDH only\n *\n */\ntypedef enum enc_mesh_event_type {\n EVT_SYNC,\n EVT_GLOBAL_HNDSHK,\n EVT_LOCAL_HNDSHK,\n EVT_INTER_GRP_BRDCST,\n EVT_FUNCTION_BARRIER_FIRST_COLL,\n EVT_FUNCTION_BARRIER_LAST_COLL,\n EVT_REDUCE_LOCAL_HNDSHK,\n EVT_INTRA_GRP_BRDCST,\n ENC_COMMON_NUM_EVENT_TYPE,\n\n ENC_MESH_NUM_EVENT_START = ENC_COMMON_NUM_EVENT_TYPE,\n EVT_REDUCE_COPY = ENC_COMMON_NUM_EVENT_TYPE,\n EVT_REDUCE_COPY_2,\n EVT_REDUCE_WRITE,\n EVT_INTER_GRP_BRDCST_2,\n EVT_LOCAL_AND_POD_GRP_BRDCST,\n EVT_LOCAL_AND_POD_GRP_BRDCST_2,\n ENC_MESH_NUM_EVENT_TYPE,\n\n ENC_A2A_NUM_EVENT_START = ENC_MESH_NUM_EVENT_TYPE,\n EVT_LOCAL_HNDSHK_1 = ENC_MESH_NUM_EVENT_TYPE,\n EVT_LOCAL_HNDSHK_2,\n EVT_GLOBAL_HNDSHK_1,\n EVT_INTER_GRP_BRDCST_1,\n EVT_INTRA_GRP_BRDCST_1,\n EVT_2DEV_BRDCST,\n EVT_2DEV_HNDSHK,\n EVT_COPY_FROM_HOST,\n ENC_A2A_NUM_EVENT_TYPE,\n\n ENC_RDH_NUM_EVENT_START = ENC_A2A_NUM_EVENT_TYPE,\n EVT_RH_STEP_0 = ENC_A2A_NUM_EVENT_TYPE,\n EVT_RH_STEP_1,\n EVT_RH_STEP_2,\n EVT_RH_STEP_3,\n EVT_RH_STEP_4,\n EVT_RH_STEP_5,\n EVT_RH_STEP_6,\n EVT_RH_STEP_7,\n EVT_RH_STEP_8,\n EVT_RH_STEP_9,\n EVT_RDH_LOCAL_HANDSHAKE = EVT_RH_STEP_9,\n EVT_RDH_AXES_HANDSHAKE,\n EVT_RD_STEP_0,\n EVT_RD_STEP_1,\n EVT_RD_STEP_2,\n EVT_RD_STEP_3,\n EVT_RD_STEP_4,\n EVT_RD_STEP_5,\n EVT_RD_STEP_6,\n EVT_RDH_AXES_HANDSHAKE_2,\n EVT_1DEV_RDH_STEP_1,\n EVT_1DEV_RDH_STEP_2,\n EVT_1DEV_RD_STEP_1,\n EVT_1DEV_RD_STEP_2,\n EVT_1DEV_RH_STEP_1,\n EVT_2DEV_RD_STEP_0,\n EVT_2DEV_RD_STEP_1,\n EVT_2DEV_RD_STEP_2,\n EVT_2DEV_RD_STEP_3,\n EVT_2DEV_RD_STEP_4,\n EVT_RDH_LOCAL_PEER_HANDSHAKE,\n ENC_RDH_NUM_EVENT_TYPE // We assume each event is used only once\n // Enforced by encd_init_mesh_event()\n} enc_mesh_event_type_t;\n\n#define ENC_MESH_MAX_NUM_EVENTS 64\n\n#define KiB (1024)\n#define MiB (1024 * KiB)\n#define GiB (1024 * MiB)\n\nstruct enc_mesh_nbr_grp {\n int *ranks;\n int ranks_n;\n};\n\nstruct enc_mesh_event {\n struct enc_mesh_nbr_grp src_neighbor_grp;\n struct enc_mesh_nbr_grp dst_neighbor_grp;\n bool valid;\n enc_mesh_event_type_t evt_type;\n};\n\ntypedef enum enc_alg_mesh_type {\n ENC_ALG_FULL_MESH,\n ENC_ALG_GROUPED_MESH,\n ENC_ALG_MESH_TRN2,\n ENC_ALG_MESH_SWITCH,\n ENC_ALG_MESH_INVALID\n} enc_alg_mesh_type_t;\n\n/* TODO: In a separate commit we will change this to a cpp\n * file so we can have classes\n */\n#define ENC_MAX_OP_TYPES (13)\nstruct enc_alg_mesh_subtype {\n struct enc_mesh_event events[ENC_MESH_MAX_NUM_EVENTS];\n int num_events;\n struct encd_alg_mesh_subtype *drv_mesh;\n struct enc_alg_mesh *mesh; /* backward reference */\n size_t op_max_limit[ENC_MAX_OP_TYPES]; /* upper limit below which we will use mesh */\n size_t op_min_limit[ENC_MAX_OP_TYPES]; /* lower limit above which we will use mesh */\n size_t op_max_limit_sbuf[ENC_MAX_OP_TYPES]; /* upper limit below which we will use mesh for 2D tensors */\n size_t op_min_limit_sbuf[ENC_MAX_OP_TYPES]; /* lower limit above which we will use mesh for 2D Tensors */\n bool no_inplace_support;\n bool is_use_chnl_buffer; /* Whether channel bufer will be used or not */\n bool is_rdh;\n bool is_single_step_mesh;\n bool is_two_step_pod_mesh;\n bool is_latency_opt;\n bool is_bw_opt;\n bool is_rmv_dst_routing;\n uint32_t alltoall_iteration;\n};\n\n#define ENC_MAX_MESH_SUBTYPES (20)\n#define ENC_MESH_MAX_NUM_DEVICES (128)\n\nstruct enc_alg_mesh {\n enc_alg_mesh_type_t mesh_type;\n\n union {\n struct {\n uint32_t devid_to_rankid[ENC_MESH_MAX_NUM_DEVICES];\n /* Whether it is a single or a multi chip mesh */\n bool is_multi_chip;\n } trn2;\n struct {\n int num_non_net_node_local_groups;\n } trn1;\n struct {\n bool root_rank;\n int num_intra_group_roots;\n int local_root_ids[ENC_MESH_MAX_NUM_DEVICES];\n int global_root_ids[ENC_MESH_MAX_NUM_DEVICES];\n } inf2;\n };\n int group_id;\n int num_groups;\n /* Mesh uses only a single channel */\n struct enc_channel channel;\n struct enc_alg_mesh_subtype mesh_subtype[ENC_MAX_MESH_SUBTYPES];\n\n /* Holds maximum amt of data a single group is allowed to deposit into\n * the channel buffer. The definition of a group varies by platform type.\n * On TRN1, TRN2 a group currently consists of all or some ranks from a\n * single chip but on INF2 it refers to a collection of chips. The concept\n * of a group exists to avoid traffic replication on the wire by combining\n * input data from multiple ranks within a group before sending it outside\n * of the group. Therefore at the destination side we only receive a single\n * chunk of data per group.\n */\n size_t max_chbuf_space_per_group;\n /* Valid only for TRN2. For TRN2 to prevent AXI deadlock we avoid on-chip\n * routing at the destination chip and deposit data in the HBM closest to\n * the entry port. So the rank owning that HBM receives data on behalf of\n * other ranks on that same chip. This is why we need to carve out dedicated\n * channel buf space for each of the other s-engines on the same chip.\n */\n size_t max_chbuf_space_per_seng;\n /* Valid only for single step mesh where we directly copy the entire input\n * buffer into another rank's channel buffer.\n */\n size_t max_chbuf_space_per_rank;\n\n /* Whether to use double buffer to skip global handshake */\n bool double_buffer;\n\n /* Whether to build RDH */\n bool build_rdh;\n bool rdh_double_buffer;\n void *rdh_devmem_res; /*intra rdh channel buffer */\n bool use_2dev_proxy;\n\n bool tokens_exchanged; /* reinitialzed tokens from old mesh config*/\n\n bool use_net; /* Whether inter-node mesh with network proxy is used or not */\n\n /* Backward references to NCCL comm and general cluster info.\n * These might come from enc_comm or enc_alg_hier\n */\n struct enc_nccl_comm_node *nccl_comm_node; /* Reference to NCCL comm */\n struct enc_comm_info *ci; /* General cluster information */\n\n struct enc_comm *comm; /* Backward reference to ENC comm */\n struct encd_alg_mesh *drv_alg;\n\n /*\n * DMA mapped memory to host dedicated for A2Av metadata available only during\n * execution.\n */\n enc_host_mem_t alltoallv_host_input;\n};\n\nstruct enc_alg_hier {\n struct {\n struct enc_nccl_comm_node *nccl_comm_node;\n struct enc_comm_info ci;\n\n struct enc_alg_metaring ring;\n struct enc_alg_metaring kangaring;\n struct enc_alg_mesh mesh;\n } intra;\n\n struct {\n struct enc_nccl_comm_node *nccl_comm_node;\n struct enc_comm_info ci;\n\n struct enc_alg_metaring ring;\n struct enc_alg_metaring rdh;\n struct enc_alg_mesh mesh;\n } inter;\n\n struct {\n struct {\n struct enc_nccl_comm_node *nccl_comm_node;\n struct enc_comm_info ci;\n\n struct enc_alg_metaring ring;\n } stage[NEC_HIER_CC_PIPELINE_STAGE_N];\n } pipeline;\n\n void* devmem_res; /* Hierarchical Reduce Scatter uses intermediate buffer */\n\n struct enc_comm *comm; /* Backward reference to ENC comm */\n struct encd_alg_hier *drv_alg;\n};\n\n/**\n * Comm info to query from NCCL\n */\ntypedef struct nccl_comm_info {\n /* General cluster information */\n uint64_t cluster_id; // randomly generated id used to identify unique clusters in log metrics\n time_t epoch; // the epoch of the initial barrier at the start of a collectives execution. used when generating core dumps so that all ranks agree on a datetime.\n\n int neuron_dev;\n int rank;\n int rank_n;\n int local_rank_n;\n int local_rack_rank_n;\n\n int node;\n int node_n;\n\n bool enable_pod;\n bool use_net; /* Whether network interface is used or not with the communicator */\n int pod;\n int pod_n;\n int pod_node;\n int pod_node_n;\n\n struct enc_peer_info *peers; /* Needs to be allocated before calling ncclGetCommInfo() or NULL if peers info is not needed */\n\n /* Ring algorithm information */\n int channel_n;\n struct enc_ring rings[NEC_MAX_CHANNELS];\n\n /* Kangaring algorithm information */\n int kangaring_channel_n;\n int* kangaring_paths[NEC_MAX_CHANNELS];\n\n /* Hamiltonian cycles of MLAs, used to construct 1-rank-per-mla rings */\n int mla_cycle_n;\n int* mla_cycles[NEC_MAX_CHANNELS];\n} nccl_comm_info_t;\n\ntypedef struct enc_nccl_comm_node {\n void *nccl_comm;\n char *key;\n size_t key_sz;\n /* Tracking the graph information in the nccl_comm. We can use\n * ncclGetCommInfo() but it's expensive. Instead, simply track the graph\n * information here. This flag can only changed from true to false. The\n * other way is not possible.\n */\n bool disable_graph;\n bool global_nccl_comm_node;\n int refcnt;\n uint32_t stream_id;\n uint32_t context_id;\n\n uint32_t num_local_participants;\n uint32_t num_local_leaders;\n uint32_t my_local_leader;\n uint32_t *local_participants;\n uint32_t *local_leaders;\n struct bp_barrier *local_barrier;\n bool intra_pod_interface; /* When intra-pod interface is used, we can't skip exeuction barrier */\n} enc_nccl_comm_node_t;\n\n/* Neuron Device information. This data structure is used to send the device information from NRT to\n * nccom for nccl communicator building.\n */\n#define ENC_PROXY_HISTOGRAM_OUTPUT_PATH_LENGTH_MAX (128)\ntypedef struct enc_proxy_histogram_config {\n bool enable;\n size_t bucket_usecs;\n size_t num_buckets;\n size_t per_neff_warmup;\n size_t warmup;\n char output_path[ENC_PROXY_HISTOGRAM_OUTPUT_PATH_LENGTH_MAX];\n} enc_proxy_histogram_config_t;\n \ntypedef struct enc_neuron_device_info {\n int nec_dev_id;\n int mla_idx;\n int tpb_idx;\n int host_device_id;\n int routing_id;\n uint64_t pod_id;\n nec_pod_type_t pod_type;\n uint32_t pod_node_id;\n uint32_t virtual_server_id;\n enc_proxy_histogram_config_t histogram_config;\n} enc_neuron_device_info_t;\n\n/**\n * Collective communicator corresponding to ncclComm structure\n *\n * enc_comm is the Collective Comm that holds all the necessary information to\n * execute an collective operation. This should be pre-set before operations are\n * posted mainly because of the topology information built upon physical\n * connectivity. Collective operations are executed on multiple channels and a\n * channel is a path for data transfer along a pre-built topology.\n */\nstruct enc_comm {\n struct enc_nccl_comm_node *nccl_comm_node; /* Reference to NCCL comm */\n struct enc_comm_info ci; /* General cluster information */\n int id;\n int stream_id;\n\n /*\n * Algorithms\n */\n struct enc_alg_metaring ring;\n struct enc_alg_metaring kangaring;\n struct enc_alg_metaring rdh;\n struct enc_alg_hier hier;\n struct enc_alg_mesh mesh;\n\n /**\n * Use these handles to share network connector buffers across NEFFs.\n * Only used in global comm. Other comms will refer to the global comm to reuse them.\n * We use net_conn_count to sequentially assign these reservations to network conectors\n * to make sure:\n * 1) different comm in a NEFF don't reuse the same buffer (for multi-stream cases)\n * 2) for each NEFF, we always start with index 0 and go up for the most overlap and\n * reusability. We reset net_conn_count to 0 in enc_load_operations\n */\n int net_conn_count;\n void* net_connector_devmem_res[NEC_MAX_NET_BUFFERS];\n\n // TODO: nr_channel_chunks and chunk_size should not be a comm property anymore\n int nr_channel_chunks; /* Channel buffer depth, applies to all channels */\n size_t chunk_size; /* Unit of transfer, applies to all channels */\n\n struct encd_comm *drv_comm; /* Reference to driver comm */\n\n char topology[1024]; /* Used for debugging purposes only to print the topology in case of an error */\n};\n\n/**\n * Global communicator\n */\nstruct enc_glb_comm {\n uint32_t g_device_id; /* Same as comm->rank */\n uint32_t g_device_cnt; /* Same as comm->rank_n */\n uint32_t vtpb_idx;\n int nec_dev_id;\n int mla_idx;\n /* Absolute neuron device hw id. This is the ID that driver\n exposes neuron device on to host system aka OS. Neuron devices\n are expesed to RT by different ID in case docker remaps\n devices */\n int host_device_id;\n int routing_id;\n uint32_t virtual_server_id;\n nec_pod_type_t pod_type;\n uint32_t pod_node_id;\n uint32_t pod_sz;\n uint64_t pod_id;\n const char *root_comm_id; /* By getenv in nrt_config */\n bool check_sigs; /* By getenv in nrt_config */\n\n uint32_t *rank_nodes; /* The node index of each rank */\n uint32_t *local_ranks; /* The intra-node rank of each rank */\n\n enc_nccl_comm_node_t nccl_comm_node; /* nccl_comm node can be used by any stream */\n\n struct bananaphone *local_rings;\n struct bp_handle *local_peer_handles;\n\n /**\n * A set of buffers containing values that are used to\n * increment semaphores over efa transactions.\n */\n uint32_t *inc_recv_sem_values_buffer;\n size_t inc_recv_sem_values_buffer_size;\n\n struct enc_comm comm;\n\n /* TODO: manage all the devmem reservations in a single place\n * Today we share the buffers under the below path:\n * enc_glb_comm->comm->ring.channels[ring_channel_id].devmem_res\n * We need to move the above reservations and the one below to a\n * singleton class e.g. enc_glb_comm->devmem_res_pool\n */\n void* inter_rdh_devmem_res[NEC_MAX_STREAM_N];\n /* TODO: manage all the devmem reservations in a single place\n * this mem res is referred by comm->rdh.rdh_devmem_res\n */\n void* intra_rdh_devmem_res[NEC_MAX_STREAM_N];\n\n void* mesh_devmem_res_per_rg[NEC_MAX_STREAM_N * NEC_MAX_COMM_N * H_COMM_MAX_ID];\n\n void* rdh_devmem_res_per_rg[NEC_MAX_STREAM_N * NEC_MAX_COMM_N];\n\n void *gateway_devmem_res[NEC_MAX_STREAM_N][NEC_MAX_CHANNELS];\n\n pthread_mutex_t gcomm_setup_mtx;\n\n void *proxy_queue; // opaque pointer to enc_proxy_queue\n\n void *device_barrier_table;\n};\n\n/**\n * Network transport FIFOs\n *\n * Host send proxy should know the EFA buffer index, offset in the buffer and the size of\n * each data tranfer to send to remote device and recv proxy\n * needs destination addresses for each data from sender to submit network receive request.\n * Send and recv proxy should know when to report the completion of using\n * EFA buffer and complete is used to notify it.\n *\n * Such information is recorded when operation is loaded and becomes available on execution. Host\n * proxy uses these APIs to query the recorded FIFO.\n */\n\n/**\n * A net_ops_info_t entry corresponds to a set of smaller operations that are defined by multiple\n * net_src_addr_t and net_dest_addr_t. These sub operations can correspond to different types of\n * actions, so store a net_addr_mark_t identifier in each net_src_addr_t or net_dest_addr_t entry\n * to denote the purpose of the sub-operation.\n */\ntypedef enum net_addr_mark {\n NET_TRANSFER, /* Will drive data transfer over EFA */\n NET_OP_COMPLETE, /* Will mark final completion of a collective operation */\n EXEC_COMPLETE /* Will mark final completion of a collective load execution */\n} net_addr_mark_t;\n\ntypedef struct net_src_addr {\n uint32_t net_op_idx;\n int complete;\n dma_addr_t dev_addr;\n void *host_addr;\n void *nccl_mhandle;\n uint32_t size;\n net_addr_mark_t mark;\n void* proxy_histogram_tag;\n /* Fields below are for mesh only */\n int dst_rank;\n /* For local RDMA read */\n void *dst_addr;\n void *dst_mhandle;\n} net_src_addr_t;\n\ntypedef struct net_dest_addr {\n uint32_t net_op_idx;\n int complete;\n dma_addr_t dev_addr;\n void *host_addr;\n void *nccl_mhandle;\n uint32_t size;\n net_addr_mark_t mark;\n /* Fields below are for mesh only */\n int src_rank;\n} net_dest_addr_t;\n\ntypedef struct net_ops_info {\n uint16_t sema_shift_offset;\n bool early_send_completion;\n bool early_recv_posting;\n volatile uint32_t *inc_send_handshake;\n volatile uint32_t *inc_send_complete;\n volatile uint32_t *inc_recv_handshake;\n volatile uint32_t *inc_recv_complete;\n uint32_t tx_entry_cnt;\n uint32_t rx_entry_cnt;\n uint32_t net_idx_loop_size;\n uint32_t initial_send_credits;\n uint32_t ending_recv_credits;\n size_t data_type_sz;\n bool is_dynamic_send_recv_sz;\n bool variable_peer;\n bool add_to_histogram;\n /*\n * proxy uses this pointer to get connector information from transaction\n * saddr/daddr fifo entry of each operation.\n */\n void *enc_channel;\n} net_ops_info_t;\n\n/**\n * API for proxy-thread to increase handshake and send/recv semaphores by writing directly to the\n * memory mapped semaphore inc register.\n * For more information, see documentation on struct enc_net_host_memory definition.\n */\nvoid nec_inc_semaphore(volatile uint32_t *sem_inc_addr, uint32_t val);\n\n/**\n * API for proxy-thread to get dynamic send and offset for the case where message\n * size is determined by data only available during execution.\n */\nsize_t nec_get_dynamic_send_size_bytes(enc_host_mem_t *dyn_input, size_t data_type_sz, int dst_rank, int rank_n);\nsize_t nec_get_dynamic_send_offset_bytes(enc_host_mem_t *dyn_input, size_t data_type_sz, int dst_rank, int rank_n);\nsize_t nec_get_dynamic_recv_offset_bytes(enc_host_mem_t *dyn_input, size_t data_type_sz, int src_rank, int rank_n);\nvoid nec_set_recv_size_bytes(enc_host_mem_t *dyn_input, size_t recv_size_bytes, size_t data_type_sz, int src_rank, int rank_n);\n\n/**\n * Qeury device information\n */\nint nec_get_device_count(int *available_devices_array, uint32_t array_size);\nint nec_get_device_pci_bdf(int neuron_dev, uint32_t *domain, uint32_t *bus_num, uint8_t *pci_slot, uint8_t *dev_func);\n\n/**\n * Query vcore size\n */\nNRT_STATUS nec_get_virtual_core_size(uint32_t *virtual_core_size);\n\ntypedef struct nec_version_info {\n uint64_t major;\n uint64_t minor;\n uint64_t patch;\n uint64_t maintenance;\n char git_hash[16];\n uint64_t compatibility_version;\n // Any new fields added needs to be here. The fields before this cannot be\n // changed to maintain backward compatibility\n uint8_t future_fields[];\n} nec_version_info_t;\n\nNRT_STATUS nec_get_version_info(nec_version_info_t *version_info);\n\nNRT_STATUS nec_build_port_and_rid_map(int local_nec_dev_id, int *mla_indexes, int *host_device_ids, int count);\n\nbool nec_is_mla_available(int local_nec_dev_id, int mla_idx);\n\nint nec_mla_idx_to_rid(int local_nec_dev_id, int mla_idx);\n\nint nec_rid_to_mla_idx(int local_nec_dev_id, int rid);\n\nint nec_get_peer_mla_idx(int local_nec_dev_id, int mla_idx, int port);\n\nint nec_get_p2p_pod_peer_node(uint32_t nec_dev_id, int node, uint32_t port_distance,\n int *peer_node);\n\nNRT_STATUS nec_pod_node_can_access_peer_node(nec_pod_type_t pod_type,\n uint32_t local_rid, uint32_t local_node_id,\n uint32_t remote_rid, uint32_t remote_node_id,\n int *can_access_peer);\n\nvoid nec_ndl_printk(char *str, uint32_t size, uint32_t action);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \n#include \n#include \n// Use quoted includes in nrt headers including other nrt headers. Most clients\n// (ptxla, jax, etc.) build with bazel, and bazel has issue with angle-brackets.\n// See https://bazel.build/docs/bazel-and-cpp#include-paths for details.\n#include \"nrt/nrt_status.h\"\n#include \"ndl/neuron_driver_shared_tensor_batch_op.h\"\n\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n/** Major and minor version of runtime. */\n#define NRT_MAJOR_VERSION 2\n#define NRT_MINOR_VERSION 0\n\ntypedef struct nrt_model nrt_model_t;\n\ntypedef struct nrt_tensor nrt_tensor_t;\n\ntypedef struct nrt_cc_context nrt_cc_context_t;\n\n/**\n * WARNING: Do not change the value of existing enums!\n * These values will be used by libnrt consumers, we\n * cannot change the defines under them, only append.\n */\ntypedef enum {\n NRT_TENSOR_PLACEMENT_DEVICE,\n NRT_TENSOR_PLACEMENT_HOST,\n NRT_TENSOR_PLACEMENT_VIRTUAL,\n} nrt_tensor_placement_t;\n\ntypedef enum {\n NRT_FRAMEWORK_TYPE_INVALID = 0, // Invalid\n NRT_FRAMEWORK_TYPE_NO_FW = 1, // Framework less execution\n NRT_FRAMEWORK_TYPE_TENSORFLOW, // Tensorflow\n NRT_FRAMEWORK_TYPE_PYTORCH, // Pytorch\n NRT_FRAMEWORK_TYPE_MXNET, // Mxnet\n NRT_FRAMEWORK_TYPE_PRECHECK, // Neuron Node Precheck\n} nrt_framework_type_t;\n\nenum {\n NRT_INSTANCE_UNKNOWN = 0,\n NRT_INSTANCE_INF1 = 1,\n NRT_INSTANCE_TRN1 = 2,\n NRT_INSTANCE_TRN1N = 3,\n NRT_INSTANCE_INF2 = 4,\n NRT_INSTANCE_TRN2 = 5,\n NRT_INSTANCE_TRN2N = 6,\n NRT_INSTANCE_INF2E = 7,\n NRT_INSTANCE_TRN2P = 8,\n NRT_INSTANCE_TRN2U = 9,\n NRT_INSTANCE_TRN2E = 10,\n NRT_INSTANCE_TRN2EU = 11,\n NRT_INSTANCE_TRN2AC = 12,\n NRT_INSTANCE_TRN2UAC = 13,\n NRT_INSTANCE_TRN3 = 14,\n NRT_INSTANCE_TRN3PDS98 = 15\n};\n\nenum {\n NRT_INSTANCE_SIZE_1XL,\n NRT_INSTANCE_SIZE_2XL,\n NRT_INSTANCE_SIZE_4XL,\n NRT_INSTANCE_SIZE_6XL,\n NRT_INSTANCE_SIZE_8XL,\n NRT_INSTANCE_SIZE_24XL,\n NRT_INSTANCE_SIZE_32XL,\n NRT_INSTANCE_SIZE_48XL,\n NRT_INSTANCE_SIZE_3XL,\n // Note: Add new sizes right above this line to prevent breaking backward compatibility\n\n NRT_INSTANCE_SIZE_UNKNOWN,\n NRT_INSTANCE_SIZE_NUM = NRT_INSTANCE_SIZE_UNKNOWN,\n};\n\ntypedef enum nrt_op_type {\n NRT_OP_ADD = 0x0,\n NRT_OP_FMA = 0x1,\n NRT_OP_MAX = 0x2,\n NRT_OP_MIN = 0x3,\n NRT_OP_INVALID = 0xF,\n} nrt_op_type_t;\n\ntypedef enum nrt_dtype {\n NRT_DTYPE_UNKNOWN = 0x0,\n NRT_DTYPE_INVALID = 0x0,\n NRT_DTYPE_FP8_E3 = 0xD,\n NRT_DTYPE_FP8_E4 = 0xE,\n NRT_DTYPE_FP8_E5 = 0xF,\n NRT_DTYPE_FLOAT16 = 0x7,\n NRT_DTYPE_BFLOAT16 = 0x6,\n NRT_DTYPE_FLOAT32 = 0xA,\n NRT_DTYPE_FP32R = 0xB,\n NRT_DTYPE_UINT8 = 0x3,\n NRT_DTYPE_UINT16 = 0x5,\n NRT_DTYPE_UINT32 = 0x9,\n NRT_DTYPE_UINT64 = 0x1,\n NRT_DTYPE_INT8 = 0x2,\n NRT_DTYPE_INT16 = 0x4,\n NRT_DTYPE_INT32 = 0x8,\n NRT_DTYPE_INT64 = 0xC,\n} nrt_dtype_t;\n\ntypedef enum nrt_cc_op_type {\n NRT_CC_ALLGATHER,\n NRT_CC_ALLREDUCE,\n NRT_CC_REDUCESCATTER\n} nrt_cc_op_type_t;\n\ntypedef struct nrt_instance_info {\n uint32_t family;\n uint32_t size;\n char arch_name[16];\n char device_revision[8];\n} nrt_instance_info_t;\n\nNRT_STATUS nrt_get_instance_info(nrt_instance_info_t *info, size_t instance_info_len);\n\n/** Initialize neuron runtime.\n *\n * @param framework[in] - Type of the framework.\n * @param fw_version[in] - Framework version as string. (eg 2.1)\n * @param fal_version[in] - Framework Abstraction Layer version as string.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_init(nrt_framework_type_t framework, const char *fw_version, const char *fal_version);\n\n/** Closes all the devices and cleans up the runtime state.\n */\nvoid nrt_close();\n\n/** Load given NEFF and place it in one or more neuron cores.\n *\n * @param neff_bytes[in] - Pointer to NEFF data.\n * @param size[in] - Length of the NEFF data.\n * @param vnc[in] - VNC index where the NEFF should be loaded(-1 means runtime would automatically load in first free VNC).\n * @param vnc_count[in] - DEPRECATED: always use -1\n * @param model[out] - Resulting model would be stored here.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_load(const void *neff_bytes, size_t size, int32_t vnc, int32_t vnc_count, nrt_model_t **model);\n\n/** Load given NEFF for collective operations and place it in one or more neuron cores.\n *\n * If global NCCL communicator was not previously created, we will create it inside this API with the assumption that\n * global device id is same as ctx_device_id and global device count is same as ctx_device_count.\n *\n * @param neff_bytes[in] - Pointer to NEFF data.\n * @param size[in] - Length of the NEFF data.\n * @param vnc[in] - VNC index where the NEFF should be loaded(-1 means runtime would automatically load in first free VNC).\n * @param vnc_count[in] - DEPRECATED: always use -1\n * @param ctx_device_id[in] - Device ID relative to the number of devices participating in this NEFF\n * @param ctx_device_count[in] - Number of devices participating in collectives operations in this NEFF\n * @param model[out] - Resulting model would be stored here.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_load_collectives(const void *neff_bytes, size_t size, int32_t vnc, int32_t vnc_count,\n uint32_t ctx_device_id, uint32_t ctx_device_count, nrt_model_t **model);\n\n/** Unload given model and free up device and host resources.\n *\n * @param model - Model to unload.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_unload(nrt_model_t *model);\n\n/** Get the number of VNCs used by a loaded model. (deprecated)\n *\n * @param model[in] - Model.\n * @param vnc_count[out] - The number of VNCs used by the model.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_model_nc_count(const nrt_model_t *model, uint32_t *vnc_count);\n\n/** Get the number of VNCs used by a loaded model. (deprecated)\n *\n * @param model[in] - Model.\n * @param vnc_count[out] - The number of VNCs used by the model.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_model_vnc_count(const nrt_model_t *model, uint32_t *vnc_count);\n\n/** Returns VirtualNeuronCores available in instance. (deprecated)\n *\n * @param vnc_count[out] - VirtualNeuronCores available in instance.\n *\n * @note This API can be called before nrt_init().\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_total_nc_count(uint32_t *vnc_count);\n\n/** Returns VirtualNeuronCores available in instance.\n *\n * @param vnc_count[out] - VirtualNeuronCores available in instance.\n *\n * @note This API can be called before nrt_init().\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_total_vnc_count(uint32_t *vnc_count);\n\n/** Returns VirtualNeuronCores visible to the application. (deprecated)\n *\n * @param vnc_count[out] - VirtualNeuronCores visible to the application.\n *\n * @note This API can be called before nrt_init().\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_visible_nc_count(uint32_t *vnc_count);\n\n/** Returns VirtualNeuronCores visible to the application.\n *\n * @param vnc_count[out] - VirtualNeuronCores visible to the application.\n *\n * @note This API can be called before nrt_init().\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_visible_vnc_count(uint32_t *vnc_count);\n\n/** A container to hold multiple tensors */\ntypedef void nrt_tensor_set_t;\n\n/** Allocates a new tensor set.\n *\n * @param result[out] - Pointer to newly allocated tensor set would be stored here.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_allocate_tensor_set(nrt_tensor_set_t **result);\n\n/** Destroys given tensor_set and frees memory.\n *\n * @param tensor_set[in] - Tensors set to be freed.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nvoid nrt_destroy_tensor_set(nrt_tensor_set_t **tensor_set);\n\n/** Add/replace given tensor to tensor set\n *\n * @param tensor_set[in] - Tensor set to which the tensor is added.\n * @param tensor_name[in] - Name of the tensor.\n * @param tensor[in] - Pointer to tensor. This pointer should be valid till nrt_destroy_tensor_set() is called.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_add_tensor_to_tensor_set(nrt_tensor_set_t *tensor_set, const char *tensor_name, nrt_tensor_t *tensor);\n\n/** Get a tensor's info from a tensor set.\n *\n * @param tensor_set[in] - Tensor set.\n * @param tensor_name[in] - Name of the tensor.\n * @param tensor[out] - Pointer to tensor would be stored here.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_tensor_from_tensor_set(nrt_tensor_set_t *tensor_set, const char *tensor_name, nrt_tensor_t **tensor);\n\n/** Execute given model with given inputs and collect outputs.\n *\n * @param model[in] - Model to execute.\n * @param input_set[in] - Set of input tensors.\n * @param output_set[in] - Set of output tensors.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_execute(nrt_model_t *model, const nrt_tensor_set_t *input_set, nrt_tensor_set_t *output_set);\n\n/** Execute given model with given inputs, repeat execution specified number of times and collect outputs.\n *\n * @param model[in] - Model to execute.\n * @param input_set[in] - Set of input tensors.\n * @param output_set[in] - Set of output tensors.\n * @param repeat_count[in] - Number of to repeat execution.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_execute_repeat(nrt_model_t *model, const nrt_tensor_set_t *input_set, nrt_tensor_set_t *output_set, int repeat_count);\n\n/** Build (initialize and setup) NCCL global communicator.\n *\n * @param vnc[in] - Local VNC (within the instance)\n * @param g_device_id[in] - Global device id\n * @param g_device_count[in] - Max world size of all neffs that will be executed\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_build_global_comm(int32_t vnc, uint32_t g_device_id, uint32_t g_device_count);\n\n/** Allocates a tensor that can be passed and used by a model for compute.\n *\n * @param tensor_placement[in] - Where the tensor would be allocated (device, host, or virtual memory)\n * @param vnc[in] - Virutal Neuron Core id to allocate the tensor on. Pass in -1 if allocating tensors on host memory.\n * @param size[in] - Size in bytes of the tensor to allocate.\n * @param name[in] - OPTIONAL. Name of the tensor.\n * @param tensor[out] - Pointer to newly created tensor will be stored here.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_allocate(nrt_tensor_placement_t tensor_placement, int vnc, size_t size, const char *name, nrt_tensor_t **tensor);\n\n/** Deallocates a tensor created by \"nrt_tensor_allocate\".\n *\n * @param tensor[in] - Deallocates given tensor.\n *\n * @return None\n */\nvoid nrt_tensor_free(nrt_tensor_t **tensor);\n\n/** Copies data from tensor to passed in buffer.\n *\n * @param tensor[in] - Tensor used to reference the tensor to read from.\n * @param buf[out] - Buffer used to store data read from the tensor.\n * @param offset[in] - Offset into the tensor to read from.\n * @param size[in] - Number of bytes to read.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_read(const nrt_tensor_t *tensor, void *buf, size_t offset, size_t size);\n\n/** Copies data from passed in buffer to tensor.\n *\n * @param tensor[in/out] - Tensor used to reference the tensor to write to.\n * @param buf[in] - Buffer used to store data to write to the tensor.\n * @param offset[in] - Offset into the tensor to write to.\n * @param size[in] - Number of bytes to write.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_write(nrt_tensor_t *tensor, const void *buf, size_t offset, size_t size);\n\n/** A batch of tensor operations on a single tensor */\n// the definition of nrt_tensor_batch_op_t is in neuron_driver_shared_tensor_batch_op.h\ntypedef struct nrt_tensor_batch {\n const nrt_tensor_t *tensor; // Tensor handle\n const nrt_tensor_batch_op_t *ops; // Array of operations for this tensor\n uint32_t num_ops; // Number of operations for this tensor\n} nrt_tensor_batch_t;\n\n/** Batch read data from multiple tensors.\n *\n * @param batches[in] - An array of batches, each of which describes operations on one tensor\n * @param num_batches[in] - Number of batches (tensors) in the array\n * @param unsafe[in] - If true, skip tensor tracking/blocking (use with caution)\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_read_batch(const nrt_tensor_batch_t *batches, uint64_t num_batches, bool unsafe);\n\n/** Batch write data to multiple tensors.\n *\n * @param batches[in] - An array of batches, each of which describes operations on one tensor\n * @param num_batches[in] - Number of batches (tensors) in the array\n * @param unsafe[in] - If true, skip tensor tracking/blocking (use with caution)\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_write_batch(const nrt_tensor_batch_t *batches, uint64_t num_batches, bool unsafe);\n\n/** Copies data between tensors.\n *\n * When copying between two device tensors, they must both be allocated on the SAME Neuron Core.\n * A NRT_INVALID will be returned in the failing case.\n *\n * @param src[in] - Tensor to copy from.\n * @param src_offset[in] - Offset into the source tensor to copy from.\n * @param dst[out] - Tensor to copy to.\n * @param dst_offset[in] - Offset into the destination tensor to copy to.\n * @param size[in] - Number of bytes to copy.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_copy(const nrt_tensor_t *src, size_t src_offset, nrt_tensor_t *dst, size_t dst_offset, size_t size);\n\n/** Gets the size of the passed in tensor.\n *\n * @param tensor[in] - Tensor used to reference the tensor to get size of.\n *\n * @return Size of the tensor.\n */\nsize_t nrt_tensor_get_size(const nrt_tensor_t *tensor);\n\n/** Set the memory + offset pointed to by tensor to value\n *\n * @param tensor[in] - allocated tensor\n * @param offset[in] - offset within the tensor\n * @param value[in] - value to set with\n * @param size[in] - size of memory to set\n *\n * @return 0 on success.\n */\nNRT_STATUS nrt_tensor_memset(nrt_tensor_t *tensor, uint64_t offset, int value, size_t size);\n\n/** Allocates an empty tensor, i.e. the tensor structure w/o any attached storage\n *\n * @param name[in] - OPTIONAL. Name of the tensor.\n * @param tensor[out] - Pointer to newly created tensor will be stored here.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_allocate_empty(const char *name, nrt_tensor_t **tensor);\n\n/** Attaches caller supplied buffer to a tensor. Any storage previously attached to the tensor is detached\n * and freed if was owned by the tensor.\n * The buffer is supplied by the caller and must persist through the entire lifetime of the tensor.\n *\n * @param tensor[in] - Tensor\n * @param buffer[in] - Caller supplied buffer to use as tensor's storage\n * @param size[in] - Buffer Size\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_tensor_attach_buffer(nrt_tensor_t *tensor, void *buffer, size_t size);\n\n/** Creates a tensor to point to a slice of another tensor\n * does not do a deep copy, just points the \"slice\" tensor storage to the \"source\" tensor storage\n *\n * @param tensor_source[in] - Tensor to point at\n * @param offset[in] - Offset from the beginning of the source tensor to point at\n * @param size[in] - Size of the slice\n * @param name[in] - Optional name for the new tensor\n * @param tensor_slice[in] - Newly allocated tensor to point to the storage of the source tensor\n *\n */\nNRT_STATUS nrt_tensor_allocate_slice( const nrt_tensor_t *tensor_source, size_t offset, size_t size, const char *name, nrt_tensor_t **tensor_slice);\n\n/** Given a tensor get the virtual address.\n *\n * @param tensor[in] - Tensor for which the VA needs to be obtained\n *\n * @return va on success, NULL on failure.\n */\nvoid *nrt_tensor_get_va(const nrt_tensor_t *tensor);\n\n/** Returns on device allocation info for a tensor\n *\n * @param tensor[in] - Tensor for which the information needs to be obtained\n * @param alloc_info[out] - On device allocation information\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\ntypedef struct nrt_tensor_device_allocation_info {\n uint64_t physical_address; // physical address in device memory space\n size_t size; // allocation size, could be larger than the tensor size\n int hbm_index; // which of the HBMs the tensor is placed\n} nrt_tensor_device_allocation_info_t;\nNRT_STATUS nrt_tensor_get_device_allocation_info(const nrt_tensor_t *tensor, nrt_tensor_device_allocation_info_t *alloc_info);\n\n/**\n * @brief A Runtime API to check if a given output tensor is fully written/complete.\n * If timeout is given as unbounded, it emits a warning at the first 30 seconds.\n *\n * @param output_tensor: The given output tensor.\n * @param timeout: The maximum total duration to wait for tensor completion in microseconds.\n * If timeout is negative, the wait is unbounded. The caller is in charge of handling the timeout behaviors.\n * o/w, it checks completion until the timeout.\n * @param expected_completion_count: The number of completions expected by the caller.\n *\n * @return NRT_STATUS: It returns NRT_SUCCESS if the tensor is complete;\n * It returns NRT_INVALID, if the output tensor is given as NULL;\n * It returns NRT_TIMEOUT if the tensor is not reaching the expected_completion_count within the timeout.\n */\nNRT_STATUS nrt_tensor_check_output_completion(const nrt_tensor_t *output_tensor,\n int64_t timeout,\n uint64_t expected_completion_count);\n\n/**\n * @brief A Runtime API to reset the completion counter inside an output tensor to 0.\n *\n * @param output_tensor: The given output tensor.\n * @return NRT_STATUS: It returns NRT_SUCCESS if reset is successful;\n * It returns NRT_INVALID, if the output tensor is given as NULL.\n */\nNRT_STATUS nrt_tensor_reset_output_completion(nrt_tensor_t *output_tensor);\n\n/**\n * @brief Get the anonymous file-descriptor of dma-buf associated with\n * a Neuron device memory region if it was registered for EFA peer direct\n *\n * @param addr[in] - Device buffer virtual address\n * @param size[in] - Device buffer size (in bytes)\n * @param fd[out] - dma-buf fd\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_get_dmabuf_fd(uint64_t va, uint64_t size, int* fd);\n\n\n/** Get the host based device id from the device id presented to runtime (which may container based device id)\n * @param neuron_dev[in] - device id\n * @param host_device_id[out] - host device id\n * @return NRT_SUCCESS if call was successful, NRT_INVALID otherwise\n */\nNRT_STATUS nrt_host_device_id_get( int neuron_dev, uint32_t *host_device_id);\n\n/** Return array of routing IDs indexed by host device ID. This is the definitive routing ID mapping provided from the driver\n * @param coutn[in/out] - [in] number of entries in the mapping table provided. [out] count of entries returned\n * @param host_did_to_rid_map[in] - table/map of routing IDs indexed by host device ID\n * @return NRT_SUCCESS if call was successful, NRT_INVALID otherwise\n */\nNRT_STATUS nrt_host_device_id_rid_map_get(uint32_t *count, uint32_t *host_did_to_rid_map);\n\n/**\n * Get the HBM virtual address and size for a specific HBM index.\n * @param device_id[in] - Device ID\n * @param hbm_idx[in] - HBM index\n * @param addr[out] - Pointer to store the virtual address\n * @param size[out] - Pointer to store the size of the HBM region\n * @return NRT_SUCCESS if call was successful and HBM region was mapped\n * NRT_INVALID_HANDLE if there are no more HBM regions to map for this device\n * NRT_INVALID if the interface isn't supported or for invalid parameters\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_get_hbm_mmap_va(int device_id, int hbm_idx, void **addr, size_t *size);\n\n\ntypedef struct nrt_vnc_memory_stats {\n size_t bytes_used;\n size_t bytes_limit;\n // NOTE: For backward compatibility, when making updates, don't delete existing fields, and\n // ALWAYS add to the end of this struct!\n} nrt_vnc_memory_stats_t;\n\n/** Get the NRT memory stats for a VNC.\n *\n * @param vnc[in] - Local VNC (within the instance)\n * @param stats[out] - Pointer to a nrt_vnc_memory_stats struct\n * @param stats_size_in[in] - Caller expected size of the nrt_vnc_memory_stats struct, for compatibility purposes\n * @param stats_size_out[out] - Library written size of the nrt_vnc_memory_stats struct, for compatibility purposes\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\n\nNRT_STATUS nrt_get_vnc_memory_stats(uint32_t vnc, nrt_vnc_memory_stats_t *stats, size_t stats_size_in, size_t *stats_size_out);\n\n/** Get BDF of the EFA device attached to a Neuron device identified by VA of HBM allocation on that device\n *\n * @param va[in] - VA of a memory allocated on a Neuron devices\n * @param efa_bdf[out] - a buffer (of sufficient size) to store BDF of the connected EFA device\n * @param len[in/out] - in: length of buffer (including NULL), out: length of string (excluding NULL)\n *\n * @return NRT_SUCCESS on success\n * NRT_RESOUCE if the buffer is not large enough to store the BDF string\n * NRT_FAILURE for other errors\n */\n\nNRT_STATUS nrt_get_attached_efa_bdf(const void *va, char *efa_bdf, size_t *len);\n\n/******************************\n * Out-of-NEFF collectives *\n ******************************/\n\ntypedef struct nrt_cc_comm {\n uint32_t *replica_group; /* a list of participants */\n uint32_t rank; /* my rank in the replica_group */\n uint32_t rank_n; /* size of replica_group */\n\n uint32_t ctx_device_id;\n uint32_t ctx_device_count;\n uint32_t vnc;\n} nrt_cc_comm_t;\n\ntypedef struct nrt_tensor_list {\n nrt_tensor_t **tensors;\n size_t num_tensors;\n} nrt_tensor_list_t;\n\n/** Build (initialize and setup) global communicator for host-driven collective operations.\n *\n * @param vnc[in] - Local VNC (within the instance)\n * @param g_device_id[in] - Global device id\n * @param g_device_count[in] - Max world size of all participating workers\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_cc_global_comm_init(uint32_t vnc, uint32_t g_device_id, uint32_t g_device_count);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_async.h", "content": "/*\n * Copyright 2025, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \n#include \n#include \n// Use quoted includes in nrt headers including other nrt headers. Most clients\n// (ptxla, jax, etc.) build with bazel, and bazel has issue with angle-brackets.\n// See https://bazel.build/docs/bazel-and-cpp#include-paths for details.\n#include \"nrt/nrt.h\"\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n// execution units\ntypedef enum {\n NRTA_XU_TENSOR_READ = 0,\n NRTA_XU_TENSOR_WRITE,\n NRTA_XU_TENSOR_OP, // For tensor ops other than read and write\n NRTA_XU_COMPUTE,\n NRTA_XU_COLLECTIVES,\n\n // For new XU types, must only add after existing ones\n NRTA_XU_TYPE_NUM\n} nrta_xu_t;\n\n\n// nrta_seq_t's are monotomically increasing ids of executions\n// The first 16 bits are a Execution Unit ID, while the last\n// 48 bits are a strictly ordered Sequence Number\ntypedef uint64_t nrta_seq_t;\ntypedef uint16_t nrta_xu_id_t;\n\n#define NRTA_SEQ_NUM_MAX ((1ull << 48) - 1)\n#define NRTA_SEQ_NUM_MASK NRTA_SEQ_NUM_MAX\n#define NRTA_SEQ_GET_SEQ_NUM(seq_id) (seq_id & NRTA_SEQ_NUM_MASK)\n#define NRTA_SEQ_GET_XU_ID(seq_id) (seq_id >> 48)\n\n\ntypedef struct nrta_error {\n nrta_seq_t seq_id;\n uint64_t error_code; // NRT_STATUS, but typed as uint64 to ensure consistent representation across compilers\n} nrta_error_t;\nstatic_assert(sizeof(nrta_error_t) == 16, \"nrta_error_t must be of size 16\");\n\n// data structure used to store errors encountered during execution\ntypedef struct nrta_error_tracker nrta_error_tracker_t;\n\n/** Enqueues a tensor write request. Copies the data from a host buffer to a\n * tensor allocated on a Neuron device. Uses TENSOR_WRITE execution unit based\n * on the LNC that allocated the tensor.\n *\n * @param tensor[in] - Destination tensor\n * @param buf[in] - Host buffer containing source data\n * @param offset[in] - Offset into the tensor\n * @param size[in] - Number of bytes to write\n * @param queue[in] - XU queue to use,\n * @param err[in] - error tracker\n * @param req_sequence[out] - Sequence number of the scheduled request\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_tensor_write(nrt_tensor_t *tensor,\n const void *buf,\n uint64_t offset,\n uint64_t size,\n int queue,\n nrta_error_tracker_t *err,\n nrta_seq_t *req_sequence);\n\n/** Enqueues a tensor read request. Copies the data from a tensor allocated on a Neuron device\n * to a host buffer. Uses TENSOR_READ execution unit based\n * on the LNC that allocated the tensor.\n *\n * @param buf[in] - Destination Host buffer\n * @param tensor[in] - Source tensor\n * @param offset[in] - Offset into the tensor\n * @param size[in] - Number of bytes to read\n * @param queue[in] - XU queue to use,\n * @param err[in] - error tracker\n * @param req_sequence[out] - Sequence number of the scheduled request\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_tensor_read(void *buf,\n nrt_tensor_t *tensor,\n uint64_t offset,\n uint64_t size,\n int queue,\n nrta_error_tracker_t *err,\n nrta_seq_t *req_sequence);\n\n/** Enqueues a tensor copy request. Copies data between two tensors allocated\n * on the same Logical Neuron Core. Uses TENSOR_OP execution unit.\n *\n * NOTE: the tensors must be allocated until the copy completes\n *\n * @param src[in] - Source tensor\n * @param src_offset[in] - Offset into the source tensor\n * @param dst[in] - Destination tensor\n * @param dst_offset[in] - Offset into the destination tensor\n * @param size[in] - Number of bytes to copy\n * @param queue[in] - XU queue to use\n * @param err[in] - error tracker\n * @param req_sequence[out] - Sequence number of the scheduled request\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_tensor_copy(nrt_tensor_t *src,\n uint64_t src_offset,\n nrt_tensor_t *dst,\n uint64_t dst_offset,\n uint64_t size,\n int queue,\n nrta_error_tracker_t *err,\n nrta_seq_t *req_sequence);\n\n/** Schedules an asynchronous request to execute a model with specified inputs\n * and outputs. Uses COMPUTE execution unit of an LNC of the loaded model.\n *\n * @param model[in] - The model to schedule for execution\n * @param input_set[in] - Set of input tensors for the model\n * @param output_set[in] - Set of tensors to receive the outputs\n * @param queue[in] - XU queue to use, must be 0\n * @param err[in] - error tracker\n * @param req_sequence[out] - Sequence number of the scheduled request\n *\n * @return NRT_SUCCESS on successful preparation, appropriate error code otherwise\n */\nNRT_STATUS nrta_execute_schedule(nrt_model_t *model,\n const nrt_tensor_set_t *input,\n nrt_tensor_set_t *output,\n int queue,\n nrta_error_tracker_t *err,\n nrta_seq_t *req_sequence);\n\n/** Prepares collective context and HW configuration needed for collectives operation.\n * Allocates a collective context handle that is returned to the caller\n * which is freed in the schedule thread post CC op execution.\n *\n * @param comm[in] - Communicator containing the replica group\n * @param input[in] - Input tensor list\n * @param output[out] - Output tensor list\n * @param dtype[in] - Data type of elements\n * @param op[in] - Reduction operation (e.g., SUM, MAX) if applicable\n * @param cc_op[in] - Collective operation (e.g., ALLREDUCE, ALLGATHER)\n * @param cc_ctx[out] - Collective context\n *\n * @return NRT_SUCCESS on successful preparation, appropriate error code otherwise\n */\nNRT_STATUS nrta_cc_prepare(nrt_cc_comm_t *comm,\n nrt_tensor_list_t *input,\n nrt_tensor_list_t *output,\n nrt_dtype_t dtype,\n nrt_op_type_t op,\n nrt_cc_op_type_t cc_op,\n nrt_cc_context_t **cc_ctx);\n\n/** Schedules an asynchronous request to execute collective operation\n *\n * @param cc_ctx[in] - Collective context\n * @param queue[in] - XU queue to use, must be 0\n * @param err[in] - error tracker\n * @param req_sequence[out] - Sequence number of the scheduled request\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrta_cc_schedule(nrt_cc_context_t **cc_ctx,\n int queue,\n nrta_error_tracker_t *err,\n nrta_seq_t *req_sequence);\n\n// completion status\n\n/** Checks completion status of a scheduled request\n *\n * @param seq[in] - Scheduled request sequence id\n * @param is_completed[out] - true if the request is completed, false otherwise\n *\n * @return NRT_SUCCESS if the request is completed, NRT_INVALID if the seq is not valid\n */\nNRT_STATUS nrta_is_completed(nrta_seq_t seq, bool *is_completed);\n\n\n/** Returns sequence number of the last completed request\n *\n * @param lnc[in] - LNC\n * @param xu[in] - XU\n * @param queue[in] - XU's queue\n * @param seq[out] - last completed sequence number\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_get_sequence(uint32_t lnc, nrta_xu_t xu, int queue, nrta_seq_t *seq);\n\n\n/** Returns a pollable file descriptor that is READABLE when the execution request\n * specified by seq is complete.\n *\n * Note that users should only use the `poll` family of functions and `close` on this file\n * descriptor. Any other FD function is invalid and can lead to undefined behavior.\n *\n * The file descriptor must be passed to `close` to free the handle once the handle is not\n * needed anymore.\n *\n * @param seq[in] - sequence to track completion\n * @param fd[out] - FD associate with the sequence.\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_get_completion_handle(nrta_seq_t seq, int *fd);\n\n\n/** Creates an error tracker list\n *\n * @param lnc_idx[in] - Logical Neuron Core this list will be used for\n * @param error_tracker[out] - Created list.\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_error_tracker_create(uint32_t lnc_idx, nrta_error_tracker_t **error_tracker);\n\n/** Frees an error tracker list\n *\n * @param error_tracker[in] - Error tracker list to free\n *\n */\nvoid nrta_error_tracker_destroy(nrta_error_tracker_t *error_tracker);\n\n/** Gets list of errors from error tracker list\n *\n * @param error_tracker[in] - Error tracker list to get errors from\n * @param list[out] - Array of errors obtained from teh error tracker\n * @param error_count[out] - Number of errors in the list\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrta_error_tracker_get_list(nrta_error_tracker_t *error_tracker, const nrta_error_t **list, size_t *error_count);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_async_sendrecv.h", "content": "#pragma once\n\n#include \"nrt/nrt.h\"\n#include \"nrt/nrt_status.h\"\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\ntypedef struct nrt_async_sendrecv_comm nrt_async_sendrecv_comm_t;\ntypedef struct nrt_async_sendrecv_request nrt_async_sendrecv_request_t;\n\n/**\n * Get the maximum number of async sendrecv communicators per logical neuron core\n *\n * @param num[out] - The maximum number of async sendrecv communicators per logical neuron core\n * @return NRT_SUCCESS on success\n * NRT_FAILURE for errors\n */\nNRT_STATUS nrt_async_sendrecv_get_max_num_communicators_per_lnc(int* num);\n\n/**\n * Get the maximum number of pending requests per async sendrecv communicator\n *\n * @param num[out] - The maximum number of pending requests per async sendrecv communicator\n * @return NRT_SUCCESS on success\n * NRT_FAILURE for errors\n */\nNRT_STATUS nrt_async_sendrecv_get_max_num_pending_request(int* num);\n\n/** Initialize asynchronous tensor send and receive on logical neuron core\n *\n * Logical neuron core ID is the absolute ID of the logical core on\n * the host machine. The ID is uneffected by device remapping via\n * docker and selection of visible logical cores.\n *\n * This function may only be called when runtime is initialized. This\n * function must have a matching call to nrt_async_sendrecv_close() before\n * nrt_close() is called.\n * This function returns error in case preceeding call to\n * nrt_async_sendrecv_close() on the logical neuron core returned error.\n *\n * @param lnc[in] - Logical neuron core ID on the current server\n * @return NRT_SUCCESS if logical core has been initialized successfully\n * NRT_FAILURE for errors\n */\nNRT_STATUS nrt_async_sendrecv_init(int lnc);\n\n/** Closes asynchronous tensor send and receive of logical neuron core and cleans up resources\n *\n * A call to this function must have a preceeding matching call to\n * nrt_async_sendrecv_init(). After this function was invoked, all sendrecv\n * communicators and requests associated with this logical neuron core\n * are closed and cannot be accessed anymore invoking functions with those\n * communicators or requests is regarded undefined behavior.\n * Cases where this function is called and one of the communicators is\n * not connected yet are considered an error. Cases where this\n * function is called and send or receive requests are still inflight\n * are considered an error.\n *\n * @param lnc[in] - Logical neuron core ID on the current server\n * @return NRT_SUCCESS if logical core has been closed successfully\n * NRT_FAILURE for errors\n */\nNRT_STATUS nrt_async_sendrecv_close(int lnc);\n\n/** Create send communicator\n *\n * Before send communicator can be used to initiate sending a tensor,\n * connection to receive communicator must be established. Use\n * function nrt_async_sendrecv_test_comm() to test whether connection is\n * established.\n * Async sendrecv for logical neuron core lnc must have been\n * initialized via call to nrt_async_sendrecv_init() before this function is\n * invoked.\n * This function is thread-safe.\n *\n * @param peer_ip[in] - IP adress of peer logical neuron core\n * @param peer_lnc[in] - Logical neuron core ID on the peer server\n * @param lnc[in] - Logical neuron core ID on the current server\n * @param send_comm[out] - Pointer to send communicator\n * @return NRT_SUCCESS if logical core has been created successfully\n * NRT_RESOURCE if the number of created communicators exceeds the limit of NRT_ASYNC_SENDRECV_MAX_NUM_COMMUNICATORS_PER_LNC\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_connect(const char* peer_ip, int peer_lnc, int lnc, nrt_async_sendrecv_comm_t** send_comm);\n\n/** Create receive communicator\n *\n * Before receive communicator can be used to initiate receiveing a tensor,\n * connection to receive communicator must be established. Use\n * function nrt_async_sendrecv_test_comm() to test whether connection is\n * established.\n * Async sendrecv for logical neuron core lnc must have been\n * initialized via call to nrt_async_sendrecv_init() before this function is\n * invoked.\n * This function is thread-safe.\n *\n * @param peer_ip[in] - IP adress of peer logical neuron core\n * @param peer_lnc[in] - Logical neuron core ID on the peer server\n * @param lnc[in] - Logical neuron core ID on the current server\n * @param recv_comm[out] - Pointer to receive communicator\n * @return NRT_SUCCESS if logical core has been created successfully\n * NRT_RESOURCE if the number of created communicators exceeds the limit of NRT_ASYNC_SENDRECV_MAX_NUM_COMMUNICATORS_PER_LNC\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_accept(const char* peer_ip, int peer_lnc, int lnc, nrt_async_sendrecv_comm_t** recv_comm);\n\n/** Test whether connection has been established\n *\n * @param comm[in] - The send or receive communicator\n * @param done[out] - True if connection to peer communicator is established\n * @return NRT_SUCCESS if test performed without error\n * NRT_INVALID_HANDLE if handle is invalid\n * NRT_TIMEOUT if the communicator fails to establish connection within time limit\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_test_comm(nrt_async_sendrecv_comm_t* comm, bool* done);\n\n/** Asynchronously send a tensor\n *\n * This is a non-blocking function.\n *\n * This function is thread-safe. This function is only allowed to be\n * invoked on a communicator that is sucessfully tested to be\n * connected via call to nrt_async_sendrecv_test_comm().\n *\n * @param tensor[in] - Tensor to receive to\n * @param offset[in] - Offset into the tensor to receive to\n * @param length[in] - Number of bytes to read\n * @param send_comm[in] - Send communicator\n * @param request[out] - Pointer to receive request\n * @return NRT_SUCCESS on success\n * NRT_INVALID_HANDLE if handle is invalid\n * NRT_RESOURCE if the number of pending requests exceeds the limit of NRT_ASYNC_SENDRECV_MAX_NUM_PENDING_REQUEST\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_send_tensor(nrt_tensor_t* tensor, size_t offset, size_t length, nrt_async_sendrecv_comm_t* send_comm, nrt_async_sendrecv_request_t** request);\n\n/** Asynchronously receive a tensor\n *\n * This is a non-blocking function.\n *\n * This function is thread-safe. This function is only allowed to be\n * invoked on a communicator that is sucessfully tested to be\n * connected via call to nrt_async_sendrecv_test_comm().\n *\n * @param tensor[in] - Tensor to receive to\n * @param offset[in] - Offset into the tensor to receive to\n * @param length[in] - Number of bytes to read\n * @param recv_comm[in] - Receive communicator\n * @param request[out] - Pointer to receive request\n * @return NRT_SUCCESS on success\n * NRT_INVALID_HANDLE if handle is invalid\n * NRT_RESOURCE if the number of pending requests exceeds the limit of NRT_ASYNC_SENDRECV_MAX_NUM_PENDING_REQUEST\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_recv_tensor(nrt_tensor_t* tensor, size_t offset, size_t length, nrt_async_sendrecv_comm_t* recv_comm, nrt_async_sendrecv_request_t** request);\n\n/** Test the completion status of a asynchronous request\n *\n * This function is thread-safe when invoked with different\n * requests. This function is not allowed to be invoked concurrently\n * by multiple threads with the same request at the same time. When\n * this function returned request to be completed, this function is\n * not allowed to be invoked again with the same request.\n *\n * @param request[in] - Request to test\n * @param done[out] - Whether the request has completed\n * @param size[out] - Number of bytes sent/received\n * @return NRT_SUCCESS on success\n * NRT_INVALID_HANDLE if handle is invalid\n * NRT_TIMEOUT if the request fails to complete data transfer within time limit\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_test_request(nrt_async_sendrecv_request_t* request, bool* done, size_t* size);\n\n/** Flush received messae to ensure full arrival in memory\n *\n * Ensure that received messages of successfully tested async sendrecv\n * receive operations prior to call to this function fully arrived in\n * memory after this function completes.\n *\n * @param lnc[in] - Receiving logical neuron core ID\n * @return NRT_SUCCESS if flush operation succeeded\n * NRT_FAILURE for other errors\n */\nNRT_STATUS nrt_async_sendrecv_flush(int lnc);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_experimental.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \n#include \n#include \"nrt/nrt_status.h\"\n#include \"nrt/nrt.h\"\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n\n/** Usage of a Tensor in the NEFF\n */\ntypedef enum nrt_tensor_usage {\n NRT_TENSOR_USAGE_INPUT = 0, // Tensor is used for ifmap\n NRT_TENSOR_USAGE_OUTPUT, // Tensor is used for ofmap\n} nrt_tensor_usage_t;\n\n#define NRT_TENSOR_NAME_MAX 256\n\ntypedef struct nrt_tensor_info {\n char name[NRT_TENSOR_NAME_MAX]; // Name of the tensor\n nrt_tensor_usage_t usage; // Type of the tensor\n size_t size; // Tensor size in bytes\n nrt_dtype_t dtype; // data type\n uint32_t *shape; // an array representing data shape\n uint32_t ndim; // the number of dimensions\n} nrt_tensor_info_t;\n\ntypedef struct nrt_tensor_info_array {\n uint64_t tensor_count; // Total number of tensors in the NEFF\n nrt_tensor_info_t tensor_array[]; // Array of tensor info\n} nrt_tensor_info_array_t;\n\n/* Function definition for async exec status callbacks */\ntypedef void (*NRT_ASYNC_EXEC_STATUS_CALLBACK)(void *params, uint32_t model_id, uint32_t vnc, uint64_t job_id, NRT_STATUS status);\n\n/** Return input/output tensor information for a given model.\n*\n* @param model[in] - Model for which tensor information needs to be extracted.\n* @param tensor_info[out] - Pointer to store the result.\n*\n* @return NRT_STATUS_SUCCESS on success.\n*/\nNRT_STATUS nrt_get_model_tensor_info(nrt_model_t *model, nrt_tensor_info_array_t **tensor_info);\n\n/** Return the instance count for this model handle (optimal number of concurrent threads that can call nrt_execute). (deprecated)\n*\n* @param model[in] - Model for the instance count needs to be returned.\n* @param instance[out] - Pointer to store the result.\n*\n* @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_model_instance_count(nrt_model_t *model, uint32_t *instance_count);\n\n\n/** Free input/output tensor information for a given model.\n*\n* @param tensor_info[in] - Pointer to store the result.\n*\n* @return NRT_STATUS_SUCCESS on success.\n*/\nNRT_STATUS nrt_free_model_tensor_info(nrt_tensor_info_array_t *tensor_info);\n\n/** Enable tracing for all VNCs visible to the app\n *\n * @param trace_mem[in] - collect memory allocation info\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_trace_start(bool trace_mem);\n\n/** Serialize all data and disable tracing\n *\n * @param filename[in] - filename to write to\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_trace_stop(const char *filename);\n\n/** temporary, to be removed. See comment in neuron_nccl.cc\n*/\nvoid *nrt_get_libnccl_net(int *err, char *err_msg, size_t err_msg_size);\n\n/** Structs to pass around ucode image info\n*/\ntypedef struct nrt_ucode_img {\n uint8_t *bin;\n size_t size;\n} nrt_ucode_img;\n\ntypedef struct nrt_ucode_info {\n nrt_ucode_img iram;\n nrt_ucode_img dram;\n} nrt_ucode_info;\n\n/** Specify pooling engine ucode iram and dram images that will get loaded by nrt_init().\n* To use this API, it MUST be called BEFORE nrt_init().\n* Swapping ucode after nrt_init() is NOT supported. Ucode images are only loaded once.\n* This API provides a temporary workaround for swapping ucode.\n*/\nNRT_STATUS nrt_set_pool_eng_ucode(const nrt_ucode_info *ucode_info);\n\n/** Copies data to memory mapped Neuron device memory\n*\n* @param dest[in] - Pointer to destination memory (mmaped device memory)\n* @param src[in] - Pointer to source memory\n* @param size[in] - Copy size\n*\n* @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_memcpy_to_device(void *dest, const void *src, size_t size);\n\n/** Register a return status callback to post exec status to when running in async exec mode.\n * Calling this multiple times will replace the previouly registered callback.\n *\n * @param callback[in] - Callback to post nrt exec status to for async execution.\n * @param params[in] - Params for the async exec thread to pass to the callback upon\n * execution completion. Can be NULL.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_register_async_exec_callback(NRT_ASYNC_EXEC_STATUS_CALLBACK callback, void *params);\n\n/** Implements a barrier by running a small all-reduce over all workers\n*\n* @param vnc[in] - local VNC (within the instance)\n* @param global_device_id[in] - global worker ID\n* @param global_device_count[in] - total number of workers\n*\n* @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_barrier(int32_t vnc, uint32_t g_device_id, uint32_t g_device_count);\n\n/** Perform all-rank AllGather\n*\n* @param vnc[in] - local VNC (within the instance)\n* @param g_device_id[in] - global worker ID\n* @param g_device_count[in] - total number of workers\n* @param rank_input_size[in] - input size\n* @param input[in] - ptr to input data from this rank\n* @param output[out] - ptr to output buffer of size (g_device_count*rank_input_size)\n*\n* @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_all_gather(int32_t vnc, uint32_t g_device_id, uint32_t g_device_count,\n uint32_t rank_input_size, void *input, void *output);\n\n/** Blocks caller until all queued executions on async worker thread are drained.\n *\n * @param vnc - VNC index to block on.\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_async_drain_queued_execs(int32_t vnc);\n\ntypedef struct nrt_model_info {\n uint32_t vnc;\n // additional fields can be added here in the future\n // do not remove previously added fields because it will cause\n // memory corruption if the caller was compiled using a different \n // version of this header.\n} nrt_model_info_t;\n/** Returns information about loaded model\n *\n * @param model [in] - the model\n * @param info [out] - the information about the model\n * @param info_size_in [in] - the size of the info structure (used for version control)\n * @param info_size_out [out] - the number of bytes written (for version control)\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_get_model_info(const nrt_model_t *model, nrt_model_info_t *info, size_t info_size_in, size_t *info_size_out);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_profile.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#include \"nrt/nrt.h\"\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n/** Enable profiling for a model\n *\n * @param model[in] - model to profile\n * @param filename[in] - output filename that will be used with nrt_profile_stop()\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_start(nrt_model_t *model, const char *filename);\n\n/** Collect results and disable profiling for a model\n *\n * @param filename[in] - output filename to save the NTFF profile to\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_stop(const char *filename);\n\n/** Options for continuous device profiling.\n *\n * Opaque struct used to preserve compatibility and enforce proper usage.\n * Use nrt_profile_continuous_options_set_* functions set options.\n * Default options:\n * - output_dir: \"./output\"\n *\n * Usage:\n * nrt_profile_continuous_options_t *options;\n * nrt_profile_continuous_options_allocate(&options);\n * nrt_profile_continuous_options_set_output_dir(options, \"./output\");\n */\ntypedef struct nrt_profile_continuous_options nrt_profile_continuous_options_t;\n\n/** Allocate memory for the nrt_profile_continuous_options_t struct and set all options to defaults.\n *\n * @param options[in] - pointer to a pointer to nrt_profile_continuous_options_t struct\n */\nNRT_STATUS nrt_profile_continuous_options_allocate(nrt_profile_continuous_options_t **options);\n\n/** Free up memory allocated for the options struct needed for continuous device profiling.\n *\n * @param options[in] - pointer to a nrt_profile_continuous_options struct\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_continuous_options_free(nrt_profile_continuous_options_t *options);\n\n/** Sets the output directory for results of continuous device profiling.\n *\n * The filename is set automatically.\n *\n * @param[in,out] options Pointer to the options struct.\n * @param[in] output_dir Path to the output directory.\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_continuous_options_set_output_dir(nrt_profile_continuous_options_t *options, const char *output_dir);\n\n/** @brief Start continuous device profiling.\n *\n * When continuous device profiling is started, profiling is enabled for every model but notifications\n * will only be serialized to disk when the user calls nrt_profile_continuous_save(). This gives\n * the user control over which profiles are saved to disk. When a profile is not saved, the overhead\n * of trace serialization and disk write is avoided. Continuous profiling is ideal for scenarios where you\n * only want to save profiles for specific executions. In this mode you do not need to call\n * nrt_profile_start() and nrt_profile_stop() because they are called internally. Continuous profiling\n * will not start if inspect device profiling is already enabled or async execution is enabled.\n *\n * @param options[in] - options to control continuous device profiling\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_continuous_start(nrt_profile_continuous_options_t *options);\n\n/** Save NTFF profile to disk for the latest model executed on requested NeuronCore.\n *\n * Output directory will be set according to the options passed into this function. The filenames of\n * NTFFs within the output directory are chosen automatically to avoid conflicts. Calling save does\n * not stop continuous profiling.\n *\n * @param vnc[in] - (start) NeuronCore id to collect profile for\n * @param options[in] - options to control continuous device profiling\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_continuous_save(uint32_t vnc, nrt_profile_continuous_options_t *options);\n\n/** Stops continuous device profiling.\n *\n * Calling stop does not save a profile.\n *\n * @return NRT_SUCCESS on success.\n */\nNRT_STATUS nrt_profile_continuous_stop();\n\n/* Begin tracing/profiling\n *\n * Users of this API must set options through environment variables:\n * \n * - NEURON_RT_INSPECT_ENABLE: Set to 1 to enable system and device profiles.\n * For control over which profile types are captured, use NEURON_RT_INSPECT_SYSTEM_PROFILE \n * and NEURON_RT_INSPECT_DEVICE_PROFILE.\n * - NEURON_RT_INSPECT_OUTPUT_DIR: The directory where captured profile data will be saved to.\n * Defaults to ./output.\n * - NEURON_RT_INSPECT_SYSTEM_PROFILE: Set to 0 to disable the capture of system profiles. \n * Defaults to 1 when NEURON_RT_INSPECT_ENABLE is set to 1.\n * - NEURON_RT_INSPECT_DEVICE_PROFILE: Set to 0 to disable the capture of device profiles.\n * Defaults to 1 when NEURON_RT_INSPECT_ENABLE is set to 1.\n * - NEURON_RT_INSPECT_ON_FAIL: Set to 1 to enable dumping of device profiles in case of an error \n * during graph execution. Defaults to 0.\n * \n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_inspect_begin();\n\n\n/* Stop tracing/profiling and dump profile data.\n * Does nothing if `duration` is given to nrt_inspect_begin() and already elapsed\n *\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_inspect_stop();\n\n\n/** @brief Options for nrt_inspect_begin_with_options API.\n *\n * Opaque struct used to preserve compatibility and enforce proper usage.\n * Use nrt_inspect_config_set_* functions to set options or \n * nrt_inspect_config_set_defaults to set use default options.\n *\n * Example Usage:\n * nrt_inspect_config_t *options;\n * nrt_inspect_config_allocate(&options);\n * nrt_inspect_config_set_output_dir(options, \"./output\");\n */\ntypedef struct nrt_inspect_config nrt_inspect_config_t;\n\n\n/** Allocate memory for the options structure which is needed to\n * start profiling using nrt_inspect_begin_with_options. This will set all options to defaults\n * \n * @param options[out] - pointer to a pointer to options nrt_inspect_config struct\n * \n */\nNRT_STATUS nrt_inspect_config_allocate(nrt_inspect_config_t **options);\n\n/** @brief all fields of the nrt_inspect_config structure to their default values.\n * \n * Default behavior after calling this function:\n * - Session ID: 1\n * - Output directory: \"./output\" (when not explicitly set)\n * - Activity types: All activity types enabled (system_profile, device_profile, host_memory, cpu_util)\n * - System trace: All NeuronCores and event types enabled for capture\n * - Inspect mode: Disabled (profiles not captured automatically)\n * - Inspect on failure: Disabled (profiles not captured on execution failures)\n * \n * @param options[in,out] - Pointer to an nrt_inspect_config structure.\n * \n * @return NRT_SUCCESS on success\n * \n * @note These default values set here are NOT influenced by the environment variables. \n * If you are using the environment variables to set the values you do not need to use this method \n * or any of the nrt_inspect_config_set_* functions.\n */\nNRT_STATUS nrt_inspect_config_set_defaults(nrt_inspect_config_t *options);\n\n/** Free up memory allocated for the options structure which is needed to\n * start profiling using nrt_inspect_begin_with_options\n * \n * @param options[in] - pointer to an options nrt_inspect_config struct\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_inspect_config_free(nrt_inspect_config_t *options);\n\n/**\n * @brief Sets the session ID for the nrt_inspect_config_t which is needed to\n * start profiling using nrt_inspect_begin_with_options\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] session_id Session ID to set.\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_inspect_config_set_session_id(nrt_inspect_config_t *options, int session_id);\n\n/**\n * @brief Sets the output directory for results of \n * profiling using nrt_inspect_begin_with_options\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] output_dir Path to the output directory. Must be a valid non-empty string \n * @return NRT_SUCCESS on success, NRT_INVALID for invalid parameters, NRT_RESOURCE for memory allocation failure.\n * \n * @note The function makes an internal copy of the string, so the caller\n * does not need to keep the original string alive.\n * @note Call nrt_inspect_config_free() to properly clean up allocated memory.\n */\nNRT_STATUS nrt_inspect_config_set_output_dir(nrt_inspect_config_t *options, const char *output_dir);\n\n/**\n * @brief Sets max number of system trace events that can be stored across all ring buffers\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] sys_trace_max_events_per_nc Max number of system trace events that can be stored across all ring buffers.\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_inspect_config_set_sys_trace_max_events_per_nc(nrt_inspect_config_t *options, uint64_t sys_trace_max_events_per_nc);\n\n/**\n * @brief Sets system trace capture enabled for a specific NeuronCore\n * ring buffers won't be allocated for disabled NeuronCores \n * \n * @param[in,out] options Pointer to the options structure.\n * @param[in] nc_idx Index of the NeuronCore.\n * @param[in] enabled Boolean value to enable or disable system trace capture.\n * @return NRT_SUCCESS on success\n */\nNRT_STATUS nrt_inspect_config_set_capture_enabled_for_nc(nrt_inspect_config_t *options, uint32_t nc_idx, bool enabled);\n\n/** \n * @brief Sets system trace capture enabled for a specific event type\n * can save memory and reduce output size\n * @param[in,out] options Pointer to the options structure.\n * @param[in] event_type Valid event types.\n * @param[in] enabled Capture enabled flag.\n * @return NRT_SUCCESS on success\n * \n * @note Event type must be a string from the list of supported event types. To get the list of supported event types, \n * use nrt_sys_trace_get_event_types in the nrt_sys_trace.h header file.\n */\nNRT_STATUS nrt_inspect_config_set_capture_enabled_for_event_type_string(nrt_inspect_config_t *options, const char *event_type, bool enabled);\n\n/**\n * @brief Enable both system and device profiling for normal execution\n * \n * When disabled (default), no profiles are captured during normal execution.\n * This flag controls whether profiles are captured automatically for each execution.\n * Note: If both enable_inspect and enable_inspect_on_fail are false, no profiling occurs.\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] enable_inspect Boolean value to enable or disable inspect profiling.\n * @return NRT_SUCCESS on success, NRT_INVALID for invalid parameters.\n */\nNRT_STATUS nrt_inspect_config_set_enable_inspect(nrt_inspect_config_t *options, bool enable_inspect);\n\n/**\n * @brief Enable dumping of device profiles in case of execution failures\n * \n * When enabled, device profiles will be captured and saved when graph execution fails.\n * This is disabled by default. If both enable_inspect and enable_inspect_on_fail are false,\n * no profiling occurs at all.\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] enable_inspect_on_fail Boolean value to enable or disable inspect on failure.\n * @return NRT_SUCCESS on success, NRT_INVALID for invalid parameters.\n */\nNRT_STATUS nrt_inspect_config_set_enable_inspect_on_fail(nrt_inspect_config_t *options, bool enable_inspect_on_fail);\n\n /**\n * Begin tracing/profiling with configurable options\n *\n * Parameters:\n * @param[in] options - A pointer to an nrt_inspect_config struct containing configuration options\n * for profiling. Use nrt_inspect_config_set_* functions to set options.\n * If NULL is passed, default options will be used.\n * @return NRT_SUCCESS on success\n * \n * @note This API ignores all the NEURON_RT_INSPECT_* environment variables.\n * If you are using the environment variables to set the values you do not need to use this method \n * or any of the nrt_inspect_config_set_* functions. Use nrt_inspect_begin() instead.\n */\nNRT_STATUS nrt_inspect_begin_with_options(nrt_inspect_config_t *options);\n\n/**\n * @brief Returns all available activity type strings\n *\n * This function allocates and returns an array of all supported activity type\n * strings. The caller is responsible for freeing both the individual strings\n * and the array itself, or can use nrt_inspect_config_free_activity_types().\n *\n * @param[out] activity_types Pointer to store the allocated array of activity type strings.\n * @param[out] count Pointer to store the number of activity types returned.\n * @return NRT_SUCCESS on success, NRT_INVALID for invalid parameters, \n * NRT_RESOURCE for memory allocation failure.\n */\nNRT_STATUS nrt_inspect_config_get_all_activity_types(const char ***activity_types, size_t *count);\n\n/**\n * @brief Returns the currently enabled activity type strings\n *\n * This function examines the enabled_activities bitmask in the configuration\n * and returns an array of strings for only the currently enabled activity types.\n * The caller is responsible for freeing both the individual strings and the array itself,\n * or can use nrt_inspect_config_free_activity_types().\n *\n * @param[in] options Pointer to the options structure.\n * @param[out] activity_types Pointer to store the allocated array of enabled activity type strings.\n * @param[out] count Pointer to store the number of enabled activity types returned.\n * @return NRT_SUCCESS on success, NRT_INVALID for invalid parameters, \n * NRT_RESOURCE for memory allocation failure.\n */\nNRT_STATUS nrt_inspect_config_get_enabled_activity_types(nrt_inspect_config_t *options, const char ***activity_types, size_t *count);\n\n/**\n * @brief Free the activity types array allocated by nrt_inspect_config_get_all_activity_types\n * or nrt_inspect_config_get_enabled_activity_types.\n * This function properly frees both the array and all individual strings.\n * \n * @param[in] activity_types Pointer to the activity types array to be freed.\n * @param[in] count Number of activity types in the array.\n */\nvoid nrt_inspect_config_free_activity_types(const char **activity_types, size_t count);\n\n/**\n * @brief Sets or clears a specific activity type in the configuration\n *\n * This function enables or disables a specific activity type by name. It converts\n * the activity type string to the corresponding enum value and updates the\n * enabled_activities bitmask accordingly.\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] activity_type String name of the activity type. Valid values are:\n * \"system_profile\", \"device_profile\", \"host_memory\", \n * \"cpu_util\", \"all\"\n * @param[in] enabled True to enable the activity, false to disable it.\n * @return NRT_SUCCESS on success, NRT_INVALID for invalid parameters or unknown activity type.\n */\nNRT_STATUS nrt_inspect_config_set_activity(nrt_inspect_config_t *options, const char *activity_type, bool enabled);\n\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_status.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n// NOTE: if making changes here please also keep\n// KaenaTools: KaenaTools: pkg/rt/rt.go in sync\n\ntypedef enum {\n NRT_SUCCESS = 0,\n NRT_FAILURE = 1, // non specific failure, don't use if there is more descriptive type\n NRT_INVALID = 2, // e.g. invalid NEFF, bad instruction, bad DMA descriptor, input tensor name/size does not match the model, etc.\n\n // TODO invalid_handle is no longer useful because handles are not passed in nrt API\n // remove\n NRT_INVALID_HANDLE = 3, // make this one explicit instead of using more generic INVALID_INPUT because it could be a common caller mistake\n NRT_RESOURCE = 4, // failed to allocate a resource for requested operation\n\n // TODO separate exec timeout from others\n NRT_TIMEOUT = 5, // operation timed out\n NRT_HW_ERROR = 6, // Hardware failure\n NRT_QUEUE_FULL = 7, // not enough space in the execution input queue\n NRT_LOAD_NOT_ENOUGH_NC = 9, // Failed to allocate enough NCs for loading a NEFF\n NRT_UNSUPPORTED_NEFF_VERSION = 10, // Unsupported version of NEFF\n\n // DO NOT USE - keep for backward compat\n NRT_FAIL_HOST_MEM_ALLOC = 11, // failed to allocate host memory\n\n // Unique retcodes to help the caller identify when nrt apis are called outside the scope of nrt_init() and nrt_close()\n NRT_UNINITIALIZED = 13,\n NRT_CLOSED = 14,\n\n NRT_QUEUE_EMPTY = 15, // Accessed a queue with no data\n\n\n NRT_EXEC_UNIT_UNRECOVERABLE = 101, // Encountered fatal error and Execution Unit is in limbo, cannot recover. Need to reset\n\n\n NRT_EXEC_BAD_INPUT = 1002, // invalid input has been submitted to exec()\n NRT_EXEC_COMPLETED_WITH_NUM_ERR = 1003, // execution was completed with numerical errors (produced NaN)\n NRT_EXEC_COMPLETED_WITH_ERR = 1004, // execution was completed with other errors,\n // either logical - event double clear, or physical - parity error\n NRT_EXEC_NC_BUSY = 1005, // the neuron core is locked (in use) by another model/process\n NRT_EXEC_OOB = 1006, // one or more indirect memcopies and/or embedding updates are out of bound\n NRT_COLL_PENDING = 1100, // collective operation is still pending\n\n // classify different types of execution hangs/timeouts. For unknown/generic hang, use NRT_TIMEOUT.\n NRT_EXEC_HW_ERR_COLLECTIVES = 1200, // Stuck in collectives op (missing notification(s)). Possibly caused by a hardware error on another worker.\n NRT_EXEC_HW_ERR_HBM_UE = 1201, // An HBM encountered an unrepairable uncorrectable error and produced incorrect results.\n NRT_EXEC_HW_ERR_NC_UE = 1202, // An on-chip memory of Neuron Core encountered a parity error and produced incorrect results.\n NRT_EXEC_HW_ERR_DMA_ABORT = 1203, // A DMA engine encountered an unrecoverable error.\n\n NRT_EXEC_SW_NQ_OVERFLOW = 1204, // Software notification queue overflow.\n NRT_EXEC_HW_ERR_REPAIRABLE_HBM_UE = 1205, // An HBM encountered an repairable uncorrectable error and produced incorrect results.\n\n NRT_NETWORK_PROXY_FAILURE = 1206, // EFA network proxy operation failed.\n} NRT_STATUS;\n\nconst char *nrt_get_status_as_str(NRT_STATUS status);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_sys_trace.h", "content": "/*\n * Copyright 2025, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n#pragma once\n#include \n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n/*\n * This is a public interface used by both the fetch api (which allows near\n * real-time querying of captured events), and inspect profiling (which saves\n * captured events to disk), as well as other profiling functions.\n */\n\n//------------------------------------------------\n// Section: System Trace Capture\n//------------------------------------------------\n\ntypedef struct nrt_sys_trace_config nrt_sys_trace_config_t;\n\n/** Allocate memory for the options structure which is needed to\n * start profiling using nrt_sys_trace_start. This will set all options to\n * defaults. The reason we use an _allocate function is so that users don't need\n * to know the size or implementation details of the config struct.\n *\n * @param options[in] - pointer to a pointer to options nrt_sys_trace_config struct\n *\n */\nNRT_STATUS nrt_sys_trace_config_allocate(nrt_sys_trace_config_t **options);\n\n/** Set all fields of the nrt_sys_trace_config structure to their default values.\n *\n * @param options[in,out] - Pointer to an nrt_sys_trace_config structure.\n */\nvoid nrt_sys_trace_config_set_defaults(nrt_sys_trace_config_t *options);\n\n/** Free up memory allocated for the options structure which is needed to\n * start profiling using nrt_sys_trace_start\n *\n * @param options[in] - pointer to an options nrt_sys_trace_config struct\n *\n */\nvoid nrt_sys_trace_config_free(nrt_sys_trace_config_t *options);\n\n/**\n * @brief Sets max number of events that can be stored across all ring buffers\n *\n * @param[in,out] options Pointer to the options structure.\n * @param[in] max_events_per_nc Max number of events that can be stored in each ring buffer.\n */\nvoid nrt_sys_trace_config_set_max_events_per_nc(nrt_sys_trace_config_t *options, uint64_t max_events_per_nc);\n\n/**\n * @brief Sets system trace capture enabled for a specific NeuronCore\n * ring buffers won't be allocated for disabled NeuronCores.\n * Can save memory, reduce output size, and speed up trace processing.\n * @param[in,out] options Pointer to the options structure.\n * @param[in] nc_idx NeuronCore index.\n * @param[in] enabled Capture enabled flag.\n */\nvoid nrt_sys_trace_config_set_capture_enabled_for_nc(nrt_sys_trace_config_t *options, uint32_t nc_idx, bool enabled);\n\n/**\n * @brief Sets system trace capture enabled for a specific event type.\n * Can save memory, reduce output size, and speed up trace processing.\n * @param[in,out] options Pointer to the options structure.\n * @param[in] event_type Event type string, possible values are from nrt_sys_trace_get_event_types\n * @param[in] enabled Capture enabled flag.\n */\nNRT_STATUS nrt_sys_trace_config_set_capture_enabled_for_event_type(nrt_sys_trace_config_t *options, const char *event_type, bool enabled);\n\n/**\n * @brief Returns an allocated array of all valid event type strings.\n * @param[out] event_types Pointer to array of const char* (allocated).\n * @param[out] count Number of event types.\n * @return NRT_SUCCESS on success, error code otherwise.\n * @note The user is responsible for freeing the array and each string, or can use\n * nrt_sys_trace_free_event_types() for convenience.\n *\n * Example usage:\n * const char **event_types = nullptr;\n * size_t count = 0;\n * NRT_STATUS status = nrt_sys_trace_get_event_types(&event_types, &count);\n * // Manual cleanup:\n * for (size_t i = 0; i < count; ++i) {\n * free((void*)event_types[i]);\n * }\n * free((void*)event_types);\n * // Or use convenience function:\n * nrt_sys_trace_free_event_types(event_types, count);\n */\nNRT_STATUS nrt_sys_trace_get_event_types(const char ***event_types, size_t *count);\n\n/**\n * @brief Free the event types array allocated by nrt_sys_trace_get_event_types.\n * This function properly frees both the array and all individual strings.\n *\n * @param[in] event_types Pointer to the event types array to be freed.\n * @param[in] count Number of event types in the array.\n */\nvoid nrt_sys_trace_free_event_types(const char **event_types, size_t count);\n\n/**\n * @brief Returns an allocated array of enabled event type strings for the given config.\n * @param[in] options Pointer to the nrt_sys_trace_config_t structure.\n * @param[out] event_types Pointer to array of const char* (allocated).\n * @param[out] count Number of enabled event types.\n * @return NRT_SUCCESS on success, error code otherwise.\n * @note The user is responsible for freeing the array and each string.\n */\nNRT_STATUS nrt_sys_trace_config_get_enabled_event_types(nrt_sys_trace_config_t *options, const char ***event_types, size_t *count);\n\n// Initiailization for system trace capture including allocating memory for event ring buffers\nNRT_STATUS nrt_sys_trace_start(nrt_sys_trace_config_t *options);\n\n// Teardown for system trace capture including freeing allocated memory for event ring buffers\nNRT_STATUS nrt_sys_trace_stop();\n\n//------------------------------------------------\n// Section: System Trace Fetch\n//------------------------------------------------\n\ntypedef struct nrt_sys_trace_fetch_options nrt_sys_trace_fetch_options_t;\nNRT_STATUS nrt_sys_trace_fetch_options_allocate(nrt_sys_trace_fetch_options_t **options);\nvoid nrt_sys_trace_fetch_options_set_defaults(nrt_sys_trace_fetch_options_t *options);\nvoid nrt_sys_trace_fetch_options_free(nrt_sys_trace_fetch_options_t *options);\n// Max number of events to fetch per NeuronCore\nvoid nrt_sys_trace_fetch_options_set_max_events_per_nc(nrt_sys_trace_fetch_options_t *options, uint64_t max_events_per_nc);\n// Fetch events only for specified NeuronCore\nvoid nrt_sys_trace_fetch_options_set_nc_idx(nrt_sys_trace_fetch_options_t *options, uint64_t nc_idx);\n\n/**\n * Fetches system trace events from process memory and returns them as a JSON-formatted string.\n * Once events are fetched, they cannot be fetched again.\n *\n * @param[out] buffer On successful return, will point to a dynamically allocated, null-terminated\n * JSON string containing the trace events. Memory for the output buffer is\n * allocated internally; therefore, the caller should not allocate or initialize\n * the buffer before calling this function. Instead, the caller must initialize\n * the buffer pointer to NULL and, after a successful call, is responsible for\n * freeing the allocated memory by calling nrt_sys_trace_buffer_free(buffer).\n *\n * @param[out] written_size A pointer to a size_t variable that will be set to the number of bytes written\n * into the allocated buffer.\n *\n * @param[in] options Pointer to options such as max number of events to fetch.\n *\n * @return NRT_SUCCESS on success.\n *\n * Usage example:\n * char *buffer;\n * size_t written_size;\n * nrt_sys_trace_fetch_options_t *options;\n * nrt_sys_trace_fetch_options_allocate(&options);\n * nrt_sys_trace_fetch_options_set_nc_idx(options, 0); // Fetch events from NeuronCore 0 only instead of all\n * nrt_sys_trace_fetch_options_set_max_events_per_nc(options, 10000); // Fetch up to 10,000 events instead of all\n * nrt_sys_trace_fetch_events(&buffer, &written_size, options);\n * // or if you want to use the default options:\n * nrt_sys_trace_fetch_events(&buffer, &written_size, NULL);\n * // finally free the buffer when the events are no longer needed:\n * nrt_sys_trace_buffer_free(buffer)\n */\nNRT_STATUS nrt_sys_trace_fetch_events(char **buffer, size_t *written_size, const nrt_sys_trace_fetch_options_t *options);\n\n/** Free the buffer allocated by nrt_sys_trace_fetch_events. Should be called after the events are no longer needed.\n *\n * @param buffer [in] - Pointer to buffer to be freed.\n *\n * @return NRT_SUCCESS on success.\n */\nvoid nrt_sys_trace_buffer_free(char *buffer);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/libnrt/include/nrt/nrt_version.h", "content": "/*\n * Copyright 2021, Amazon.com, Inc. or its affiliates. All Rights Reserved\n */\n\n#pragma once\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n#define RT_VERSION_DETAIL_LEN 128\n#define GIT_HASH_LEN 64\n\ntypedef struct nrt_version {\n uint64_t rt_major;\n uint64_t rt_minor;\n uint64_t rt_patch;\n uint64_t rt_maintenance;\n char rt_detail[RT_VERSION_DETAIL_LEN];\n char git_hash[GIT_HASH_LEN];\n} nrt_version_t;\n\n/** Get the NRT library version\n *\n * @param ver[out] - Pointer to nrt version struct\n * @param size[in] - Length of the data needed to be filled in the nrt_version_struct\n *\n * @return NRT_STATUS_SUCCESS on success.\n */\nNRT_STATUS nrt_get_version(nrt_version_t *ver, size_t size);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/neuron-gatherinfo/LICENSE", "content": "Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this\nsoftware and associated documentation files (the \"Software\"), to deal in the Software\nwithout restriction, including without limitation the rights to use, copy, modify,\nmerge, publish, distribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,\nINCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A\nPARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT\nHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\nSOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n" }, { "path": "src/neuron-gatherinfo/clear_params_tfpb.py", "content": "import re\nimport copy\nimport argparse\nimport tensorflow as tf\nimport numpy as np\nimport string\n\nfrom google.protobuf import text_format\nfrom tensorflow.core.framework import node_def_pb2\nfrom tensorflow.core.framework import attr_value_pb2\nfrom tensorflow.python.framework import tensor_util\nfrom tensorflow.tools.graph_transforms import TransformGraph\n\ndef zero_const(node):\n val = tf.make_ndarray(node.attr.get(\"value\").tensor)\n new_val = val * 0.0\n new_tensor = tensor_util.make_tensor_proto(new_val, new_val.dtype, new_val.shape)\n node.attr[\"value\"].CopyFrom(attr_value_pb2.AttrValue(tensor=new_tensor))\n\ndef ZeroAllConst(graphdef):\n sess = tf.compat.v1.Session(graph=tf.import_graph_def(graphdef))\n const_by_name = {}\n node_by_name = {}\n for node in graphdef.node:\n node_by_name[node.name] = node \n if node.op == \"Const\":\n const_by_name[node.name] = node \n if node.op == \"BiasAdd\" or node.op == \"MatMul\" \\\n or node.op.startswith(\"Conv\") \\\n or node.op.startswith(\"FusedBatchNorm\"):\n for i in node.input: \n i_node = node_by_name[i]\n if i_node.op == \"Const\":\n zero_const(i_node)\n if i_node.op == \"Identity\":\n x_node = node_by_name[i_node.input[0]]\n if x_node.op == \"Const\":\n zero_const(x_node)\n return graphdef\n\ndef load_graph(model_file):\n graph_def = tf.compat.v1.GraphDef()\n\n with open(model_file, \"rb\") as f:\n graph_def.ParseFromString(f.read())\n return graph_def\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Zero-out parameters of BiasAdd, MatMul, Conv*, and FusedBatchNorm of TensorFlow frozen graph.\")\n parser.add_argument(\"--graph\", help=\"File name of frozen graph to be converted\",\n required=True)\n parser.add_argument(\"--out_graph\", help=\"File name to save converted frozen graph\",\n required=True)\n args = parser.parse_args()\n\n graph_orig = load_graph(args.graph)\n graph_mod = ZeroAllConst(graph_orig)\n with tf.io.gfile.GFile(args.out_graph, \"wb\") as f:\n f.write(graph_mod.SerializeToString())\n #with tf.io.gfile.GFile(args.out_graph + \"txt\", 'w') as f:\n # f.write(text_format.MessageToString(graph_mod))\n" }, { "path": "src/neuron-gatherinfo/mx_neuron_check_model.py", "content": "import os\nimport json\nimport sys\nimport struct\nimport argparse\nimport subprocess\nfrom collections import Counter\n \nclass neuron_parser:\n def __init__(self):\n self.parser = argparse.ArgumentParser()\n self.parser.add_argument('model_path', type=str, help='path prefix to MXNet model (the part before -symbol.json).')\n self.parser.add_argument('--show_names', action='store_true', help='list operation by name instead of summarizing by type (caution: this option will generate many lines of output for a large model).')\n self.parser.add_argument('--expand_subgraph', action='store_true', help='show subgraph operations.')\n self.parser_args = self.parser.parse_args()\n self.neuronop_info = {}\n self.total_pipeline_cores = 0\n self.min_required_pipeline_cores = 0\n path = self.parser_args.model_path\n\n if os.path.exists(path + '-symbol.json'):\n self.load_mxnet_model(path)\n elif os.path.isdir(path):\n self.load_tensorflow_model(path)\n else:\n raise RuntimeError('Cannot determine framework type from model path argument.')\n self.supported = self.get_neuron_supported()\n self.supported.extend(self.addl_support)\n for name, executable, (sg_nodetypes, sg_nodenames) in self.neuron_nodes:\n num_cores, requested_cores, _ = self.get_cores_from_executable(executable)\n self.neuronop_info[name] = (num_cores, requested_cores, sg_nodetypes, sg_nodenames)\n self.total_pipeline_cores += num_cores\n if num_cores > self.min_required_pipeline_cores:\n self.min_required_pipeline_cores = num_cores\n\n def get_neuron_supported(self):\n exec_cmd = [\"neuron-cc\", \"list-operators\", \"--framework\", self.framework]\n oplist = subprocess.check_output(' '.join(exec_cmd), shell=True)\n oplist = str(oplist, 'utf-8')\n oplist = oplist.split(\"\\n\")\n return oplist[:-1] # Remove the last element which is ''\n \n def get_tf_subgraph_types_names(self, node):\n from tensorflow.core.framework import graph_pb2\n graph_def = graph_pb2.GraphDef()\n graph_def.ParseFromString(node.attr['graph_def'].s)\n sg_nodes = graph_def.node\n sg_nodes = [sg_node for sg_node in sg_nodes if sg_node.op not in self.excl_types]\n nodetypes = [sg_node.op for sg_node in sg_nodes]\n nodenames = [sg_node.name for sg_node in sg_nodes]\n return nodetypes, nodenames\n\n def load_tensorflow_model(self, path):\n import tensorflow as tf\n import tensorflow_hub as hub\n self.framework = 'TENSORFLOW'\n self.neuron_optype = \"NeuronOp\"\n self.excl_types = ['Placeholder', 'PlaceholderWithDefault', 'NoOp', 'Const', 'Identity', 'IdentityN', 'VarHandleOp', 'VarIsInitializedOp', 'AssignVariableOp', 'ReadVariableOp', 'StringJoin', 'ShardedFilename', 'SaveV2', 'MergeV2Checkpoints', 'RestoreV2']\n self.addl_support = ['FusedBatchNormV3', 'BatchMatMulV2', 'AddV2', 'StopGradient', self.neuron_optype]\n model = hub.load(path)\n graph_def = model.graph.as_graph_def()\n nodes = graph_def.node\n nodes = [node for node in nodes if node.op not in self.excl_types]\n self.nodetypes = [node.op for node in nodes]\n self.nodenames = [node.name for node in nodes]\n self.neuron_nodes = [(node.name, node.attr['executable'].s, self.get_tf_subgraph_types_names(node)) for node in nodes if node.op == self.neuron_optype]\n\n def get_mx_subgraph_types_names(self, node):\n nodetypes = []\n nodenames = []\n for sg in node['subgraphs']:\n filtered_nodes = [sg_node for sg_node in sg['nodes'] if sg_node['op'] not in self.excl_types]\n nodetypes.extend([sg_node['op'] for sg_node in filtered_nodes])\n nodenames.extend([sg_node['name'] for sg_node in filtered_nodes])\n return nodetypes, nodenames\n\n def load_mxnet_model(self, path): \n import mxnet as mx\n if mx.__version__ != \"1.5.1\":\n try:\n import mx_neuron as mxn\n except:\n raise \"Please install mxnetneuron package.\"\n self.framework = 'MXNET'\n self.neuron_optype = \"_neuron_subgraph_op\"\n self.excl_types = ['null']\n self.addl_support = [self.neuron_optype]\n sym, args, auxs = mx.model.load_checkpoint(path, 0)\n nodes = json.loads(sym.tojson())[\"nodes\"]\n nodes = [node for node in nodes if node['op'] not in self.excl_types]\n self.nodetypes = [node['op'] for node in nodes]\n self.nodenames = [node['name'] for node in nodes]\n neuron_nodes_tmp = [node for node in nodes if node['op'] == self.neuron_optype]\n self.neuron_nodes = [(node['name'], bytearray(args[node['name']+\"_neuronbin\"].asnumpy()), self.get_mx_subgraph_types_names(node)) for node in neuron_nodes_tmp]\n\n @staticmethod\n def get_cores_from_executable(executable):\n _NC_HEADER_SIZE = 544\n header = executable[:_NC_HEADER_SIZE]\n info = list(struct.unpack('168xI304xI64B', header))\n numCores = info.pop(0)\n numCoresRequested = info.pop(0)\n coresPerNode = info\n return numCores, numCoresRequested, coresPerNode\n\n # Display table of operation type or name and whether supported or not\n def print_node_type_info(self):\n self.cnt_total = len(self.nodetypes)\n self.cnt_supported = 0\n if self.parser_args.show_names:\n widthn = max(max(map(len, self.nodenames)), 8)\n widtht = max(max(map(len, self.nodetypes)), 8)\n format_str = \"{:<\" + str(widthn) + \"} {:<\" + str(widtht) + \"} {:<4}\"\n pp = lambda x: print(format_str.format(*x))\n pp(['Op Name', 'Op Type', 'Neuron Supported ?'])\n pp(['-------', '-------', '------------------'])\n for idx, opname in enumerate(self.nodenames):\n optype = self.nodetypes[idx]\n if optype in self.supported:\n pp([opname, optype, 'Yes'])\n self.cnt_supported += 1\n for idx, opname in enumerate(self.nodenames):\n optype = self.nodetypes[idx]\n if optype not in self.supported:\n pp([opname, optype, 'No'])\n else:\n count = Counter(self.nodetypes)\n width = max(max(map(len, self.nodetypes)), 8)\n format_str = \"{:<\" + str(width) + \"} {:<14} {:<4}\"\n pp = lambda x: print(format_str.format(*x))\n pp(['Op Type', 'Num Instances', 'Neuron Supported ?'])\n pp(['-------', '-------------', '------------------'])\n for key in count:\n if key in self.supported:\n pp([key, count[key], 'Yes'])\n self.cnt_supported += count[key]\n for key in count:\n if key not in self.supported:\n pp([key, count[key], 'No'])\n print()\n\n def print_subgraph_ops(self, sg_nodetypes, sg_nodenames):\n if self.parser_args.show_names:\n widthn = max(max(map(len, sg_nodenames)), 8)\n widtht = max(max(map(len, sg_nodetypes)), 8)\n format_str = \"{:<\" + str(widthn) + \"} {:<\" + str(widtht) + \"}\"\n pp = lambda x: print(' ', format_str.format(*x))\n pp(['Op Name', 'Op Type'])\n pp(['-------', '-------'])\n for idx, opname in enumerate(sg_nodenames):\n optype = sg_nodetypes[idx]\n pp([opname, optype])\n else:\n count = Counter(sg_nodetypes)\n width = max(max(map(len, sg_nodetypes)), 8)\n format_str = \"{:<\" + str(width) + \"} {:<14}\"\n pp = lambda x: print(' ', format_str.format(*x))\n pp(['Op Type', 'Num Instances'])\n pp(['-------', '-------------'])\n for key in count:\n pp([key, count[key]])\n\n def print_neuron_node_info(self):\n idx = 0\n width = max(max(map(len, self.neuronop_info)), 14) \n format_str = \"{:<\" + str(width) + \"} {:<14}\"\n pp = lambda x: print(format_str.format(*x))\n pp(['Subgraph Name', 'Num Pipelined NeuronCores'])\n pp(['-------------', '-------------------------'])\n core_cnt_list = []\n for name, (num_cores, _, sg_nodetypes, sg_nodenames) in self.neuronop_info.items():\n pp([name, num_cores])\n core_cnt_list.append(num_cores)\n idx += 1\n if self.parser_args.expand_subgraph:\n self.print_subgraph_ops(sg_nodetypes, sg_nodenames)\n print()\n\n def print_neuron_support_stats(self):\n print(\"* Total inference operations: {}\".format(self.cnt_total))\n print(\"* Total Neuron supported inference operations: {}\".format(self.cnt_supported))\n if self.cnt_total > 0:\n perc = self.cnt_supported / self.cnt_total * 100\n else:\n perc = 0\n print(\"* Percent of total inference operations supported by Neuron: {:.1f}\".format(perc))\n print()\n\n def print_common_desc(self):\n if self.parser_args.show_names:\n print(\"* Each line shows an operation name and whether the type of that operation is supported in Neuron.\")\n else:\n print(\"* Each line shows an operation type, the number of instances of that type within model,\\n\" \\\n \"* and whether the type is supported in Neuron.\")\n print(\"* Some operation types are excluded from table because they are no-operations or training-related operations:\\n\", \\\n self.excl_types, \"\\n\")\n\n def run(self):\n if len(self.neuronop_info) > 0:\n print(\"\\n* Found {} Neuron subgraph(s) ({}(s)) in this compiled model.\\n\" \\\n \"* Use this tool on the original uncompiled model to see Neuron supported operations.\\n\" \\\n \"* The following table shows all operations, including Neuron subgraphs.\".format(len(self.neuronop_info), self.neuron_optype))\n self.print_common_desc()\n self.print_node_type_info()\n print('* Please run this model on Inf1 instance with at least {} NeuronCore(s).'.format(self.min_required_pipeline_cores))\n print(\"* The following list show each Neuron subgraph with number of pipelined NeuronCores used by subgraph\\n\"\\\n \"* (and subgraph operations if --expand_subgraph is used):\\n\")\n self.print_neuron_node_info()\n else:\n print(\"\\n* The following table shows the supported and unsupported operations within this uncompiled model.\")\n self.print_common_desc()\n self.print_node_type_info()\n self.print_neuron_support_stats()\n \nif __name__=='__main__':\n toolkit = neuron_parser()\n toolkit.run()\n" }, { "path": "src/neuron-gatherinfo/neuron-gatherinfo.py", "content": "#!/usr/bin/env python3\n# coding=utf-8\n\n\n\"\"\" Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n\n SPDX-License-Identifier: MIT-0\n\n Program to gather information from a system\n\"\"\"\nimport sys\nimport os\nimport argparse\nimport shutil\nimport subprocess\nimport re\n\nACTUAL_CMD = os.path.realpath(sys.argv[0])\n\nUSAGE_MSG = \"\"\"\n Usage: {} [options]\n This program is used to gather information from this system for analysis\n and debugging\n \"\"\".format(ACTUAL_CMD)\n\nEXCLUDE_FILES_BY_NAME = \"weight files, model, NEFF (Neuron Executable File Format)\"\n\nHELP_CC_FILES = \"\"\" Location of the neuron-cc generated files \"\"\"\nDEFAULT_CCFILES_LOCATION = \"~/bin\"\n\nSYSLOG_SEARCH_PATTERNS = r\"nrtd|neuron|kernel:\"\n\nEXTERNAL_CMDS = [\"lscpu\", \"lshw\",\n \"lspci | grep -i Amazon\",\n \"neuron-cc --version\",\n \"neuron-ls\",\n \"top -b -n 1\",\n \"uname -a\", \"uptime\",\n ]\n\nPROC_FILES = [\"/proc/cmdline\",\n \"/proc/cpuinfo\",\n \"/proc/filesystems\",\n \"/proc/interrupts\",\n \"/proc/iomem\",\n \"/proc/loadavg\",\n \"/proc/meminfo\",\n \"/proc/modules\",\n \"/proc/mtrr\",\n \"/proc/version\",\n ]\n\nHELP_ADDITIONAL_FILE_OR_DIR = \"\"\" Additional file or directory that the user wants to provide in\n the archive. The user can sanitize this file or directory before sharing \"\"\"\n\nINCLUDE_MSG = \"\"\"\n By default, only the lines containing (grep) patterns like '{}' from the syslog are copied.\n Other lines are excluded. Using this option allows the timestamp section of other lines\n to be included. The rest of the contents of the line itself are elided. Providing the\n timestamp section may provide time continuity while viewing the copied syslog file\n \"\"\".format(SYSLOG_SEARCH_PATTERNS)\n\nHELP_RT_FILES = \"\"\" Location of the neuron runtime generated files \"\"\"\nMISCINFO_FILE = 'miscinfo.txt'\n\nHELP_VERBOSE = \"\"\" Verbose mode displays commands executed and any additional information\n which may be useful in debugging the tool itself\n \"\"\"\n\nINCLUDE_EXTNS = ('.pb')\n\nHELP_INCLUDE_EXTN_FILES = \"\"\" Include files with these extensions from the compiler work\n directory in the archive:\n {}\n \"\"\".format(INCLUDE_EXTNS)\n\nHELP_STDOUT = \"\"\" The file where the stdout of the compiler run was saved \"\"\"\n\nHELP_OUTDIR_MSG = \"\"\"\n The output directory where all the files and other information will be stored.\n The output will be stored as an archive as well as the actual directory where all the\n contents are copied. This will allow a simple audit of the files, if necessary.\n *** N O T E ***: Make sure that this directory has enough space to hold the files\n and resulting archive\n \"\"\"\n\nUSERCMDFILE = \"how-the-user-executed-the-script-{}.txt\".format(os.path.basename(ACTUAL_CMD))\n\nNEURONDUMPPROGRAM = \"/opt/aws/neuron/bin/neuron-dump.py\"\nNEURONDUMPFILE = os.path.splitext(os.path.basename(NEURONDUMPPROGRAM))[0]\n\nNEURON_ERRMSG = \"Error: File {} doesn't exist, aws-neuron-tool package isn't installed?\".format(\n NEURONDUMPPROGRAM)\n\nNEURON_INFO_TARBALL = \"{}\".format(os.path.splitext(os.path.basename(ACTUAL_CMD))[0])\nNEURONTMPDIR = NEURON_INFO_TARBALL\n\nARCHIVE_MSG = \"\\n\\n\\t******\\n\\tArchive created at:\\n\\t\\t{}\\n\\tFrom directory:\\n\\t\\t{}\\n\\t******\\n\\n\"\n\nNOT_IMPLEMENTED_MSG = \", nothing to see here, folks (not implemented as yet)\"\n\n# these are the only compiler-generated files that are included by default\nCOMPILER_FILES = ['graph_def.neuron-cc.log', 'all_metrics.csv', 'hh-tr-operand-tensortensor.json']\n\nCOMPILER_FILES_USER_OPT_IN = ['exp_and_others.json', 'graph_def.neff', 'graph_def.pb',\n 'hh-spilled.json', 'hh-tr-accDN2virtDN.json',\n 'hh-tr-external-move.json', 'hh-tr-internal-move.json',\n 'hh-tr-removeDN.json', 'hh-transforms.json', 'wavegraph.json',\n 'hh.json', 'pass03_scheduling.json',\n 'relay_graph_opt_pre_color.txt', 'relay_graph_post_opt_kelp.txt',\n 'relay_graph_post_opt_unit_level.txt', 'relay_graph_pre_opt.txt',\n 'saved_model.pb', 'sch.json', 'sch_tmp.json',\n 'schedule_trace.json',\n 'wavegraph-bin.json']\n\nMODEL_DATA_MSG = \"\"\"\n By using this option, the entire compiler work directory's contents will be\n included (excluding the {} files, unless an additional option is used). This would\n include model information, etc.\n The files that are included, by default, are these: {}\n\n \"\"\".format(INCLUDE_EXTNS, \", \".join(COMPILER_FILES))\n\nMODEL_DATA_MSG_INFO = \"\"\"\n\\t**************************\n\\tBased on your command line option, we're also packaging these files:\n\n\\t\\t{}\n\n\\tAnd this directory: {}\n\n\\t**************************\n\"\"\"\n\ndef get_os_version():\n\n ''' function to obtain the Linux version\n Args:\n\n Output:\n\n Returns:\n string with value 'Ubuntu' or 'RedHat'\n '''\n\n try:\n with open(\"/proc/version\") as fdin:\n data = fdin.read()\n if data.find('Ubuntu') == -1:\n osver = 'RedHat'\n else:\n osver = 'Ubuntu'\n except FileNotFoundError:\n osver = 'Ubuntu'\n\n return osver\n\n\ndef get_files(*, basedir, matchfiles, verbose):\n ''' function to get the files based on a base directory and file extension\n\n Args:\n basedir : base directory where files reside\n matchfiles : set of files to match\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n\n Returns:\n list of files found\n\n '''\n\n myfiles = list()\n for dpath, _, files in os.walk(basedir):\n for mfile in files:\n if mfile in matchfiles:\n mfile = os.path.realpath(os.path.join(dpath, mfile))\n if os.path.isfile(mfile):\n myfiles.append(mfile)\n else:\n if verbose:\n print(\"Warning: {} is not a file\".format(mfile))\n\n return myfiles\n\n\ndef dump_compiler_info(*, outdir, location, allowmodel=False, addfldir=None, verbose=False):\n ''' function to gather the following information:\n Framework:\n - TensorFlow\n - MXNet\n - PyTorch\n Compiler:\n Args:\n outdir : output directory\n location : location of compiler-generated files\n allowmodel : if True, allow gathering of additional files\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output: compiler-generated files copied to outdir\n\n Returns:\n '''\n\n if location is not None:\n if allowmodel: # copy the entire directory\n try:\n shutil.copytree(location, os.path.join(outdir, os.path.basename(location)),\n ignore_dangling_symlinks=True)\n except shutil.Error:\n pass\n else:\n fileset = set(COMPILER_FILES)\n l1data = get_files(basedir=location, matchfiles=fileset, verbose=verbose)\n copy_files(outdir=outdir, basedir=location, filelist=l1data, verbose=verbose)\n\n if addfldir is not None:\n if os.path.isfile(addfldir):\n shutil.copy(addfldir, outdir)\n else: # directory copy\n try:\n shutil.copytree(addfldir, os.path.join(outdir, os.path.basename(addfldir)),\n ignore_dangling_symlinks=True)\n except shutil.Error:\n pass\n\n # print(\"Function: \", sys._getframe().f_code.co_name, # pylint: disable=W0212\n # NOT_IMPLEMENTED_MSG)\n\n\ndef copy_stdout(*, outdir, stdout, verbose):\n ''' function to copy the stdout file to the destination location\n\n Args:\n outdir : destination location (output directory)\n stdout : file containing the output of running neuron-cc\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n\n Returns:\n '''\n\n if verbose:\n print(\"Copying {} to {}\".format(stdout, outdir))\n\n shutil.copy(stdout, outdir)\n\n\ndef copy_syslog(*, outdir, include_flag=False, verbose):\n '''\n function to copy contents of the syslog to the output directory\n\n Args:\n outdir : output directory location where the syslog's contents\n are to be copied\n include_flag : if True, include lines that do not match\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n copy of syslog's contents with just \"Neuron-specific\" lines\n\n Returns:\n '''\n\n # syslog looks like this:\n # 2019-11-21T19:32:50.347183+00:00 ink neuron-rtd[17977]: nrtd[17977]: \n # The first regex (regex1) is used to match lines that we want to see in our copy\n\n regex1 = re.compile(r'^(\\S+)\\s.*?({})'.format(SYSLOG_SEARCH_PATTERNS))\n regex2 = re.compile(r'^(\\S+)\\s')\n\n osver = get_os_version()\n if osver == 'Ubuntu':\n syslog = '/var/log/syslog'\n else:\n syslog = '/var/log/messages'\n\n try:\n with open(syslog) as fdin,\\\n open(os.path.join(outdir, 'copy-of-syslog'), 'w') as fdout:\n for line in fdin:\n match = regex1.search(line)\n if match is not None:\n fdout.write(line)\n else:\n if include_flag:\n match = regex2.match(line)\n if match is not None:\n # exclude the rest of the line\n fdout.write(match.group(1) + ' XXX contents elided XXX\\n')\n else:\n print(\"Error in parsing this line: {}\".format(line))\n except FileNotFoundError:\n print(\"Error, /var/log/syslog not found\")\n\n\ndef dump_rt_info(*, location, verbose):\n ''' function to dump the following information:\n - runtime\n - Framework (??)\n Args:\n location: location of runtime files\n verbose : flag to indicate if verbose messages need to be displayed\n Returns:\n list of info\n '''\n\n # l1data = get_files(basedir=location, file_extn=('.sh'))\n print(\"Function: \", sys._getframe().f_code.co_name, # pylint: disable=W0212\n NOT_IMPLEMENTED_MSG)\n\n\ndef allow_capture_of_files():\n '''\n function to allow the capture of files from the customer's environment\n This is OFF by default and has to be explicitly enabled by the command-line\n option by the user\n\n Args:\n\n Output:\n\n Returns:\n\n '''\n\n print(\"Function: \", sys._getframe().f_code.co_name, # pylint: disable=W0212\n NOT_IMPLEMENTED_MSG)\n\n\ndef add_additional_filters(filterfile):\n '''\n function to apply additional filters to files that are being captured\n\n Args:\n filterfile : text file with patterns (regexs), one per line, to use as filters\n\n\n Output:\n\n Returns:\n\n '''\n\n print(\"Function: \", sys._getframe().f_code.co_name, # pylint: disable=W0212\n NOT_IMPLEMENTED_MSG)\n\n\ndef dump_miscinfo(*, outdir, verbose):\n ''' function to dump miscellaneous information, including:\n - system info (uname -a)\n - package info (??? list of packages installed)\n - neuron-ls\n - neuron-top\n\n Args:\n outdir : output directory\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n Creates various reports in the outdir location\n\n Returns:\n\n '''\n\n osver = get_os_version()\n if osver == 'Ubuntu':\n pkgcmds = [\"apt list | egrep '^aws'\",\n \"pip list | egrep '^neuron|^numpy|^tensor|^scipy'\"]\n else:\n pkgcmds = [\"rpm -qa | egrep '^aws|^neuron|^numpy|^tensor|^scipy'\"]\n\n cmds = EXTERNAL_CMDS + pkgcmds\n\n for cmd in cmds:\n cmdname = cmd.split(' ')[0] # get just the command name for creating the file\n cmdfile = os.path.join(outdir, \"report-{}.txt\".format(cmdname))\n\n with open(cmdfile, \"w\") as fdout:\n\n if verbose:\n print(\"Running cmd: {} and capturing output in file: {}\".format(cmd, cmdfile))\n\n try:\n res = subprocess.Popen(cmd, stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT, universal_newlines=True,\n shell=True)\n stdout, stderr = res.communicate()\n if stderr is not None:\n fdout.write(\"Error in executing cmd: {}\\nError: {}\\n\".format(cmd, str(stderr)))\n else:\n fdout.write(\"Output from executing cmd: {}\\n\\n{}\\n\".format(cmd, str(stdout)))\n except (OSError, ValueError) as err:\n fdout.write(\"Error in executing cmd: {}\\nError: {}\\n\".format(cmd, err))\n\n\ndef dump_proc_info(*, outdir, verbose):\n '''\n function to dump information related to \"/proc\"\n\n Args:\n outdir : output directory\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n Creates various reports in the outdir location\n\n Returns:\n\n '''\n\n for procfile in PROC_FILES:\n fname = procfile.split('/') # use the 2nd and 3rd items from this (canonical form)\n pfile = os.path.join(outdir, \"report-{}-{}.txt\".format(fname[1], fname[2]))\n if verbose:\n print(\"Copying contents of: {} to: {}\".format(procfile, pfile))\n\n try:\n with open(pfile, \"w\") as fdout, open(procfile) as fdin:\n fdout.write(\"Contents of {}\\n\\n\".format(procfile))\n fdout.write(fdin.read())\n except FileNotFoundError:\n print(\"Error: file {} not found\\n\".format(procfile))\n\n\ndef sanity_check(options):\n '''\n function to check if command-line arguments are valid\n\n Args:\n options : options from argparse parser\n\n Output:\n\n Returns:\n 0 : success\n 1 : failure\n '''\n\n # the script has to be run as root or \"sudo\"\n if os.getuid() != 0:\n print(\"*** Rerun this script as user 'root' or as sudo **\\n\\n\")\n return 1\n\n outdir = options.outdir\n\n retval = 0\n if os.path.isfile(outdir) or os.path.isdir(outdir):\n print(\"Error: {} already exists, please provide a non-existing directory\".format(outdir))\n retval = 1\n\n if not os.path.isfile(options.stdout):\n print(\"Error: {} doesn't exist, please provide an existing file\".format(options.stdout))\n retval = 1\n\n if options.addfldir is not None:\n if not os.path.isfile(options.addfldir) and not os.path.isdir(options.addfldir):\n print(\"Error: {} isn't a file nor a directory\".format(options.addfldir))\n retval = 1\n\n for mydir in [options.ccdir, options.rtdir]:\n if mydir is not None and not os.path.isdir(mydir):\n print(\"Error: {} is not a directory, please provide a directory\".format(mydir))\n retval = 1\n\n if options.allowmodel and options.ccdir is None:\n print(\"Error: you need to specify a compiler work directory along with the 'm' option\")\n retval = 1\n return retval\n\n\ndef copy_files(*, outdir, basedir, filelist, verbose):\n '''\n function to copy files from the original source area\n into the destination. This is also the place for any\n massaging or eliding of file contents\n\n Args:\n outdir : destination location\n basedir : base directory from where the files are to be copied\n filelist: list of files to be copied\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n Copy of files (possibly altered) from the source\n\n Returns:\n\n '''\n\n for thisfile in filelist:\n myfile = '.' + thisfile[len(basedir):]\n mydir = os.path.dirname(os.path.join(outdir, myfile))\n if not os.path.isdir(mydir):\n os.makedirs(mydir)\n shutil.copy(thisfile, mydir, follow_symlinks=True)\n\n\ndef write_miscinfo(*, outdir, data):\n '''\n function to write out the contents of the miscellaneous commands\n\n Args:\n outdir : destination location\n data : list of strings to be stored in a file\n\n Output:\n MISCINFO_FILE created with the contents of the output of the various\n commands\n '''\n\n flname = os.path.join(outdir, MISCINFO_FILE)\n\n with open(flname, \"w\") as fdout:\n fdout.write(\"\\n\".join(data))\n\n\ndef run_neuron_dump(outdir, verbose):\n '''\n function to call the existing neuron-dump.py tool\n\n Args:\n outdir : destination location\n verbose : flag to indicate if verbose messages need to be displayed\n\n Output:\n tarball created by this tool\n\n Returns:\n\n '''\n\n if not os.path.isfile(NEURONDUMPPROGRAM):\n print(NEURON_ERRMSG)\n return\n\n cmd = \"{} -o {}\".format(NEURONDUMPPROGRAM, os.path.join(outdir, NEURONDUMPFILE))\n\n if verbose:\n print(\"Executing command: {}\".format(cmd))\n\n try:\n res = subprocess.Popen(cmd, stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT, universal_newlines=True,\n shell=True)\n stdout, stderr = res.communicate()\n if stderr is not None:\n print(\"Error in executing cmd: {}\\nError: {}\\n\".format(cmd, str(stderr)))\n except (OSError, ValueError) as err:\n print(\"Error in executing cmd: {}\\nError: {}\\n\".format(cmd, err))\n\n if verbose:\n print(\"Output of cmd: {}\\n{}\".format(cmd, stdout))\n\n\ndef package_tarball(*, outdir, allowmodel, ccdir, verbose):\n '''\n function to package everything into a tarball\n\n Args:\n outdir : output directory\n allowmodel : flag to indicate whether the user has allowed\n gathering of model data\n\n Output:\n A tar ball created in directory one level above outdir\n this would be the directory provided by the user\n\n Returns:\n '''\n\n mytarball = os.path.join(os.path.split(outdir)[0], NEURON_INFO_TARBALL)\n\n if verbose:\n print(\"Creating archive: {}\".format(mytarball))\n\n archivefile = shutil.make_archive(mytarball, 'gztar', outdir)\n print(ARCHIVE_MSG.format(archivefile, outdir))\n\n if allowmodel:\n print(MODEL_DATA_MSG_INFO.format(\"\\n\\t\\t\".join(COMPILER_FILES),\n ccdir))\n\n\ndef add_cmdline_args():\n '''\n function to add the command line arguments and options\n\n Args:\n\n Output:\n\n Returns:\n parser for cmd line\n\n '''\n\n parser = argparse.ArgumentParser(\n formatter_class=argparse.RawDescriptionHelpFormatter,\n description=USAGE_MSG)\n\n parser.add_argument('--additionalfileordir',\n dest='addfldir',\n help=HELP_ADDITIONAL_FILE_OR_DIR,\n default=None)\n\n parser.add_argument('-c', '--compileroutdir',\n dest='ccdir',\n help=HELP_CC_FILES,\n default=None)\n\n parser.add_argument('-i', '--include',\n dest='includemismatch',\n help=INCLUDE_MSG,\n action='store_true',\n default=False)\n\n parser.add_argument('-f', '--filter',\n dest='filterfile',\n default=None)\n\n parser.add_argument('-m', \"--modeldata\", # data related to model, etc. will be gathered\n dest='allowmodel',\n action='store_true',\n help=MODEL_DATA_MSG,\n default=False)\n\n parser.add_argument('-o', '--out',\n dest='outdir',\n help=HELP_OUTDIR_MSG,\n required=True)\n\n parser.add_argument('-r', '--runtimeoutdir',\n dest='rtdir',\n help=HELP_RT_FILES,\n default=None)\n\n parser.add_argument('-s', '--stdout',\n dest='stdout',\n help=HELP_STDOUT,\n required=True)\n\n parser.add_argument('-v', '--verbose',\n dest='verbose',\n help=HELP_VERBOSE,\n action='store_true',\n default=False)\n\n return parser\n\n\ndef main():\n \"\"\" main function\n creates command-line option parser, sanity checks, and then executes code\n based on command-line options\n \"\"\"\n\n parser = add_cmdline_args()\n\n if len(sys.argv) == 1:\n parser.print_help()\n sys.exit(1)\n\n options = parser.parse_args()\n # append the directory where we'll create files to what the user provides\n options.outdir = os.path.realpath(os.path.join(options.outdir, NEURONTMPDIR))\n\n if options.ccdir is not None:\n options.ccdir = os.path.realpath(options.ccdir)\n\n if options.addfldir is not None:\n options.addfldir = os.path.realpath(options.addfldir)\n\n if options.rtdir is not None:\n options.rtdir = os.path.realpath(options.rtdir)\n\n options.stdout = os.path.realpath(options.stdout)\n\n if sanity_check(options):\n parser.print_help()\n sys.exit(1)\n\n # create the base directory\n try:\n os.makedirs(options.outdir)\n except FileNotFoundError:\n print(\"Error in creating directory {}\".format(options.outdir))\n sys.exit(1)\n\n # if options.allow:\n # allow_capture_of_files()\n\n if options.filterfile is not None:\n add_additional_filters(os.path.realpath(options.filterfile))\n\n # record the command as executed by the user\n with open(os.path.join(options.outdir, USERCMDFILE), \"w\") as fdout:\n fdout.write(\"Command executed as: {}\\n\".format(\" \".join(sys.argv)))\n\n dump_compiler_info(outdir=options.outdir, location=options.ccdir,\n allowmodel=options.allowmodel,\n addfldir=options.addfldir,\n verbose=options.verbose)\n\n # Not being used now. neuron-dump.py would do this\n # dump_rt_info(location=options.rtdir, verbose=options.verbose)\n\n dump_miscinfo(outdir=options.outdir, verbose=options.verbose)\n dump_proc_info(outdir=options.outdir, verbose=options.verbose)\n\n copy_stdout(outdir=options.outdir, stdout=options.stdout, verbose=options.verbose)\n copy_syslog(outdir=options.outdir, include_flag=options.includemismatch,\n verbose=options.verbose)\n\n # run the existing tool neuron-dump.py as well\n run_neuron_dump(outdir=options.outdir, verbose=options.verbose)\n\n package_tarball(outdir=options.outdir, allowmodel=options.allowmodel,\n ccdir=options.ccdir, verbose=options.verbose)\n\n # change permissions for the directory and output\n os.system(\"chown -R {} {}\".format(os.getlogin(), os.path.split(options.outdir)[0]))\n\n # write_miscinfo(outdir=options.outdir, data=l3)\n\n\nif __name__ == \"__main__\":\n\n main()\n" }, { "path": "src/neuron-gatherinfo/tf_neuron_check_model.py", "content": "import os\nimport json\nimport sys\nimport struct\nimport argparse\nimport subprocess\nfrom collections import Counter\n \nclass neuron_parser:\n def __init__(self):\n self.parser = argparse.ArgumentParser()\n self.parser.add_argument('model_path', type=str, help='a TensorFlow SavedModel directory (currently supporting TensorFlow v1 SaveModel only).')\n self.parser.add_argument('--show_names', action='store_true', help='list operation by name instead of summarizing by type (caution: this option will generate many lines of output for a large model).')\n self.parser.add_argument('--expand_subgraph', action='store_true', help='show subgraph operations.')\n self.parser_args = self.parser.parse_args()\n self.neuronop_info = {}\n self.total_pipeline_cores = 0\n self.min_required_pipeline_cores = 0\n path = self.parser_args.model_path\n if os.path.exists(path + '-symbol.json'):\n self.load_mxnet_model(path)\n elif os.path.isdir(path):\n self.load_tensorflow_model(path)\n else:\n raise RuntimeError('Cannot determine framework type from model path argument.')\n self.supported = self.get_neuron_supported()\n self.supported.extend(self.addl_support)\n for name, executable, (sg_nodetypes, sg_nodenames) in self.neuron_nodes:\n num_cores, requested_cores, _ = self.get_cores_from_executable(executable)\n self.neuronop_info[name] = (num_cores, requested_cores, sg_nodetypes, sg_nodenames)\n self.total_pipeline_cores += num_cores\n if num_cores > self.min_required_pipeline_cores:\n self.min_required_pipeline_cores = num_cores\n\n def get_neuron_supported(self):\n exec_cmd = [\"neuron-cc\", \"list-operators\", \"--framework\", self.framework]\n oplist = subprocess.check_output(' '.join(exec_cmd), shell=True)\n oplist = str(oplist, 'utf-8')\n oplist = oplist.split(\"\\n\")\n return oplist[:-1] # Remove the last element which is ''\n \n def get_tf_subgraph_types_names(self, node):\n from tensorflow.core.framework import graph_pb2\n graph_def = graph_pb2.GraphDef()\n graph_def.ParseFromString(node.attr['graph_def'].s)\n sg_nodes = graph_def.node\n sg_nodes = [sg_node for sg_node in sg_nodes if sg_node.op not in self.excl_types]\n nodetypes = [sg_node.op for sg_node in sg_nodes]\n nodenames = [sg_node.name for sg_node in sg_nodes]\n return nodetypes, nodenames\n\n def load_tensorflow_model(self, path):\n import tensorflow as tf\n import tensorflow_hub as hub\n self.framework = 'TENSORFLOW'\n self.neuron_optype = \"NeuronOp\"\n self.excl_types = ['Placeholder', 'PlaceholderWithDefault', 'NoOp', 'Const', 'Identity', 'IdentityN', 'VarHandleOp', 'VarIsInitializedOp', 'AssignVariableOp', 'ReadVariableOp', 'StringJoin', 'ShardedFilename', 'SaveV2', 'MergeV2Checkpoints', 'RestoreV2']\n self.addl_support = ['FusedBatchNormV3', 'BatchMatMulV2', 'AddV2', 'StopGradient', self.neuron_optype]\n model = hub.load(path)\n graph_def = model.graph.as_graph_def()\n nodes = graph_def.node\n nodes = [node for node in nodes if node.op not in self.excl_types]\n self.nodetypes = [node.op for node in nodes]\n self.nodenames = [node.name for node in nodes]\n self.neuron_nodes = [(node.name, node.attr['executable'].s, self.get_tf_subgraph_types_names(node)) for node in nodes if node.op == self.neuron_optype]\n\n def get_mx_subgraph_types_names(self, node):\n nodetypes = []\n nodenames = []\n for sg in node['subgraphs']:\n filtered_nodes = [sg_node for sg_node in sg['nodes'] if sg_node['op'] not in self.excl_types]\n nodetypes.extend([sg_node['op'] for sg_node in filtered_nodes])\n nodenames.extend([sg_node['name'] for sg_node in filtered_nodes])\n return nodetypes, nodenames\n\n def load_mxnet_model(self, path): \n import mxnet as mx\n if mx.__version__ != \"1.5.1\":\n try:\n import mxnetneuron as mxn\n except:\n raise \"Please install mxnetneuron package.\"\n self.framework = 'MXNET'\n self.neuron_optype = \"_neuron_subgraph_op\"\n self.excl_types = ['null']\n self.addl_support = [self.neuron_optype]\n sym, args, auxs = mx.model.load_checkpoint(path, 0)\n nodes = json.loads(sym.tojson())[\"nodes\"]\n nodes = [node for node in nodes if node['op'] not in self.excl_types]\n self.nodetypes = [node['op'] for node in nodes]\n self.nodenames = [node['name'] for node in nodes]\n neuron_nodes_tmp = [node for node in nodes if node['op'] == self.neuron_optype]\n self.neuron_nodes = [(node['name'], bytearray(args[node['name']+\"_neuronbin\"].asnumpy()), self.get_mx_subgraph_types_names(node)) for node in neuron_nodes_tmp]\n\n @staticmethod\n def get_cores_from_executable(executable):\n _NC_HEADER_SIZE = 544\n header = executable[:_NC_HEADER_SIZE]\n info = list(struct.unpack('168xI304xI64B', header))\n numCores = info.pop(0)\n numCoresRequested = info.pop(0)\n coresPerNode = info\n return numCores, numCoresRequested, coresPerNode\n\n # Display table of operation type or name and whether supported or not\n def print_node_type_info(self):\n self.cnt_total = len(self.nodetypes)\n self.cnt_supported = 0\n if self.parser_args.show_names:\n widthn = max(max(map(len, self.nodenames)), 8)\n widtht = max(max(map(len, self.nodetypes)), 8)\n format_str = \"{:<\" + str(widthn) + \"} {:<\" + str(widtht) + \"} {:<4}\"\n pp = lambda x: print(format_str.format(*x))\n pp(['Op Name', 'Op Type', 'Neuron Supported ?'])\n pp(['-------', '-------', '------------------'])\n for idx, opname in enumerate(self.nodenames):\n optype = self.nodetypes[idx]\n if optype in self.supported:\n pp([opname, optype, 'Yes'])\n self.cnt_supported += 1\n for idx, opname in enumerate(self.nodenames):\n optype = self.nodetypes[idx]\n if optype not in self.supported:\n pp([opname, optype, 'No'])\n else:\n count = Counter(self.nodetypes)\n width = max(max(map(len, self.nodetypes)), 8)\n format_str = \"{:<\" + str(width) + \"} {:<14} {:<4}\"\n pp = lambda x: print(format_str.format(*x))\n pp(['Op Type', 'Num Instances', 'Neuron Supported ?'])\n pp(['-------', '-------------', '------------------'])\n for key in count:\n if key in self.supported:\n pp([key, count[key], 'Yes'])\n self.cnt_supported += count[key]\n for key in count:\n if key not in self.supported:\n pp([key, count[key], 'No'])\n print()\n\n def print_subgraph_ops(self, sg_nodetypes, sg_nodenames):\n if self.parser_args.show_names:\n widthn = max(max(map(len, sg_nodenames)), 8)\n widtht = max(max(map(len, sg_nodetypes)), 8)\n format_str = \"{:<\" + str(widthn) + \"} {:<\" + str(widtht) + \"}\"\n pp = lambda x: print(' ', format_str.format(*x))\n pp(['Op Name', 'Op Type'])\n pp(['-------', '-------'])\n for idx, opname in enumerate(sg_nodenames):\n optype = sg_nodetypes[idx]\n pp([opname, optype])\n else:\n count = Counter(sg_nodetypes)\n width = max(max(map(len, sg_nodetypes)), 8)\n format_str = \"{:<\" + str(width) + \"} {:<14}\"\n pp = lambda x: print(' ', format_str.format(*x))\n pp(['Op Type', 'Num Instances'])\n pp(['-------', '-------------'])\n for key in count:\n pp([key, count[key]])\n\n def print_neuron_node_info(self):\n idx = 0\n width = max(max(map(len, self.neuronop_info)), 14) \n format_str = \"{:<\" + str(width) + \"} {:<14}\"\n pp = lambda x: print(format_str.format(*x))\n pp(['Subgraph Name', 'Num Pipelined NeuronCores'])\n pp(['-------------', '-------------------------'])\n core_cnt_list = []\n for name, (num_cores, _, sg_nodetypes, sg_nodenames) in self.neuronop_info.items():\n pp([name, num_cores])\n core_cnt_list.append(num_cores)\n idx += 1\n if self.parser_args.expand_subgraph:\n self.print_subgraph_ops(sg_nodetypes, sg_nodenames)\n print()\n\n def print_neuron_support_stats(self):\n print(\"* Total inference operations: {}\".format(self.cnt_total))\n print(\"* Total Neuron supported inference operations: {}\".format(self.cnt_supported))\n if self.cnt_total > 0:\n perc = self.cnt_supported / self.cnt_total * 100\n else:\n perc = 0\n print(\"* Percent of total inference operations supported by Neuron: {:.1f}\".format(perc))\n print()\n\n def print_common_desc(self):\n if self.parser_args.show_names:\n print(\"* Each line shows an operation name and whether the type of that operation is supported in Neuron.\")\n else:\n print(\"* Each line shows an operation type, the number of instances of that type within model,\\n\" \\\n \"* and whether the type is supported in Neuron.\")\n print(\"* Some operation types are excluded from table because they are no-operations or training-related operations:\\n\", \\\n self.excl_types, \"\\n\")\n\n def run(self):\n if len(self.neuronop_info) > 0:\n print(\"\\n* Found {} Neuron subgraph(s) ({}(s)) in this compiled model.\\n\" \\\n \"* Use this tool on the original uncompiled model to see Neuron supported operations.\\n\" \\\n \"* The following table shows all operations, including Neuron subgraphs.\".format(len(self.neuronop_info), self.neuron_optype))\n self.print_common_desc()\n self.print_node_type_info()\n print('* Please run this model on Inf1 instance with at least {} NeuronCore(s).'.format(self.min_required_pipeline_cores))\n print(\"* The following list show each Neuron subgraph with number of pipelined NeuronCores used by subgraph\\n\"\\\n \"* (and subgraph operations if --expand_subgraph is used):\\n\")\n self.print_neuron_node_info()\n else:\n print(\"\\n* The following table shows the supported and unsupported operations within this uncompiled model.\")\n self.print_common_desc()\n self.print_node_type_info()\n self.print_neuron_support_stats()\n \nif __name__=='__main__':\n toolkit = neuron_parser()\n toolkit.run()\n" }, { "path": "src/neuronperf/LICENSE", "content": "AWS Neuron License Agreement\n\nTHIS IS AN AGREEMENT BETWEEN YOU AND AMAZON WEB SERVICES, INC. (WITH ITS AFFILIATES, \"AWS\" OR \"WE\") THAT GOVERNS YOUR USE OF THE AWS NEURON SOFTWARE (TOGETHER WITH ANY UPDATES AND UPGRADES TO IT, AND ACCOMPANYING DOCUMENTATION, THE “SOFTWARE”) THAT WE MAKE AVAILABLE TO YOU. IF YOU DOWNLOAD, INSTALL, OR USE THE SOFTWARE, YOU ACCEPT AND AGREE TO BE BOUND BY THIS AGREEMENT AND REPRESENT THAT YOU HAVE THE AUTHORITY TO BIND YOURSELF OR THE ENTITY YOU REPRESENT TO THIS AGREEMENT.\n\n1.\tUse of the Software\nWe hereby grant you a personal, limited, nonexclusive, non-transferable, non-sublicenseable, revocable, royalty-free, worldwide license during the term of this Agreement to install and use the Software in connection with AWS Services. You may not use the Software if you do not have an account in good standing with AWS. Some components of the Software (whether developed by AWS or third parties) may also be governed by applicable open source software licenses located in the software component's source code. Your license rights with respect to these individual components are defined by the applicable open source software license, and nothing in this Agreement will restrict, limit, or otherwise affect any rights or obligations you may have, or conditions to which you may be subject, under such open source software licenses. “AWS Services” means each of the services made available by AWS as may be updated by AWS from time to time in its sole discretion at https://aws.amazon.com/service-terms/ and are subject to your AWS Customer Agreement or AWS Enterprise Agreement.\n\n2.\tLimitations\n\nYou may not, and you will not encourage, assist or authorize any other person to (a) sell, rent, lease, lend, loan, distribute, act as a service bureau, publicly communicate, transform, or sub-license the Software or otherwise assign any rights to the Software in whole or in part, (b) modify, alter, tamper with, repair, or otherwise create derivative works of the Software, (c) reverse engineer, disassemble, or decompile the Software or apply any other process or procedure to derive the source code of any software included in the Software, or (d) access or use the Software or the AWS Service in a way intended to avoid incurring fees or exceeding usage limits or quotas. All rights granted to you are conditioned on your continued compliance with this Agreement, and will immediately and automatically terminate if you do not comply with any term or condition of this Agreement or the AWS Customer Agreement or AWS Enterprise Agreement, including any failure to remit timely payment for the Software or the AWS Service. You will not use the Software with any software or other materials that are subject to licenses or restrictions (e.g., open source software licenses) that, when combined with the Software, would require us to disclose, license, distribute or otherwise make all or any part of such Software available to anyone. You will not remove, modify, or obscure any copyright, patent, trademark or other proprietary or attribution notices on or in any Software.\n\n3.\tReservation of Rights\n\nYou may not use the Software for any illegal purpose. The Software is the intellectual property of AWS or its licensors. The structure, organization, and code of the Software are valuable trade secrets and AWS confidential information. The Software is protected by applicable law, including without limitation copyright laws and international treaty provisions. Except for the rights expressly granted to you in this Agreement, all right, title and interest in the Software are reserved and retained by AWS and our licensors. You do not acquire any intellectual property or other rights in the Software as a result of downloading, installing, or using the Software.\n\n4.\tUpdates\n\nIn order to keep the Software up-to-date, we may offer automatic or manual updates at any time. If we elect to provide maintenance or support of any kind, we may terminate that maintenance or support at any time without notice to you.\n\n5.\tTermination\n\nYou may terminate this Agreement at any time by uninstalling and destroying all copies of the Software that are in your possession or control. This Agreement (including any rights granted to you under this Agreement) will immediately and automatically terminate without notice from us if (a) you fail to comply with any term or condition of this Agreement or any other agreement you have with AWS, or (b) you fail to make timely payment for any AWS Service. In the case of termination, you must cease all downloading, installation, and use of the Software and uninstall and destroy all copies of the Software that are in your possession or control. We may modify, suspend, discontinue, or terminate your right to use part or all of the Software at any time without notice to you, and in that event we may modify the Software to make it inoperable. AWS will not be liable to you should it exercise those rights. Our failure to insist upon or enforce your strict compliance with this Agreement will not constitute a waiver of any of our rights. No waiver of any provision of this Agreement shall be effective unless in writing.\n\n6.\tDisclaimer of Warranties and Limitation of Liability\n\na. YOU EXPRESSLY ACKNOWLEDGE AND AGREE THAT INSTALLATION AND USE OF, AND ANY OTHER ACCESS TO, THE SOFTWARE IS AT YOUR SOLE RISK. THE SOFTWARE IS DELIVERED TO YOU “AS IS” WITH ALL FAULTS AND WITHOUT WARRANTY OF ANY KIND, AND AWS, ITS LICENSORS AND DISTRIBUTORS, AND EACH OF THEIR RESPECTIVE AFFILIATES AND SUPPLIERS (COLLECTIVELY, THE “RELEASED PARTIES”) DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, ACCURACY, QUIET ENJOYMENT, AND NON-INFRINGEMENT. NO ORAL OR WRITTEN INFORMATION OR ADVICE GIVEN BY A RELEASED PARTY OR AN AUTHORIZED REPRESENTATIVE OF A RELEASED PARTY WILL CREATE A WARRANTY. THE LAWS OF CERTAIN JURISDICTIONS DO NOT ALLOW THE DISCLAIMER OF IMPLIED WARRANTIES. IF THESE LAWS APPLY TO YOU, SOME OR ALL OF THE ABOVE DISCLAIMERS, EXCLUSIONS, OR LIMITATIONS MAY NOT APPLY TO YOU, AND YOU MAY HAVE ADDITIONAL RIGHTS.\n\nb. TO THE EXTENT NOT PROHIBITED BY LAW, NO RELEASED PARTY WILL BE LIABLE TO YOU FOR ANY INCIDENTAL OR CONSEQUENTIAL DAMAGES FOR BREACH OF ANY EXPRESS OR IMPLIED WARRANTY, BREACH OF CONTRACT, NEGLIGENCE, STRICT LIABILITY, OR ANY OTHER LEGAL THEORY RELATED TO THE SOFTWARE, INCLUDING WITHOUT LIMITATION ANY DAMAGES ARISING OUT OF LOSS OF PROFITS, REVENUE, DATA, OR USE OF THE APPLICATION, EVEN IF A RELEASED PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN ANY CASE, ANY RELEASED PARTY’S AGGREGATE LIABILITY UNDER THE AGREEMENT WILL BE LIMITED TO $50.00. THE LAWS OF CERTAIN JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES. IF THESE LAWS APPLY TO YOU, SOME OR ALL OF THE ABOVE EXCLUSIONS OR LIMITATIONS MAY NOT APPLY TO YOU, AND YOU MAY HAVE ADDITIONAL RIGHTS.\n\n7.\tIndemnification\n\nYou are liable for and will defend, indemnify, and hold harmless the Released Parties and their officers, directors, agents, and employees, from and against any liability, loss, damage, cost, or expense (including reasonable attorneys’ fees) arising out of your use of the Software, violation of the Agreement, violation of applicable law, or violation of any right of any person or entity, including without limitation intellectual property rights.\n\n8.\tCompliance with Laws; Export Regulations\n\nYou will comply with all export and re-export restrictions and regulations of the United States Department of Commerce and other United States and foreign agencies and authorities that may apply to the Software, and not to transfer, or encourage, assist, or authorize the transfer of the Software to a prohibited country or otherwise in violation of any applicable restrictions or regulations.\n\n9.\tU.S. Government End Users\n\nThe Software is provided to the U.S. Government as “commercial items,” “commercial computer software,” “commercial computer software documentation,” and “technical data” with the same rights and restrictions generally applicable to the Software. If you are using the Software on behalf of the U.S. Government and these terms fail to meet the U.S. Government’s needs or are inconsistent in any respect with federal law, you will immediately discontinue your use of the Software. The terms “commercial item,” “commercial computer software,” “commercial computer software documentation,” and “technical data” are defined in the Federal Acquisition Regulation and the Defense Federal Acquisition Regulation Supplement.\n\n10.\tAmendment\n\nWe may amend this Agreement at our sole discretion by posting the revised terms on the AWS website (aws.amazon.com) or within the Software. Your continued use of the Software after any amendment's effective date evidences your agreement to be bound by it. If you do not agree to a change, you must stop using the Software and terminate this Agreement.\n\n13.\tConflicts\n\nIn the event of any conflict or inconsistency among the terms and conditions of this Agreement and the existing AWS Customer Agreement or your AWS Enterprise Agreement, such conflict or inconsistency will be resolved by giving precedence to this Agreement.\n\n14.\tEntire Agreement and Severability\n\nThis is the entire agreement between AWS and you regarding the Software and supersedes all prior understandings regarding such subject matter (including any Evaluation Agreement). If any term or condition of this Agreement is deemed invalid, void, or for any reason unenforceable, that part will be deemed severable and will not affect the validity and enforceability of any remaining term or condition.\n" }, { "path": "src/neuronperf/README.md", "content": "# NeuronPerf\n\nA library for benchmarking machine learning models on accelerators.\n\n## Documentation\n\nhttps://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuronperf/index.html" }, { "path": "src/neuronperf/build.sh", "content": "#!/bin/bash\n\nset -ex\n\npython3 -m pytest -vv \\\n --verbose \\\n --ignore=build/private \\\n --cov=neuronperf \\\n --cov-report term-missing \\\n --cov-report html:build/brazil-documentation/coverage \\\n --cov-report xml:build/brazil-documentation/coverage/coverage.xml \\\n --color=yes \\\n -x \\\n test \\\n -m \"sanity or slow\"\n\npython3 setup.py bdist_wheel --dist-dir build/pip/public/neuronperf\n" }, { "path": "src/neuronperf/conf.py", "content": "\"\"\"Sphinx configuration.\"\"\"\n\nimport datetime\nimport os\nimport shutil\n\nfrom amazon_doc_utils import brazil_info\n\n# Get metadata from brazil\nbrazil_version, intersphinx_factory = brazil_info.get(\n [brazil_info.PackageVersion, brazil_info.IntersphinxFactory]\n)\n\n\ndef run_apidoc(app):\n \"\"\"Generate doc stubs using sphinx-apidoc.\"\"\"\n module_dir = os.path.join(app.srcdir, \"../src/\")\n output_dir = os.path.join(app.srcdir, \"_apidoc\")\n excludes = []\n\n # Ensure that any stale apidoc files are cleaned up first.\n if os.path.exists(output_dir):\n shutil.rmtree(output_dir)\n\n cmd = [\n \"--separate\",\n \"--module-first\",\n \"--doc-project=API Reference\",\n \"-o\",\n output_dir,\n module_dir,\n ]\n cmd.extend(excludes)\n\n try:\n from sphinx.ext import apidoc # Sphinx >= 1.7\n\n apidoc.main(cmd)\n except ImportError:\n from sphinx import apidoc # Sphinx < 1.7\n\n cmd.insert(0, apidoc.__file__)\n apidoc.main(cmd)\n\n\ndef setup(app):\n \"\"\"Register our sphinx-apidoc hook.\"\"\"\n app.connect(\"builder-inited\", run_apidoc)\n\n\n# Sphinx configuration below.\nproject = brazil_version.name\nversion = brazil_version.mv\nrelease = brazil_version.full_version\ncopyright = \"{}, Amazon.com\".format(datetime.datetime.now().year)\n\nintersphinx_mapping = intersphinx_factory.get_mapping()\n\nextensions = [\n \"sphinx.ext.autodoc\",\n \"sphinx.ext.intersphinx\",\n \"sphinx.ext.napoleon\",\n \"sphinx.ext.todo\",\n \"sphinx.ext.viewcode\",\n]\n\nsource_suffix = \".rst\"\nmaster_doc = \"index\"\n\nautoclass_content = \"class\"\nautodoc_member_order = \"bysource\"\ndefault_role = \"py:obj\"\n\nhtml_theme = \"haiku\"\nhtmlhelp_basename = \"{}doc\".format(project)\n\nnapoleon_use_rtype = False\n" }, { "path": "src/neuronperf/model_neuron_b1.csv", "content": "n_models,workers_per_model,pipeline_size,batch_size,throughput_avg,throughput_peak,latency_ms_p0,latency_ms_p50,latency_ms_p90,latency_ms_p95,latency_ms_p99,latency_ms_p100,load_avg_ms,warmup_avg_ms,e2e_avg_ms,input_avg_ms,preprocess_avg_ms,postprocess_avg_ms,infer_avg_ms,worker_avg_s,total_infs,total_s,status,model_filename,multiprocess,multiinterpreter,device_type,instance_type\n1,1,1,1,31346.0,31408.0,0.03,0.03,0.031,0.032,0.037,0.732,62.217,2.625,0.031,0.001,0.0,0.0,0.028,4.93,154704,5.0,finished,model_neuron_b1.pt,True,False,neuron,inf1.6xlarge\n16,16,1,1,380604.75,380923.0,0.03,0.032,0.054,0.054,0.057,0.938,293.806,3.266,0.043,0.001,0.0,0.0,0.039,4.7,1799549,5.0,finished,model_neuron_b1.pt,True,False,neuron,inf1.6xlarge\n1,2,1,1,51178.0,51319.0,0.035,0.036,0.037,0.039,0.047,1.13,114.118,2.713,0.037,0.001,0.0,0.0,0.033,4.88,248984,5.0,finished,model_neuron_b1.pt,True,False,neuron,inf1.6xlarge\n16,32,1,1,381098.75,383905.0,0.03,0.058,0.067,0.073,0.121,48.07,303.916,4.42,0.08,0.001,0.0,0.0,0.074,4.69,1804925,5.0,finished,model_neuron_b1.pt,True,False,neuron,inf1.6xlarge\n" }, { "path": "src/neuronperf/pyproject.toml", "content": "[tool.black]\nline-length = 100\n\n[tool.isort]\nknown_first_party = [\"neuronperf\"]\n\n[tool.pytest.ini_options]\nmarkers = [\n \"sanity\",\n \"slow\",\n]\n\n# required for compatibility with black:\nprofile = \"black\"\n\n# To maintain consistency with other settings\nline_length = 100\n" }, { "path": "src/neuronperf/src/neuronperf/__init__.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nNeuronPerf Library\n~~~~~~~~~~~~~~~~~~\n\nA library for benchmarking machine learning models on accelerators.\n\n:copyright: (c) 2022 Amazon Inc.\n:license: See LICENSE.\n\"\"\"\n\nfrom .__version__ import __title__, __description__, __url__, __version__\nfrom .__version__ import __author__, __author_email__, __license__\nfrom .__version__ import __copyright__\n\n# setup logging first\nimport logging\n\n_log_level = logging.DEBUG\nlog = logging.getLogger(__name__)\nlog.setLevel(_log_level)\n\nfrom .logging import _get_stream_handlers\n\nfor handler in _get_stream_handlers(_log_level):\n log.addHandler(handler)\n\nfrom .benchmarking import compile, benchmark, set_verbosity\nfrom .cpu import cpu\nfrom .cpu.cpu import DummyModel\nfrom .reporting import CSV_COLS, PRINT_COLS, get_reports, print_reports, write_csv, write_json\nfrom .timing import timestamp_convert, Timer\n" }, { "path": "src/neuronperf/src/neuronperf/__version__.py", "content": "__title__ = \"neuronperf\"\n__description__ = \"A benchmarking library for machine learning accelerators.\"\n__url__ = \"https://awsdocs-neuron.readthedocs-hosted.com/en/neuronperf\"\n__version__ = \"0.0.0.0\"\n__author__ = \"AWS\"\n__author_email__ = \"neuronperf@amazon.com\"\n__license__ = \"Proprietary\"\n__copyright__ = \"Copyright Amazon Web Services and its Affiliates. All rights reserved.\"\n" }, { "path": "src/neuronperf/src/neuronperf/benchmarking.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.benchmarking\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides utility functions and classes that underlie the framework benchmarkers.\n\"\"\"\n\nfrom typing import Any, Callable, Dict, List, Union\n\nimport collections\nimport concurrent\nimport concurrent.futures\nimport copy\nimport functools\nimport logging\nimport multiprocessing\nimport os\nimport psutil\nimport subprocess\nimport sys\nimport tempfile\nimport threading\nimport time\nimport traceback\n\nimport dill\n\n\nfrom . import model_index\nfrom .compile_constants import NEURONCORE_PIPELINE_CORES, FAST_MATH, FAST_MATH_OPTIONS\nfrom .reporting import get_reports\nfrom .scripts import run_benchmark_file\nfrom .timing import Timer\n\n\nlog = logging.getLogger(__name__)\n\n# Wrapper for sending back subprocess failure info. Needs to be at top level for pickle.\nBenchmarkerErrorWrapper = collections.namedtuple(\"BenchmarkerErrorWrapper\", \"trace\")\n\nERROR = \"error\"\nSUPPORTED_DEVICE_TYPES = [\"neuron\", \"cpu\", \"cuda\", \"gpu\"] # TODO: \"tpu\"]\nBENCHMARK_SECS = 120\n\n\nclass Benchmarker(threading.Thread):\n r\"\"\"\n :class:`benchmarking:Benchmarker` benchmarks a single model.\n\n This class is a `threading.Thread`. Call `start` to launch a non-blocking\n benchmarking thread. Calling `stop` will end the benchmarking and block\n until all subroutines complete.\n\n An object of this class may be serialized and sent to multiple subprocesses\n for parallel use. After benchmarking, results can be obtained with\n `results`.\n \"\"\"\n\n def __init__(\n self,\n id: int,\n device_id: int,\n load_fn: Callable[[str], Any],\n model_filename: str,\n inputs,\n workers_per_model: int,\n env_setup_fn: Callable[[int, Dict, Any], None] = None,\n setup_fn: Callable[[int, Dict, Any], None] = None,\n preprocess_fn: Callable[[Any], Any] = None,\n postprocess_fn: Callable[[Any], Any] = None,\n dataset_loader_fn: Callable[[Any, int], Any] = None,\n model_class_name: str = None,\n model_class_file: str = None,\n ):\n super().__init__()\n\n self.id = id\n self.device_id = device_id\n self.load_fn = load_fn\n self.model_filename = model_filename\n self.inputs = inputs\n self.input_iter = None # Prepared in setup()\n self.input_lock = threading.Lock()\n self.workers_per_model = workers_per_model\n self.env_setup_fn = env_setup_fn\n self.setup_fn = setup_fn\n self.preprocess_fn = preprocess_fn\n self.postprocess_fn = postprocess_fn\n self.dataset_loader_fn = dataset_loader_fn\n self.model_class_name = model_class_name\n self.model_class_file = model_class_file\n\n # Mutable internal state.\n self.model = None\n self.benchmark_timer = Timer()\n self.env_setup_timer = Timer()\n self.setup_timer = Timer()\n self.load_timer = Timer()\n self.warmup_timer = Timer()\n self.input_timer = Timer()\n self.preprocess_timers = [Timer() for _ in range(workers_per_model)]\n self.infer_timers = [Timer() for _ in range(workers_per_model)]\n self.postprocess_timers = [Timer() for _ in range(workers_per_model)]\n self.e2e_timers = [Timer() for _ in range(workers_per_model)]\n self.worker_timers = [Timer() for _ in range(workers_per_model)]\n self.n_infs = [0] * workers_per_model\n self.process_id = 0 # set at launch time\n self.benchmarking = False\n self.benchmarking_lock = threading.Lock()\n self.status_lock = threading.Lock()\n self.status = \"ready\"\n self.error = None\n\n def _status(self, status, error=None):\n \"\"\"Update internal status, unless a previous error has occurred.\"\"\"\n with self.status_lock:\n if self.status == ERROR:\n return\n self.status = status\n if error:\n self.error = error\n\n def next_input(self):\n self.input_lock.acquire()\n self.input_timer.start()\n try:\n return next(self.input_iter)\n finally:\n self.input_timer.stop()\n self.input_lock.release()\n\n def prepare_inputs(self):\n \"\"\"Prepares input iterator; runs an optional custom setup function.\"\"\"\n if self.dataset_loader_fn:\n\n def input_iter():\n dataset_loader = self.dataset_loader_fn(self.inputs, self.workers_per_model)\n while True:\n inputs = next(dataset_loader)\n yield inputs if isinstance(inputs, tuple) else (inputs,)\n\n self.input_iter = input_iter()\n else:\n\n def input_iter():\n inputs = self.inputs if isinstance(self.inputs, tuple) else (self.inputs,)\n while True:\n yield inputs\n\n self.input_iter = input_iter()\n\n def load(self):\n \"\"\"Loads the model that will be used for benchmarking.\"\"\"\n with self.load_timer:\n self.model = self.load_fn(self.model_filename, device_id=self.device_id)\n\n def warmup(self):\n \"\"\"Warmup the model with a single e2e inference.\"\"\"\n with self.warmup_timer:\n inputs = self.next_input()\n if self.preprocess_fn:\n inputs = self.preprocess_fn(*inputs)\n outputs = self.model(*inputs if isinstance(inputs, tuple) else inputs)\n if self.postprocess_fn:\n self.postprocess_fn(outputs)\n self.n_infs[0] += 1 # track warmup infs in worker 0\n\n def setup(self):\n \"\"\"Perform all setup work prior to benchmarking.\"\"\"\n self.prepare_inputs()\n\n if self.env_setup_fn:\n with self.env_setup_timer:\n self.env_setup_fn()\n\n self.load()\n\n if self.setup_fn:\n with self.setup_timer:\n self.setup_fn(self.model)\n\n self.warmup()\n\n def infer(self, worker_id) -> tuple:\n \"\"\"Execute a single inference.\"\"\"\n with self.e2e_timers[worker_id]:\n inputs = self.next_input()\n if self.preprocess_fn:\n with self.preprocess_timers[worker_id]:\n inputs = self.preprocess_fn(*inputs)\n with self.infer_timers[worker_id]:\n outputs = self.model(*inputs if isinstance(inputs, tuple) else inputs)\n if self.postprocess_fn:\n with self.postprocess_timers[worker_id]:\n outputs = self.postprocess_fn(outputs)\n return outputs\n\n def worker_thread(self, worker_id):\n \"\"\"A single worker thread that runs inference until signalled to stop.\"\"\"\n n_infs = 0\n try:\n log.debug(f\"Benchmarker {self.id}, Worker {worker_id} started.\")\n with self.worker_timers[worker_id]:\n while self.benchmarking and self.status != ERROR:\n self.infer(worker_id)\n n_infs += 1\n if self.status == ERROR:\n log.debug(\n f\"Benchmarker {self.id}, Worker {worker_id} stopped early due to an error after {n_infs} inferences.\"\n )\n except StopIteration:\n pass\n except:\n trace = \"\".join(traceback.format_exception(*sys.exc_info()))\n log.error(\n f\"Benchmarker {self.id}, Worker {worker_id} encountered an error during benchmarking:\\n{trace}\"\n )\n self._status(ERROR, BenchmarkerErrorWrapper(trace))\n finally:\n self.n_infs[worker_id] += n_infs\n log.debug(\n f\"Benchmarker {self.id}, Worker {worker_id} finished after {self.n_infs[worker_id]} inferences.\"\n )\n\n def run(self):\n with self.benchmarking_lock:\n if self.benchmarking:\n raise RuntimeError(\n f\"Benchmarker {self.id} can't start because it is already running.\"\n )\n self.benchmarking = True\n self._status(\"running\")\n\n # Set our process id, now that we are launched.\n self.process_id = os.getpid()\n\n # Launch all workers and begin benchmarking.\n # If any individual worker reports an error, self.status will reflect\n # that after this method.\n with self.benchmark_timer:\n try:\n self.setup()\n except:\n trace = \"\".join(traceback.format_exception(*sys.exc_info()))\n log.error(f\"Benchmarker {self.id} encountered an error during prep:\\n{trace}\")\n self._status(ERROR, BenchmarkerErrorWrapper(trace))\n else:\n with concurrent.futures.ThreadPoolExecutor(max_workers=self.workers_per_model) as exe:\n for worker_id in range(self.workers_per_model):\n exe.submit(self.worker_thread, worker_id)\n\n # There are three ways to reach the next section:\n # 1. We ran out of benchmarking examples in a provided dataset (graceful quit on StopIteration).\n # 2. We were asked to stop().\n # 3. We encountered an error.\n\n # In cases 1 and 3, we can acquire the lock, update our state if necessary, and quit.\n # In case 2, we already hold the lock, so we can skip this section and let stop() handle cleanup.\n if self.benchmarking_lock.acquire(blocking=False):\n try:\n self.benchmarking = False\n self._status(\"finished\")\n finally:\n self.benchmarking_lock.release()\n\n def stop(self):\n # Setting self.benchmarking = False triggers workers to terminate gracefully.\n # We must hold the benchmarking_lock until the thread has joined to ensure\n # consistent use of the self.benchmarking flag.\n with self.benchmarking_lock:\n if not self.benchmarking:\n return\n self._status(\"stopping\")\n self.benchmarking = False\n self.join()\n self._status(\"finished\")\n\n def results(self) -> dict:\n with self.benchmarking_lock:\n if self.benchmarking:\n raise RuntimeError(\"Cannot produce results until benchmarking has completed.\")\n return {\n \"id\": self.id,\n \"device_id\": self.device_id,\n \"workers_per_model\": self.workers_per_model,\n \"n_infs\": sum(self.n_infs),\n \"status\": self.status,\n \"process_id\": self.process_id,\n \"total_s\": self.benchmark_timer.total_duration(\"s\"),\n \"timers\": {\n \"env_setup\": [self.env_setup_timer],\n \"setup\": [self.setup_timer],\n \"load\": [self.load_timer],\n \"input\": [self.input_timer],\n \"warmup\": [self.warmup_timer],\n \"preprocess\": self.preprocess_timers,\n \"infer\": self.infer_timers,\n \"postprocess\": self.postprocess_timers,\n \"e2e\": self.e2e_timers,\n \"worker\": self.worker_timers,\n },\n }\n\n\nclass StatsThread(threading.Thread):\n \"\"\"A thread to collect some system metrics duirng benchmarking.\"\"\"\n\n def __init__(self, interval: float):\n super().__init__()\n self.interval = interval # interval (in seconds) to collect metrics\n self.cpu_percents = []\n self.mem_percents = []\n self.running = True\n\n def run(self):\n while self.running:\n cpu_percent = psutil.cpu_percent(interval=self.interval, percpu=False)\n mem_percent = psutil.virtual_memory()[2]\n self.cpu_percents.append(cpu_percent)\n self.mem_percents.append(mem_percent)\n\n def join(self, **kwargs):\n self.running = False\n super().join(**kwargs)\n\n\ndef _combine_results(results: List[dict]) -> dict:\n \"\"\"Combines the results of multiple benchmarkers into a single results structure.\"\"\"\n combined_results = {}\n for result in results:\n # workers_per_model should be the same across all benchmarkers, so we only need it once.\n combined_results.setdefault(\"workers_per_model\", result[\"workers_per_model\"])\n # If an error occurred anywhere, preserve it.\n combined_results[\"status\"] = (\n result[\"status\"] if combined_results.get(\"status\", \"\") != ERROR else ERROR\n )\n combined_results[\"n_infs\"] = combined_results.get(\"n_infs\", 0) + result[\"n_infs\"]\n # Keep the longest subprocess duration.\n combined_results[\"total_s\"] = max(combined_results.get(\"total_s\", 0), result[\"total_s\"])\n # Concatenate all timing info.\n timers = combined_results.get(\"timers\", {})\n for k, v in result[\"timers\"].items():\n timer_list = timers.get(k, [])\n timer_list.extend(v)\n timers[k] = timer_list\n combined_results[\"timers\"] = timers\n return combined_results\n\n\ndef _get_num_workers(pipeline_size: int) -> int:\n \"\"\"Returns a best-guess number of worker threads for a single benchmarking process.\"\"\"\n return 2 if pipeline_size == 1 else pipeline_size - 1\n\n\ndef get_instance_type() -> str:\n \"\"\"Try to obtain the maximum number of NeuronCores available on this instance.\"\"\"\n try:\n import urllib.request\n\n with urllib.request.urlopen(\n \"http://169.254.169.254/latest/meta-data/instance-type\"\n ) as response:\n instance_type = response.read().decode(\"utf-8\")\n log.debug(\"Automatically determined instance type: {}\".format(instance_type))\n return instance_type\n except:\n return None\n\n\ndef _get_cost_per_hour(instance_type: str) -> float:\n # Hourly rates\n instancetype_to_cost = {\n \"inf1.xlarge\": 0.228,\n \"inf1.2xlarge\": 0.362,\n \"inf1.6xlarge\": 1.18,\n \"inf1.24xlarge\": 4.721,\n }\n try:\n return instancetype_to_cost[instance_type]\n except:\n # Just ignore unknown instance types for now\n return None\n\n\ndef _get_max_neuroncores(instance_type: str = None) -> int:\n \"\"\"Try to obtain the maximum number of NeuronCores available on this instance.\"\"\"\n instancetype_to_neuroncores = {\n \"inf1.xlarge\": 4,\n \"inf1.2xlarge\": 4,\n \"inf1.6xlarge\": 16,\n \"inf1.24xlarge\": 64,\n }\n try:\n if not instance_type:\n instance_type = get_instance_type()\n return instancetype_to_neuroncores[instance_type]\n except:\n num_cores = 2\n log.warning(f\"Unknown Neuron device size. Assuming {num_cores} NeuronCores is the maximum.\")\n return num_cores\n\n\ndef _get_num_gpus(instance_type: str = None) -> int:\n \"\"\"Try to obtain the maximum number of NeuronCores available on this instance.\"\"\"\n instancetype_to_gpus = {\n \"g4dn.xlarge\": 1,\n \"g4dn.2xlarge\": 1,\n \"g4dn.4xlarge\": 1,\n \"g4dn.8xlarge\": 1,\n \"g4dn.16xlarge\": 1,\n \"g4dn.12xlarge\": 4,\n \"g4dn.metal\": 8,\n \"g4ad.xlarge\": 1,\n \"g4ad.2xlarge\": 1,\n \"g4ad.4xlarge\": 1,\n \"g4ad.8xlarge\": 2,\n \"g4ad.16xlarge\": 4,\n \"p4d.24xlarge\": 8,\n }\n try:\n if not instance_type:\n instance_type = get_instance_type()\n return instancetype_to_gpus[instance_type]\n except:\n log.warning(\"Unknown GPU device size. Assuming 1 GPU is available.\")\n return 1\n\n\ndef _get_num_devices(device_type: str, instance_type: str = None) -> int:\n \"\"\"This is a stub, to be populated later for other instance types.\"\"\"\n if device_type == \"neuron\":\n return _get_max_neuroncores(instance_type)\n elif device_type == \"cpu\":\n return multiprocessing.cpu_count()\n elif device_type == \"cuda\" or device_type == \"gpu\":\n return _get_num_gpus(instance_type)\n else:\n log.warning(\"An unknown device_type was passed: {}\".format(device_type))\n return None\n\n\ndef _sanitize_inputs(inputs, batch_sizes: Union[int, List[int]], dataset_inputs=False) -> List[int]:\n \"\"\"Return inputs and batch_sizes with matching lengths, or throw an error.\"\"\"\n if not isinstance(inputs, list):\n inputs = [inputs]\n if isinstance(batch_sizes, int):\n batch_sizes = [batch_sizes]\n if not batch_sizes:\n log.warning(\n \"Batch sizes were not provided, so assuming 1 and only the first input will be benchmarked.\"\n )\n batch_sizes = [1]\n if not dataset_inputs:\n if len(batch_sizes) < len(inputs):\n delta = len(inputs) - len(batch_sizes)\n log.warning(\n \"Received {} inputs, but only {} batch sizes. Discarding last {} inputs.\".format(\n len(inputs), len(batch_sizes), delta\n )\n )\n inputs = inputs[: len(batch_sizes)]\n elif len(inputs) < len(batch_sizes):\n delta = len(batch_sizes) - len(inputs)\n log.warning(\n \"Received {} batch sizes, but only {} inputs. Discarding last {} batch sizes.\".format(\n len(batch_sizes), len(inputs), delta\n )\n )\n batch_sizes = batch_sizes[: len(inputs)]\n return inputs, batch_sizes\n\n\ndef set_verbosity(verbosity: int):\n r\"\"\"\n Controls the verbosty of NeuronPerf logging.\n\n :param int verbosity: 0 = error, 1 = info, 2 = debug\n \"\"\"\n if 0 == verbosity:\n log.setLevel(logging.ERROR)\n elif 1 == verbosity:\n log.setLevel(logging.INFO)\n else:\n log.setLevel(logging.DEBUG)\n\n\ndef compile(\n compile_fn,\n model,\n inputs,\n batch_sizes: Union[int, List[int]] = None,\n pipeline_sizes: Union[int, List[int]] = None,\n performance_levels: Union[str, List[int]] = None,\n models_dir: str = \"models\",\n model_name: str = None,\n filename: str = None,\n compiler_args: dict = None,\n verbosity: int = 1,\n **kwargs,\n) -> str:\n r\"\"\"\n Compiles the provided model with each provided example input, pipeline size, and performance level.\n\n :param model: The model to compile.\n :param list inputs: A list of example inputs.\n :param Union[int, List[int]] batch_sizes: A list of batch sizes that correspond to the example inputs.\n :param Union[int, List[int]] pipeline_sizes: A list of pipeline sizes to use. See :ref:`neuroncore-pipeline`.\n :param Union[int, List[int]] performance_levels: A list of performance levels to try. Options are: 0 (max accuracy), 1, 2, 3 (max performance, default). See :ref:`mixed-precision`.\n :param str models_dir: The directory where compilation artifacts will be stored.\n :param str model_name: An optional model name tag to apply to compiled artifacts.\n :param str filename: The name of the model index to write out. If not provided, a name will be generated and returned.\n :param dict compiler_args: Additional compiler arguments to be forwarded with every compilation.\n :param int verbosity: 0 = error, 1 = info, 2 = debug\n :return: A model index filename. If a configuration fails to compile, it will not be included in the index and an error will be logged.\n :rtype: str\n \"\"\"\n # Set NeuronPerf logging verbosity.\n set_verbosity(verbosity)\n\n # Standardize arguments.\n if not pipeline_sizes:\n pipeline_sizes = [1]\n if not performance_levels:\n performance_levels = []\n if not compiler_args:\n compiler_args = {}\n if not model_name:\n if isinstance(model, str):\n model_name = model\n else:\n try:\n model_name = model.__name__\n except AttributeError:\n log.warning(\"Unable to determine a model name, using 'Model'.\")\n model_name = \"Model\"\n if isinstance(pipeline_sizes, int):\n pipeline_sizes = [pipeline_sizes]\n if isinstance(performance_levels, int):\n performance_levels = [performance_levels]\n\n inputs, batch_sizes = _sanitize_inputs(inputs, batch_sizes)\n\n # Sanity check and sanitize compiler_args.\n if NEURONCORE_PIPELINE_CORES in compiler_args:\n if pipeline_sizes:\n log.warning(\n (\n \"You provided NeuronCore Pipeline Core sizes using both \"\n \"compiler_args and pipeline_sizes. Ignoring flag in compiler_args.\"\n )\n )\n else:\n pipeline_sizes = [compiler_args[NEURONCORE_PIPELINE_CORES]]\n del compiler_args[NEURONCORE_PIPELINE_CORES]\n\n if FAST_MATH in compiler_args:\n if performance_levels:\n log.warning(\n (\n f\"You provided performance_levels and {FAST_MATH}. \"\n \"Ignoring flag in compiler_args.\"\n )\n )\n del compiler_args[FAST_MATH]\n\n # Check if performance levels are within expected bounds.\n max_performance = max(FAST_MATH_OPTIONS)\n performance_levels_invalid = list(\n filter(\n lambda level: level < min(FAST_MATH_OPTIONS) or level > max_performance,\n performance_levels,\n )\n )\n if performance_levels_invalid:\n log.warning(\n \"You provided some invalid performance_levels. Ignoring: {}\".format(\n performance_levels_invalid\n )\n )\n performance_levels = [\n level\n for level in performance_levels\n if (level in performance_levels) and (level not in performance_levels_invalid)\n ]\n\n # If we still have no values, set default to max performance.\n if not performance_levels:\n performance_levels.append(max_performance)\n\n # Create standard output dir, if it doesn't exit.\n os.makedirs(models_dir, exist_ok=True)\n\n # Compile all requested model combinations.\n model_idxs = []\n\n # TODO: Support appending to existing index by filtering already-compiled configs.\n def make_index():\n \"\"\"Create a model index file that contains info about all compiled models.\"\"\"\n index = model_index.append(*model_idxs)\n # Return the name of the new index file.\n return model_index.save(index, filename=filename)\n\n compile_idx = 1\n n_compiles = len(inputs) * len(pipeline_sizes) * len(performance_levels)\n for input_idx, example_input in enumerate(inputs):\n batch_size = batch_sizes[input_idx]\n for pipeline_size in pipeline_sizes:\n for performance_level in performance_levels:\n _compiler_args = copy.copy(compiler_args)\n _compiler_args[FAST_MATH] = FAST_MATH_OPTIONS[performance_level]\n if pipeline_size != 1:\n _compiler_args[NEURONCORE_PIPELINE_CORES] = str(pipeline_size)\n\n # Construct a more informative model name with some config info\n model_name_ex = \"{}_b{}_p{}_{}\".format(\n model_name,\n batch_size,\n pipeline_size,\n model_index.generate_id(),\n )\n log.info(\n (\n f\"Compiling batch size {batch_size} for {pipeline_size} NeuronCore(s) with performance level \"\n f\"{performance_level}/{max_performance}. [{compile_idx}/{n_compiles}]\"\n )\n )\n status = \"ready\"\n timer = Timer()\n with timer:\n try:\n model_filename = compile_fn(\n model,\n example_input,\n models_dir,\n model_name_ex,\n compiler_args=_compiler_args,\n **kwargs,\n )\n status = \"finished\"\n except KeyboardInterrupt:\n status = \"error\"\n model_filename = None\n log.error(\"Compilation interrupted, terminating.\")\n return make_index()\n except:\n status = \"error\"\n model_filename = None\n log.exception(\n (\n f\"Failed to compile input={input_idx}, \"\n f\"batch_size={batch_size}, \"\n f\"pipeline_size={pipeline_size}, \"\n f\"performance_level={performance_level}.\"\n )\n )\n finally:\n model_idx = model_index.create(\n model_filename,\n model_name=model_name,\n batch_size=batch_size,\n pipeline_size=pipeline_size,\n performance_level=performance_level,\n compile_s=round(timer.total_duration(\"s\"), 2),\n status=status,\n )\n model_idxs.append(model_idx)\n filename = make_index()\n compile_idx += 1\n return filename\n\n\ndef run_benchmarker(benchmarker, duration, pipe=None):\n def _send(results):\n if pipe:\n pipe.send(results)\n pipe.close()\n else:\n return results\n\n try:\n log.debug(f\"Benchmarker {benchmarker.id} started.\")\n check_freq = 0.1 # Check progress every 0.1 seconds.\n start_time = time.time()\n benchmarker.start()\n elapsed = 0\n while (elapsed < duration) and benchmarker.benchmarking:\n elapsed = time.time() - start_time\n remaining = max(0, duration - elapsed)\n time.sleep(min(check_freq, remaining))\n benchmarker.stop()\n except:\n trace = \"\".join(traceback.format_exception(*sys.exc_info()))\n error = BenchmarkerErrorWrapper(trace)\n return _send(error)\n else:\n results = benchmarker.results() if benchmarker.status != ERROR else benchmarker.error\n return _send(results)\n finally:\n log.debug(f\"Benchmarker {benchmarker.id} finished.\")\n\n\ndef _run_benchmarker_new_interpreter(benchmarker, duration):\n \"\"\"\n This function is a workaround for frameworks that cannot be safely forked.\n The premise is to launch a new Python interpreter and run benchmarking\n from within the new interpreter. It works by writing serialized benchmarkers\n to temporary files, and then launching run_benchmark_file.py. The script\n writes back serialized results.\n \"\"\"\n\n # Temporary serialization workaround. This attribute is inherited from Thread.\n # TODO: Separate data from benchmarking.\n setattr(benchmarker, \"_stderr\", None)\n\n script = run_benchmark_file.__file__\n\n # Serialize the benchmarker to a file.\n f = tempfile.NamedTemporaryFile(delete=False)\n log.debug(\"Dumping Benchmarker {} to file '{}'.\".format(benchmarker.id, f.name))\n try:\n dill.dump(benchmarker, f)\n except dill.PicklingError:\n raise dill.PicklingError(\n (\n \"NeuronPerf was unable to serialize the benchmarker. This is probably becuause your model \"\n \"could not be serialized. Make sure to use top-level classes instead of locals. You may \"\n \"need to wrap your model and manually load it using Python's importlib.\"\n )\n )\n f.close()\n\n # Run the benchmarking script in a clean Python process.\n command = [\n sys.executable,\n script,\n f.name,\n str(duration),\n ]\n\n # If we are manually loading a model class file in subprocesses, we need to let them know.\n if benchmarker.model_class_name and benchmarker.model_class_file:\n command.append(f\"--model_class_name={benchmarker.model_class_name}\")\n command.append(f\"--model_class_file={benchmarker.model_class_file}\")\n\n proc = subprocess.Popen(\n command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding=\"utf-8\"\n )\n\n # Interpreter and framework overhead add a delay to processing. We should ensure\n # that during multiinterpreter benchmarking, sufficient time is waited for results.\n timeout = 60 + duration\n\n try:\n outs, errs = proc.communicate(timeout=timeout)\n with open(f.name, \"rb\") as fp:\n result = dill.load(fp)\n if isinstance(result, BenchmarkerErrorWrapper):\n raise ChildProcessError(\n \"Benchmarker {} encountered an error:\\n{}\".format(benchmarker.id, result.trace)\n )\n if isinstance(result, Benchmarker):\n # If we still have a benchmarker object instead of results, something\n # went wrong that wasn't handled by the benchmarker routine.\n from pathlib import Path\n\n path = Path(f.name)\n logs = os.path.join(path.parent, \"neuronperf_error_{}\".format(str(path.stem)))\n if os.path.exists(logs):\n with open(logs, \"rt\") as logs_fp:\n err_logs = logs_fp.readlines()\n os.unlink(logs)\n raise ChildProcessError(\n \"Benchmarker {} failed. Logs from child process:\\n{}\".format(\n benchmarker.id, \"\".join(err_logs)\n )\n )\n else:\n raise ChildProcessError(\n (\n \"Benchmarker {} failed and no error logs were found. A child process may have \"\n \"aborted. To obtain a stack trace, try running a single configuration inside a \"\n \"single process by passing multiprocess=False, multiinterpreter=False\"\n )\n )\n\n return result\n except subprocess.TimeoutExpired:\n proc.kill()\n raise ChildProcessError(\n \"Benchmarker {} stopped responding after {} seconds.\".format(benchmarker.id, timeout)\n )\n finally:\n os.unlink(f.name)\n\n\ndef _run_benchmarkers_multiprocess(\n benchmarkers: List[Benchmarker], duration: int, benchmark_func=run_benchmarker\n) -> dict:\n results = []\n # Hand each benchmarker object to a subprocess.\n pipes, procs = [], []\n for benchmarker in benchmarkers:\n parent_pipe, child_pipe = multiprocessing.Pipe()\n pipes.append(parent_pipe)\n proc = multiprocessing.Process(\n target=benchmark_func, args=(benchmarker, duration, child_pipe)\n )\n procs.append(proc)\n # Launch benchmarking.\n for proc in procs:\n proc.start()\n # Collect results.\n for id, (pipe, proc) in enumerate(zip(pipes, procs)):\n try:\n proc_result = pipe.recv()\n if isinstance(proc_result, BenchmarkerErrorWrapper):\n log.error(\"Child process encountered an error:\\n{}\".format(proc_result.trace))\n raise ChildProcessError()\n proc.join()\n results.append(proc_result)\n except KeyboardInterrupt:\n log.error(\"Benchmarking interrupted, terminating.\")\n for proc in procs:\n proc.terminate()\n raise KeyboardInterrupt()\n except EOFError:\n log.error(\n (\n f\"Child process {id} was killed by the host OS during benchmarking.\\n\"\n \"You may have run out of memory.\\n\"\n \"Verify that your model can perform inference without NeuronPerf or try n_models=1.\"\n )\n )\n return _combine_results(results)\n\n\ndef _run_benchmarkers_multithreaded(\n benchmarkers: List[Benchmarker], duration: int, benchmark_func=run_benchmarker\n) -> dict:\n results = []\n timeout = 60 + duration # Add some time for setup overhead and cleanup.\n try:\n args = ((benchmarker, duration) for benchmarker in benchmarkers)\n with concurrent.futures.ThreadPoolExecutor(max_workers=len(benchmarkers)) as exe:\n results.extend(exe.map(lambda arg: benchmark_func(*arg), args, timeout=timeout))\n for result in results:\n if isinstance(result, BenchmarkerErrorWrapper):\n raise RuntimeError(\"Worker thread encountered an error:\\n{}\".format(result.trace))\n except concurrent.futures.TimeoutError:\n log.error(\"Benchmarking timed out after {} seconds.\".format(timeout))\n except KeyboardInterrupt:\n raise KeyboardInterrupt(\"Benchmarking interrupted, terminating.\")\n return _combine_results(results)\n\n\ndef run_benchmarkers(\n benchmarkers: List[Benchmarker],\n duration: int,\n stats_interval: float = 0.5,\n multiprocess: bool = True,\n multiinterpreter: bool = False,\n) -> dict:\n results = {}\n\n # Launch a background thread to collect system stats during benchmarking.\n stats_thread = StatsThread(stats_interval)\n stats_thread.start()\n\n try:\n if multiinterpreter:\n if not sys.executable:\n raise ValueError(\n (\n \"Unable to benchmark in multi-interpreter mode because \"\n \"the Python interpreter cannot be located (sys.executable is empty).\"\n )\n )\n # We can safely re-use the multithreaded path here by using a custom benchmarking\n # function that spawns fresh interpreters.\n results = _run_benchmarkers_multithreaded(\n benchmarkers, duration, benchmark_func=_run_benchmarker_new_interpreter\n )\n elif multiprocess:\n results = _run_benchmarkers_multiprocess(benchmarkers, duration)\n else:\n results = _run_benchmarkers_multithreaded(benchmarkers, duration)\n finally:\n stats_thread.join()\n results[\"cpu_percents\"] = stats_thread.cpu_percents\n results[\"mem_percents\"] = stats_thread.mem_percents\n\n return results\n\n\ndef _get_env_setup_fn(benchmarker_id: int, benchmarker_config: dict, env_setup_fn):\n \"\"\"Wrap an environment setup function with device-specific requirements.\"\"\"\n device_type = str(benchmarker_config[\"device_type\"]).lower().strip()\n legacy = bool(os.environ.get(\"NEURONCORE_GROUP_SIZES\"))\n if \"neuron\" == device_type:\n\n @functools.wraps(env_setup_fn)\n def _env_setup_fn():\n import os\n\n id = benchmarker_id\n config = benchmarker_config\n pipeline_size = config[\"pipeline_size\"]\n if config[\"multiprocess\"] or config[\"multiinterpreter\"]:\n # In multiprocess mode, need to specify the exact cores for the process.\n min_core = pipeline_size * id\n max_core = min_core + (pipeline_size - 1)\n visible_cores = f\"{min_core}-{max_core}\"\n\n if legacy:\n os.environ[\"NEURONCORE_GROUP_SIZES\"] = str(pipeline_size)\n else:\n os.environ[\"NEURON_RT_VISIBLE_CORES\"] = visible_cores\n else:\n # In multithreaded mode, all required cores are allocated in this process.\n n_models = config[\"n_models\"]\n if legacy:\n os.environ[\"NEURONCORE_GROUP_SIZES\"] = \",\".join([str(pipeline_size)] * n_models)\n else:\n os.environ[\"NEURON_RT_VISIBLE_CORES\"] = \"0-{}\".format(\n n_models * pipeline_size - 1\n )\n\n # Finally, call any additional custom setup function provided.\n if env_setup_fn:\n env_setup_fn(id, config)\n\n return _env_setup_fn\n elif device_type == \"cpu\":\n return env_setup_fn\n elif device_type == \"cuda\" or device_type == \"gpu\":\n\n @functools.wraps(env_setup_fn)\n def _env_setup_fn():\n import os\n\n os.environ[\"CUDA_VISIBLE_DEVICES\"] = str(benchmarker_id)\n\n if env_setup_fn:\n env_setup_fn(benchmarker_id, benchmarker_config)\n\n return _env_setup_fn\n else:\n log.warning(\n (\n f\"NeuronPerf does not implement a proper environment setup for {device_type}. \"\n \"You may need to provide your own.\"\n )\n )\n return env_setup_fn\n\n\ndef _get_setup_fn(benchmarker_id: int, benchmarker_config: dict, setup_fn):\n \"\"\"Wraps a customer provided setup function with additional info from the benchmarker.\"\"\"\n if not setup_fn:\n return None\n\n @functools.wraps(setup_fn)\n def _setup_fn(model):\n setup_fn(benchmarker_id, benchmarker_config, model)\n\n return _setup_fn\n\n\ndef _get_device_id(benchmarker_id: int, benchmarker_config: dict):\n \"\"\"Calculate an appropriate device id for a benchmarker object.\"\"\"\n device_id = benchmarker_id\n device_type = str(benchmarker_config[\"device_type\"]).lower().strip()\n if device_type in SUPPORTED_DEVICE_TYPES:\n if not (benchmarker_config[\"multiprocess\"] or benchmarker_config[\"multiinterpreter\"]):\n device_id = benchmarker_id * benchmarker_config[\"pipeline_size\"]\n return device_id\n else:\n log.warning(\n \"Assuming device_id={} for benchmarker_id={} for unknown device_type={}\".format(\n device_id, benchmarker_id, device_type\n )\n )\n return device_id\n\n\ndef benchmark(\n load_fn: Callable[[str, int], Any],\n model_filename: str,\n inputs: Any,\n batch_sizes: Union[int, List[int]] = None,\n duration: float = BENCHMARK_SECS,\n n_models: Union[int, List[int]] = None,\n pipeline_sizes: Union[int, List[int]] = None,\n performance_levels: Union[int, List[int]] = None,\n workers_per_model: Union[int, None] = None,\n env_setup_fn: Callable[[int, Dict], None] = None,\n setup_fn: Callable[[int, Dict, Any], None] = None,\n preprocess_fn: Callable[[Any], Any] = None,\n postprocess_fn: Callable[[Any], Any] = None,\n dataset_loader_fn: Callable[[Any, int], Any] = None,\n multiprocess: bool = True,\n multiinterpreter: bool = False,\n return_timers: bool = False,\n stats_interval: float = 0.5,\n device_type: str = \"neuron\",\n cost_per_hour: float = None,\n model_name: str = None,\n model_class_name: str = None,\n model_class_file: str = None,\n verbosity: int = 1,\n) -> List[Dict]:\n r\"\"\"\n Benchmarks the model index or individiual model using the provided inputs.\n If a model index is provided, additional fields such as ``pipeline_sizes`` and\n ``performance_levels`` can be used to filter the models to benchmark. The default\n behavior is to benchmark all configurations in the model index. Any additional\n compiler_args passed will be forwarded to the compiler on every invocation.\n\n :param Callable[[str, int], Any] load_fn: A function that accepts a model filename and device id, and returns a loaded model. This is automatically passed through the subpackage calls (e.g. ``neuronperf.torch.benchmark``).\n :param str model_filename: A path to a model index from compile or path to an individual model. For CPU benchmarking, a class should be passed that can be instantiated with a default constructor (e.g. ``MyModelClass``).\n :param list inputs: A list of example inputs. If the list contains tuples, they will be destructured on inference to support multiple arguments.\n :param Union[int, List[int]] batch_sizes: A list of ints indicating batch sizes that correspond to the inputs. Assumes 1 if not provided.\n :param duration float: The number of seconds to benchmark each model.\n :param n_models Union[int, List[int]]: The number of models to run in parallel. Default behavior runs 1 model and the max number of models possible, determined by a best effort from ``device_type``, instance size, or other environment state.\n :param Union[int, List[int]] pipeline_sizes: A list of pipeline sizes to use. See :ref:`neuroncore-pipeline`.\n :param Union[int, List[int]] performance_levels: A list of performance levels to try. Options are: 0 (max accuracy), 1, 2, 3 (max performance, default). See :ref:`mixed-precision`.\n :param Union[int, List[int]] workers_per_model: The number of workers to use per model loaded. If ``None``, this is automatically selected.\n :param Callable[[int, Dict], None] env_setup_fn: A custom environment setup function to run in each subprocess before model loading. It will receive the benchmarker id and config.\n :param Callable[[int, Dict, Any], None] setup_fn: A function that receives the benchmarker id, config, and model to perform last minute configuration before inference.\n :param Callable[[Any], Any]: preprocess_fn: A custom preprocessing function to perform on each input before inference.\n :param Callable[[Any], Any]: postprocess_fn: A custom postprocessing function to perform on each input after inference.\n :param bool multiprocess: When True, model loading is dispatched to forked subprocesses. Should be left alone unless debugging.\n :param bool multiinterpreter: When True, benchmarking is performed in a new python interpreter per model. All parameters must be serializable. Overrides multiprocess.\n :param bool return_timers: When True, the return of this function is a list of tuples ``(config, results)`` with detailed information. This can be converted to reports with ``get_reports(results)``.\n :param float stats_interval: Collection interval (in seconds) for metrics during benchmarking, such as CPU and memory usage.\n :param str device_type: This will be set automatically to one of the ``SUPPORTED_DEVICE_TYPES``.\n :param float cost_per_hour: The price of this device / hour. Used to estimate cost / 1 million infs in reports.\n :param str model_name: A friendly name for the model to use in reports.\n :param str model_class_name: Internal use.\n :param str model_class_file: Internal use.\n :param int verbosity: 0 = error, 1 = info, 2 = debug\n :return: A list of benchmarking results.\n :rtype: List[Dict]\n \"\"\"\n # Set NeuronPerf logging verbosity.\n set_verbosity(verbosity)\n\n # --------------------------------------------\n # Input validation\n # --------------------------------------------\n # Validate that enough information was provided.\n if not load_fn:\n raise ValueError(\n \"You should call benchmark() through a framework submodule, e.g. neuronperf.torch.benchmark().\"\n )\n if not isinstance(model_filename, str):\n raise ValueError(\n \"You must provide the path to a saved model or the path to a model index from neuronperf.compile().\"\n )\n\n # Useful for debugging.\n if not multiprocess and not multiinterpreter:\n log.warning(\"Benchmarking in a single process.\")\n\n # Standardize inputs.\n dataset_inputs = dataset_loader_fn is not None\n if (not dataset_inputs) and (not isinstance(inputs, list)):\n inputs = [inputs]\n if isinstance(n_models, int):\n n_models = [n_models]\n if isinstance(pipeline_sizes, int):\n pipeline_sizes = [pipeline_sizes]\n if isinstance(performance_levels, int):\n performance_levels = [performance_levels]\n if workers_per_model is None:\n workers_per_model = []\n elif isinstance(workers_per_model, int):\n workers_per_model = [workers_per_model]\n if duration < BENCHMARK_SECS:\n log.warning(\"Results may be unreliable with short test durations.\")\n\n # If the model_filename is JSON, attempt to interpret it as a model index.\n index = None\n if model_filename.endswith(model_index.MODEL_INDEX_SUFFIX):\n index = model_index.load(model_filename)\n\n # If we loaded a model_index, ensure provided inputs are compatible\n # and use it to refine the benchmarking combinations we will run.\n if index:\n # Extract a model name from the index, if possible.\n if not model_name:\n model_name = index[\"model_name\"]\n\n # If batch_sizes, pipeline_sizes and/or performance_levels were provided,\n # treat them as filters on the index. A value of None is treated as no filter.\n # See the docs for model_index.filter().\n index = model_index.filter(\n index,\n status=\"finished\", # only take compiled models\n batch_size=batch_sizes, # select all requested batch sizes\n pipeline_size=pipeline_sizes,\n performance_level=performance_levels,\n )\n\n if 0 == len(index[\"model_configs\"]):\n raise ValueError(\n \"No models were found in the model index matching requested criteria. Check that compilation succeeded.\"\n )\n\n # If a model index was provided without batch_sizes, extract the sizes from the index.\n if not batch_sizes:\n # Select unique batch_sizes in model index.\n batch_sizes = set(config[\"batch_size\"] for config in index[\"model_configs\"])\n batch_sizes = sorted(list(batch_sizes))\n\n # Validate batch sizes after attempting to extract from the model index.\n inputs, batch_sizes = _sanitize_inputs(inputs, batch_sizes, dataset_inputs)\n\n # If we still don't have a model name, use the filename.\n if not model_name:\n model_name = model_filename\n\n # If no pipeline_sizes are provided, we'll assume it's 1 for a single model unless told otherwise.\n if not pipeline_sizes:\n log.debug(\"Pipeline size was not specified, assuming 1.\")\n pipeline_sizes = [1]\n\n # Assume max performance is desired.\n if not performance_levels:\n max_performance = max(FAST_MATH_OPTIONS)\n log.debug(f\"Performance level was not specified, assuming {max_performance}.\")\n performance_levels = [max_performance]\n\n # If a model was provided directly without a model index, build a dummy model index.\n # A single model can not possibly have been compiled for more than 1 configuration,\n # hence why we can assume index [0].\n if not index:\n index = model_index.create(\n filename=model_filename,\n model_name=model_name,\n batch_size=batch_sizes[0],\n pipeline_size=pipeline_sizes[0],\n performance_level=performance_levels[0],\n )\n\n model_configs = index[\"model_configs\"]\n\n # --------------------------------------------\n # Benchmarking\n # --------------------------------------------\n\n # Estimate time remaining based on configs requested to run.\n # If n_models wasn't provided, the default benchmarks [min, max].\n n_models_est = 2 if not n_models else len(n_models)\n # If workers_per_model wasn't provided, the default benchmarks [1, 2].\n n_models_est *= 2 if not workers_per_model else len(workers_per_model)\n secs_remaining = len(model_configs) * n_models_est * duration\n mins_remaining = None if secs_remaining < 60 else round(secs_remaining / 60.0, 1)\n etr = f\"{mins_remaining} minutes\" if mins_remaining else f\"{int(round(secs_remaining))} seconds\"\n log.info(\"Benchmarking '{}', ~{} remaining.\".format(model_filename, etr))\n\n # Try to determine instance type.\n instance_type = get_instance_type()\n if not instance_type:\n instance_type = \"unknown\"\n\n # Try to automatically determine the maximum number of devices available.\n max_devices = _get_num_devices(device_type, instance_type)\n log.debug(\"Automatically determined number of devices: {}\".format(max_devices))\n\n # Try to detect cost / hour for this device.\n if not cost_per_hour:\n cost_per_hour = _get_cost_per_hour(instance_type)\n\n # Run through all requested combinations and generate a report.\n # This will produce a list of tuples, (config, results).\n all_results = []\n\n def make_reports():\n \"\"\"Helper to generate reports from available results.\"\"\"\n # If all_results was set, we return the unmodified benchmarking results.\n return all_results if return_timers else get_reports(all_results, cost_per_hour)\n\n for model_config in model_configs:\n batch_size = model_config[\"batch_size\"]\n pipeline_size = model_config[\"pipeline_size\"]\n\n # Determine the number of model copies for each benchmarking session.\n model_counts = n_models\n # If the user didn't provide n_models, choose reasonable defaults.\n if not model_counts:\n # Try to run a single model and the max models supported on this hardware.\n if max_devices and (max_devices // pipeline_size > 1):\n model_counts = [1, max_devices // pipeline_size]\n else:\n model_counts = [1]\n # If the user provided model counts and we determine they are too large, emit a warning.\n else:\n if max_devices:\n model_counts_too_large = list(\n filter(\n lambda model_count: model_count * pipeline_size > max_devices, model_counts\n )\n )\n if model_counts_too_large:\n log.warning(\n (\n \"Some values of n_models exceed the number of devices available: \"\n f\"{model_counts_too_large} > {max_devices}\"\n )\n )\n\n # Compute number of workers for this pipeline size, if not specified.\n n_workers = workers_per_model\n if not n_workers:\n n_workers = [_get_num_workers(pipeline_size)]\n # 1 worker thread == min latency\n if 1 not in n_workers:\n n_workers.insert(0, 1)\n\n for _workers_per_model in n_workers:\n # We now know everything we need to benchmark.\n # 1. Build a comprehensive benchmarker config,\n # 2. build one benchmarker per model,\n # 3. run the benchmarkers in parallel,\n # 4. and collect the results for this configuration.\n for model_count in model_counts:\n # 1. Benchmarker config\n config = {\n \"model_filename\": model_config[\"filename\"],\n \"model_name\": model_name,\n \"device_type\": device_type,\n \"instance_type\": instance_type,\n \"batch_size\": batch_size,\n \"n_models\": model_count,\n \"workers_per_model\": _workers_per_model,\n \"pipeline_size\": pipeline_size,\n \"n_devices\": model_count * pipeline_size,\n \"performance_level\": model_config[\"performance_level\"],\n \"multiprocess\": multiprocess,\n \"multiinterpreter\": multiinterpreter,\n \"stats_interval\": str(stats_interval),\n \"start_dts\": time.strftime(\"%Y%m%d-%H%M%S\"),\n \"duration\": str(duration),\n }\n\n # 2. Build the benchmarkers\n benchmarkers = []\n for benchmarker_id in range(model_count):\n benchmarker = Benchmarker(\n id=benchmarker_id,\n device_id=_get_device_id(benchmarker_id, config),\n load_fn=load_fn,\n model_filename=model_config[\"filename\"],\n inputs=inputs if dataset_inputs else inputs[batch_sizes.index(batch_size)],\n workers_per_model=_workers_per_model,\n env_setup_fn=_get_env_setup_fn(benchmarker_id, config, env_setup_fn),\n setup_fn=_get_setup_fn(benchmarker_id, config, setup_fn),\n preprocess_fn=preprocess_fn,\n postprocess_fn=postprocess_fn,\n dataset_loader_fn=dataset_loader_fn,\n model_class_name=model_class_name,\n model_class_file=model_class_file,\n )\n benchmarkers.append(benchmarker)\n\n # 3. Run benchmarkers in parallel\n log.debug(\"Running model config: {}\".format(config))\n try:\n results = run_benchmarkers(\n benchmarkers,\n duration,\n stats_interval=stats_interval,\n multiprocess=multiprocess,\n multiinterpreter=multiinterpreter,\n )\n\n # 4. Collect results\n config[\"stop_dts\"] = time.strftime(\"%Y%m%d-%H%M%S\")\n all_results.append((config, results))\n except KeyboardInterrupt:\n # If we are interrupted, return whatever we have on hand.\n return make_reports()\n except:\n # If something else goes wrong with the model, we should\n # log this configuration and move on.\n log.exception(\"Failure benchmarking config: {}\".format(config))\n\n return make_reports()\n" }, { "path": "src/neuronperf/src/neuronperf/compile_constants.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.compile_constants\n~~~~~~~~~~~~~~~~~~~~~~~\nHolds constants used at compile time.\n\"\"\"\n\nNEURONCORE_PIPELINE_CORES = \"--neuroncore-pipeline-cores\"\nFAST_MATH = \"--fast-math\"\nFAST_MATH_OPTIONS = {\n 0: \"none\",\n 1: \"fp32-cast-matmult no-fast-relayout\",\n 2: \"fp32-cast-matmult\",\n 3: \"all\",\n}\n" }, { "path": "src/neuronperf/src/neuronperf/cpu/__init__.py", "content": "from neuronperf.cpu.cpu import benchmark\n" }, { "path": "src/neuronperf/src/neuronperf/cpu/cpu.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.cpu\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides CPU support.\n\"\"\"\n\nimport functools\nimport logging\n\nfrom .. import benchmarking\n\n\nlog = logging.getLogger(__name__)\n\n\nclass DummyModel:\n def __call__(self, x):\n x *= 5\n x += 3\n return x\n\n\ndef benchmark(model_class, inputs, *args, **kwargs):\n if not isinstance(model_class, type):\n raise TypeError(\"For CPU benchmarking, you must provide a class to instantiate.\")\n\n device_type = kwargs.pop(\"device_type\", \"cpu\")\n multiinterpreter = kwargs.pop(\"multiinterpreter\", False)\n if multiinterpreter:\n log.warning(\n \"CPU + multiinterpreter is not yet fully supported. You need to provide a custom load_fn that can import your class and instantiate it.\"\n )\n\n # Create a custom load_fn that instantiates the model.\n def load_fn(*args, **kwargs):\n return model_class()\n\n kwargs[\"device_type\"] = device_type\n kwargs[\"multiinterpreter\"] = multiinterpreter\n\n return benchmarking.benchmark(\n load_fn,\n model_class.__name__,\n inputs,\n *args,\n **kwargs,\n )\n" }, { "path": "src/neuronperf/src/neuronperf/logging.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.logging\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides logging utility functions.\n\"\"\"\n\nimport logging\n\n\nFORMAT_STRING = '%(levelname)s:%(name)s - %(message)s'\n\n\ndef _get_stream_handlers(level = logging.DEBUG):\n formatter = logging.Formatter(FORMAT_STRING)\n sh = logging.StreamHandler()\n sh.setLevel(logging.DEBUG)\n sh.setFormatter(formatter)\n return [sh]\n" }, { "path": "src/neuronperf/src/neuronperf/model_index.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.model_index\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides utilities for working with model indexes.\n\"\"\"\n\nfrom typing import Any, List, Union\n\nimport builtins\nimport copy as copy_module\nimport itertools\nimport json\nimport logging\nimport os\nimport pathlib\nimport random\nimport shutil\n\n\nfrom .__version__ import __version__\nfrom .compile_constants import FAST_MATH_OPTIONS\n\n\nlog = logging.getLogger(__name__)\n\nMODEL_INDEX_SUFFIX = \".json\"\n\n\ndef generate_id(length: int = 8):\n \"\"\"Generate a random-enough sequence to append to model names and prevent collisions.\"\"\"\n id_chars = \"abcdefghijklmnopqrstuvwxyz0123456789\"\n new_id = [id_chars[random.randrange(len(id_chars))] for _ in range(length)]\n return \"\".join(new_id)\n\n\ndef generate_name(model_name: str):\n \"\"\"Generate a model index name from a model name.\"\"\"\n return model_name + \"_\" + generate_id() + MODEL_INDEX_SUFFIX\n\n\ndef _create(model_name: str, compile_info: list) -> dict:\n if not isinstance(compile_info, list):\n log.exception(\n \"Expected a list of compile info dicts, received '{}'.\".format(str(type(compile_info)))\n )\n model_index = {\n \"NeuronPerf_version\": __version__,\n \"model_name\": model_name,\n \"model_configs\": compile_info,\n }\n return model_index\n\n\ndef create(\n filename: str,\n model_name: str = None,\n batch_size: int = 1,\n pipeline_size: int = 1,\n performance_level: int = max(FAST_MATH_OPTIONS),\n compile_s: float = None,\n status: str = \"finished\",\n) -> dict:\n r\"\"\"\n Create a new model index from a pre-compiled model.\n\n :param str filename: The path to the compiled model.\n :param str model_name: A friendly name for the model. Will default to filename.\n :param int batch_size: The batch size at compilation for this model.\n :param int pipeline_size: The pipeline size used at compilation for this model.\n :param int performance_level: The performance level this model was compiled with.\n :param float compile_s: Seconds spent compiling.\n :param str status: A string describing compilation result. Can be \"finished\" or \"error\".\n :return: A new dictionary representing a model index.\n :rtype: dict\n \"\"\"\n if not model_name:\n model_name = filename\n compile_info = [\n {\n \"filename\": filename,\n \"batch_size\": batch_size,\n \"pipeline_size\": pipeline_size,\n \"performance_level\": performance_level,\n \"compile_s\": compile_s,\n \"status\": status,\n }\n ]\n return _create(model_name, compile_info)\n\n\ndef delete(filename: str):\n \"\"\"Deletes the model index and all associated models referenced by the index.\"\"\"\n if not os.path.exists(filename):\n log.warning(\"Asked to delete '{}', but it can't be located.\".format(filename))\n return\n\n # Load the index\n configs = load(filename)[\"model_configs\"]\n\n # Remove all referenced models\n model_filenames = map(lambda x: x[\"filename\"], itertools.chain(configs))\n for model_filename in model_filenames:\n log.debug(f\"Deleting '{model_filename}'.\")\n if os.path.exists(model_filename):\n if os.path.isdir(model_filename):\n shutil.rmtree(model_filename)\n else:\n os.remove(model_filename)\n\n # Finally, remove the model index itself\n log.debug(f\"Deleting '{filename}'\")\n os.remove(filename)\n\n\ndef copy(old_index: Union[str, dict], new_index: str, new_dir: str) -> str:\n r\"\"\"\n Copy an index to a new location. Will rename ``old_index``\n to ``new_index`` and copy all model files into ``new_dir``,\n updating the index paths.\n\n This is useful for pulling individual models out of a pool.\n\n Returns the path to the new index.\n \"\"\"\n os.makedirs(new_dir, exist_ok=True)\n index = _sanitize(old_index)[0].copy()\n\n configs = index[\"model_configs\"]\n for config in configs:\n path = pathlib.Path(config[\"filename\"])\n config[\"filename\"] = str(shutil.copy2(path, new_dir))\n\n return save(index, new_index)\n\n\ndef move(old_index: str, new_index: str, new_dir: str) -> str:\n \"\"\"This is the same as ``copy`` followed by ``delete`` on the old index.\"\"\"\n index = copy(old_index, new_index, new_dir)\n delete(old_index)\n return index\n\n\ndef _sanitize(*model_indexes: Union[str, dict]) -> List[dict]:\n r\"\"\"\n Helper function to load indexes if strings are provided.\n If already loaded, this is a no-op.\n \"\"\"\n if not model_indexes:\n raise ValueError(\"No model indexes were provided.\")\n indexes = []\n # Load any paths provided and sanity check all inputs.\n for index in model_indexes:\n if not index:\n raise ValueError(\"An empty value was received, but expected a model index.\")\n if isinstance(index, str):\n index = load(index)\n if not isinstance(index, dict):\n raise TypeError(\"Expected a model index, but received '{}'.\".format(str(type(None))))\n if not len(index) > 0:\n raise ValueError(\"Received an empty model index.\")\n indexes.append(index)\n # Check versions are all the same, and emit a warning if they aren't.\n versions = set(map(lambda x: x[\"NeuronPerf_version\"], indexes))\n if len(versions) > 1:\n log.warning(\"Received model with different versions: '{}'.\".format(str(versions)))\n model_name = indexes[0][\"model_name\"]\n # Ensure model names are matching.\n if not all(model_name == index[\"model_name\"] for index in indexes):\n model_names = list(set(map(lambda x: x[\"model_name\"], indexes)))\n log.warning(\"Received model indexes with different model names: {}\".format(model_names))\n return indexes\n\n\ndef append(*model_indexes: Union[str, dict]) -> dict:\n r\"\"\"\n Appends the model indexes non-destructively into a new model index, without\n modifying any of the internal data.\n\n This is useful if you have benchmarked multiple related models and wish to\n combine their respective model indexes into a single index.\n\n Model name will be taken from the first index provided.\n Duplicate configs will be filtered.\n\n :param Union[str, dict] model_indexes: Model indexes or paths to model indexes to combine.\n :return: A new dictionary representing the combined model index.\n :rtype: dict\n \"\"\"\n indexes = _sanitize(*model_indexes)\n # Extract the model configs from the indexes\n config_iter = map(lambda index: copy_module.deepcopy(index[\"model_configs\"]), indexes)\n # Combine the model configs\n combined = list(itertools.chain.from_iterable(config_iter))\n # Split unique and duplicate configs\n duplicate = []\n unique = []\n for config in combined:\n if config in unique:\n duplicate.append(config)\n else:\n unique.append(config)\n if len(duplicate) > 0:\n log.warning(\n (\n f\"There were {len(duplicate)} duplicate model configs \"\n \"filtered. The duplicates were:\\n\"\n \"{}\".format(\"\\n\".join(map(lambda c: str(c), duplicate)))\n )\n )\n # Build new index from configs\n return _create(indexes[0][\"model_name\"], unique)\n\n\ndef save(model_index: dict, filename: str = None, root_dir=None) -> str:\n r\"\"\"Save a NeuronPerf model index to a file.\"\"\"\n if not filename:\n model_name = model_index[\"model_name\"]\n filename = generate_name(model_name)\n if not filename.lower().endswith(MODEL_INDEX_SUFFIX):\n filename += MODEL_INDEX_SUFFIX\n if not root_dir:\n root_dir = \".\"\n try:\n with open(os.path.join(root_dir, filename), \"w\") as fp:\n json.dump(model_index, fp)\n except OSError:\n log.exception(\"Failed to write '{}'.\".format(filename))\n return filename\n\n\ndef load(filename) -> dict:\n \"\"\"Load a NeuronPerf model index from a file.\"\"\"\n model_index = None\n try:\n with open(filename, \"r\") as fp:\n model_index = json.load(fp)\n except OSError:\n # file is probably not a model index\n log.exception(\"Failed to load model index '{}'\".format(filename))\n else:\n from distutils.version import LooseVersion\n\n try:\n if LooseVersion(model_index[\"NeuronPerf_version\"]) > LooseVersion(__version__):\n log.warning(\n \"Model index newer than NeuronPerf (version {} > {}). Try updating NeuronPerf.\".format(\n model_index[\"NeuronPerf_version\"], __version__\n )\n )\n except TypeError:\n log.warning(\n \"Couldn't compare model index version ({}) to NeuronPerf version ({}), continuing anyway.\".format(\n model_index[\"NeuronPerf_version\"], __version__\n )\n )\n\n return model_index\n\n\ndef filter_configs(configs, filter_name, filter_values) -> List:\n \"\"\"Filters provided configs on specified filter and value and returns a new config list.\"\"\"\n if filter_values is None:\n return configs.copy()\n # Filter on configs that have the filter_name and value is in filter_values\n if not isinstance(filter_values, list):\n filter_values = [filter_values]\n return list(\n builtins.filter(\n lambda config: filter_name in config and config[filter_name] in filter_values, configs\n )\n )\n\n\ndef filter(index: Union[str, dict], **kwargs) -> dict:\n r\"\"\"\n Filters provided model index on provided criteria and returns a new index.\n Each kwarg is a standard (k, v) pair, where k is treated as a filter name\n and v may be one or more values used to filter model configs.\n \"\"\"\n index = _sanitize(index)[0].copy()\n\n # Filter each config on provided kwargs pairs.\n configs = index[\"model_configs\"]\n for k, v in kwargs.items():\n configs = filter_configs(configs, k, v)\n\n index[\"model_configs\"] = configs\n return index\n" }, { "path": "src/neuronperf/src/neuronperf/mxnet/__init__.py", "content": "from neuronperf.mxnet.mxnet import benchmark, compile\n" }, { "path": "src/neuronperf/src/neuronperf/mxnet/mxnet.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.mxnet\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides Apache MXNet support.\n\"\"\"\n\nimport contextlib\nimport functools\nimport os\nimport threading\n\n# handle different API versions of mxnet\nimport mxnet as mx\nfrom distutils.version import LooseVersion\n\nif LooseVersion(mx.__version__) >= LooseVersion(\"1.8\"):\n _mx_version = 1.8\n import mx_neuron as neuron\nelse:\n _mx_version = 1.5\n from mxnet.contrib import neuron\n\nfrom .. import benchmarking\n\n\nclass _MXNetModelWrapper:\n def __init__(self, device_id, sym, args, aux):\n self.device_id = device_id\n self.sym = sym\n self.args = args\n self.aux = aux\n self.ctx = None\n self.exes = {}\n self.lock = threading.Lock()\n\n def __call__(self, inputs):\n # on the first inference, do prep work\n if not self.ctx:\n self.ctx = mx.neuron(self.device_id)\n\n # prepare inputs for model\n for k, v in inputs.items():\n inputs[k] = mx.nd.array(v)\n self.args.update(inputs)\n\n # obtain an executor for this thread\n thread_id = threading.get_ident()\n if thread_id not in self.exes:\n with self.lock:\n exe = self.sym.bind(\n ctx=self.ctx, args=self.args, aux_states=self.aux, grad_req=\"null\"\n )\n self.exes[thread_id] = exe\n else:\n exe = self.exes[thread_id]\n\n # run inference\n outputs = exe.forward(**inputs)\n mx.nd.waitall()\n return outputs[0]\n\n\n@contextlib.contextmanager\ndef change_dir(new_dir):\n old_dir = os.getcwd()\n os.chdir(os.path.join(old_dir, new_dir))\n try:\n yield\n finally:\n os.chdir(old_dir)\n\n\ndef _load_fn(model_filename, **kwargs):\n device_id = kwargs.get(\"device_id\", 0)\n sym, args, aux = mx.model.load_checkpoint(model_filename, 0)\n return _MXNetModelWrapper(device_id, sym, args, aux)\n\n\ndef _compile_fn(model, example_inputs, models_dir, model_name, **kwargs):\n _sym, _args, _aux = model\n model_filename = os.path.join(models_dir, model_name)\n compiler_args = kwargs.pop(\"compiler_args\", {})\n\n # MXNet passes additional kwargs directly to compiler\n _sym, _args, _aux = neuron.compile(\n _sym,\n _args,\n _aux,\n example_inputs,\n **compiler_args,\n )\n\n with change_dir(models_dir):\n mx.model.save_checkpoint(model_name, 0, _sym, _args, _aux)\n return model_filename\n\n\ndef compile(model, inputs, *args, **kwargs):\n return benchmarking.compile(_compile_fn, model, inputs, *args, **kwargs)\n\n\ndef benchmark(model_filename, inputs, *args, **kwargs):\n env_setup_fn = kwargs.pop(\"env_setup_fn\", lambda *_: None)\n\n # Use a custom setup function to handle MXNet concurrency requirements.\n @functools.wraps(env_setup_fn)\n def _env_setup_fn(id, config):\n workers_per_model = str(config[\"workers_per_model\"])\n os.environ[\"MXNET_CPU_TEMP_COPY\"] = workers_per_model\n os.environ[\"MXNET_EXEC_NUM_TEMP\"] = workers_per_model\n os.environ[\"MXNET_CPU_WORKER_NTHREADS\"] = workers_per_model\n os.environ[\"MXNET_MP_WORKER_NTHREADS\"] = workers_per_model\n\n # Remember to call any additional custom setup provided.\n env_setup_fn(id, config)\n\n kwargs[\"env_setup_fn\"] = _env_setup_fn\n\n return benchmarking.benchmark(_load_fn, model_filename, inputs, *args, **kwargs)\n" }, { "path": "src/neuronperf/src/neuronperf/py.typed", "content": "# Marker file that indicates this package supports typing\n" }, { "path": "src/neuronperf/src/neuronperf/reporting.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.reporting\n~~~~~~~~~~~~~~~~~~~~\nProvides utilities for producing reports from benchmarking results.\n\"\"\"\n\nfrom typing import List\n\nimport csv\nimport itertools\nimport json\nimport logging\nimport time\n\nimport numpy as np\n\nfrom . import __version__\n\n\nlog = logging.getLogger(__name__)\n\nCSV_COLS = [\n \"model_name\",\n \"n_models\",\n \"workers_per_model\",\n \"pipeline_size\",\n \"batch_size\",\n \"throughput_avg\",\n \"throughput_peak\",\n \"latency_ms_p0\",\n \"latency_ms_p50\",\n \"latency_ms_p90\",\n \"latency_ms_p95\",\n \"latency_ms_p99\",\n \"latency_ms_p100\",\n \"cpu_avg_percent\",\n \"cpu_percent_p50\",\n \"mem_avg_percent\",\n \"mem_percent_p50\",\n \"e2e_avg_ms\",\n \"infer_avg_ms\",\n \"total_infs\",\n \"total_s\",\n \"performance_level\",\n \"model_filename\",\n \"device_type\",\n \"instance_type\",\n \"cost_per_1m_inf\",\n]\n\nPRINT_COLS = [\n \"throughput_avg\",\n \"latency_ms_p50\",\n \"latency_ms_p99\",\n \"n_models\",\n \"pipeline_size\",\n \"workers_per_model\",\n \"batch_size\",\n \"model_filename\",\n]\n\nREQUIRED_CONFIG_KEYS = [\n \"multiprocess\",\n \"multiinterpreter\",\n \"device_type\",\n \"batch_size\",\n \"model_filename\",\n \"model_name\",\n \"n_models\",\n \"pipeline_size\",\n]\n\nREQUIRED_RESULTS_KEYS = [\n \"workers_per_model\",\n \"status\",\n \"timers\",\n \"n_infs\",\n \"total_s\",\n]\n\n\ndef _validate_config(config):\n for required_key in REQUIRED_CONFIG_KEYS:\n if required_key not in config:\n raise ValueError(\n (\n f\"Model config is missing required key '{required_key}'. \"\n \"Something probably went wrong during benchmarking. Provided:\\n{config}\"\n )\n )\n\n\ndef _validate_results(results):\n for required_key in REQUIRED_RESULTS_KEYS:\n if required_key not in results:\n raise ValueError(\n (\n f\"Benchmarking results are missing required key '{required_key}'. \"\n \"Something probably went wrong during benchmarking. Provided:\\n{results}\"\n )\n )\n\n\ndef _get_report_name(model_name: str) -> str:\n return \"{}.results-{}\".format(model_name, time.strftime(\"%Y%m%d-%H%M%S\"))\n\n\ndef get_report(\n benchmark_results, cost_per_hour: float = None, window_size: int = 1, verbosity: int = 0\n) -> dict:\n r\"\"\"Get a performance report from benchmarker results.\n\n :param benchmark_results: Results from a :class:`benchmarking:Benchmarker` object.\n :param float cost_per_hour: The cost / hour for this device.\n :param int window_size: Window size in seconds used to measure throughput.\n :param int verbosity: Controls logging during report generation. Use 0 (default), 1, or 2.\n :returns: A dictionary containing performance information.\n \"\"\"\n report = {}\n config, results = benchmark_results\n _validate_config(config)\n _validate_results(results)\n try:\n report[\"NeuronPerf_version\"] = __version__\n\n # copy benchmarker info from config into report\n for k, v in config.items():\n report[k] = v\n\n # number of intervals is the same across all stats, so we can use this as a proxy\n report[\"n_stats_intervals\"] = len(results[\"cpu_percents\"])\n\n report[\"workers_per_model\"] = results[\"workers_per_model\"]\n report[\"status\"] = results[\"status\"]\n\n # timing stats\n report[\"load_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"load\"]), float\n ).mean()\n report[\"input_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"input\"]), float\n ).mean()\n report[\"warmup_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"warmup\"]), float\n ).mean()\n report[\"env_setup_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"env_setup\"]), float\n ).mean()\n report[\"setup_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"setup\"]), float\n ).mean()\n report[\"preprocess_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"preprocess\"]), float\n ).mean()\n report[\"infer_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"infer\"]), float\n ).mean()\n report[\"postprocess_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"postprocess\"]), float\n ).mean()\n report[\"e2e_avg_ms\"] = np.fromiter(\n (t.avg(\"ms\") for t in results[\"timers\"][\"e2e\"]), float\n ).mean()\n report[\"worker_avg_s\"] = round(\n np.fromiter((t.avg(\"s\") for t in results[\"timers\"][\"worker\"]), float).mean(), 2\n )\n report[\"total_infs\"] = results[\"n_infs\"] * config[\"batch_size\"]\n report[\"total_s\"] = round(results[\"total_s\"], 2)\n\n percentiles = [0, 50, 90, 95, 99, 100]\n\n cpu_percents = np.fromiter(results[\"cpu_percents\"], float)\n if cpu_percents.size > 2:\n cpu_percentiles = np.percentile(cpu_percents[1:-1], percentiles)\n report[\"cpu_avg_percent\"] = cpu_percentiles.mean()\n for i, p in enumerate(percentiles):\n report[f\"cpu_percent_p{p}\"] = cpu_percentiles[i]\n\n mem_percents = np.fromiter(results[\"mem_percents\"], float)\n if mem_percents.size > 2:\n mem_percentiles = np.percentile(mem_percents[1:-1], percentiles)\n report[\"mem_avg_percent\"] = mem_percentiles.mean()\n for i, p in enumerate(percentiles):\n report[f\"mem_percent_p{p}\"] = mem_percentiles[i]\n\n # latency\n latencies = np.fromiter(\n itertools.chain.from_iterable(t.durations(\"ms\") for t in results[\"timers\"][\"e2e\"]),\n float,\n )\n latency_percentiles = np.percentile(latencies, percentiles)\n for i, p in enumerate(percentiles):\n report[\"latency_ms_p{}\".format(p)] = latency_percentiles[i]\n\n # bucketize ending timestamps\n end_timestamps = np.fromiter(\n itertools.chain.from_iterable(t.end_timestamps(\"s\") for t in results[\"timers\"][\"e2e\"]),\n float,\n )\n bucket_ends = np.floor(end_timestamps / window_size)\n # group timestamps by window and correct for batch size\n _, bucket_counts = np.unique(bucket_ends, return_counts=True)\n bucket_counts *= config[\"batch_size\"]\n # find max and normalize by window size\n report[\"throughput_peak\"] = bucket_counts.max() / window_size\n report[\"throughput_avg\"] = bucket_counts[1:-1].mean() / window_size\n\n if verbosity > 0:\n report[\"throughput_hist\"] = bucket_counts\n if verbosity > 1:\n report[\"e2e_durations_ms\"] = np.fromiter(\n (t.durations(\"ms\") for t in results[\"timers\"][\"e2e\"]), float\n )\n\n # Try to estimte cost / inference\n if cost_per_hour:\n try:\n infs_per_hour = 3600 * report[\"throughput_avg\"]\n report[\"cost_per_1m_inf\"] = cost_per_hour * (1_000_000 / infs_per_hour)\n except:\n # We'll ignore this, as it's caused by a missing field that would have\n # already generated an earlier error log. We should continue producing\n # a report nonetheless.\n pass\n\n # Truncate floats to 3 places for readability.\n for key, value in report.items():\n if isinstance(value, float):\n report[key] = round(value, 3)\n\n except:\n log.exception(\n (\n \"Failed to produce a report from benchmarking results. \"\n \"Something probably went wrong during benchmarking.\"\n )\n )\n return report\n\n\ndef get_reports(results, cost_per_hour: float = None) -> List[dict]:\n r\"\"\"\n Summarizes and combines the detailed results from\n ``neuronperf.benchmark``, when run with ``return_timers=True``.\n One report dictionary is produced per model configuration benchmarked.\n The list of reports can be fed directly to other reporting utilities,\n such as ``neuronperf.write_csv``.\n\n :param results: Benchmarker results.\n :param float cost_per_hour: The cost / hour for this device.\n \"\"\"\n reports = []\n for idx, (config, result) in enumerate(results):\n try:\n _validate_config(config)\n _validate_results(result)\n except ValueError:\n log.exception(f\"Result {idx} is missing required information, skipping.\")\n continue\n report = get_report((config, result), cost_per_hour)\n reports.append(report)\n return reports\n\n\ndef print_reports(reports: List[dict], cols=PRINT_COLS, sort_by=\"throughput_peak\", reverse=False):\n r\"\"\"Print a subset of report cols to the terminal.\n\n :param reports: Results from `get_reports`.\n :param cols: The columns in the report to be displayed.\n :param sort_by: Sort the cols by the specified key.\n :param reverse: Sort order.\n \"\"\"\n if not reports:\n print(\"No reports were found. Did benchmarking succeed?\")\n return\n # Print headers.\n col_width = max(map(lambda col: len(col), cols)) + 1\n row_format = \"{{:<{}}}\".format(col_width) * len(cols)\n print(row_format.format(*cols))\n # Extract all rows.\n rows = []\n for report in reports:\n row = []\n for col in cols:\n row.append(report[col] if col in report else \"N/A\")\n rows.append(row)\n # Sort rows by the specified key, if the key exists.\n if sort_by in cols:\n sort_index = cols.index(sort_by)\n rows = sorted(rows, key=lambda row: row[sort_index], reverse=reverse)\n # Print all rows.\n for row in rows:\n print(row_format.format(*row))\n\n\ndef write_csv(reports: List[dict], filename: str = None, cols=CSV_COLS):\n r\"\"\"Write a benchmarking report to CSV file.\n\n :param reports: Results from `get_reports`.\n :param filename: File name to write out. If not provided, generated from model_name in report and current timestamp.\n :param cols: The columns in the report to be kept.\n \"\"\"\n if not filename:\n filename = \"{}.csv\".format(_get_report_name(reports[0][\"model_name\"]))\n try:\n with open(filename, \"w\", newline=\"\", encoding=\"utf-8\") as csvfile:\n writer = csv.writer(csvfile)\n writer.writerow(cols)\n for idx, report in enumerate(reports):\n row = []\n for col in cols:\n if col in report:\n row.append(report[col] if report[col] is not None else \"N/A\")\n else:\n log.debug(f\"Report {idx} is missing field '{col}'.\")\n row.append(\"N/A\")\n writer.writerow(row)\n return filename\n except OSError:\n log.exception(f\"Failed to write '{filename}'. Check that you have write permissions.\")\n\n\ndef write_json(reports: List[dict], filename: str = None):\n if not filename:\n filename = \"{}.json\".format(_get_report_name(reports[0][\"model_name\"]))\n try:\n with open(filename, \"w\", encoding=\"utf-8\") as jsonfile:\n json.dump(reports, jsonfile)\n return filename\n except OSError:\n log.exception(\n (\n f\"Failed to write '{filename}'. Check that the report \"\n \"contains data and that you have write permissions.\"\n )\n )\n" }, { "path": "src/neuronperf/src/neuronperf/scripts/__init__.py", "content": "" }, { "path": "src/neuronperf/src/neuronperf/scripts/run_benchmark_file.py", "content": "import argparse\nimport dill\nimport neuronperf\n\n\ndef main():\n parser = argparse.ArgumentParser(\n prog=\"benchmark\",\n description=\"Run a serialized Benchmarker for a given `duration`. Upon \"\n \"success overwrite `filename` with the updated Benchmarker\",\n )\n parser.add_argument(\"filename\", type=str, help=\"The serialized Benchmarker\")\n parser.add_argument(\"duration\", type=float, help=\"The duration of each config (seconds)\")\n parser.add_argument(\"--model_class_name\", type=str, help=\"The name of a model class to load\")\n parser.add_argument(\"--model_class_file\", type=str, help=\"Path to Python module defining model_class_name\")\n args = parser.parse_args()\n\n try:\n # If we were provided with a model class to import before deserialization, we need\n # to handle that now. The class will be manually imported.\n if args.model_class_name and args.model_class_file:\n import importlib.util\n\n spec = importlib.util.spec_from_file_location(\n args.model_class_name, args.model_class_file\n )\n module = importlib.util.module_from_spec(spec)\n spec.loader.exec_module(module)\n globals()[args.model_class_name] = getattr(module, args.model_class_name)\n\n # Load the benchmarker object\n with open(args.filename, \"rb\") as f:\n benchmarker = dill.load(f)\n\n # Execute the benchmarker\n result = neuronperf.benchmarking.run_benchmarker(benchmarker, args.duration)\n\n # Write the result back to the same file\n with open(args.filename, \"wb\") as f:\n dill.dump(result, f)\n except:\n # Dump traceback to a file for debugging.\n import os\n import sys\n import traceback\n from pathlib import Path\n\n path = Path(args.filename)\n filename = os.path.join(path.parent, \"neuronperf_error_{}\".format(path.stem))\n trace = \"\".join(traceback.format_exception(*sys.exc_info()))\n with open(filename, \"wt\") as err_fp:\n err_fp.write(trace)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "src/neuronperf/src/neuronperf/tensorflow/__init__.py", "content": "from neuronperf.tensorflow.tensorflow import benchmark, compile\n" }, { "path": "src/neuronperf/src/neuronperf/tensorflow/tensorflow.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.tensorflow\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides TensorFlow support.\n\"\"\"\n\nimport itertools\nimport logging\nimport os\nimport threading\n\n\nfrom .. import benchmarking\n\n\nlog = logging.getLogger(__name__)\n_lock = threading.Lock()\n\n\ndef _load_fn(model_file, **kwargs):\n with _lock:\n import tensorflow as tf\n\n if tf.__version__.startswith(\"1\"):\n return tf.contrib.predictor.from_saved_model(model_file)\n else:\n import tensorflow.keras as keras\n\n return keras.models.load_model(model_file)\n\n\ndef _compile_fn(model, inputs, models_dir, model_name, **kwargs):\n import tensorflow as tf\n import tensorflow.neuron as tfn\n\n model_filename = os.path.join(models_dir, model_name)\n\n # NeuronPerf provides compiler_args as a dictionary, but framework expects a different format.\n compiler_args = kwargs.pop(\"compiler_args\", {})\n\n if tf.__version__.startswith(\"1\"):\n compiler_args_flattened = list(itertools.chain.from_iterable(compiler_args.items()))\n kwargs[\"compiler_args\"] = compiler_args_flattened\n kwargs[\"model_feed_dict\"] = inputs\n\n # For TF 1.x, the saved model path is expected instead of a loaded model.\n tfn.saved_model.compile(model, model_filename, **kwargs)\n else:\n if compiler_args:\n compiler_args_flattened = \" \".join(\n [\"{}={}\".format(k, v) for k, v in compiler_args.items()]\n )\n os.environ[\"NEURON_CC_FLAGS\"] = compiler_args_flattened\n else:\n os.environ[\"NEURON_CC_FLAGS\"] = \"\"\n\n model_neuron = tfn.trace(model, inputs, **kwargs)\n model_neuron.save(model_filename)\n return model_filename\n\n\ndef compile(model, inputs, *args, **kwargs):\n return benchmarking.compile(_compile_fn, model, inputs, *args, **kwargs)\n\n\ndef benchmark(model_filename, inputs, *args, **kwargs):\n # Tensorflow-neuron is not currently fork safe, so we workaround this during benchmarking\n # by spawning a fresh interpreter session for each model we benchmark.\n if \"multiinterpreter\" in kwargs and not kwargs[\"multiinterpreter\"]:\n log.warning(\n \"Setting multiinterpreter=False is not safe with TensorFlow. Use at your own risk.\"\n )\n else:\n kwargs[\"multiinterpreter\"] = True\n\n return benchmarking.benchmark(_load_fn, model_filename, inputs, *args, **kwargs)\n" }, { "path": "src/neuronperf/src/neuronperf/timing.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf._timing\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides utility functions for timing and time unit conversions.\n\"\"\"\n\nfrom typing import Any, Callable\n\nimport sys\nimport time\nimport typing\n\nimport numpy as np\n\n\ntime_unit_ratios = {\n 'ns': { 'ns': 1, 'us': 1e-3, 'ms': 1e-6, 's': 1e-9 },\n 'us': { 'ns': 1e3, 'us': 1, 'ms': 1e-3, 's': 1e-6 },\n 'ms': { 'ns': 1e6, 'us': 1e3, 'ms': 1, 's': 1e-3 },\n 's': { 'ns': 1e9, 'us': 1e6, 'ms': 1e3, 's': 1 }\n}\n\n\nsupported_time_units = time_unit_ratios.keys()\n\n\ndef timestamp_convert(timestamps,\n input_time_unit: str,\n output_time_unit: str):\n \"\"\"Convert timestamp(s) from one time unit to another.\n\n :param ts: A timestamp or iterable of timestamps.\n :param input_time_unit: A string specifying the input time unit.\n :param output_time_unit: A string specifying the output time unit.\n :returns: A single timestamp or container of timestamps in the output time unit.\n \"\"\"\n try:\n ratio = time_unit_ratios[input_time_unit][output_time_unit]\n except:\n raise ValueError(f\"Can't convert {input_time_unit} to {output_time_unit}\")\n\n return timestamps * ratio\n\n\nclass Timer():\n def __init__(self,\n timer_fn: Callable[[], Any] = time.perf_counter,\n timer_unit: str = 's'):\n self.timer_fn = timer_fn\n self.timer_unit = timer_unit\n self._start = []\n self._end = []\n\n def __enter__(self):\n self.start()\n\n def __exit__(self, type, value, traceback):\n self.stop()\n\n def __delitem__(self, index):\n del self._start[index]\n del self._end[index]\n\n def __getitem__(self, index):\n # it's possible that start and end won't match if negative indices are used,\n # b/c timer may have started and not stopped yet\n if index < 0: index = index % len(self._end)\n return self._start[index], self._end[index]\n\n def __iter__(self):\n return zip(self._start, self._end)\n\n def __len__(self):\n return len(self._end)\n\n def __str__(self):\n return str(self.timestamps())\n\n def start(self):\n # If we've already started, consider this a request to restart.\n # This also handles partial timestamps due to a Timer-unrelated error.\n if len(self._start) > len(self._end): self._start.pop()\n self._start.append(self.timer_fn())\n\n def stop(self):\n # if we haven't started, ignore this\n if 0 == len(self._start): return\n self._end.append(self.timer_fn())\n\n def next(self):\n \"\"\"Manually advance the timer to the next timestamp measurement.\"\"\"\n self.stop()\n self.start()\n\n def reset(self):\n self._start.clear()\n self._end.clear()\n\n def insert(self, timestamps: tuple, time_unit: str):\n \"\"\"Manually insert a timestamp pair. Does not affect ongoing timing.\n\n :param timestamps: Timestamp pair to insert.\n :param time_unit: The time unit of the incoming timestamps.\n \"\"\"\n if len(timestamps) != 2 or not time_unit: raise ValueError()\n timestamps = timestamp_convert(np.array(timestamps), time_unit, self.timer_unit)\n self._start.insert(0, timestamps[0])\n self._end.insert(0, timestamps[1])\n\n def start_timestamps(self, time_unit: str = None):\n if not time_unit: return np.array(self._start)\n return timestamp_convert(np.array(self._start), self.timer_unit, time_unit)\n\n def end_timestamps(self, time_unit: str = None):\n if not time_unit: return np.array(self._end)\n return timestamp_convert(np.array(self._end), self.timer_unit, time_unit)\n\n def timestamps(self, time_unit: str = None):\n \"\"\"Returns a list of pairs of timestamps (start, end).\n\n :param time_unit: The time unit of the output timestamp(s). `None` will use the timer's native unit.\n \"\"\"\n starts, ends = self.start_timestamps(time_unit), self.end_timestamps(time_unit)\n return np.stack((starts[:len(ends)], ends), axis=-1)\n\n def durations(self, time_unit: str = None):\n \"\"\"Returns an `ndarray` of timestamp deltas, optionally converted into a provided time unit.\n\n :param time_unit: The time unit of the output timestamp(s). `None` will use the timer's native unit.\n :returns: An `ndarray` of timestamp deltas.\n \"\"\"\n starts, ends = self.start_timestamps(), self.end_timestamps()\n return timestamp_convert(ends - starts[:len(ends)], self.timer_unit, time_unit)\n\n def total_duration(self, time_unit: str = None):\n \"\"\"Returns total duration of all time measurements, optionally converted into a provided time unit.\n\n :param time_unit: The time unit of the output timestamp(s). `None` will use the timer's native unit.\n :\n \"\"\"\n starts, ends = self.start_timestamps(), self.end_timestamps()\n total = np.sum(ends - starts[:len(ends)])\n return total if not time_unit else timestamp_convert(total, self.timer_unit, time_unit)\n\n def avg(self, time_unit: str = None):\n \"\"\"Returns average duration, optionally converted into a provided time unit.\n\n :param time_unit: The time unit of the output timestamp(s). `None` will use the timer's native unit.\n :returns: The average duration.\n \"\"\"\n return self.durations(time_unit).mean() if len(self._end) > 0 else 0\n" }, { "path": "src/neuronperf/src/neuronperf/torch/__init__.py", "content": "from neuronperf.torch.torch import benchmark, compile\n" }, { "path": "src/neuronperf/src/neuronperf/torch/torch.py", "content": "# -*- coding: utf-8 -*-\n\n\"\"\"\nneuronperf.torch\n~~~~~~~~~~~~~~~~~~~~~~~\nProvides PyTorch support.\n\"\"\"\n\nimport functools\nimport itertools\nimport logging\nimport math\nimport os\nimport types\n\nimport torch\n\nfrom .. import benchmarking\n\n\nlog = logging.getLogger(__name__)\n\n\ndef _compile_fn(model, example_inputs, models_dir, model_name, **kwargs):\n import torch_neuron\n\n \"\"\"Compiles a model for Neuron.\"\"\"\n model_filename = os.path.join(models_dir, \"{}.pt\".format(model_name))\n model.eval()\n\n # NeuronPerf provides compiler_args as a dictionary, but framework expects a different format.\n compiler_args = kwargs.get(\"compiler_args\", {})\n compiler_args_flattened = list(itertools.chain.from_iterable(compiler_args.items()))\n kwargs[\"compiler_args\"] = compiler_args_flattened\n\n model_neuron = torch.neuron.trace(\n model,\n example_inputs,\n **kwargs,\n )\n model_neuron.save(model_filename)\n return model_filename\n\n\ndef _load_fn(model_filename, **kwargs):\n import torch_neuron\n\n model = torch.jit.load(model_filename)\n model.eval()\n return model\n\n\ndef _class_load_fn(model_class, **kwargs):\n model = model_class()\n model.eval()\n return model\n\n\ndef compile(model, inputs, *args, **kwargs):\n return benchmarking.compile(_compile_fn, model, inputs, *args, **kwargs)\n\n\n# See: https://pytorch.org/docs/stable/data.html#dataset-types\ndef _get_dataset_loader_fn(dataset, loop):\n def _worker_init_fn(worker_id):\n # This function will be called for each worker by torch.\n worker_info = torch.utils.data.get_worker_info()\n worker_id = worker_info.id\n num_workers = worker_info.num_workers\n dataset = worker_info.dataset # the dataset copy in this worker process\n per_worker = int(math.ceil(len(dataset) / float(num_workers)))\n start = worker_id * per_worker\n end = min(start + per_worker, len(dataset))\n log.debug(\n \"worker_id={}, num_workers={}, per_worker={}, start={}, end={}\".format(\n worker_id, num_workers, per_worker, start, end\n )\n )\n\n # We monkey-patch the dataset __iter__ function to support a multi-worker config.\n def _iter(self, start, end, loop):\n if loop:\n return itertools.cycle(range(start, end))\n else:\n return iter(range(start, end))\n\n __iter__ = functools.partial(_iter, start, end, loop)\n dataset.__iter__ = types.MethodType(__iter__, dataset)\n\n def dataset_loader_fn(dataset, num_workers):\n return iter(\n torch.utils.data.DataLoader(\n dataset, num_workers=num_workers, worker_init_fn=_worker_init_fn\n )\n )\n\n return dataset_loader_fn\n\n\ndef benchmark(model_filename, inputs, *args, dataset_inputs=False, loop_dataset=False, **kwargs):\n # These functions may need to be overridden or wrapped, depending upon config requested.\n load_fn = _load_fn\n setup_fn = kwargs.get(\"setup_fn\", lambda *args, **kwargs: None)\n preprocess_fn = kwargs.get(\"preprocess_fn\", lambda *args: (*args,))\n\n # If cuda is requested, ensure it's available and provide smart wrappers for CUDA device loading.\n device_type = kwargs.get(\"device_type\", None)\n use_cuda = device_type and (\"cuda\" in device_type.lower() or \"gpu\" == device_type.lower())\n if use_cuda:\n if not torch.cuda.is_available():\n raise ValueError(\n \"You requested CUDA benchmarking, but torch is unable to locate a CUDA device.\"\n )\n\n # Must use multiinterpreter for CUDA.\n if \"multiinterpreter\" in kwargs and not kwargs[\"multiinterpreter\"]:\n log.warning(\n (\n \"You set multiinterpreter to False, but it is required for safe CUDA benchmarking.\\n\"\n \"Your preference has been overridden so that benchmarking may continue.\"\n )\n )\n kwargs[\"multiinterpreter\"] = True\n\n # If we received a non-string, use class-based load function\n if not isinstance(model_filename, str):\n # In GPU benchmarking, a model class is expected. This line is for clarity.\n model_class = model_filename\n if not isinstance(model_class, type):\n raise TypeError(\"GPU benchmarking expects a model class to be provided instead of a filename.\")\n\n # We must also know the name of the file to import from, so that serialization can succeed.\n import inspect\n\n try:\n model_class_file = inspect.getfile(model_class)\n kwargs[\"model_class_file\"] = model_class_file\n kwargs[\"model_class_name\"] = model_class.__name__\n except:\n raise ValueError(\n (\n \"Your model class must be defined in a Python module so that it can be serialized properly.\\n\"\n \"Please add your model to a simple Python file along with any required imports.\"\n )\n )\n\n @functools.wraps(_class_load_fn)\n def load_fn(*args, **kwargs):\n return _class_load_fn(model_class, **kwargs)\n\n # Now swap the class object for its name so the benchmarker still receives a string.\n model_filename = model_class.__name__\n\n # Wrap setup_fn so that it moves the model to CUDA device.\n @functools.wraps(setup_fn)\n def _setup_fn(id, config, model):\n setup_fn(id, config, model)\n model.to(\"cuda\")\n\n kwargs[\"setup_fn\"] = _setup_fn\n\n # Wrap preprocess_fn with one that moves inputs to CUDA.\n @functools.wraps(preprocess_fn)\n def _preprocess_fn(*inputs):\n inputs = preprocess_fn(*inputs)\n for input in inputs:\n input.to(\"cuda\")\n return (*inputs,)\n\n kwargs[\"preprocess_fn\"] = _preprocess_fn\n\n # When custom datasets are used, a loader function will need to be available in subprocesses.\n dataset_loader_fn = None\n if dataset_inputs:\n dataset_loader_fn = _get_dataset_loader_fn(example_inputs, loop_dataset)\n kwargs[\"dataset_loader_fn\"] = dataset_loader_fn\n\n with torch.no_grad():\n return benchmarking.benchmark(\n load_fn,\n model_filename,\n inputs,\n *args,\n **kwargs,\n )\n" }, { "path": "src/neuronperf/test/test_neuronperf.py", "content": "# -*- coding: utf-8 -*-\n\nimport json\nimport os\nimport pathlib\nimport shutil\nimport time\n\nimport numpy as np\nimport pytest\n\nimport neuronperf\n\n\n@pytest.mark.sanity\ndef test_timer():\n timer = neuronperf.Timer()\n with timer:\n time.sleep(1)\n\n # sanity check\n assert timer.total_duration(\"s\") > 0.5 and timer.total_duration(\"s\") < 1.5\n\n # check conversions are functional\n assert (\n timer.total_duration(\"ns\")\n > timer.total_duration(\"us\")\n > timer.total_duration(\"ms\")\n > timer.total_duration(\"s\")\n )\n\n # check timestamp deltas are close to total\n assert timer.total_duration(\"s\") == pytest.approx(timer.durations(\"s\").sum())\n\n # check iteration functions\n for _ in range(10):\n with timer:\n time.sleep(0.01)\n assert len(timer) > 10\n\n # check that timer always returns pairs\n timestamps = timer.timestamps()\n for pair in timestamps:\n assert 2 == len(pair)\n assert pair[1] > pair[0]\n\n # check that len is functional\n assert len(timer) == len(timestamps)\n\n\n@pytest.mark.sanity\ndef test_timestamp_convert():\n # test scalar behavior\n assert 1000 == pytest.approx(neuronperf.timestamp_convert(1, \"s\", \"ms\"))\n assert 1.5 == pytest.approx(neuronperf.timestamp_convert(1500, \"ms\", \"s\"))\n assert 2.3e6 == pytest.approx(neuronperf.timestamp_convert(2.3, \"s\", \"us\"))\n\n # test array behavior\n times = np.array([1, 2, 3])\n times_ms = neuronperf.timestamp_convert(times, \"s\", \"ms\")\n assert 1000 == pytest.approx(times_ms[0])\n\n\n@pytest.mark.sanity\ndef test_model_index_create_from_file():\n filename = \"dummy_model.ext\"\n model_name = \"dummy\"\n index = neuronperf.model_index.create(filename, model_name=model_name)\n assert index[\"model_name\"] == model_name\n assert len(index[\"model_configs\"]) == 1\n assert index[\"model_configs\"][0][\"filename\"] == filename\n\n\n@pytest.mark.sanity\ndef test_model_index_create_delete_save_load():\n filename = \"dummy_index.json\"\n if os.path.exists(filename):\n neuronperf.model_index.delete(filename)\n\n model_name = \"Dummy\"\n model_filename = os.path.join(\"models\", \"dummy.model\")\n model_index = neuronperf.model_index.create(model_filename, model_name=model_name)\n neuronperf.model_index.save(model_index, filename=filename)\n assert os.path.exists(filename)\n\n model_index_loaded = neuronperf.model_index.load(filename)\n assert model_index_loaded == model_index\n assert model_index_loaded[\"model_name\"] == model_name\n assert model_index_loaded[\"model_configs\"][0][\"batch_size\"] == 1\n\n neuronperf.model_index.delete(filename)\n assert not os.path.exists(filename)\n\n\n@pytest.mark.sanity\ndef test_model_index_copy():\n filename = \"dummy_index.json\"\n if os.path.exists(filename):\n neuronperf.model_index.delete(filename)\n\n model_filename = os.path.join(\"models\", \"dummy.model\")\n os.makedirs(\"models\", exist_ok=True)\n pathlib.Path(model_filename).touch()\n model_name = \"Dummy\"\n model_index = neuronperf.model_index.create(model_filename, model_name=model_name)\n neuronperf.model_index.save(model_index, filename=filename)\n\n # Test copy API using a pre-loaded model inndex\n neuronperf.model_index.copy(model_index, \"new_index.json\", \"new_models\")\n assert os.path.exists(\"models\")\n assert os.path.exists(model_filename)\n assert os.path.exists(\"new_index.json\")\n assert os.path.exists(os.path.join(\"new_models\", \"dummy.model\"))\n\n new_index = neuronperf.model_index.load(\"new_index.json\")\n assert new_index[\"model_configs\"][0][\"filename\"] == os.path.join(\"new_models\", \"dummy.model\")\n\n neuronperf.model_index.delete(filename)\n neuronperf.model_index.delete(\"new_index.json\")\n shutil.rmtree(\"new_models\")\n shutil.rmtree(\"models\")\n\n\n@pytest.mark.sanity\ndef test_model_index_copy_2():\n filename = \"dummy_index.json\"\n if os.path.exists(filename):\n neuronperf.model_index.delete(filename)\n\n model_filename = os.path.join(\"models\", \"dummy.model\")\n os.makedirs(\"models\", exist_ok=True)\n pathlib.Path(model_filename).touch()\n model_name = \"Dummy\"\n model_index = neuronperf.model_index.create(model_filename, model_name=model_name)\n neuronperf.model_index.save(model_index, filename=filename)\n\n # Test copy API using a file\n neuronperf.model_index.copy(filename, \"new_index.json\", \"new_models\")\n assert os.path.exists(\"models\")\n assert os.path.exists(model_filename)\n assert os.path.exists(\"new_index.json\")\n assert os.path.exists(os.path.join(\"new_models\", \"dummy.model\"))\n\n new_index = neuronperf.model_index.load(\"new_index.json\")\n assert new_index[\"model_configs\"][0][\"filename\"] == os.path.join(\"new_models\", \"dummy.model\")\n\n neuronperf.model_index.delete(filename)\n neuronperf.model_index.delete(\"new_index.json\")\n shutil.rmtree(\"new_models\")\n shutil.rmtree(\"models\")\n\n\n@pytest.mark.sanity\ndef test_model_index_move():\n filename = \"dummy_index.json\"\n if os.path.exists(filename):\n neuronperf.model_index.delete(filename)\n\n model_filename = os.path.join(\"models\", \"dummy.model\")\n os.makedirs(\"models\", exist_ok=True)\n pathlib.Path(model_filename).touch()\n model_name = \"Dummy\"\n model_index = neuronperf.model_index.create(model_filename, model_name=model_name)\n neuronperf.model_index.save(model_index, filename=filename)\n\n neuronperf.model_index.move(filename, \"new_index.json\", \"new_models\")\n assert not os.path.exists(filename)\n assert not os.path.exists(model_filename)\n assert os.path.exists(\"new_index.json\")\n assert os.path.exists(os.path.join(\"new_models\", \"dummy.model\"))\n\n new_index = neuronperf.model_index.load(\"new_index.json\")\n assert new_index[\"model_configs\"][0][\"filename\"] == os.path.join(\"new_models\", \"dummy.model\")\n\n neuronperf.model_index.delete(\"new_index.json\")\n shutil.rmtree(\"new_models\")\n shutil.rmtree(\"models\")\n\n\n@pytest.mark.sanity\ndef test_model_index_append():\n model_indexes = [\n neuronperf.model_index.create(f\"Dummy_{x}\", model_name=\"Dummy\") for x in range(10)\n ]\n combined_index = neuronperf.model_index.append(*model_indexes)\n # Assert that combination apparently did happen.\n assert len(combined_index[\"model_configs\"]) == len(model_indexes)\n # Check that batch_sizes haven't been modified.\n assert all(1 == config[\"batch_size\"] for config in combined_index[\"model_configs\"])\n\n # Test for duplicate filtering behavior\n model_indexes = [neuronperf.model_index.create(\"Dummy\") for _ in range(10)]\n combined_index = neuronperf.model_index.append(*model_indexes)\n assert len(combined_index[\"model_configs\"]) == 1\n\n\n@pytest.mark.sanity\ndef test_model_index_filter():\n idx_1 = neuronperf.model_index.create(\"fake\", performance_level=2, compile_s=1)\n idx_2 = neuronperf.model_index.create(\"fake2\", compile_s=2)\n idx = neuronperf.model_index.append(idx_1, idx_2)\n\n filtered = neuronperf.model_index.filter(idx, filename=\"fake\")\n print(filtered)\n assert 1 == len(filtered[\"model_configs\"])\n assert \"fake\" == filtered[\"model_name\"]\n\n filtered = neuronperf.model_index.filter(idx, performance_level=2)\n assert 1 == len(filtered[\"model_configs\"])\n assert \"fake\" == filtered[\"model_name\"]\n\n # None key should filter nothing\n filtered = neuronperf.model_index.filter(idx, compile_s=None)\n assert 2 == len(filtered[\"model_configs\"])\n\n\n@pytest.mark.sanity\n@pytest.mark.slow\ndef test_benchmarker():\n dummy_model = lambda x: None\n dummy_load = lambda path, device_id: dummy_model\n b = neuronperf.benchmarking.Benchmarker(\n id=0, device_id=0, load_fn=dummy_load, model_filename=\"test\", inputs=[], workers_per_model=2\n )\n b.start()\n time.sleep(1.5)\n b.stop()\n\n assert b.status == \"finished\"\n assert all(n_infs > 100 for n_infs in b.n_infs)\n\n\n@pytest.mark.slow\ndef test_benchmark_multithread():\n benchmarker_results = neuronperf.cpu.benchmark(\n neuronperf.DummyModel,\n [np.array([1, 2, 3, 4])],\n duration=2,\n n_models=4,\n multiprocess=False,\n multiinterpreter=False,\n verbosity=2,\n return_timers=True,\n )\n\n # Return value is a list of tuples:\n # [(config, results), (config, results), ...]\n # Each config is a dict. Each result is a dict.\n\n # A single configuration without workers_per_model set will produce 2 results\n assert len(benchmarker_results) == 2\n\n for benchmarker_result in benchmarker_results:\n config, results = benchmarker_result\n assert \"cpu_percents\" in results\n assert \"mem_percents\" in results\n assert not config[\"multiprocess\"]\n assert not config[\"multiinterpreter\"]\n assert results[\"status\"] == \"finished\"\n assert results[\"n_infs\"] > 100\n\n\n@pytest.mark.slow\ndef test_benchmark_multithread_2():\n dummy_model = lambda x: None\n dummy_load = lambda path, device_id: dummy_model\n reports = neuronperf.benchmark(\n load_fn=dummy_load,\n model_filename=\"dummy_filename\",\n inputs=[[1]],\n duration=2,\n n_models=4,\n multiprocess=False,\n multiinterpreter=False,\n verbosity=2,\n )\n\n # A single configuration without workers_per_model set will produce 2 results\n assert len(reports) == 2\n report = reports[0]\n assert not report[\"multiprocess\"]\n assert not report[\"multiinterpreter\"]\n assert report[\"status\"] == \"finished\"\n assert report[\"total_infs\"] > 100\n\n\n@pytest.mark.slow\ndef test_benchmark_multiprocess():\n n_models = 16\n benchmarker_results = neuronperf.cpu.benchmark(\n neuronperf.DummyModel,\n inputs=[np.array([1, 2])],\n batch_sizes=[1],\n duration=2,\n n_models=n_models,\n multiprocess=True,\n multiinterpreter=False,\n verbosity=2,\n return_timers=True,\n )\n\n # A single configuration will produce a single result tuple\n assert len(benchmarker_results) == 2\n # Extract the benchmarker results\n config, results = benchmarker_results[0]\n # Confirm that there is least 1 timer / model for each benchmarker\n assert len(next(iter(results[\"timers\"].values()))) >= n_models\n assert config[\"multiprocess\"]\n assert not config[\"multiinterpreter\"]\n assert results[\"status\"] == \"finished\"\n assert results[\"n_infs\"] > 100\n\n\n@pytest.mark.slow\ndef test_benchmark_multiinterpreter():\n benchmarker_results = neuronperf.cpu.benchmark(\n neuronperf.DummyModel,\n inputs=[np.array([1, 2])],\n duration=2.5,\n n_models=2,\n multiprocess=False,\n multiinterpreter=True,\n verbosity=2,\n return_timers=True,\n )\n\n # A single configuration without workers_per_model set will produce 2 results\n assert len(benchmarker_results) == 2\n # Extract the benchmarker results\n config, results = benchmarker_results[0]\n assert config[\"multiinterpreter\"]\n assert results[\"status\"] == \"finished\"\n assert results[\"n_infs\"] > 100\n\n\n@pytest.mark.slow\ndef test_reporting():\n benchmarker_results = neuronperf.cpu.benchmark(\n neuronperf.DummyModel,\n inputs=[np.array([1, 2, 3, 4])],\n n_models=[1, 4],\n duration=2,\n verbosity=2,\n return_timers=True,\n )\n\n assert len(benchmarker_results) == 4\n reports = neuronperf.get_reports(benchmarker_results)\n assert len(reports) == len(benchmarker_results)\n assert all(\"total_infs\" in report for report in reports)\n\n neuronperf.print_reports(reports)\n csv_file = neuronperf.write_csv(reports)\n os.remove(csv_file)\n json_file = neuronperf.write_json(reports)\n with open(json_file, \"rt\") as fp:\n json.load(fp)\n os.remove(json_file)\n" }, { "path": "static/google673a8c4fbaa024d8.html", "content": "google-site-verification: google673a8c4fbaa024d8.html" }, { "path": "static/robots.txt", "content": "User-agent: *\n\nDisallow: /en/v2.24.0/\n\nDisallow: /en/v2.23.0/\n\nDisallow: /en/v2.22.1/\n\nDisallow: /en/v2.22.0/\n\nDisallow: /en/v2.21.1/\n\nDisallow: /en/v2.21.0/\n\nDisallow: /en/v2.20.2/\n\nDisallow: /en/v2.20.1/\n\nDisallow: /en/v2.20.0/\n\nDisallow: /en/v2.19.1/\n\nDisallow: /en/v2.19.0/\n\nDisallow: /en/v2.18.2/\n\nDisallow: /en/v2.18.1/\n\nDisallow: /en/v2.18.0/\n\nDisallow: /en/v2.17.0/\n\nDisallow: /en/v2.16.1/\n\nDisallow: /en/v2.16.0/\n\nDisallow: /en/v2.15.2/\n\nDisallow: /en/v2.15.1/\n\nDisallow: /en/v2.15.0/\n\nDisallow: /en/v2.14.1/\n\nDisallow: /en/v2.14.0/\n\nDisallow: /en/v2.13.2/\n\nDisallow: /en/v2.13.1/\n\nDisallow: /en/v2.13.0/\n\nDisallow: /en/v2.12.2/\n\nDisallow: /en/v2.12.1/\n\nDisallow: /en/v2.12.0/\n\nDisallow: /en/v2.11.0/\n\nDisallow: /en/v2.10.0/\n\nDisallow: /en/v2.9.0/\n\nDisallow: /en/v2.8.0/\n\nDisallow: /en/v2.7.0/\n\nDisallow: /en/v2.6.0/\n\nDisallow: /en/v2.5.0/\n\nDisallow: /en/v2.4.0/\n\nDisallow: /en/v2.3.0/\n\nDisallow: /en/v1.19.2/\n\nDisallow: /en/v1.19.1/\n\nDisallow: /en/v1.19.0/\n\nDisallow: /en/v1.18.0/\n\nDisallow: /en/v1.17.2/\n\nDisallow: /en/v1.17.1/\n\nDisallow: /en/v1.17.0/\n\nDisallow: /en/v1.16.3/\n\nDisallow: /en/v1.16.2/\n\nDisallow: /en/v1.16.1/\n\nDisallow: /en/v1.16.0/\n\nDisallow: /en/v1.15.2/\n\nDisallow: /en/1.15.1/\n\nDisallow: /en/1.15.0/\n\nDisallow: /en/1.14.2/\n\nDisallow: /en/1.14.1/\n\nDisallow: /en/1.14.0/\n\nDisallow: /en/1.13.0/\n\nDisallow: /en/1.12.2/\n\nDisallow: /en/1.12.1/\n\nDisallow: /en/1.12.0/\n\nDisallow: /en/1.11.0/\n\n\nSitemap: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/sitemap1.xml\n" }, { "path": "static/sitemap1.xml", "content": "\n\n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/index.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/misc-customops.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/third-party-solutions.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/dlami/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/nki_faq.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/torch-neuron-ubuntu20.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/mxnet-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/jax-neuronx.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/setup-troubleshooting.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/multiframework-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/setup-rocky-linux-9.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/troubleshooting.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/ecs-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/third-party-solutions.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/dlc-then-customize-devflow.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/eks-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/aws-batch-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/sagemaker-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/parallelcluster-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/ec2-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/index.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/releasecontent.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/2.29.0.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-developer-guide.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-configurable-parameters.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/configuration-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/rn.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-troubleshoot.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/faq.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/dlc-then-ecs-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/dlc-then-customize-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/neo-then-hosting-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/dlc-then-k8s-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorial-docker-runtime1.0.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/dlc-then-ec2-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/faq-troubleshooting-releasenote.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/neuron-dra.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/ec2.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/dlc-then-eks-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/container-deployment-flows.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/ec2-then-ec2-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/k8.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/troubleshooting.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/kubernetes-getting-started.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/developerflows.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/locate-neuron-dlc-image.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/faq.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/neuron-plugins.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/container-sm-hosting-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/getting-started.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/beta-participation.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/amazonq-getstarted.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/whats-new.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/troubleshooting.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/profiling-tools.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/monitoring-tools.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/sdk-policy.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/what-is-neuron.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/security.html\n 2026-02-13\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuron-cc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuronx-cc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/tensorflow_serving_tutorial.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuronx-cc/how-to-convolution-in-unet.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuronx-cc/developer-guide.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuronx-cc/faq.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuron-cc/api-reference-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuron-cc/command-line-reference.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuron-cc/developer-guide.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuron-cc/faq.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF005.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF011.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/ESPP047.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF010.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF004.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF006.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF007.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF013.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF017.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EBVF030.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF016.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EHCA005.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EARG001.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF015.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF001.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EBIR023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EUOC002.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EXTP004.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EOOM001.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/ESPP004.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EOOM002.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF018.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF024.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF031.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF019.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF022.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EVRF009.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/EXSP001.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/error-codes/ESFH002.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/compiler/neuronx-cc/api-reference-guide/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/calculator/neuron-calculator.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/neuron2-intro-faq.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/contributing-faq.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/onnx-faq.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/index.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/models/index.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/models/inference-inf1-samples.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/models/training-trn1-samples.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/models/inference-inf2-trn1-samples.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/benchmarks/index.html\n 2025-11-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/glossary.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/oss/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/mxnet-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/torch-neuron-tab-training.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/tensorflow-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/docs-quicklinks.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/user-guide-quickstart.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/github-samples.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/tab-inference-tensorflow-neuron.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/inference-quickstart.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/quick-start/training-quickstart.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/news-and-blogs/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trn1-arch.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inferentia2.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium3.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inf1-arch.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inferentia.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trn2-arch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trn3-arch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/neuron-core-v4.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/neuron-core-v1.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inf2-arch.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/neuron-core-v2.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/neuron-core-v3.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/rounding-modes.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/data-types.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/neuron-caching.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/custom-c++-operators.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/neuroncore-pipeline.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/neuroncore-batching.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/logical-neuroncore-config.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/benchmarks/inf2/inf2-performance.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/benchmarks/trn1/trn1-training-performance.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/benchmarks/trn1/trn1-inference-performance.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/benchmarks/inf1/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-transition-pytorch-trainium.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-multiframework-dlamis-inf1.html\n 2025-10-17\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-maintenance-nxdt-nxd-core-training.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-pytorch-2-7-2-8.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-tensorflow-2-8-9.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pytorch-1-1-3.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-python38-no-longer-support.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/github-changes.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/sm-training-trn1-introduce.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-nemo-megatron.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-nemo.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-beta-pytorch-neuroncore-placement-apis.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pt2.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-nxdi-changes.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eol-megatron-lm.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-device-version.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-torch-neuronx-nki-jit.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-xla-bf16.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-llama3-2-checkpoint.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-jax-neuronx-nki-call.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-ubuntu-20-base.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-eos-tensorflow-tutorial-inf.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-end-of-support-pytorch-2-6.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-maintenance-nxdi-nxd-core-inference.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-neurondevice.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-eol-nemo-arg.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-eos-tnx.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-python38.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-pytorch-113.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-nki-library-kernel-migration.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eol-ubuntu-18.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neuron-profiler-v230.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pytorch-1-9.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-al2.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-u20-dlamis.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/sm-training-dlc-2.9.1.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-inf1-virtual-environments.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-maintenance-mxnet.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-nki-library-namespace-changes.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pytorch-profiling-api.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/neuron2-intro.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-nxd-examples.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-dlami-ubuntu-22-04.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/gpg-expiration.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-tensorflow-inf2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-torch-neuron-versions.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-tensorboard-tools.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-component-change.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-nxd-examples.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-tensorflow1-x.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-package-change.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-mllama-checkpoint.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-end-of-support-nxdt-nxd-core.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-eos-pt2-6.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pytorch-2-7-2-8-v229.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neuron-driver-support-inf1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/dlami-neuron-2.10.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-end-of-support-neuronxcc-nki.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-tensorflow-inf2.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pt-versions.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/dlami-pytorch-introduce.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-torch-neuron.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/dlami-neuron-2.12.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neuron-profiler.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-tf-versions.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-pytorch-2-1.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neuron-profiler-2.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/neuron250-packages-changes.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-neuron-det.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-vllm-v0.html\n 2026-02-26\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-tensorboard-plugin.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neurondevice.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-end-of-support-vllm-v0.html\n 2026-02-26\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-deprecation-nxd-path-trace-api.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-nki-jit-torch.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-maintenance-tf.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-python-3-9-eol.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-deprecation-transformer-flag.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neuron-det.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-nki-library-namespace-changes-2-28.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-tensorflow2-10.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-moving-samples.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-eos-opt.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-megatronlm-2-13.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-neurondevice-version.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-block-dimension-nki.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pytorch-2-1.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announcement-end-of-support-parallel-model-trace.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-probuf.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-deprecation-containers-rtd.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-correction-neuron-driver-support-inf1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/end-of-support-pt2.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/neuron230-packages-changes.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-al2.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-nxdt-nxd-core-training.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-maintenance-tnx.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-support-tensorflow1-x.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/neuron-rtd-eol.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/release-neuron2.4.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-no-longer-support-u20-dlc-dlami.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-jax-neuronx-nki-call.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eol-python-3-7.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-bf16-vars.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-pytorch-2-7-2-8.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-nki-namespace-migration.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-eos-dlami.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron2.x/announce-intent-eos-pt-version.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/eol-pt-15.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/announce-eol-pt-before-1-8.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/eol-tf-21-24.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/announce-eol-pt-1-5.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/announce-eol-mx-before-1-5.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/announce-eol-tf-before-2-5.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/announcements.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/eol-ncgs-env_2.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/announcements/neuron1.x/announce-eol-tf-before-2-7.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-7.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/torch-neuronx-graph-partitioner-app-note.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-6.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/torch-neuronx-dataparallel-app-note.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-8.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-9.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/introducing-pytorch-2-x.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuronx/migration-from-xla-downcast-bf16.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/mxnet-neuron/flex-eg.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/transformers-neuronx/generative-llm-inference-with-neuron.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/neuronx-cc/neuronx-cc-training-mixed-precision.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/neuron-cc/mixed-precision.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/neuronx-distributed/introducing-nxd-inference.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/neuronx-distributed/introducing-nxdt-training.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuron/index.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuron/bucketing-app-note.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuron/torch-neuron-dataparallel-app-note.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/torch-neuron/rcnn-app-note.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/neuron1x/introducing-libnrt.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/perf/neuron-cc/performance-tuning.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/appnotes/perf/neuron-cc/parallel-ncgs.html\n 2025-10-20\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/training/neuron-training.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/inference/neuron-faq.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/faq/inference/trouble-shooting-faq.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/inference-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/guide-torch-neuron-vs-torch-neuronx-inference.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/pytorch-native-overview.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/training-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/jax/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/jax/setup/jax-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/jax/setup/jax-neuronx-known-issues.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/jax/api-reference-guide/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/jax/api-reference-guide/neuron-envvars.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-disable-dynamic-batching.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/training-troubleshooting.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-dynamic-batching.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/pytorch-neuron-supported-operators.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-default.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/misc-inference-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/additional-examples-training.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/additional-examples-inference-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-dim-neq-zero.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/torch-neuronx-dataparallel-example-specify-ncs.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup-trn1-multi-node-execution.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/misc-training.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/about/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-u24.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u24.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/note-setup-general.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install-prev-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-al2-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-neuronx-install-cxx11.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-install-prev-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/pytorch-update-u20-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/torch-neuronx-profiling-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/torch-neuronx-profiling-dev-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/analyze_for_training.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/tutorials-training-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/finetune_hftrainer.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/mlp.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/zero1_gpt2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/bert.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/inference/tutorials-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/inference/tutorial-torchserve-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/training/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/training/pytorch-neuron-programming-guide.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/training/pytorch-neuron-debug.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/inference/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/inference/core-placement.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/inference/autobucketing-dev-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/programming-guide/inference/trace-vs-xla-lazytensor.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/training/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/training/pytorch-neuron-parallel-compile.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/training/torch-neuron-envvars.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-async-lazy-load.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-replace-weights.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-core-placement.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/inference-api-guide-torch-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-trace.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-data-parallel.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/api-reference-guide/inference/api-torch-neuronx-analyze.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.8.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.7.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/setup/prev-releases/neuronx-2.9.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/.git/logs/refs/remotes/origin/VRF004.html\n 2026-01-27\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/how-to/how-to-ultraserver.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/files/index-dra.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/tutorial-oci-hook.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-device-plugin.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-monitor.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/build-run-neuron-container.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-multiple-scheduler.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-scheduler.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-problem-detector-and-recovery.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/tutorial-docker-env-setup.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-problem-detector-and-recovery-irsa.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-default-scheduler.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-helm-chart.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-setup.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-prerequisite.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/k8s-neuron-scheduler-flow.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/get-started/quickstart-configure-deploy-dlc.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/get-started/quickstart-pytorch-inference-dlc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/training/Dockerfile-trainium-dlc.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/training/mlp.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/inference/config-properties.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/inference/torchserve-neuron.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/inference/dockerd-libmode-entrypoint.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/inference/Dockerfile-tf-serving.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/inference/Dockerfile-libmode.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/inference/Dockerfile-inference-dlc.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/v1/inference/Dockerfile-torch-neuron.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/v1/inference/Dockerfile-app-rt-same.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/v1/inference/Dockerfile-app-rt-diff.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/v1/inference/dockerd-entrypoint-app-rt-same.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/docker-example/v1/inference/Dockerfile-neuron-rtd.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/training/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/training/tutorial-training.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/training/k8s_mlp_train_demo.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/inference/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/inference/k8s_rn50_demo.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/tutorials/inference/tutorial-infer.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/about/index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/about/core-dump.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/about/collectives.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/compute-comm-overlap.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/work-with-neff-files.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/direct-hbm-tensor-alloc.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/core-dump-deep-dive.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/intranode-collective-comm.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/device-memory.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/runtime-performance-tips.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/explore/internode-collective-comm.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_async_sendrecv.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_status.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt-async-api-best-practices.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_async.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/ndebug_stream.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_sys_trace.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/debug-stream-api.html\n 2026-02-05\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt-async-api-examples.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/ndl.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt-async-api-overview.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nec.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/neuron_driver_shared.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_experimental.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/neuron_ds.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_profile.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/neuron_driver_shared_tensor_batch_op.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/api/nrt_version.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/content.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.28.1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.28.0.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/rn.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/mxnet-neuron.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorboard-neuron.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/libneuronxla.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/index.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/runtime.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/nxd-inference.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/nki-lib.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/dlamis.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/dev-tools.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/nki.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/containers.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/nxd-training.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/compiler.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/jax.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/pytorch.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/components/nxd-core.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/nemo/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/nemo/neuronx-nemo.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/customcxxps/gpsimd-tools.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/customcxxps/gpsimd-customop-lib.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron-cc/neuron-cc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron1/prev/content.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron1/prev/rn.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron1/neuronrelease/previous-content.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorflow/tensorflow-neuron/tensorflow-neuron.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorflow/tensorflow-neuron/tensorflow-neuron-v2.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorflow/tensorflow-modelserver-neuron/tensorflow-modelserver-neuron.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorflow/tensorflow-modelserver-neuron/tensorflow-modelserver-neuronx.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorflow/tensorflow-modelserver-neuron/tensorflow-modelserver-neuron-v2.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/tensorflow/tensorflow-neuronx/tensorflow-neuronx.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron-cc/neuron-cc-ops/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron-cc/neuron-cc-ops/neuron-cc-ops-pytorch.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron-cc/neuron-cc-ops/neuron-cc-ops-xla.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron-cc/neuron-cc-ops/neuron-cc-ops-tensorflow.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/archive/neuron-cc/neuron-cc-ops/neuron-cc-ops-mxnet.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/nx-jax.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/runtime.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/dlami.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/nxd-inference.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/nx-pytorch.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/tools.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/docs-and-samples.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/containers.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/nxd-training.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/compiler.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.25.0/nxd-core.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/nx-jax.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/runtime.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/dlami.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/nxd-inference.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/nx-pytorch.html\n 2025-12-19\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/tools.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/nki.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/containers.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.26.0/nxd-core.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/runtime.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/dlami.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/nxd-inference.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/nki-lib.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/nx-pytorch.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/tools.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/nki.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/containers.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/2.27.0/compiler.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/plugins/npd-ecs-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/dlc-then-ecs-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/aws-batch-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/sagemaker-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/parallelcluster-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/ec2-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/setup/ecs-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/setup/eks-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/dlc-then-ecs-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/byoc-hosting-devflow-inf2.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/neo-then-hosting-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/dlc-then-k8s-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/dlc-then-ec2-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/aws-batch-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/dlc-then-eks-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/ec2-then-ec2-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/sagemaker-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/dev-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/byoc-hosting-devflow.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/parallelcluster-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/env-setup-text.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/ec2-then-ec2-devflow-inf2.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/ec2-flows.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/inference/container-sm-hosting-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/sm-devflow/sm-training-devflow.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/batch/batch-training.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/parallelcluster/parallelcluster-training.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/devflows/training/ec2/ec2-training.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nemo-megatron/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/transformers-neuronx/index.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/index.html\n 2026-02-26\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/overview-index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/nxdi-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/neuron-inference-overview.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/api-reference-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/overview.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/misc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer-guide.html\n 2025-11-11\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api-reference-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tp_developer_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tensor_parallelism_overview.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/context_parallelism_overview.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/pp_developer_guide.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/model_optimizer_wrapper_developer_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/activation_memory_reduction_developer_guide.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api-reference-guide-training.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/neuronx_distributed_inference_developer_guide.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/developer-guide-inference.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api-reference-guide-inference.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/model_builder_v2_api_reference.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/developer-guide-training.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/lora_finetune_developer_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/app_notes.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/pipeline_parallelism_overview.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/standard_mixed_precision.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/ptl_developer_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/index-training.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/developer-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/activation_memory_reduction.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/save_load_developer_guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/api_guide.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/index-inference.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/neuronx-distributed-misc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/setup/index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/inference.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/training_tutorials.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/training_llama_tp_pp.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/training_llama_tp_zero1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/inference_tutorials.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/finetune_llama3_8b_ptl_lora.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/neuronx-distributed/tutorials/training.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/general/config_overview.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/general/features.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/general/installation_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/general/known_issues.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-tp-appnote.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-cp-appnote.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-amr-appnote.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/app_notes/nxd-training-pp-appnote.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/hf_llama3_70B_pretraining.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/hf_llama3_8B_DPO_ORPO.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/hf_llama3_8B_SFT.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/hf_llama3_8B_SFT_LORA.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/checkpoint_conversion.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/tutorials/hf_llama3_8B_pretraining.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/index.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/migration_nnm_nxdt.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/cpu_mode_developer_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/optimizer_lr_scheduler_flow.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/new_model_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/new_dataloader_guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-training/developer_guides/migration_nemo_nxdt.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/misc/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/misc/nxdi-troubleshooting.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/app-notes/parallelism.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/app-notes/index.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/vllm/index.html\n 2026-02-26\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/vllm/quickstart-vllm-offline-serving.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/vllm/quickstart-vllm-online-serving.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/models/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/disaggregated-inference-tutorial.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/trn3-gpt-oss-120b-tutorial.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/trn2-llama3.3-70b-tutorial.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/sd-inference-tutorial.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/trn2-llama3.1-405b-tutorial.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/trn2-llama3.3-70b-fp8.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/trn2-llama3.1-405b-speculative-tutorial.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/tutorials/disaggregated-inference-tutorial-1p1d.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/index.html\n 2026-02-26\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/nxd-examples-migration-guide.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/model-reference.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/weights-sharding-guide.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/vllm-user-guide-v1.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/performance-cli-params.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/moe-arch-deep-dive.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/onboarding-models.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/vllm-user-guide.html\n 2026-02-26\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/llm-inference-benchmarking-guide.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/disaggregated-inference.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/custom-quantization.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/migrate-from-tnx-to-nxdi.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/how-to-use-fpem.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/accuracy-eval-with-datasets.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/writing-tests.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/feature-guide.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/api-guides/index.html\n 2025-11-12\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/api-guides/api-guide.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/models/llama3/llama_33_70b.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/models/qwen3/qwen3_moe_235b.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/al2-python.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/launch-trn1-dlami.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/legacy-inf1/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/legacy-inf1/pytorch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/setup-jupyter-notebook-steps-troubleshooting.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/notebook/running-jupyter-notebook-as-script.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/update-dlc.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/manual.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/update-manual.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/update-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/pytorch/dlc.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/jax/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/jax/dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/jax/manual.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/jax/dlc.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf2/note-setup-libnrt-warning.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf2/launch-inf2-dlami.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf2/dlami-enable-neuron-pytorch.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/trn1/dlami-notes.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/neuron-pip-install.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/note-setup-cntr.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/launch-inf1-dlami-aws-cli.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/launch-inf1-dlami.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/launch-inf1-ami.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/note-setup-general.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/neuron-pip-setup.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/compile_mode.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/develop_mode.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/tensorboard-plugin-neuron-pip-install.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/dlami-enable-neuron-mxnet.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/note-setup-libnrt-warning.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/dlami-enable-neuron-pytorch.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/setup/install-templates/inf1/deploy_mode.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-dge.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/index.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-dynamic-loops.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-compiler.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/mxfp-matmul.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/use-neuron-profile.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki_block_dimension_migration_guide.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-dma-bandwidth-guide.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-beta2-migration-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-aps.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-hbm-crc-hashing.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki-0-3-0-update-guide.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/deep-dives/nki_perf_guide.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/how-to-scheduling-apis.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/index.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/nki_simulator.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/framework_custom_op.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.isa.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.collectives.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.api.shared.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.language.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.simulate.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/nki.language.tile_size.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/quickstart-implement-run-kernel.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/nki-language-guide.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/setup-env.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/index.html\n 2026-04-08\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/tiling-overview.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/indexing-overview.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/data-representation-overview.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/lnc.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/nki-dma-overview.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/memory-hierarchy-overview.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.set_rng_seed.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.memset.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_n_gather.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.gelu_apprx_tanh.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.gelu_apprx_sigmoid.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.bn_stats.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.erf_dx.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.ceil.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.is_hbm.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float32.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.dge_mode.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.rms_norm.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.arctan.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.dma_engine.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.greater.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.mish.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.store.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.maximum.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.activation.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sequential_range.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.dropout.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.register_store.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.shared_identity_matrix.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_scalar_cumulative.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.trunc.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.rng.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.dma_transpose.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.tan.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.affine_select.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.matmul_perf_mode.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.VirtualRegister.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.exp.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.dma_compute.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.less_equal.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_matmul_mx.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.gelu.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.register_move.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.gelu_apprx_sigmoid_dx.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.broadcast_to.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.sendrecv.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.max.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.logical_not.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.reduce_scatter.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.collective_permute.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.softplus.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.static_range.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.all_gather.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.subtract.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_transpose.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float8_e5m2.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.load.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.shared_constant.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.bitwise_or.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.quantize_mx.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float4_e2m1fn_x4.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.collective_permute_implicit_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.softmax.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.program_id.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.int8.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.greater_equal.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_version.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_scalar.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.invert.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.uint32.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.core_barrier.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sign.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.shared_hbm.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.negative.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.affine_range.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nonzero_with_count.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.tfloat32.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.zeros.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.oob_mode.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.collective_permute_implicit.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.square.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.is_on_chip.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.ndarray.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.matmul.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.ones.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float8_e4m3.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.num_programs.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.rank_id.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.exponential.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.where.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.is_psum.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float8_e4m3fn.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.power.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_stream_shuffle.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.erf.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.register_alloc.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.rand_set_state.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.rand.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.reciprocal.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.load_transpose2d.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.int32.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sbuf.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sum.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.log.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.get_nc_version.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.equal.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.select_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.dma_copy.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.register_load.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.int16.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.engine.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.less.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float8_e4m3fn_x4.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.scalar_tensor_tensor.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.bfloat16.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_copy_predicated.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.private_hbm.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.not_equal.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_copy.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.logical_and.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.logical_or.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.multiply.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.uint8.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_find_index8.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.max8.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.floor.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.right_shift.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float8_e5m2_x4.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.prod.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.all_to_all.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.rsqrt.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.bool_.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.mean.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.expand_dims.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.min.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.left_shift.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.add.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.hbm.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.rand_get_state.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.jit.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.relu.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.range_select.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.ReplicaGroup.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_match_replace8.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_partition_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.all_to_all_v.html\n 2026-04-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.logical_xor.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.uint16.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.nc_matmul.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_tensor_scan.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.program_ndim.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.local_gather.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.tanh.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.bn_aggr.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sigmoid.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.bitwise_and.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_tensor.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.is_sbuf.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.transpose.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.simulate.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.ds.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.reduce_cmd.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.all_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.no_reorder.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.tensor_scalar_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.cos.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.copy.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.activation_reduce.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.device_print.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.full.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.silu_dx.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.collectives.collective_permute_implicit_current_processing_rank_id.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.psum.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.tile_size.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.empty_like.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sin.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.silu.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.iota.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.sequence_bounds.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.minimum.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.var.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.abs.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.reciprocal.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.gather_flattened.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.bitwise_xor.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.random_seed.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.isa.rand2.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.zeros_like.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.sqrt.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.gelu_dx.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.dropout.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.dynamic_range.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.all.html\n 2026-04-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/api/generated/nki.language.float16.html\n 2026-02-24\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/architecture/index.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/architecture/trainium2_arch.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/architecture/trainium3_arch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/architecture/trainium_inferentia2_arch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/matrix_multiplication.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/spmd_multiple_nc_tensor_addition.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/fused_mamba.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/kernel-optimization.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/transpose2d.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/average_pool2d.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/tutorials/spmd_tensor_addition.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/kernel-utils/index.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/kernel-utils/tiled-range.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/kernel-utils/tensor-view.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/kernel-utils/allocator.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/kernel-utils/stream-shuffle-broadcast.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/specs/index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/specs/design-rmsnorm-quant.html\n 2026-02-17\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/about/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/transformer-tkg.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/index.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/cross-entropy.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/conv1d.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/find-nonzero-indices.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/output-projection-cte.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/dynamic-elementwise-add.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/blockwise-mm-backward.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/mlp.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/fgcc.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/attention-cte.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/depthwise-conv1d.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/rmsnorm-quant.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/qkv.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/moe-tkg.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/output-projection-tkg.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/attention-block-tkg.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/fg-allgather.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/topk-reduce.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/sb2sb-allgather.html\n 2026-04-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/cumsum.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/moe-cte.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/attention-tkg.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/router-topk.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/library/api/rope.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/api-reference-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/mxnet-neuron-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/misc-mxnet-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/neo-then-hosting-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/ec2-then-ec2-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/api-compilation-python-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/developer-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/troubleshooting-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/inference-mxnet-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_terminology.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_benchmark_guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_troubleshooting.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_overview.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_examples.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_model_index_guide.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_evaluate_guide.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_faq.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/rn.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_compile_guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_framework_notes.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/neuronperf/neuronperf_install.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/api-reference-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/index.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/transformers-neuronx-tutorials.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/transformers-neuronx-developer-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/transformers-neuronx-api-reference.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/transformers-neuronx-misc.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/developer-guide.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/transformers-neuronx-developer-guide-for-continuous-batching.html\n 2025-10-28\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/torch-neuron-dataparallel-example-specify-ncs.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/torch-neuron-dataparallel-example-dynamic-batching.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/api-reference-guide-torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/torch-neuron-dataparallel-example-disable-dynamic-batching.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/inference-torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/api-torch-neuron-dataparallel-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/developer-guide-torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/torch-neuron-dataparallel-example-dim-neq-zero.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/additional-examples-inference-torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/api-compilation-python-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/torch-neuron-dataparallel-example-default.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/troubleshooting-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/api-core-placement.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/misc-inference-torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/helper-tools/index.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/helper-tools/tutorial-neuron-check-model.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/helper-tools/tutorial-neuron-gatherinfo.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/setup-legacy-inf1-tensorflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron-inference.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx-inference.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorboard/getting-started-tensorboard-neuron-plugin.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/training-gpt-neox.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/gpt3_neuronx_nemo_megatron_pretraining.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/training_llama2_tp_pp_ptl.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/megatron_gpt_pretraining.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/training-gpt-neox-20b.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/finetuning_llama2_7b_ptl.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/finetune_t5.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/multinode-training-model-profiling.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/training_codegen25_7b.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tutorials/ssd300_demo/ssd300_demo.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/api-reference-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/api-tracing-python-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tensorflow2-accelerated-ops.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/additional-examples.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/dlc-then-ecs-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/neo-then-hosting-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/dlc-then-ec2-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/api-auto-replication-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/dlc-then-eks-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/ec2-then-ec2-devflow.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/misc-tensorflow-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tf2_faq.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/api-tfn-analyze-model-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/api-compilation-python-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/api-reference-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/tfnx-analyze-model-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/tf-neuronx-auto-replication-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/tfneuronx-python-tracing-api.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/misc-tensorflow-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-install-prev-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-neuronx-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u20-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/tensorflow-update-al2-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/tutorials/tutorial-tensorflowx-serving-NeuronRT-Visible-Cores.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/tutorials/tutorials-tensorflow-neuronx.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/prev-releases/neuronx-2.9.0-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuronx/setup/prev-releases/neuronx-2.8.0-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-install-prev-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-update.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-update-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/tensorflow-update-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tutorials/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-nlp.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tutorials/tensorflow-tutorial-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tutorials/tutorials-tensorflow-utilizing-neuron-capabilities.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/tutorials/bert_demo/bert_demo.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.17.0-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.17.2-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.16.3-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.17.1-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.15.2-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.15.1-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.18.0-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.19.0-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.15.0-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/tensorflow/tensorflow-neuron/setup/prev-releases/neuron-1.14.2-tensorflow-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-update-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-update-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-update-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install-prev.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-update.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install-prev-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install-prev-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install-prev-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install-cxx11.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-update-al2-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-install-prev-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/pytorch-update-u20-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/guides/torch-lstm-support.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/tutorials-torch-neuron-nlp.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/transformers-marianmt.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/tutorials-utilizing-neuron-capabilities.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/tutorials-torch-neuron-computervision.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/neuroncore_pipeline_pytorch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/tutorial-libtorch.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/pytorch-tutorial-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/tutorial-torchserve.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/tutorials/tutorials-inference-torch-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/guides/core-placement/torch-core-placement.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.19.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.17.2-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-2.4.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.15.2-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.15.1-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.15.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.18.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-2.3.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.16.1-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-2.5.0-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.16.2-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.16.3-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/torch-neuron/setup/prev-releases/neuron-1.14.2-pytorch-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/transformers-neuronx/setup/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-update-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-neuron-al2-base-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-neuron-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-neuron-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-neuron-ubuntu22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-install-prev-al2.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-install-prev-u20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-install-prev-u22.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-neuron-ubuntu20.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-neuron-ubuntu20-base-dlami.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-install-prev-al2023.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/mxnet-update.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/tutorials/tutorials-mxnet-utilizing-neuron-capabilities.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/tutorials/tutorials-mxnet-computervision.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/tutorials/tutorial-model-serving.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/tutorials/tutorials-mxnet-nlp.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/tutorials/tutorials-mxnet-neuron.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/tutorials/mxnet-tutorial-setup.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.17.2-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.15.2-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.14.2-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.19.0-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.16.3-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.18.0-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.15.0-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/archive/mxnet-neuron/setup/prev-releases/neuron-1.15.1-mxnet-install.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/index.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/nccom-test.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-top-user-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-sysfs-user-guide.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-monitor-user-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-ls.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tensorboard/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tensorboard/getting-started-tensorboard-neuronx-plugin.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tutorials/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tutorials/torch-neuronx-profiling-with-tb.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tutorials/tutorial-neuron-monitor-mnist.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tutorials/tutorial-tensorboard-scalars-mnist.html\n 2025-10-09\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/tutorials/performance-profiling-vllm.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/profiler/neuron-profile-user-guide.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/profiler/neuron-profiler-2-0-beta-user-guide.html\n 2025-12-01\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/index.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-system-profiles.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-hierarchy-view.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-database-viewer.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/how-to-link-view-source-code.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/migration-faq.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-device-profiles.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/get-started.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/how-to-profile-workload.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-summary-page.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-memory-viewer.html\n 2026-04-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/view-perfetto.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-tensor-viewer.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-explorer/overview-ai-recommendations.html\n 2026-02-25\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/api-reference-guide/api-reference-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/api-reference-guide/custom-ops-ref-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/programming-guide/custom-c++-operators-devguide.html\n 2026-02-03\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/programming-guide/programming-guide.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/tutorials/customop-mlp-perf-opt.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/tutorials/tutorials.html\n 2025-10-07\n \n \n https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-customops/tutorials/customop-mlp-training.html\n 2025-10-07\n \n" }, { "path": "tools/index.rst", "content": ".. _neuron-tools:\n\n.. meta::\n :description: Developer tools for profiling, monitoring, and analyzing machine learning workloads on AWS Neuron devices.\n :keywords: AWS Neuron, developer tools, profiler, monitoring, analysis, TensorBoard, visualization, debugging, optimization\n :date-modified: 12/02/2025\n\nDeveloper Tools\n================\n\nAWS Neuron provides a comprehensive suite of developer tools for optimizing, monitoring, and debugging machine learning workloads on AWS Inferentia and Trainium accelerators. These tools enable developers to gain deep insights into model performance, system utilization, and hardware behavior to maximize the efficiency of ML applications running on Neuron-enabled instances.\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: Neuron Explorer\n :link: /tools/neuron-explorer/index\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n \n Neuron Explorer is a suite of tools designed to support ML engineers throughout their development journey on AWS Trainium, from model development through debugging, profiling, analysis, and optimization.\n\n .. grid-item-card:: Neuron Profiler 2.0\n :link: /tools/profiler/neuron-profiler-2-0-beta-user-guide\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n \n Neuron Profiler 2.0 offers a user-friendly experience for capturing and analyzing application performance through both high-level system profiles and detailed device-level profiles.\n\n .. grid-item-card:: Neuron Profiler\n :link: /tools/profiler/neuron-profile-user-guide\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n \n The Neuron Profiler is a tool to profile and analyze performance of a ML model compiled with the Neuron compiler and run on NeuronDevices.\n\n .. grid-item-card:: System Tools\n :link: /tools/neuron-sys-tools/index\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n \n Command-line utilities for monitoring, debugging, and managing AWS Neuron devices, including neuron-monitor, neuron-top, neuron-ls, and more.\n\n .. grid-item-card:: Third Party Tools\n :link: /tools/third-party-solutions\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n \n Third-party tools and integrations that support the AWS Neuron development experience, including monitoring, visualization, and optimization solutions.\n\n..\n .. grid-item-card:: AP Visualizer\n :link: ap-visualizer/ap-visualizer.html\n :link-type: url\n :class-header: sd-bg-primary sd-text-white\n \n Visualize access patterns of tensors on Neuron devices.\n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Tutorials\n :link: /tools/tutorials/index\n :link-type: doc\n :class-header: sd-bg-secondary sd-text-white\n\n Tutorials for how to utilize all Neuron Tools.\n\n .. grid-item-card:: Release Notes\n :link: /release-notes/components/dev-tools\n :link-type: doc\n :class-header: sd-bg-secondary sd-text-white\n\n Latest updates, new features, and improvements to Neuron Tools and Neuron Explorer.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n Neuron Profiler 2.0 \n Neuron Profiler \n System Tools \n Third-party Tools \n Tutorials \n Release Notes \n" }, { "path": "tools/neuron-explorer/get-started.rst", "content": ".. meta::\n :description: Setup and get started guide for new Neuron SDK profiler\n :date_updated: 12/02/2025\n\n.. _new-neuron-profiler-setup:\n\nGet Started with Neuron Explorer\n========================================\n\nIn this guide, you'll learn how to set up and launch Neuron Explorer, including the web-based UI for interactive analysis. By the end of this guide, you'll be able to visualize and analyze performance data for your models directly in your browser.\n\nOverview\n---------\n\nIn this guide, you'll launch an AWS Trainium or Inferentia EC2 instance using the AWS Deep Learning AMI (DLAMI) for Neuron, install and verify Neuron Explorer, start both the API and UI servers, and set up secure SSH tunneling to view the Neuron Explorer interface in your local browser.\n\nUse this tool when you want to collect, inspect, and visualize Neuron profiling data from model training or inference jobs running on Neuron-compatible instances. At a high level, you will:\n\n1. Launch a Neuron DLAMI instance\n2. Verify Neuron Explorer installation\n3. Start the Neuron Explorer servers\n4. Configure SSH tunneling\n5. Access the Neuron Explorer UI locally\n\n\nPrerequisites\n--------------\n\n* An AWS account with permissions to launch EC2 instances.\n* Access to an AWS Trainium or Inferentia instance type (such as trn1.2xlarge, inf2.xlarge).\n* AWS Neuron DLAMI with the latest Neuron SDK preinstalled.\n* SSH key pair (``.pem`` file) to securely connect to your EC2 instance.\n* Local machine with SSH client and web browser installed.\n\n\nBefore you begin\n-----------------\n\nComplete these steps before starting the task in this document:\n\n1. Make sure you have an active AWS account and `a default VPC available in your region `_. \n2. Create or locate your SSH key pair (``.pem`` file) that allows access to your EC2 instance.\n\nInstructions\n-------------\n\n1. Launch a Neuron-compatible EC2 instance\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nLaunch an EC2 instance with either a Trainium or Inferentia instance type using the AWS Neuron DLAMI.\nYou can do this from the AWS Management Console or CLI. For more instructions on how to launch an instance with Neuron DLAMI, refer to the instructions here.\n\n**Expected outcome**\n\nYour instance should start and appear in the EC2 dashboard as \"Running.\"\n\n\n2. Verify that Neuron Explorer is installed\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOnce you've connected to your EC2 instance with SSH, verify that Neuron Explorer and the associated tools are installed:\n\n.. code-block:: bash\n\n apt list --installed | grep neuronx-tools\n\n**Expected outcome**\n\nYou should see neuronx-tools listed among the installed packages, confirming that Neuron Explorer is available on your instance.\n\n\n3. Launch the API and UI SPA servers\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nStart the Neuron Explorer web servers using the following command:\n\n.. code-block:: bash\n\n neuron-explorer view -v 2 --data-path ./parquet_files\n\n\nThis command starts:\n\n* The UI SPA (Single Page Application) server (default port: 3001)\n* The API server (default port: 3002)\n\n\n**Expected outcome**\n\nYou'll see terminal logs confirming that both the UI and API servers are running.\n\n\n4. Set up SSH tunneling\n^^^^^^^^^^^^^^^^^^^^^^^^\n\nBy default, Neuron Explorer runs locally on the EC2 instance. To securely access it from your local computer, you must create SSH tunnels for ports 3001 and 3002.\n\nRun the following command from your local machine terminal (replace placeholders such as ``your-key`` and ``public_ip_address_of_your_instance``):\n\n.. code-block:: bash\n\n ssh -i ~/your-key.pem -L 3001:localhost:3001 -L 3002:localhost:3002 ubuntu@[public_ip_address_of_your_instance_] -fN\n\n**Explanation:**\n\n* ``-L 3001:localhost:3001`` forwards the UI server.\n* ``-L 3002:localhost:3002`` forwards the API server.\n* ``-fN`` keeps the tunnel open in the background.\n\n\n**Expected outcome**\n\nNo error messages should appear, indicating that your SSH tunnels are active.\n\n.. note::\n Replace ``ubuntu`` with the appropriate username for your AMI (for example, ``ec2-user`` on Amazon Linux).\n\n5. Connect to the Neuron Explorer UI\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOnce your tunnel is active, open your preferred web browser and navigate to:\n\n.. code-block:: text\n\n http://localhost:3001\n\n\n**Expected outcome**\n\nThe Neuron Explorer UI loads in your browser, displaying an interactive dashboard for exploring profiling data.\n\nConfirm your work\n------------------\n\nYou've successfully set up Neuron Explorer! To confirm everything is working:\n\n1. The browser should display the Neuron Explorer interface.\n2. The terminal running the profiler command should show log activity when interacting with the UI.\n3. You can explore profiling sessions from your ``./parquet_files`` directory.\n\nIf all these checks pass, you are ready to begin analyzing performance data using Neuron Explorer.\n\nCommon issues\n---------------\n\nIf you encounter an error or other issue while working through this task, here are some commonly encountered issues and how to address them:\n\n* **Neuron Explorer UI doesn't load**: Check that your SSH tunnel is configured correctly. Make sure ports 3001 and 3002 are forwarded using the ``-L`` flags in your SSH command, and verify the EC2 instance is running.\n* **No profiling data displayed**: Double-check that the directory passed to ``--data-path`` contains valid .parquet profiling files generated by a prior Neuron profiling run.\n* **neuron-profile command not found**: Ensure that Neuron SDK is installed. Please ensure that you have launched your instance with Neuron DLAMI or you have set up your instance based on the instructions mentioned here.\n* **Connection refused on port 3001 or 3002**: Confirm that your EC2 security group allows outbound traffic and that the SSH tunnel was created from your local machine, not from inside the instance.\n" }, { "path": "tools/neuron-explorer/how-to-link-view-source-code.rst", "content": ".. meta::\n :description: Learn how to use source code linking in Neuron Explorer to understand code performance and optimize your applications\n :date-modified: 11/21/2025\n\n.. _neuron-explorer-source-code:\n\nSource Code Viewer\n====================\n\nIn this guide, you'll learn how to use Neuron Explorer's source code linking feature to visualize connections between your application code and device performance. Discover how to navigate between source code and device instructions, highlight performance-critical sections, view framework stack traces, and leverage interactive code decorations to optimize your AWS Neuron applications for maximum efficiency.\n\nOverview\n--------\n\nSource code linking helps you understand how your code changes affect device performance and identify ways to optimize it. This feature creates interactive connections between source code files and other Neuron Explorer widgets. You can zoom to device instructions from selected code lines, navigate between instructions and source code, and highlight instructions for specific loop iterations. You can use source code linking in both the VS Code extension and standalone web application. This gives you flexibility for different developer workflows.\n\nThe Framework Stack Trace feature shows up in the Event Details when an instruction on the device profile is clicked. This feature is used to map the device instructions back to framework level code in JAX or PyTorch to better understand what part of the application code resulted in a particular device instruction.\n\n.. image:: /tools/profiler/images/view-link-1.gif\n\nInstructions\n-------------\n\nTo enable the addition of the \"NKI Source Location\" field to a profile enable set this environment variable: ``NEURON_FRAMEWORK_DEBUG=1``\n\nTo enable tracking of the stack trace information, you set these environment variables before compiling your NEFF:\n\n.. code-block:: bash\n\n export XLA_IR_DEBUG=1\n export XLA_HLO_DEBUG=1\n\nOnce you have the NEFF, you can simply capture the profile as usual. To view your source code while viewing the profile, use the ``--framework-source-root`` flag to pass the path to framework source files. This is optional and is only needed if you want to view your code alongside the displayedprofile.\n\n.. code-block:: bash\n\n neuron-explorer view -n file.neff -s profile.ntff --framework-source-root /path/to/framework/source/files\n\nCode Viewer Widget\n-------------------\n\nHighlighting Instructions\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSelect source code lines to highlight their corresponding instructions in the profiler view. You can select individual lines or multiple lines through block selection or multiple cursors.\n\n.. image:: /tools/profiler/images/view-link-2.png\n\nNavigating to Source Code\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n(Ctrl/Cmd)+Click any instruction to jump to it's location in source code. If there are multiple matches, you will be prompted to select which file to navigate to.\n\n.. image:: /tools/profiler/images/view-link-3.png\n\nSource Code Decorations\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nPerformance metrics appear as decorations directly in your source code, updating automatically with the instruction profiler's time range. \n\nConfigure which metrics to display and in the settings panel. Currently only instruction count and PE element count are supported.\n\n.. image:: /tools/profiler/images/view-link-4.png\n\nNavigating to Instructions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSelect lines in your source code and navigate to their corresponding instructions using Ctrl+Shift+G, the context menu, or the \"Zoom into Instructions\" command from the command palette. \n\nThe Device Trace Viewer will then zoom to show all instructions associated with your selection.\n\n.. image:: /tools/profiler/images/view-link-5.png\n\nDependency Annotations\n~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen enabled, selecting an instruction will highlight its dependent source code lines. The selected instruction's line will be highlighted in one color, with its dependencies shown in a different color.\n\n.. image:: /tools/profiler/images/view-link-6.png\n" }, { "path": "tools/neuron-explorer/how-to-profile-workload.rst", "content": ".. meta::\n :description: Learn how to capture a profile, launch the Neuron Explorer UI, and use the Profile Manager to analyze your workload performance.\n :date-modified: 12/02/2025\n\nCapture and View Profiles in Neuron Explorer\n================================================\n\nCapturing Profiles\n------------------\nIn this guide, you'll learn how to capture a profile, launch the Neuron Explorer, use the Profile Manager, and view Neuron Explorer in your IDE.\n\nTo get a better understanding of your workload's performance, you must collect the raw device traces and runtime metadata in the form of an NTFF (Neuron Trace File Format) which you can then correlate with the compiled NEFF (Neuron Executable File Format) to derive insights.\n\nSet the following environment variables before compiling to capture more descriptive layer names and stack frame information.\n\n.. code-block:: bash\n\n export XLA_IR_DEBUG=1\n export XLA_HLO_DEBUG=1\n\nFor NKI developers, set ``NEURON_FRAMEWORK_DEBUG`` in addition to the two above to enable kernel source code tracking:\n\n.. code-block:: bash\n\n export NEURON_FRAMEWORK_DEBUG=1\n\nIf profiling was successful, you will see NEFF (``.neff``) and NTFF (``.ntff``) artifacts in the specified output directory similar to the following:\n\n.. code-block:: bash\n\n output\n └── i-0ade06f040a13f2bf_pid_210229\n ├── 395760075800974_instid_0_vnc_0.ntff\n └── neff_395760075800974.neff\n\nDevice profiles for the first execution of each NEFF per NeuronCore are captured, and NEFF/NTFF pairs with the same prefix (for PyTorch) or unique hash (for JAX or CLI) must be uploaded together. See the section on :ref:`uploading profiles ` for more details.\n\nJAX Profiling API\n~~~~~~~~~~~~~~~~~\n\nWhen using the JAX context-managed profiling API, set two extra environment variables to signal the profile plugin to begin capturing device profile data when the profiling API is invoked.\n\n.. code-block:: python\n\n os.environ[\"NEURON_RT_INSPECT_DEVICE_PROFILE\"] = \"1\"\n os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"] = \"./output\"\n\nThen, profile a block of code:\n\n.. code-block:: python\n\n with jax.profiler.trace(os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"]):\n\nFull code example:\n\n.. code-block:: python\n\n from functools import partial\n import os\n import jax\n import jax.numpy as jnp\n\n from jax.sharding import Mesh, NamedSharding, PartitionSpec as P\n from jax.experimental.shard_map import shard_map\n from time import sleep\n from functools import partial\n\n os.environ[\"NEURON_RT_INSPECT_DEVICE_PROFILE\"] = \"1\"\n os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"] = \"./output\"\n\n jax.config.update(\"jax_default_prng_impl\", \"rbg\")\n\n mesh = Mesh(jax.devices(), ('i',))\n\n def device_put(x, pspec):\n return jax.device_put(x, NamedSharding(mesh, pspec))\n\n lhs_spec = P('i', None)\n lhs = device_put(jax.random.normal(jax.random.key(0), (128, 128)), lhs_spec)\n\n rhs_spec = P('i', None)\n rhs = device_put(jax.random.normal(jax.random.key(1), (128, 16)), rhs_spec)\n\n\n @jax.jit\n @partial(shard_map, mesh=mesh, in_specs=(lhs_spec, rhs_spec),\n out_specs=rhs_spec)\n def matmul_allgather(lhs_block, rhs_block):\n rhs = jax.lax.all_gather(rhs_block, 'i', tiled=True)\n return lhs_block @ rhs\n\n with jax.profiler.trace(os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"]):\n out = matmul_allgather(lhs, rhs)\n for i in range(10):\n with jax.profiler.TraceAnnotation(\"my_label\"+str(i)):\n out = matmul_allgather(lhs, rhs)\n sleep(0.001)\n\n\n expected = lhs @ rhs\n with jax.default_device(jax.devices('cpu')[0]):\n equal = jnp.allclose(jax.device_get(out), jax.device_get(expected), atol=1e-3, rtol=1e-3)\n print(\"Tensors are the same\") if equal else print(\"Tensors are different\")\n\n\n.. _neuron-explorer-capture-environment-variables:\n.. _neuron-explorer-non-framework-user-experience:\n\nEnvironment Variables\n~~~~~~~~~~~~~~~~~~~~~\n\nYou can also control profiling with environment variables. This is useful when you can’t easily change your \napplication code, such as when running an executable which calls the Neuron Runtime or in a containerized \nenvironment where the application code is built into the container image.\n\n.. _neuron-explorer-core-control-variables:\n\nCore Control Variables\n^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Variable\n - Description\n - Default behavior\n * - ``NEURON_RT_INSPECT_ENABLE``\n - Set to ``1`` to enable profiling\n - Enables system profiling and disables device profiling. To control which profile types are captured, see :ref:`Profile type selection `\n * - ``NEURON_RT_INSPECT_OUTPUT_DIR``\n - Directory for profile data output\n - Default directory for captured profile data is ``./output``\n\n.. _neuron-explorer-profile-type-selection:\n\nDevice or System Profile Type Selection\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. note:: \n \n When ``NEURON_RT_INSPECT_ENABLE`` set to ``1``, ``NEURON_RT_INSPECT_SYSTEM_PROFILE`` is enabled by default (set to 1) and ``NEURON_RT_INSPECT_DEVICE_PROFILE`` is disabled by default (set to ``0``).\n\nWhen ``NEURON_RT_INSPECT_ENABLE`` = 1, two different profile types are available:\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Variable\n - Profile type\n - Description\n - Enable capture\n - Disable capture\n * - ``NEURON_RT_INSPECT_SYSTEM_PROFILE``\n - System-level\n - Captures runtime system events and operations\n - Set to ``1``\n - Set to ``0``\n * - ``NEURON_RT_INSPECT_DEVICE_PROFILE``\n - Device-level\n - Captures detailed NeuronCore hardware metrics\n - Set to ``1``\n - Set to ``0``\n\n.. note::\n\n These variables have no effect if ``NEURON_RT_INSPECT_ENABLE`` is not set to ``1``.\n\n.. _neuron-explorer-advanced-config-vars:\n \nAdvanced configuration for System Profiles\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Variable\n - Profile type\n - Description\n - Default behavior\n * - ``NEURON_RT_INSPECT_SYS_TRACE_MAX_EVENTS_PER_NC``\n - System-level\n - Maximum trace events per NeuronCore before oldest events are overwritten\n - 1,000,000\n\n.. note:: \n \n Increasing the event limit will consume more host memory.\n\nCapture using nccom-test with Environment Variables\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nProfiling can be enabled using environment variables. For simplicity, we have a quick way to generate a Neuron workload through using :ref:`nccom-test `. nccom-test is a benchmarking tool which is already available with Neuron AMI.\n\n.. code-block:: shell\n\n export NEURON_RT_INSPECT_ENABLE=1\n export NEURON_RT_INSPECT_OUTPUT_DIR=./output\n nccom-test allr allg -b 512kb -e 512kb -r 32 -n 10 -d fp32 -w 1 -f 512\n\n.. note::\n If you have problems with nccom-test add the --debug flag.\n If using a trn1.2xlarge instance, change -r 32 to -r 2 to use fewer neuron cores.\n\nTo understand the profiling output see this section: :ref:`Inspect Output `\n\nCapture with EKS\n^^^^^^^^^^^^^^^^\n\nCapturing a profile on EKS is most easily done through setting of environment variables as described in the section \n:ref:`Non-framework specific User Experience `. By using environment \nvariables, users do not need to change application code in their container image or modify their run commands. \n\nUpdate the deployment yaml to include the ``NEURON_RT_INSPECT_ENABLE`` and ``NEURON_RT_INSPECT_OUTPUT_DIR`` \nenvironment variables. For distributed workloads, it’s important that ``NEURON_RT_INSPECT_OUTPUT_DIR`` points to a \ndirectory on a shared volume which all workers have access to.\n\n.. code-block:: yaml\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: trn1-mlp\n spec:\n restartPolicy: Never\n schedulerName: default-scheduler\n nodeSelector:\n beta.kubernetes.io/instance-type: trn1.32xlarge\n containers:\n - name: trn1-mlp\n env:\n - name: NEURON_RT_INSPECT_ENABLE\n value: \"1\"\n - name: NEURON_RT_INSPECT_OUTPUT_DIR\n value: \"/shared/output\"\n command: ['torchrun']\n args:\n - '--nnodes=1'\n - '--nproc_per_node=32'\n - 'train_torchrun.py'\n image: ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:mlp\n imagePullPolicy: IfNotPresent\n resources:\n limits: \n aws.amazon.com/neuron: 16\n\n\n.. note::\n\n EKS users running PyTorch and JAX applications are still free to change their application code \n and use the PyTorch or JAX Python profiling APIs if they want finer-grained control over profiling. \n However, using the environment variables conveniently allows profiling without modifying the \n container image or application code.\n\n\nCLI\n~~~\n\nIn certain cases, you may want to profile the application without requiring code modifications such as when deploying a containerized application through EKS. Note that when capturing with the CLI, profiling will be enabled for the entire lifetime of the application. If more granular control is required for profiling specific sections of the model, it is recommended to use the PyTorch or JAX APIs.\n\nTo enable profiling without code change, run your workload with the following environment variables set:\n\n.. code-block:: bash\n\n export NEURON_RT_INSPECT_ENABLE=1\n export NEURON_RT_INSPECT_DEVICE_PROFILE=1\n export NEURON_RT_INSPECT_OUTPUT_DIR=./output\n python train.py\n\nCLI reference for System Profiles\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIn addition to controlling profiling with environment variables, you can use the ``neuron-explorer inspect`` command line interface \nfor profiling applications. This provides the same functionality as environment variables but helps you avoid typos, invalid arguments, \nand provides a useful ``--help`` command to explain available options.\n\n.. code-block:: shell\n\n Usage:\n neuron-explorer [OPTIONS] inspect [inspect-OPTIONS] [userscript...]\n\n Application Options:\n -v, --version Show version and exit\n\n Help Options:\n -h, --help Show this help message\n\n [inspect command options]\n -o, --output-dir= Output directory for the inspection results (default: .)\n -n, --num-trace-events= Maximum number of trace events to capture when profiling. Once hitting this limit, old events are dropped\n\n [inspect command arguments]\n userscript: Run command/script that launches a Neuron workload. E.g. 'python app.py' or './runscript.sh'\n\nExample of using System Profiles CLI\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUser can provide any type of their own script to generate a Neuron workload such as Pytorch to the System Profiles CLI. \nFor simplicity, we have a quick way to generate a Neuron workload \nthrough using ``nccom-test``. ``nccom-test`` is a benchmarking tool which is already available with Neuron AMI and ``aws-neuronx-tools`` package.\n\n.. code-block:: shell\n\n ubuntu@ip-172-31-63-210:~$ neuron-explorer inspect -o inspect-output-nccom-test nccom-test allg -b 512kb -e 512kb -r 32 -n 10 -d fp32 -w 1 -f 512\n INFO[0000] Running command \"nccom-test allg -b 512kb -e 512kb -r 32 -n 10 -d fp32 -w 1 -f 512\" with profiling enabled\n size(B) count(elems) type time:avg(us) algbw(GB/s) busbw(GB/s)\n 524288 131072 fp32 24.15 21.71 21.03\n Avg bus bandwidth: 21.0339GB/s\n\n.. note::\n If you have problems with nccom-test add the --debug flag.\n If using a trn1.2xlarge instance, change -r 32 to -r 2 to use fewer neuron cores.\n\n.. _neuron-explorer-inspect-output:\n\n``neuron-explorer inspect`` Output\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe above command traces a Neuron workload execution and saves the output to the ``inspect-output-nccom-test`` directory. \nYou will see the output directory contains a single NEFF file and a device profile (NTFF) for all Neuron Cores which executed that NEFF. \nYou will also see ``ntrace.pb`` and ``trace_info.pb`` files storing the system profile data.\nBelow showing what the outputs will look like:\n\n.. code-block:: shell\n\n ubuntu@ip-172-31-63-210:~$ tree inspect-output-nccom-test\n inspect-output-nccom-test\n ├── i-012590440bb9fd263_pid_98399\n │ ├── 14382885777943380728_instid_0_vnc_0.ntff\n │ ├── 14382885777943380728_instid_0_vnc_1.ntff\n │ ├── 14382885777943380728_instid_0_vnc_10.ntff\n │ ├── 14382885777943380728_instid_0_vnc_11.ntff\n ...\n │ ├── 14382885777943380728_instid_0_vnc_8.ntff\n │ ├── 14382885777943380728_instid_0_vnc_9.ntff\n │ ├── cpu_util.pb\n │ ├── host_mem.pb\n │ ├── neff_14382885777943380728.neff\n │ ├── ntrace.pb\n │ └── trace_info.pb\n └──\n\n 2 directories, 74 files\n\n\nTo view a summary of the captured profile data run the command\n\n.. code-block:: shell\n\n neuron-explorer view -d inspect-output-nccom-test --output-format summary-text\n\n\n.. _neuron-explorer-filtering-system-profiles:\n\nCapture-time Filtering\n----------------------\n\n**Capture-time filtering** reduces memory usage and trace file size by only collecting specific events, but filtered data cannot be recovered later.\nConfigure filters before trace capture using environment variables or API functions. \nYou can use NeuronCore filters to only capture events for specific NeuronCores (for example only events associated with NeuronCore 0 or all the NeuronCores on a specific NeuronDevice). \nYou can use event type filters to only capture specific events (for example model execute or collectives events). \nIt is possible to combine both NeuronCore and event type filters.\n\nNeuronCore\n~~~~~~~~~~\n\nIf capture is enabled for a NeuronCore then a ring buffer will be allocated in host memory for storing those core's events. Thus filtering by NeuronCore decreases host memory usage during capture.\n\nDefault Behavior\n^^^^^^^^^^^^^^^^\n\nBy default, all visible NeuronCores are enabled for capture. \n\nUsing Environment Variables\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code-block:: shell\n\n # Filter to capture events only from NeuronCore 0\n export NEURON_RT_INSPECT_EVENT_FILTER_NC=0\n\n # Filter to capture events from NeuronCores 0, 2, and 4\n export NEURON_RT_INSPECT_EVENT_FILTER_NC=0,2,4\n\n # Filter to capture events from a range of NeuronCores (0 through 3)\n export NEURON_RT_INSPECT_EVENT_FILTER_NC=0-3\n\n # Reset to default behavior\n unset NEURON_RT_INSPECT_EVENT_FILTER_NC # Back to capturing all visible cores\n\nUsing API Functions\n^^^^^^^^^^^^^^^^^^^\n\n.. code-block:: c\n\n #include \n\n // Allocate and configure trace options\n nrt_sys_trace_config_t *config;\n nrt_sys_trace_config_allocate(&config);\n nrt_sys_trace_config_set_defaults(config);\n\n // Enable capture only for specific NeuronCores\n\n // Disable all cores since by default they are all enabled\n int num_cores = 128;\n for (int i=0; i\n\n // Get all available event types\n const char **event_types = nullptr;\n size_t count = 0;\n NRT_STATUS status = nrt_sys_trace_get_event_types(&event_types, &count);\n\n if (status == NRT_SUCCESS) {\n printf(\"Available event types:\\n\");\n for (size_t i = 0; i < count; ++i) {\n printf(\" %s\\n\", event_types[i]);\n }\n \n // Free the event types array\n for (size_t i = 0; i < count; ++i) {\n free((void*)event_types[i]);\n }\n free((void*)event_types);\n }\n\nUsing Environment Variables\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe ``NEURON_RT_INSPECT_EVENT_FILTER_TYPE`` environment variable supports:\n\n* **Default**: If not set, all event types are captured\n* **Specific event types**: Use exact event names from ``nrt_sys_trace_get_event_types()``\n* **Event categories**: Use ``hardware`` or ``software`` to filter by category\n* **Exclusion**: Use ``^`` prefix to exclude specific events from a category\n\n.. code-block:: shell\n\n # Filter to capture only specific event types\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=model_load,nrt_execute,runtime_execute\n\n # Filter to capture all hardware events\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=hardware\n\n # Filter to capture all software events\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=software\n\n # Filter to capture all hardware events EXCEPT cc_exec\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=hardware,^cc_exec\n\n # Filter to capture all software events EXCEPT model_load\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=software,^model_load\n\n # Mix categories and specific events\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=hardware,tensor_read,tensor_write\n\n # Reset to default behavior\n unset NEURON_RT_INSPECT_EVENT_FILTER_TYPE # Back to capturing all event types\n\nThe ``hardware`` group contains events that are executed on the NeuronCore. \nThese are ``nc_exec_running``, ``cc_running``, ``cc_exec_barrier``, ``numerical_err``, ``nrt_model_switch``, ``timestamp_sync_point``, ``hw_notify``.\nThe ``software`` group contains all other events.\n\nUsing API Functions\n^^^^^^^^^^^^^^^^^^^\n\nUse the ``nrt_sys_trace_config_set_capture_enabled_for_event_type`` API to filter by event type.\n\n.. code-block:: c\n\n #include \n\n // Configure trace options\n nrt_sys_trace_config_t *config;\n nrt_sys_trace_config_allocate(&config);\n nrt_sys_trace_config_set_defaults(config); // By default, all event types are enabled\n\n // Disable specific event types (others remain enabled)\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, \"device_exec\", false);\n\n // Or disable all first, then enable only specific ones\n const char **all_event_types = nullptr;\n size_t all_count = 0;\n nrt_sys_trace_get_event_types(&all_event_types, &all_count);\n\n // Disable all event types first\n for (size_t i = 0; i < all_count; ++i) {\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, all_event_types[i], false);\n }\n\n // Enable only specific event types\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, \"model_load\", true);\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, \"nrt_execute\", true);\n\n // Verify which event types are enabled\n const char **enabled_types = nullptr;\n size_t enabled_count = 0;\n nrt_sys_trace_config_get_enabled_event_types(config, &enabled_types, &enabled_count);\n printf(\"Enabled event types: %zu\\n\", enabled_count);\n for (size_t i = 0; i < enabled_count; ++i) {\n printf(\" %s\\n\", enabled_types[i]);\n }\n\n // Clean up memory (caller is responsible)\n for (size_t i = 0; i < enabled_count; ++i) {\n free((void*)enabled_types[i]);\n }\n free((void*)enabled_types);\n\n for (size_t i = 0; i < all_count; ++i) {\n free((void*)all_event_types[i]);\n }\n free((void*)all_event_types);\n\n // Start tracing\n nrt_sys_trace_start(config);\n\n // Your application code here...\n\n // Cleanup\n nrt_sys_trace_stop();\n nrt_sys_trace_config_free(config);\n\n\nProcessing-time Filtering\n--------------------------\n\n**Processing-time filtering** preserves the complete trace and allows flexible analysis with different filters, but requires more memory and storage during capture.\nApply filters when viewing or processing already captured profiles. This approach allows you to \nanalyze the same trace data in different ways without recapturing. The filters can be used for any \n``neuron-explorer`` output format including ``--output-format json`` and ``--output-format perfetto``.\n\nNeuronCore\n~~~~~~~~~~\n\nUse the ``--system-trace-filter-neuron-core`` to only process events for specific NeuronCores. The IDs are local to the instance and not global IDs. \n\nIf the ``--system-trace-filter-neuron-core`` argument is not set then events from all NeuronCores will be included in the processed trace.\n\n\n**Single neuron core**\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --system-trace-filter-neuron-core \"0\"\n\n**Multiple neuron cores**\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --system-trace-filter-neuron-core \"0,1,2,3\"\n\nEvent Type\n~~~~~~~~~~\nUse the ``--system-trace-filter-event-type`` to only process specific trace events types.\n\nIf the ``--system-trace-filter-event-type`` argument is not set then all event types will be included in the processed trace.\n\n**Single event type**\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --system-trace-filter-event-type \"nrt_execute\"\n\n**Multiple event type**\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --system-trace-filter-event-type \"nrt_execute,nrt_load\"\n\nInstance ID\n~~~~~~~~~~~\n\nUse the ``--system-trace-filter-instance-id`` to only process events for specific ec2 instances.\n\nIf the ``--system-trace-filter-instance-id`` argument is not set then events from all instances will be included in the processed trace.\n\n**Single instance**\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --system-trace-filter-instance-id \"i-abc123\"\n\n**Multiple instances**\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --system-trace-filter-instance-id \"i-abc123,i-def456,i-ghi789\"\n\nProcessing only system or device profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYou can reduce processing times by skipping the processing of system or device profiles. Choose this when you are interested in only a specific profile, or when you want to start with a limited set of profiling data before exploring the full profile.\n\nTo skip processing of device profiles use the ``--ignore-device-profile`` option. To skip processing of system profiles use the ``--ignore-system-profile`` option. These options can be used with the ``--output-format`` values ``parquet`` (default), ``perfetto``, or ``json``.\n\nFor example:\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --ignore-device-profile --output-format perfetto\n\n\nView Profiles\n-------------\n\nTo view a profile in Neuron Explorer, follow these steps:\n\n1. **Start the Neuron Explorer UI and API servers** using the ``neuron-explorer`` tool from ``aws-neuronx-tools``:\n\n .. code-block:: bash\n\n neuron-explorer view --data-path /absolute/path/to/db\n\n By default, the UI will be launched on port 3001 and the API server will be launched on port 3002.\n\n2. **Set up port-forwarding** (if running on a remote EC2 instance) to enable local viewing:\n\n .. code-block:: bash\n\n ssh -i @ -L 3001:localhost:3001 -L 3002:localhost:3002\n\n note::\n it is necessary to forward both 3001 (for the UI server) and 3002 (for the data server)\n\n3. **Open the UI** by navigating to ``localhost:3001`` in your browser.\n\n4. **Upload your profile** by clicking the **\"Upload Profile\"** button in the Profile Manager page. You can either:\n\n * Upload the NEFF (``.neff``) and NTFF (``.ntff``) files individually using the \"Individual Files\" upload mode, or\n * Upload the folder containing the NEFF and NTFF files using the \"Directory Upload\" mode.\n\nNeuron Explorer Browser UI\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. _neuron-explorer-profile-manager:\n\nProfile Manager\n^^^^^^^^^^^^^^^\n\nProfile Manager is a page for uploading artifact (NEFF, NTFF and source code) and selecting profiles to access.\n\n.. image:: /tools/profiler/images/profile-workload-3.png\n\n.. _profile-manager-upload-profile:\n\n\n\nClick on the \"Upload Profile\" button to open the Upload Profile modal.\n\n\n**Device Profile Upload**\n\nSelect \"Individual Files\" upload mode to upload NEFF, NTFF, and source code individually.\n\nSelect \"Directory Upload\" to upload profile files from a directory.\n\n.. note::\n * \"Profile name\" is a required field. You cannot upload a profile with existing name unless the option \"Force Upload\" is checked at the bottom. Force Upload currently will overwrite the existing profile with the same name.\n * For uploading source code, the UI only supports the upload of folders, individual files, or compressed files in the gzipped tar ``.tar.gz`` archive format.\n\n.. image:: /tools/neuron-explorer/images/device-profile-upload-ui.png\n\n\n.. _profile-manager-system-profile-upload:\n\n**System Profile Upload**\n\nSelect \"Directory Upload\", then in the Profile Directory drag and drop area, select the directory containing the system profile files.\n\nThe directory should contain instance sub-directories with the following: ``ntrace.pb``, ``trace_info.pb``, ``cpu_util.pb``, and ``host_mem.pb``.\nFor an example see the output in :ref:`neuron-explorer inspect `\n\n.. note::\n System Profile uploads only support \"Directory Upload\".\n\n.. image:: /tools/neuron-explorer/images/system-profile-upload-ui.png\n\n\n**Processing Status**\n\nAfter uploading a profile, the processing task is shown under \"User Uploaded\" table. Use the \"Refresh\" button in the top-right to fetch the latest processing status and verify completion.\n\n\n**Listing profiles**\n\nAll uploaded profiles are provided in the Profile Manager page with details such as the processing status and upload time, along with various quick access actions.\n\n.. image:: /tools/profiler/images/profile-workload-5.png\n\n* **Pencil button**: Rename a profile.\n* **Star button**: Mark this profile as favorite profile. This profile will be shown in the User's favorites list.\n* **Bulb button**: Navigate to the summary page of this profile. For more details on the summary page, see :doc:`this overview of the Neuron Explorer Summary Page `.\n\nClicking on the name of profile takes you to its corresponding profile page.\n\nNeuron Explorer for Visual Studio Code\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe UI is also available as a VSCode extension, enabling better native integration for features such as code linking.\n\nInstall the Neuron Explorer extension from the Visual Studio Code Marketplace. Open the Extensions view in VSCode by pressing **Ctrl+Shift+X** (Windows/Linux) or **CMD+Shift+X** (MacOS), and search for ``AWS Neuron Explorer`` or ``amazonwebservices.neuron-explorer``. Select the extension published by **Amazon Web Services** in the sidebar, then click the blue **Install** button.\n\n.. image:: /tools/profiler/images/profile-workload-1.png\n\nEnsure the SSH tunnel is established by following the steps above. Otherwise, specify a custom endpoint by selecting the extension in the left activity bar. Then, navigate to the \"Endpoint\" action on the bottom bar of your VSCode session and select \"Custom endpoint\", and enter ``localhost:3002``. \n\n.. image:: /tools/profiler/images/profile-workload-2.png\n\nFrom there, navigate to the **Profile Manager** page through the extension UI in the left activity bar.\n\nJSON Output\n~~~~~~~~~~~\n\nThe ``--output-format`` json option writes processed profile data to human-readable JSON that can be used for scripting and manual inspection.\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --output-format json\n\nThis will generate a ``system_profile.json`` file containing the system profile data and a ``device_profile_model_.json`` file for each unique compiled model that was executed on a Neuron Device. \n\nThe system_profile.json JSON contains the following data types:\n\n* ``trace_events``: Neuron Runtime API trace events and Framework/Application trace events containing timestamps, durations, names, and the ec2 instance-id to differentiate between events from different compute nodes in a distributed workload.\n\n.. code-block:: json\n\n {\n \"Neuron_Runtime_API_Event\": {\n \"duration\": 27094,\n \"group\": \"nrt-nc-000\",\n \"id\": 1,\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"lnc_idx\": \"0\",\n \"name\": \"nrt_tensor_write\",\n \"parent_id\": 0,\n \"process_id\": \"1627711\",\n \"size\": \"4\",\n \"tensor_id\": \"4900392441224765051\",\n \"tensor_name\": \"_unknown_\",\n \"thread_id\": 1627711,\n \"timestamp\": 1729888371056597613,\n \"type\": 11\n },\n \"Framework_Event\": {\n \"duration\": 3758079,\n \"group\": \"framework-80375131\",\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"name\": \"PjitFunction(matmul_allgather)\",\n \"process_id\": \"701\",\n \"thread_id\": 80375131,\n \"timestamp\": 1729888382798557372,\n \"type\": 99999\n }\n }\n\n* ``mem_usage``: sampled host memory usage \n\n.. code-block:: json\n\n {\n \"duration\": 1,\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"percent_usage\": 9.728179797845964,\n \"timestamp\": 1729888369286687792,\n \"usage\": 51805806592\n }\n\n* ``cpu_util``: sampled CPU utilization. Results are provided per core and per ec2 instance involved in a distributed workload\n\n.. code-block:: json\n\n {\n \"cpu_id\": \"47\",\n \"duration\": 1,\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"timestamp\": 1729888371287337243,\n \"util\": 2.3255813\n },\n\n\nView in Perfetto\n~~~~~~~~~~~~~~~~\n\nUsers can view their Neuron Explorer profiles in Perfetto. Please see :doc:`view-perfetto` for more information.\n\n.. note::\n New Neuron Explorer features released in 2.27 and onwards may not be supported in Perfetto. For the full user experience and features set, please use the Neuron Explorer UI or VSCode Integration.\n\n\nTroubleshooting\n---------------\n\nIncomplete JAX Profiles\n~~~~~~~~~~~~~~~~~~~~~~~\n\nIf your JAX profile has fewer events than expected or lacks the Runtime API trace, check whether \n``jax.profiler.stop_trace`` is being called inside a ``with jax.profiler.trace`` context block. \nThis can prematurely stop tracing. Use ``jax.profiler.stop_trace`` only when profiling was started \nwith ``jax.profiler.start_trace``, not when using the context-managed ``with jax.profiler.trace`` API.\n\nAlso when using ``jax.profiler`` within your script ensure that the \nenvironment variable ``NEURON_RT_INSPECT_ENABLE`` is not set to 1. \nAdditionally, ensure that ``NEURON_RT_INSPECT_OUTPUT_DIR`` is set to \nthe correct output directory and this is the output directory passed to \n``with jax.profiler.trace``.\n\nDropped Events in System Profile\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen processing a system profile, you may see a warning indicating that some trace events were dropped during capture.\n\n.. code-block:: shell\n\n WARN[0000] Warning: 1001 trace events were dropped during capture (stored 530560 out of 531561 total events). Consider increasing buffer size, reducing trace duration, or filtering events.\n\nThis means during capture the trace event buffers filled and oldest events were overwritten. If you need to avoid dropping events for the full duration of your workload consider the following adjustments:\n\n* Increase buffer size by setting ``NEURON_RT_INSPECT_SYS_TRACE_MAX_EVENTS_PER_NC`` (see :ref:`Profile Capture Environment Variables `). This will increase host memory usage.\n* Apply capture-time filters (NeuronCores / event types) (see :ref:`Filtering System Profiles `.)\n* Shorten profiled region: limit the code span under the profiling context / runtime.\n" }, { "path": "tools/neuron-explorer/index.rst", "content": ".. meta::\n :description: Neuron Explorer documentation for performance profiling, debugging, and optimization of ML workloads on AWS Trainium and Inferentia.\n :date-modified: 12/02/2025\n\n.. _neuron-explorer-home:\n\nNeuron Explorer\n=================\n\n.. important::\n\n Neuron Explorer is the recommended profiling tool for AWS Neuron workloads. It provides end-to-end profiling support along with the latest features and an improved user experience. \n \n **Note:** Neuron will end support for :ref:`Neuron Profiler 2.0 ` and :ref:`Neuron Profiler ` in Neuron 2.29 release. Users are encouraged to migrate to Neuron Explorer. Please see :doc:`migration-faq` and :ref:`neuron-explorer-faq` for more details.\n \nNeuron Explorer is a suite of tools designed to support ML engineers throughout their development journey on AWS Trainium. Neuron Explorer helps developers maintain context, iterate efficiently, and focus on building and optimizing high-performance models. Developers can access Neuron Explorer from CLI, UI, or inside their IDE through VSCode integration.\n\nProfiling Viewers\n--------------------\n\nNeuron Explorer enables ML performance engineers to trace execution from source code down to hardware operations, enabling detailed analysis of model behavior at every layer of the stack. The suite of tools supports both single-node and distributed applications, allowing developers to analyze workloads at scale. \n\nGetting Started\n---------------\n\n.. grid:: 1 2 2 2\n :gutter: 3\n\n .. grid-item-card:: Get Started\n :link: get-started\n :link-type: doc\n :class-card: sd-border-1\n\n Set up Neuron Explorer, launch the web UI, and configure SSH tunneling for secure access to profiling data.\n\n .. grid-item-card:: Capture and View Profiles\n :link: how-to-profile-workload\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to capture and view profiles in the Neuron Explorer UI or directly in your IDE via VSCode Integration.\n\nVisualization and Analysis\n---------------------------\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: Device Trace Viewer\n :link: overview-device-profiles\n :link-type: doc\n :class-card: sd-border-1\n\n Explore hardware-level execution with timeline view, operator table, event details, annotations, dependency highlighting, search, and more analysis features.\n\n .. grid-item-card:: System Trace Viewer\n :link: overview-system-profiles\n :link-type: doc\n :class-card: sd-border-1\n\n Explore system-level execution with timeline view and more analysis features.\n\n\n.. grid:: 1 2 2 2\n :gutter: 3\n\n .. grid-item-card:: Hierarchy Viewer\n :link: overview-hierarchy-view\n :link-type: doc\n :class-card: sd-border-1\n\n Visualize the entire execution from model layers down to hardware execution, supporting interactivity with device viewer and source code linking.\n\n .. grid-item-card:: Source Code Viewer\n :link: how-to-link-view-source-code\n :link-type: doc\n :class-card: sd-border-1\n\n Navigate between NKI and PyTorch source code and profile data with bidirectional linking and highlighting.\n\n .. grid-item-card:: Summary Viewer\n :link: overview-summary-page\n :link-type: doc\n :class-card: sd-border-1\n\n Get streamlined performance insights and optimization recommendations with high-level metrics and visualizations.\n\n .. grid-item-card:: Database Viewer\n :link: overview-database-viewer\n :link-type: doc\n :class-card: sd-border-1\n\n Develop your own analyses, examine profiling data stored in database tables, or run ad-hoc queries during performance analysis. \n\n .. grid-item-card:: Tensor Viewer\n :link: overview-tensor-viewer\n :link-type: doc\n :class-card: sd-border-1\n\n Viewing tensor information including names, sizes, shapes, and memory usage details.\n\n .. grid-item-card:: Memory Viewer\n :link: overview-memory-viewer\n :link-type: doc\n :class-card: sd-border-1\n\n Analyze memory allocation, usage patterns, and potential inefficiencies across SBUF partitions.\n\n .. grid-item-card:: AI Recommendation Viewer\n :link: overview-ai-recommendations\n :link-type: doc\n :class-card: sd-border-1\n\n Get AI powered bottleneck analysis and optmization recommendations for NKI profiles.\n\nTutorials\n----------\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: Profile a NKI Kernel\n :link: /nki/guides/use-neuron-profile\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to profile a NKI kernel with Neuron Explorer.\n\n.. grid:: 1 2 2 2\n :gutter: 3\n\n .. grid-item-card:: vLLM Performance\n :link: /tools/tutorials/performance-profiling-vllm\n :link-type: doc\n :class-card: sd-border-1\n\n Capture and analyze system-level and device-level profiles for vLLM inference workloads on Trainium.\n\n\nAdditional Resources\n--------------------\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: Viewing Profiles with Perfetto\n :link: view-perfetto\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to view Neuron Explorer profiles using the Perfetto UI for trace analysis.\n\n.. _download-neuron-explorer-vscode:\n\nNeuron Explorer for Visual Studio Code\n------------------------------------------------\n\nThe Neuron Explorer VSCode extension is available on the Visual Studio Code Extension Marketplace.\n\nTo install the extension, open the Extensions view in VSCode by pressing **Ctrl+Shift+X** (Windows/Linux) or **CMD+Shift+X** (MacOS), and search for ``AWS Neuron Explorer`` or ``amazonwebservices.neuron-explorer``. Select the extension published by **Amazon Web Services** in the sidebar, then click the blue **Install** button.\n\nYou can also install the extension directly from the `Visual Studio Code Marketplace `_.\n\n.. _neuron-explorer-faq:\n\nNeuron Explorer FAQ\n-------------------\n\nWhat can I expect from the Neuron Explorer?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNeuron Explorer provides a comprehensive profiling experience with both device-level and system-level profiling support. Neuron Explorer features an enhanced profiling experience with hierarchical profiling, bidirectional code linking, AI-powered recommendations, IDE integration, and more. In future releases, Neuron Explorer will continue to expand with additional profiling viewers and features, debugging capabilities, and enhanced recommendation and analysis tools to support the entire ML development journey on Trainium.\n\nWhat is the difference between device-level and system-level profiling?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDevice-level profiling captures hardware execution data from NeuronCores, including compute engine instructions, DMA operations, and hardware utilization. Use device-level profiling to analyze hardware performance, identify compute or memory bottlenecks, and optimize kernel implementations.\n\nSystem-level profiling captures software execution data, including framework operations, Neuron Runtime API calls, CPU utilization, and memory usage. Use system-level profiling to analyze framework overhead, identify CPU bottlenecks, and debug runtime issues.\n\nIs Neuron Explorer going to replace Neuron Profiler and Neuron Profiler 2.0?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nYes. Neuron Explorer is the recommended profiling tool and replaces both Neuron Profiler and Profiler 2.0.\n\nNeuron Profiler and Profiler 2.0 are supported for one final release. In Neuron 2.29 release, they will enter end-of-support and will no longer receive updates or technical support, though they will remain accessible through the ``neuron-profile`` package in previous releases. Users should migrate to Neuron Explorer now.\n\nAre my existing profiles compatible with Neuron Explorer?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nYes. Neuron Explorer is backwards compatible with profile data captured using Neuron Profiler or Profiler 2.0. Existing profile files must be reprocessed before viewing in Neuron Explorer, but you do not need to recapture them. See :ref:`new-neuron-profiler-setup`.\n\nFor detailed migration guidance, including CLI command mappings and feature comparisons, see the :doc:`migration-faq`.\n\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n\n Get Started \n Neuron Profiler to Neuron Explorer Migration Guide \n Capture and View Profiles \n Device Trace Viewer \n System Trace Viewer \n Hierarchy Viewer \n Source Code Viewer \n Summary Viewer \n Database Viewer \n Tensor Viewer \n Memory Viewer \n AI Recommendation Viewer \n View Profiles with Perfetto \n \n" }, { "path": "tools/neuron-explorer/migration-faq.rst", "content": ".. _neuron-profiler-migration-guide:\n\nMigration Guide from Neuron Profiler to Neuron Explorer\n========================================================\n\nThis guide provides detailed information for migrating from Neuron Profiler or Neuron Profiler 2.0 to Neuron Explorer.\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nOverview\n--------\n\nNeuron Explorer is the recommended profiling tool for AWS Neuron workloads, replacing both Neuron Profiler and Neuron Profiler 2.0. This guide helps you transition your profiling workflows to Neuron Explorer.\n\nKey Differences\n---------------\n\nThe following table summarizes the key differences between Neuron Profiler/Profiler 2.0 and Neuron Explorer:\n\n.. list-table::\n :widths: 30 35 35\n :header-rows: 1\n :align: left\n\n * - Feature\n - Neuron Profiler / Profiler 2.0\n - Neuron Explorer\n * - CLI tool\n - ``neuron-profile``\n - ``neuron-explorer``\n * - Device Profiling\n - Yes\n - Yes (enhanced)\n * - System Profiling\n - Yes (Profiler 2.0 only)\n - Yes\n * - Hierarchy Viewer\n - No\n - Yes\n * - Source Code Viewer\n - Yes (Device profiles)\n - Yes (Device profiles)\n * - AI Recommendation Viewer\n - No\n - Yes (for NKI profiles)\n * - IDE Integration\n - No\n - Yes (VSCode Extension)\n * - Database Viewer\n - No\n - Yes\n * - Tensor Viewer\n - No\n - Yes\n * - Additional Installation Requirements\n - InfluxDB installation required\n - None\n\n\nUpdate CLI Commands\n--------------------\n\nReplace ``neuron-profile`` with ``neuron-explorer`` in your scripts and workflows. The following commands are subject to change before GA:\n\n.. list-table::\n :widths: 50 50\n :header-rows: 1\n :align: left\n\n * - Neuron Profiler Command\n - Neuron Explorer Command\n * - ``neuron-profile view -d ./output``\n - ``neuron-explorer view -d ./output``\n * - ``neuron-profile view -n file.neff -s profile.ntff``\n - ``neuron-explorer view -n file.neff -s profile.ntff``\n * - ``neuron-profile capture -n file.neff -s profile.ntff``\n - ``neuron-explorer capture -n file.neff -s profile.ntff``\n\n\nFrequently Asked Questions\n--------------------------\n\nDo I need to install InfluxDB for Neuron Explorer?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNo. Unlike Neuron Profiler, Neuron Explorer requires no external installation or setup.\n\nHow do I view existing profiles captured with Neuron Profiler?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nExisting NEFF and NTFF files captured with Neuron Profiler are fully compatible with Neuron Explorer. To view them:\n\n.. code-block:: bash\n\n # View a single device profile\n neuron-explorer view -n file.neff -s profile.ntff\n\nThe profiles will be reprocessed using Neuron Explorer's processing pipeline, which may provide additional insights not available in the original Neuron Profiler view.\n\nHow do I capture profiles with Neuron Explorer?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNeuron Explorer provides the ``neuron-explorer capture`` command for standalone NEFF profiling, similar to ``neuron-profile capture``:\n\n.. code-block:: bash\n\n # Capture a device profile\n neuron-explorer capture -n file.neff -s profile.ntff\n\nYou can also use the framework profiling APIs or environment variables to capture profiles during your actual workload execution. For NKI kernel profiling, continue using the ``nki.benchmark`` or ``nki.profile`` APIs as documented in the :ref:`NKI profiling guide `.\n\nWhat new features does Neuron Explorer provide?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNeuron Explorer introduces several new capabilities:\n\n- **Hierarchy Viewer**: Visualize execution from model layers down to hardware operations. See :doc:`overview-hierarchy-view`.\n- **Source Code Viewer**: Navigate between source code and profile data. See :doc:`how-to-link-view-source-code`.\n- **AI Recommendation Viewer**: Get AI-powered optimization suggestions for NKI profiles. See :doc:`overview-ai-recommendations`.\n- **Database Viewer**: Run custom queries on profiling data. See :doc:`overview-database-viewer`.\n- **Memory Viewer**: Get insight into memory allocation, usage patterns, and potential memory usage inefficiencies.\n- **Tensor Viewer**: Examine tensor information including shapes and memory usage. See :doc:`overview-tensor-viewer`.\n- **VSCode Extension**: View profiles directly in your IDE with native code linking support.\n- **System Trace Viewer**: Enhanced system-level profiling visualization. See :doc:`overview-system-profiles`.\n\nHow do I get help during migration?\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n- Review the :doc:`get-started` guide for initial setup\n- See :doc:`how-to-profile-workload` for detailed capture and viewing instructions\n- Check submitted issues and file new issues via the `AWS Neuron GitHub issues `_\n" }, { "path": "tools/neuron-explorer/overview-ai-recommendations.rst", "content": ".. meta::\n :description: AI Recommendation feature helps identify and understand bottlenecks and optimization opportunities for NKI kernels through AI-powered analysis\n :date-modified: 11/21/2025\n\nAI Recommendation Viewer\n=========================\n\nIn this guide, you'll learn how to use the AI Recommendation Viewer to identify and understand bottlenecks and optimization opportunities for NKI kernels through AI-powered analysis of the user's profile and source code. Users receive actionable recommendations through the Neuron Explorer UI, CLI, or via their IDE. Each report provides the top 2-3 optimization opportunities ranked by effort and impact, including the symptom with quantified metrics, the optimization with implementation guidance, expected speedup estimates, and implementation tradeoffs. \n\nThe feature is entirely opt-in and only enabled for profiles that the user explicitly requests a recommendation for.\n\n.. warning:: \n * Responses in this AWS Bedrock-powered feature are AI-generated. Verify accuracy and appropriateness before use. \n * This feature is available in US Regions only. Neuron may securely transmit data across Regions within your geography for processing. \n * Your AWS account will be billed for Bedrock usage. Each time you generate an AI Recommendation for a profile, a single Bedrock request is made with up to 30,000 input tokens and 10,000 output tokens. \n * At the moment, this feature may only be used with Claude Sonnet 4.5.\n\n.. _local_setup_directions:\n\nLocal setup directions\n----------------------------------------------------\n\nAI Recommendations use Amazon Bedrock. To enable this feature, you must configure AWS credentials on the system you are running neuron-explorer on. The AWS credentials should have bedrock:InvokeModel permissions and access to Claude Sonnet 4.5. For information on configuring Bedrock access, refer to the `AWS Bedrock model access documentation `_.\n\nGetting an AI Recommendation From the UI\n----------------------------------------------------\n\nTo generate an AI Recommendation from the UI open your profile, click the \"Add Widget\" dropdown, and select **AI Recommendation**.\n\n.. image:: /tools/profiler/images/recommendation-button.png\n\nGo to the **AI Recommendation** widget box and click the **Get AI Recommendation** button. This will perform additional analysis and send the recommendation request to AWS Bedrock and can take up to a minute to generate. Avoid refreshing the page during this time.\n\n.. image:: /tools/profiler/images/recommendation-widget.png\n\nOnce the recommendation has been generated it will be displayed in the widget box. For each recommendation you will see the performance inefficiency symptoms that were observed, the suggested optimization to make, and potential tradeoffs to look out for when implementing the optimizations.\n\n.. image:: /tools/profiler/images/recommendation-view.png\n\nGetting an AI Recommendation from the CLI\n----------------------------------------------------\n\nUsers may also get AI recommendations with the ``neuron-explorer recommend`` CLI command. \n\nBefore you start, ensure that you have followed the :ref:`local setup directions ` to enable Bedrock access on your configured AWS account. ``neuron-explorer`` uses the default AWS credentials you have configured. If you will use other credentials, you can specify an AWS profile to use by setting environment variables: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html.\n\nTo generate a recommendation, provide the following to the ``neuron-explorer recommend`` command:\n\n* A NEFF file for your compiled NKI kernel\n* An NTFF file for your captured profile\n* The location where your NKI source files can be found\n\nExample:\n\n.. code-block::\n\n neuron-explorer recommend -n -s --nki-source-root \n\nRunning this command processes the profile and prints the AI-generated recommendation to the console in Markdown format. You can save this output to a file and view it in any text editor or Markdown viewer.\n" }, { "path": "tools/neuron-explorer/overview-database-viewer.rst", "content": ".. meta::\n :description: Learn about the Database Viewer tool in Neuron Explorer for querying and exploring profiling data using SQL or natural language queries.\n :date-modified: 01/27/2026\n\n.. _database-viewer-overview:\n\nDatabase Viewer\n=====================\n\nThe Database Viewer offers an interactive interface providing visibility to all underlying data that the Neuron Explorer\nprocesses from a :doc:`NEFF ` and NTFF. \nUse this tool to develop your own analyses, examine profiling data stored in database tables, or run ad-hoc queries during performance analysis. \nYou can access this data through natural language queries or raw SQL.\n\n\n.. image:: /tools/profiler/images/database-viewer.png\n\nTable Selection and Schema Inspection\n-------------------------------------\n\nWhen the tool loads, it fetches the list of available database tables. Select a table from the dropdown to view its schema.\n\nThe schema table displays:\n\n* **Field Name** - Column name (hover for description tooltip).\n* **Data Type** - The data type of the field.\n* **Required** - Whether the field is required.\n* **Unit** - Measurement unit (if applicable).\n* **Example** - Example value for the field.\n\nQuerying Data\n-------------\n\nThe query input supports two modes:\n\n1. **SQL queries** - Write standard SQL starting with ``SELECT``.\n2. **Natural language queries** - Describe what you want in plain English.\n\nExamples:\n\nNatural language query to get the first 5 rows::\n\n Get the first 5 rows\n\nSQL query to filter with conditions::\n\n SELECT field_name FROM table_name WHERE condition\n\nPress **Enter** or click **Execute Query** to run. Use **Shift+Enter** for multi-line input.\n\nQuery Results\n-------------\n\nResults appear below the query input in reverse chronological order (newest first). Each result shows:\n\n* The original query text.\n* The generated SQL (for natural language queries).\n* A scrollable results table.\n\nClick **Export CSV** to download any result set as a CSV file.\n\n.. image:: /tools/profiler/images/database-viewer-query-result.png" }, { "path": "tools/neuron-explorer/overview-device-profiles.rst", "content": ".. meta::\n :description: Learn about Neuron Explorer widgets for device profiling including timeline views, event details, annotations, and performance analysis tools.\n :date-modified: 12/02/2025\n\nDevice Trace Viewer\n===================\n\nThe Neuron Device Trace Viewer displays a hardware instruction level granularity of execution on a NeuronCore. Neuron Explorer collects the timestamped start and end events that occur on the device into a NTFF. As a post-processing step, the profiler will correlate these events with information in the compiled NEFF to generate a detailed report of the hardware performance. The Neuron Explorer UI provides several different tools for an extensible and customizable workflow.\n\n.. image:: /tools/profiler/images/device-profile-1.png\n\nTools\n------\n\nDevice Trace Viewer\n~~~~~~~~~~~~~~~~~~~~~\n\nThe Device Trace Viewer presents a timeline view of the device execution, including activity on the DMA and compute engines, Hardware FLOPs Utilization (HFU) and device memory utilization over time, and more.\n\n.. image:: /tools/profiler/images/device-profile-2.png\n\nHover\n^^^^^\n\n.. image:: /tools/profiler/images/device-profile-3.png\n\nHover on events in the timeline to see important identifying information at a glance, such as the time window, the hierarchy, and the hardware instruction that was executed.\n\nFor more details, clicking the event will display the full details in the Event Details widget.\n\nColor Scheme\n^^^^^^^^^^^^\n\n.. list-table::\n :header-rows: 0\n :widths: 50 50\n\n * - .. image:: /tools/profiler/images/device-profile-4.png\n :width: 100%\n - .. image:: /tools/profiler/images/device-profile-5.png\n :width: 100%\n\nInstructions are color-coded according to their associated PyTorch operator. All instructions derived from the same PyTorch operator share an identical color.\n\n.. note::\n In future releases, we will introduce more customizable options for color-coding.\n\nPanning\n^^^^^^^\n\n.. image:: /tools/profiler/images/device-profile-6.gif\n\nPanning is supported in a couple of ways:\n\n* Left-clicking the x-axis and dragging it\n* Spinning scroll-wheel while holding down shift\n* With the keyboard:\n * A/D keys for left/right movement\n * Left/right arrow keys for left/right movement\n\nThe amount panned depends on the current zoom level.\n\nEvent Details\n~~~~~~~~~~~~~\n\nUpon clicking an event in the Device Trace Viewer, all details related to the event will appear in the Event Details. The information shown will be a superset of the information available on hover, allowing us to dive deeper into what is happening on the hardware.\n\n* The Event Details table will populate with field data from clicked events from the instruction widget.\n* When filtering by fields through Search, all matching events will be rendered as pages in the Event Details. Users can navigate through each page to analyze data for each matching event.\n \n.. image:: /tools/profiler/images/device-profile-7.png\n\nAnnotations\n~~~~~~~~~~~\n\nUsers can create annotations by right-clicking in the Device Trace Viewer. These annotations can be moved by clicking and dragging the vertical line, and will snap to the closest events when applicable.\n\nThe annotations tab will show more details on all available annotations in the profile, such as the time difference and summary metrics that occur between two markers. The option of which two annotations to compare is configurable in the diff vs column. You can also quickly zoom in to the region between two annotations by selecting the checkbox on the left. Users can rename, delete, save, and load annotations for better readability and collaboration.\n\n.. image:: /tools/profiler/images/device-profile-8.png\n\nOperator Table\n~~~~~~~~~~~~~~\n\nThe Operator Table aggregates the hardware level metrics into framework layers and operations, such as the MFU and amount of data being moved. Users can progressively expand each row to get a further breakdown of each nested operator.\n\nFilters can be applied and columns can be sorted for more streamlined viewing.\n\n.. image:: /tools/profiler/images/device-profile-9.png\n\nOverall Summary\n~~~~~~~~~~~~~~~\n\nThe Overall Summary displays performance metrics across the entire profile run, with metrics broken down into different categories such as by the NeuronCore engines. These can be used for quick insights into how well the model performed.\n\n.. image:: /tools/profiler/images/device-profile-10.png\n\nCurrent Selection Summary\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe Current Selection Summary provides metrics for the current time window. Zooming in and out in the Device Trace Viewer will update the summary. This can be used in conjunction with the zoom feature of Annotations for easy access to a region of interest.\n\n.. image:: /tools/profiler/images/device-profile-11.png\n\n.. _box-selection-summary:\n\nBox Selection Summary\n~~~~~~~~~~~~~~~~~~~~~\n\nThe Box Selection Summary provides metrics within a bounding box region. Select and drag regions within the timeline widget to update the summary.\n\n.. image:: /tools/profiler/images/box-select.gif\n\nBox selection is supported in a couple of ways:\n\n* Toggling the box selection button within the timeline widget\n* Clear selection with `esc` key\n\nCorreponding summary information of the selected region is displayed within the box selection selection widget.\n\nCode Viewer\n~~~~~~~~~~~\n\nProfiles that are uploaded with source code files enable users to quickly navigate between NKI and application level source code and the corresponding hardware level instructions.\n\nIn the Device Trace Viewer, we can click on an event to highlight the source code line in the Code Viewer. A (Ctrl/Cmd) + click on the event will scroll to the corresponding source code line.\n\nIn the Code Viewer, clicking on a line in the source code will automatically highlight all associated events in the Device Trace Viewer. Similarly, highlighting multiple lines of the source code will also highlight all events in the timeline.\n\n.. image:: /tools/profiler/images/device-profile-12.png\n\nSee :ref:`neuron-explorer-source-code` for instructions on how to enable source code viewing.\n\nLayout Customization\n~~~~~~~~~~~~~~~~~~~~\n\nUnderstanding and optimizing performance with the profiler can be overwhelming given the amount of information being processed and displayed. As part of preparing for optimization work, you can cross-reference different information, such as the Device Trace Viewer with the application source code. With the widget-based UI, you can customize the layout to best fit a specific workflow. Each widget can be added, removed, dragged around, and resized. Once you are happy with the layout, you can save it through the Layout dropdown at the top right. The layouts are not tied to a specific profile, so they can be loaded and re-used for future profiles as well.\n\n.. image:: /tools/profiler/images/device-profile-13.png\n" }, { "path": "tools/neuron-explorer/overview-hierarchy-view.rst", "content": ".. meta::\n :description: Learn about the Hierarchy View in Neuron Explorer for analyzing framework layers and HLO operations with zooming, highlighting, and display options.\n :date-modified: 12/02/2025\n\nHierarchy Viewer\n===================\n\nThe Hierarchy Viewer shows an up-leveled representation of the hardware execution organized by the framework layers and HLO operations. It enables you to progressively drill down into nested layers or operators and map the execution of application level constructs to the Neuron device. This view interacts with other tools such as the Device Trace Viewer.\n\n.. image:: /tools/profiler/images/hierarchy-view-1.gif\n\n\nZooming\n-------\n\n.. image:: /tools/profiler/images/hierarchy-view-2.png\n\nYou can zoom in on the Hierarchy Viewer in a couple of ways:\n\n* Click-drag your mouse across the graph (support in both directions)\n* Scroll down using your mouse wheel, with the mouse cursor on the x-axis\n* Zoom in and out buttons in the top-right corner\n* With the keyboard:\n \n * W and S for zooming in and out, respectively\n * Up and down arrow keys for zooming in and out, respectively\n\nTo zoom out, simply scroll up with your mouse wheel when you place your mouse cursor on the x-axis.\n\nChange Displayed Layers\n-----------------------\n\n.. image:: /tools/profiler/images/hierarchy-view-3.png\n\nThe display options menu, accessed with the button in the top-right corner, allows you to selectively show or hide different layers. For instance, in the example shown above, the framework layer is hidden while displaying the hierarchy starting from HLO.\n\nHighlighting\n------------\n\n.. image:: /tools/profiler/images/hierarchy-view-4.png\n\nRight-clicking on an operator in Hierarchy Viewer will highlight all the corresponding instructions in the Device Trace Viewer for the operator using the same color. Multiple operators can be highlighted at once.\n\n.. image:: /tools/profiler/images/hierarchy-view-5.png\n\n" }, { "path": "tools/neuron-explorer/overview-memory-viewer.rst", "content": ".. meta::\n :description: Learn about the Memory View in Neuron Explorer for analyzing all the memory allocations on SBUF.\n :date-modified: 03/24/2026\n\nMemory Viewer\n===================\n\nThe Memory Viewer in Neuron Explorer offers deep, low-level insight into memory allocation, usage patterns, and potential inefficiencies — going well beyond surface-level metrics. With comprehensive visibility into how memory is consumed across the device, it enables kernel and performance engineers to make informed optimization decisions, reduce debugging time, and improve overall system performance.\n\n.. image:: /tools/neuron-explorer/images/memory_viewer_overview.png\n :alt: Memory Viewer overview showing memory allocation patterns across SBUF partitions\n\n\nEnable Memory Viewer during Profile Upload\n--------------------------------------------\n\nTo enable the Memory Viewer feature, check the option 'Enable Memory Viewer' when you upload your profile:\n\n.. image:: /tools/neuron-explorer/images/memory_viewer_enable.png\n\nView the Memory Viewer Widget\n------------------------------\n\nOnce your profile finishes processing and is ready to view, click the Add Widget button and select 'Memory Viewer':\n\n.. image:: /tools/neuron-explorer/images/memory_viewer_add_widget.png\n\n\nBy hovering your mouse over each allocation, you can see the detailed information about this allocation. For allocations triggered by instructions, hover informations includes: \n* Start time and end time\n* Duration\n* Start address and end address \n* Opcode\n* Operands \n\nFor allocations triggered by DMAs, hover information includes: \n* Partition number \n* Start time and end time \n* Duration\n* Start address and end address \n* DMA queue name \n* Block ID\n\nBy analyzing memory allocations, you can address memory fragmentation by identifying sparse allocation patterns and potentially rescheduling instructions or DMAs to different addresses to maintain memory compactness. Additionally, you can perform spill/reload analysis to identify opportunities for reducing spills by relocating allocations to available space at alternative addresses.\n\nYou can also use the dropdown menu to inspect the memory allocations on different partitions and NC cores: \n\n.. image:: /tools/neuron-explorer/images/memory_viewer_hover.png\n" }, { "path": "tools/neuron-explorer/overview-summary-page.rst", "content": ".. meta::\n :description: Learn how to use the Neuron Explorer summary page to quickly identify performance issues, view key metrics, and get actionable optimization recommendations for your profiles.\n :date-modified: 03/20/2026\n\nSummary Viewer\n================\n\nThe Neuron Explorer summary viewer provides a streamlined view of your profile's most critical performance insights, enabling quick identification of issues and optimization opportunities without navigating through detailed data.\n\n.. image:: /tools/profiler/images/explorer-summary-page.png\n\nBenefits\n--------\n\nBoth new and experienced users benefit from this streamlined view of profiling data.\n\n* Identify performance issues quickly\n* Understand your profile's most critical metrics at a glance\n* Get actionable recommendations for optimization\n\nHow to use\n-------------\n\n1. **Open your profile** - The Summary Viewer is accessible via the Profile Manager or Neuron Explorer UI.\n2. **Examine key metrics** - Review the metrics and graphs to understand your profile's performance characteristics.\n3. **Review recommendations** - Start with the **Performance Insights & Recommendations** section. This section highlights the most important performance issues.\n4. **Select specific time regions** - Use the \"Region Selection\" menu to view specific timeslices corresponding to network layers. This helps you drill down into specific sections of your profile. You can generate custom time regions using the \"Add Region\" button.\n5. **Take action** - Apply the recommended optimizations to your model or workload.\n\nUnderstanding region-level insights\n-----------------------------------\n\nWhen you work with profiles from entire networks or network subgraphs, different regions will have different performance characteristics. The landing page enables performance analysis on a per-layer basis and provides:\n\n* Layer-specific recommendations\n* Time-range indication of where problems occur\n* More accurate insights for complex profiles\n\nUse the 'Region Selection' menu to navigate between different layers and view their individual performance data.\n\nWhat the landing page displays\n------------------------------\n\nPerformance Insights and Recommendations\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis section shows 2-4 recommendations to help you improve performance. The profiler analyzes your data and identifies the most important issues to address. The profiler prioritizes recommendations by criticality and shows you the most critical ones first.\n\nExample recommendations\n^^^^^^^^^^^^^^^^^^^^^^^\n\n.. list-table::\n :header-rows: 1\n :widths: 30 35 35\n\n * - Condition\n - Root Cause\n - Recommended Action\n * - Low Model FLOPS relative to Active FLOPS (< 50%)\n - Tensor engine is active but not performing useful matrix operations\n - Ensure instructions use the entire tensor engine and are pipelined correctly\n * - NKI instruction coverage < 50% on tensor, vector, or scalar engine\n - Compiler-generated instructions dominate the engine\n - Write NKI kernel code for the network operations present in that profile section\n * - Active FLOPS throttling detected\n - FLOPS lost due to throttling during active tensor engine periods\n - Investigate the root cause of throttling to recover tensor engine utilization\n * - Transpose FLOPS > 10% of total hardware FLOPS\n - Excessive data movement within the tensor engine\n - Improve memory layout to reduce transpose operations\n * - Collective operation outliers detected\n - Significantly underperforming collective operations relative to their group median\n - Check for overlapping instructions that might be causing delays\n * - Spill reload bytes > 25% of total HBM reads\n - Excessive spill/reload operations consuming memory bandwidth\n - Check for data dependencies causing excessive spill/reload operations\n\nKey Metrics\n~~~~~~~~~~~\n\nThis section displays tables and graphs that summarize your profile's performance metrics.\n\nCompute Performance Statistics\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n* **total_time** - Total duration of on-device time for the run in seconds. This doesn't include host-device data movement overhead or host runtime/framework overhead.\n* **mm_arithmetic_intensity** - The ratio of regular Matrix Multiplication (MATMUL) Floating Point Operations (FLOPs) to total Dynamic Random Access Memory (DRAM) transfer size. This metric helps you determine if your workload is memory-bound or compute-bound.\n* **hfu_estimated_percent** - Hardware FLOPs Utilization reflects the Tensor Engine utilization calculated from all Tensor Engine instructions.\n* **mfu_estimated_percent** - Model FLOPs Utilization reflects the Tensor Engine utilization for useful compute (matrix multiplications from your model definition).\n\nMemory Bandwidth Utilization\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n* **total_bandwidth_available** - The total bytes possible to be transferred within the given time region for the current Neuron hardware specification.\n* **mbu_estimated_percent** - Memory Bandwidth Utilization (MBU) shows the achieved (as running on the current Neuron hardware) High Bandwidth Memory (HBM) bandwidth utilization.\n* **average_dma_size** - The average DMA transfer size (higher is better).\n* **useful_read_percent** - The fraction of HBM reads that are useful (``hbm_read_bytes`` - ``hbm_reload_bytes``) / hbm_read_bytes). Note that \"useful\" is related to an inherent property of the memory itself, but a measurement of how efficiently the memory is being utilized by a specific workload or application. Low numbers may indicate inefficient memory access patterns and suboptimal layouts.\n\nFLOPs Utilization\n^^^^^^^^^^^^^^^^^\n\nFor each compute engine (tensor, vector, scalar, gpsimd), displays how well utilized the engine is. You can view all cores simultaneously or select a specific Neuron Core from the dropdown.\n\nTensor Engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe Tensor engine has a detailed breakdown of how the FLOPs are being used:\n\n* **model_flops**: The percentage of tensor flops spent performing useful matrix operations, contributing to model progress\n* **transpose_flops**: The percentage of tensor flops spent performing transpose operations / data movement\n* **active_flops** - Percentage of tensor flops that correspond to the active_time of the tensor engine, but where the engine was not effectively utilized.\n* **throttled_flops (active and inactive)** - Percentage of FLOPs wasted due to throttling, either during active or inactive tensor engine periods.\n\nThere are a few key things to look for in this graph:\n\n1. **model_flops relative to active_flops**. Large differences could indicate that the tensor engine is being poorly utilized with small tensor sizes, or that operations are not being pipelined effectively.\n2. **model_flops relative to transpose_flops**. It is desired to have little-to-no ``transpose_flops`` consuming tensor engine utilization. Ideally the ``model_flops`` amount is much larger than the amount of transposes.\n3. **active_throttled_flops**: FLOPs lost due to throttling during active periods is undesirable. It is worth identifying the root cause for the throttling if there is indication of this happening.\n\nOther Engines (Scalar, Vector, GpSimd)\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThese engines do not yet have detailed FLOP utilization breakdowns, they only show the active period of operation for the engine.\n\n* **active_flops** - Percentage of FLOPs when the engine processes at least one instruction (excluding semaphore waits).\n\nNKI Engine Statistics\n^^^^^^^^^^^^^^^^^^^^^\n\nThis chart shows the instruction count breakdown between NKI-generated instructions and compiler-generated instructions for each compute engine (tensor, vector, scalar). The stacked bar chart helps you understand how much of your workload is running NKI kernel code versus compiler-generated code.\n\nHovering over a bar displays a detailed breakdown of instruction counts by opcode for that engine and source type.\n\nWhen NKI instruction coverage is below 50% for a given engine, the summary page generates a recommendation to write NKI kernel code for the network operations in that profile section.\n\nDMA Utilization\n^^^^^^^^^^^^^^^\n\nThis chart shows how the DMA engines are being utilized, displayed as a percentage of the total available bandwidth. Two dropdown menus control the chart's aggregation:\n\n* **Outer aggregation** - Choose between viewing data per DMA engine (\"All Engines\") or per Neuron Core (\"Neuron Cores\").\n* **Inner aggregation** - Choose between grouping by data type or source type:\n\n * **Data Type** groups transfers into Instruction, IO, Weights, and Dynamic categories.\n * **Source Type** groups transfers into Static (compiler-generated), Software Dynamic (GpSimd-generated), and Hardware Dynamic (DGE hardware-generated) categories.\n\nEach category shows two bar segments: a solid bar representing bandwidth utilization and a striped bar representing active time utilization beyond the bandwidth portion. This helps distinguish between time spent transferring data and time the DMA engine is active but not fully utilizing bandwidth.\n\nMemory Bandwidth Breakdown\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nShows how the available HBM memory bandwidth was used as a doughnut chart:\n\n* HBM Read — effective read bytes (excluding spill reloads)\n* HBM Write — effective write bytes (excluding spill saves)\n* SBUF Spill Reload — bytes reloaded from HBM due to state buffer spills\n* SBUF Spill Save — bytes saved to HBM due to state buffer spills\n* Unused — remaining available bandwidth\n\nCollective Operations Duration\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nDisplays the duration of each collective operation in the profile, grouped by operation type and size. Two visualization modes are available via a dropdown:\n\n* **Scatter** - Shows individual operation durations as scatter points, with each operation type on a separate row. Hovering over a point displays detailed information including algorithm, operation, duration, start/end timestamps, element count, input/output sizes, and trigger engine. Clicking a point pins the tooltip for easy text selection.\n* **Box Plot** - Shows the statistical distribution (min, Q1, median, mean, Q3, max, variance, count) of operation durations per operation type. This is useful for quickly identifying the spread and central tendency of each operation group.\n\nBoth modes are useful for identifying outliers in collective runtime, which can be used to investigate specific sections of the profile more deeply. It is possible to filter out datasets by clicking on the datasets in the legend of the graph.\n\nSystem Information\n^^^^^^^^^^^^^^^^^^\n\nDisplays metadata about the system and software versions used during profiling:\n\n* Instance Type\n* Compiler Version\n* Explorer Version\n* Driver Version\n* Runtime Version\n* Collectives Version\n\nSystem Profile Summary\n======================\n\nWhen a system profile is loaded, the Summary Viewer automatically switches to the System Profile Summary view. System profiles capture data across multiple devices, processes, and instances, providing a holistic view of distributed workload performance.\n\nOverview\n--------\n\nThe System Profile Summary provides:\n\n* A high-level overview of the entire system's profiling session\n* HBM memory usage trends across logical NeuronCores\n* A detailed table of all device profiles with key performance metrics\n* The ability to drill down into individual device profiles for detailed analysis\n\nSystem Overview Card\n--------------------\n\nDisplays aggregate information about the profiling session:\n\n* **Instances** - Number of unique instances captured in the profile\n* **Processes** - Number of unique processes captured\n* **System Profile Time** - Total wall-clock duration of the system profiling session\n* **Total Device Runtime** - Cumulative on-device execution time across all device profiles\n* **Total Device Profiles** - Number of individual device profiles in the system profile\n\nHBM Memory Usage Chart\n-----------------------\n\nA line chart showing HBM memory usage over time. When per-NeuronCore data is available, the chart displays a separate line for each logical NeuronCore (HBM index), color-coded for easy identification. When only aggregate data is available, a single filled area chart shows total HBM usage.\n\nThe x-axis shows time (in the profiling session's time domain) and the y-axis shows memory usage in bytes. Hovering over the chart displays the exact timestamp and memory usage for each NeuronCore.\n\nDevice Profiles Table\n---------------------\n\nA table listing all device profiles captured in the system profile. The table supports:\n\n* **Process filtering** - Use the dropdown to filter profiles by process ID, or select \"All Processes\" to view everything.\n* **Expandable rows** - Click the expand arrow on any row to see additional per-profile metrics including tensor/vector/scalar engine active time percentages, DMA active time, and HBM read/write bytes.\n* **Column tooltips** - Hover over column headers to see descriptions of each metric from the profile schema.\n\nTable columns:\n\n* **Profile Name** - Clickable link that navigates to the detailed device profile view\n* **LNC** - Logical NeuronCore ID\n* **Neuron Cores** - Number of physical NeuronCores used by this profile\n* **Total Duration** - Total on-device execution time for this profile's events\n* **Calls** - Number of execution events for this profile\n* **Duration** - Total profiled time for this device profile\n* **MFU** - Model FLOPs Utilization\n* **HFU** - Hardware FLOPs Utilization\n* **MBU** - Memory Bandwidth Utilization\n* **CC Active** - Collective communication active time percentage\n\nDevice Profile Detail View\n--------------------------\n\nClicking a device profile name in the table navigates to a detail view that embeds the standard Summary Viewer for that specific device profile. This provides the full set of per-device metrics, charts, and recommendations described in the sections above.\n\nA \"Back to System Overview\" button at the top returns you to the system-level summary.\n" }, { "path": "tools/neuron-explorer/overview-system-profiles.rst", "content": ".. meta::\n :description: Learn about the System Profile in Neuron Explorer for analyzing system-level execution across instances and workers with runtime and hardware events.\n :date-modified: 01/30/2026\n\nSystem Profile\n================\n\nThe Neuron System Profile show a system-level granularity of execution across instances and workers in your workload. This provides visibility into Neuron Runtime API calls and ML framework function calls (PyTorch or JAX) to help identify bottlenecks in distributed workloads. The Neuron Explorer UI provides system-level widgets for an extensible and customizable workflow.\n\n.. image:: /tools/neuron-explorer/images/neuron-explorer-system-viewer.png\n\nSystem Trace Viewer\n---------------------\n\nThe System Trace Viewer provides an interactive timeline interface with time range selection, configurable event grouping, system event details on hover, and linking of hardware events to Device Trace Viewer widgets.\n\nYou can see events in the Neuron Runtime and correlate them with hardware execution events on the Neuron Devices.\n\n.. image:: /tools/neuron-explorer/images/system-timeline-widget.png\n\nYou can also see the device memory (HBM) allocations for each Neuron device over time. Hovering over these memory usage events shows a breakdown by usage category.\n\n.. image:: /tools/neuron-explorer/images/system-timeline-widget-hbm-usage.png\n\n\nAdding Widgets\n---------------\nThe System Profile supports both System and Device widgets, enabling multi-profile analysis, for example comparing annotated device events across different devices.\n\nTo add a widget:\n\n1. Click the **Add Widget** button to open the Add Widget modal.\n2. Select a Device or System widget.\n3. Click a widget tile to load it with the selected profile. Each tile is tagged with its supported profile type (system, device, or both).\n\nTo load multiple instances of the same widget type for different profiles, repeat the steps above and select a different profile each time.\n\n.. image:: /tools/neuron-explorer/images/system-timeline-add-widget.gif\n\n\nAfter adding a widget, you can switch to a different profile by using the profile dropdown at the top of the widget.\n\n.. image:: /tools/neuron-explorer/images/widget_switch_profiles.png\n\n.. note::\n\n Adding duplicate widgets for the same profile is not currently supported.\n\n\n\nSettings\n----------\n\nThe System Trace Viewer supports multiple grouping modes to organize events for different analysis perspectives.\nYou can switch between the following grouping modes in the settings to focus your analysis on different aspects of system performance:\n\n.. list-table:: Grouping Options\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Grouping Option\n - Description\n - Example\n * - CPU vs Device Grouping (Default)\n - Groups events by event source (CPU or Neuron device events)\n - Runtime events: ``i-0b1ea78ca2865fd32/PID:1765325/TID:0/neuron_rt``, Hardware events: ``i-0b1ea78ca2865fd32/PID:1765325/Worker:0/neuron_hw``\n * - NeuronCore Grouping\n - Groups events by individual NeuronCore\n - ``i-0b1ea78ca2865fd32/NC:0``, ``i-0b1ea78ca2865fd32/NC:1``\n * - Thread Grouping\n - Groups events by thread identifier\n - ``i-0b1ea78ca2865fd32/PID:1765325/TID:0``\n * - Process Grouping\n - Groups events by process identifier\n - ``i-0b1ea78ca2865fd32/PID:1765325``\n * - Instance Grouping\n - Groups all events by instance only\n - ``i-0b1ea78ca2865fd32``\n\n.. image:: /tools/neuron-explorer/images/system-timeline-settings.png\n\nEvent Details\n--------------\n\nClicking on trace events in the timeline populates the Event Details widget with a list of properties for the system trace event.\n\n.. image:: /tools/neuron-explorer/images/system-event-details.png\n\nDevice Profile Linking\n------------------------\n\nThe System Trace Viewer links hardware events to the Device Trace Viewer, which renders the corresponding device traces.\n\nNavigating from the System Trace Viewer to a Device Trace Viewer can be accomplished in two ways:\n\nOpen the Device Profile List Modal\n------------------------------------\n\nTo see a list of all device profiles captured during your workload:\n\n1. **Click the \"Device Profiles List\" button** in the top right action bar of the System Trace Viewer to open a modal containing a list of device profiles\n2. **Select a Device Profile and click Submit** to open the Device Trace Viewer with the selected device profile\n\n.. image:: /tools/neuron-explorer/images/system-timeline-device-profiles-list-modal.png\n\nDrill-down from Hardware Events\n---------------------------------\n\nTo drill-down from a hardware event to the Device Trace Viewer:\n\n1. Find a hardware event such as ``nc_exec_running``\n2. Click on the hardware event\n3. Wait for the Device Trace Viewer to open\n\nThis will open a new Device Trace Viewer with the selected device profile showing detailed hardware events. To learn about device profiles, see :doc:`Device Profiles in Neuron Explorer `.\n\n.. image:: /tools/neuron-explorer/images/system-timeline-hardware-event-linking.gif\n" }, { "path": "tools/neuron-explorer/overview-tensor-viewer.rst", "content": ".. meta::\n :description: Learn about the Tensor Viewer in Neuron Explorer for viewing tensor information including names, sizes, shapes, and memory usage details.\n :date-modified: 01/27/2026\n\n.. _tensor-viewer-overview:\n\nTensor Viewer\n=================\n\nThe Tensor Viewer contains the following information about all tensors in the NEFF file:\n\n* **variable_name** - The tensor name.\n* **type** - How the system uses the tensor. Examples include input tensor, output tensor, or weight tensor.\n* **format** - How the tensor arranges in memory. For example, \"NHWC\" shows a specific dimension arrangement. Letters include N (batch size), H (height), W (width), C (channel).\n* **shape** - The tensor's multi-dimensional shape.\n* **size** - The tensor's total size in bytes.\n* **node** - NEFF node.\n* **pcore_idx** - Index of the physical NeuronCore within a Logical NeuronCore (LNC). A Logical NeuronCore groups physical NeuronCores. For LNC2, this field shows either 0 or 1.\n* **load_to_sbuf_avg_size_bytes** - The average size in bytes of each DMA transfer when the system loads this tensor into the State Buffer.\n* **load_to_sbuf_total_size_bytes** - The total size in bytes of all DMA transfers when the system loads this tensor into the State Buffer.\n* **load_to_sbuf_dma_count** - The total number of DMAs that loaded this tensor into the State Buffer.\n* **load_to_sbuf_repeat_factor** - How many times the system loaded this tensor into the State Buffer. A value of 1 means one load, 2 means two loads, and so on.\n\n.. image:: /tools/profiler/images/tensor-viewer-table.png\n\nYou can use this data to match with framework-level instructions or for kernel development. You can also use it to search for instructions in the Device Timeline Viewer. \nThe SBUF loading information in the table can help you verify tensors are loaded efficiently.\n\nSearching\n---------\n\nYou can use the Tensor Viewer with the Device Timeline Viewer and Search tool to match tensor information in the table with instructions that run on the device. \nEnter the variable_name from the table, into the DMA search field to see all DMA instructions that relate to that tensor.\nThe example below shows a complete search for the tensor token_position_to_id:\n\n.. image:: /tools/profiler/images/tensor-viewer-search-example.png\n" }, { "path": "tools/neuron-explorer/view-perfetto.rst", "content": ".. meta::\n :description: Learn about using Neuron Explorer with Perfetto\n :date-modified: 02/05/2026\n\nViewing Profiles with Perfetto\n==============================\n\n.. note::\n New Neuron Explorer features released in 2.27 and onwards may not be supported in Perfetto. For the full user experience and features set, please use the Neuron Explorer UI or VSCode Integration.\n\nPerfetto is an open-source trace analysis toolkit with a powerful UI for visualizing and analyzing trace data.\nUsers of Neuron Profiler have the option of viewing their profiles in the Perfetto UI.\n\nThe ``--output-format perfetto`` option writes processed data to Perfetto's native protobuf-based tracing format which can be visualized in the Perfetto UI at https://ui.perfetto.dev/.\n\nExample:\n\n.. code-block:: shell\n\n neuron-explorer view -d ./output --output-format perfetto\n\nThis will generate a ``system_profile.pftrace`` file for the system profile and a ``device_profile_model_.pftrace`` file for each unique compiled model that was executed on a Neuron Device.\n\nTo view the system profile, go to https://ui.perfetto.dev/ and open the ``system_profile.pftrace`` file.\n\n.. note::\n When loading trace files in the Perfetto UI, your data is processed locally and not uploaded to Perfetto’s servers.\n\n|neuron-explorer-perfetto-timeline|\n\nTo view a device profile go to https://ui.perfetto.dev/ and open the ``device_profile_model_.pftrace`` file. This will show a detailed view of hardware activity on the NeuronCore during execution of this graph.\n\n|neuron-explorer-perfetto-device-timeline|\n\n.. note::\n Your browser may run out of memory when viewing ``*.pftrace`` (Perfetto trace) files that are more than a few hundred MB. See the section :ref:`Viewing Large Profiles in Perfetto ` for directions on how to view large traces using the trace processor.\n\n\nPerfetto Output View Options\n----------------------------\n\nWhen outputting to Perfetto it is possible to group your traces by different attributes. This is useful for\nlarger profiles involving many NeuronCores and instances. The following options are available:\n\n.. list-table:: Perfetto output view options\n :header-rows: 1\n :widths: 30 70\n\n * - CLI option\n - Description\n * - ``--system-trace-primary-group``\n - First-order grouping of trace events (maps to a Perfetto process / process group of rows). Provide a comma-delimited list of field names. Allowed fields: ``instance_id``, ``thread_id``, ``lnc_idx``, ``process_id``. Default: ``instance_id,process_id``.\n * - ``--system-trace-secondary-group``\n - Second-order grouping of trace events (maps to a Perfetto thread / single row). Provide a comma-delimited list of field names. Allowed fields: ``instance_id``, ``worker_gid``, ``thread_id``, ``lnc_idx``, ``process_id``. Default: ``worker_gid,lnc_idx, thread_id``.\n\n\nFor example, the following profile uses ``neuron-explorer view --output-format=perfetto --system-trace-primary-group=instance_id,process_id --system-trace-secondary-group=lnc_idx,thread_id`` to group the system profile first by unique combinations\nof instance_id and process_id, and then in each of those groups there are rows of events with unique combinations of lnc_idx and thread_id.\n\n|neuron-explorer-perfetto-grouping|\n\nGrouping By Global Worker ID\n----------------------------\n\nBy default, Perfetto traces are grouped by ``worker_gid`` which is a unique global identifier for each NeuronCore across all instances in a distributed workload.\nWhen clicking on an event in the trace you will see fields for both ``lnc_idx`` (local NeuronCore index on that process) and ``worker_gid`` (global NeuronCore index across all instances).\nIt is possible for ``lnc_idx`` to be the same for different processes on the same instance or across different instances in a distributed workload. However, ``worker_gid`` is unique for each NeuronCore across all instances.\nThe image below shows how to correlate the naming of tracks (rows) in the Perfetto UI to both ``lnc_idx`` and ``worker_gid``.\n\n|neuron-explorer-perfetto-gid|\n\n\n.. |neuron-explorer-perfetto-timeline| image:: /images/neuron-profiler2-perfetto-timeline.png\n.. |neuron-explorer-perfetto-device-timeline| image:: /images/neuron-profiler2-perfetto-device-timeline.png\n.. |neuron-explorer-perfetto-grouping| image:: /images/neuron-profiler2-perfetto-grouping.png\n.. |neuron-explorer-perfetto-gid| image:: /images/neuron-profiler2-perfetto-gid.png\n" }, { "path": "tools/neuron-sys-tools/index.rst", "content": "System Tools\n============\n\nNeuron system tools provide essential utilities for monitoring, debugging, and managing AWS Neuron devices and workloads. These command-line tools offer real-time insights into device utilization, process management, hardware health, and performance metrics across Neuron instances.\n\n.. toctree:: \n :maxdepth: 1\n :hidden:\n\n Neuron-Monitor User Guide \n Neuron-Top User Guide \n Neuron-LS User Guide \n Neuron-Sysfs User Guide \n NCCOM-TEST User Guide \n TensorBoard \n\n.. grid:: 1 1 2 2\n :gutter: 3\n\n .. grid-item-card:: Neuron-Monitor User Guide\n :link: /tools/neuron-sys-tools/neuron-monitor-user-guide\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Real-time monitoring tool for tracking NeuronCore utilization, memory usage, and thermal metrics across Neuron devices with customizable output formats.\n\n .. grid-item-card:: Neuron-Top User Guide\n :link: /tools/neuron-sys-tools/neuron-top-user-guide\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Interactive process viewer similar to htop that displays running processes on Neuron devices with real-time resource consumption metrics.\n\n .. grid-item-card:: Neuron-LS User Guide\n :link: /tools/neuron-sys-tools/neuron-ls\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Device discovery and listing tool that provides detailed information about available Neuron devices, their capabilities, and current status.\n\n .. grid-item-card:: Neuron-Sysfs User Guide\n :link: /tools/neuron-sys-tools/neuron-sysfs-user-guide\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Low-level system interface tool for accessing Neuron device information through the Linux sysfs filesystem interface.\n\n .. grid-item-card:: NCCOM-TEST User Guide\n :link: /tools/neuron-sys-tools/nccom-test\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Collective communication testing and benchmarking tool for validating and measuring performance of multi-device communication patterns.\n\n .. grid-item-card:: TensorBoard\n :link: /tools/tensorboard/index\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n TensorBoard Neuron plugin for Trn1 instances, including installation, configuration, and advanced visualization features.\n\n .. grid-item-card:: Tutorials\n :link: /tools/tutorials/index\n :link-type: doc\n :class-header: sd-bg-secondary sd-text-white\n\n Tutorials for how to utilize the Neuron system tools suite.\n\n .. grid-item-card:: What's New\n :link: /release-notes/prev/2.27.0/index\n :link-type: doc\n :class-header: sd-bg-secondary sd-text-white\n\n Latest updates, new features, and improvements to the Neuron system tools suite.\n" }, { "path": "tools/neuron-sys-tools/nccom-test.rst", "content": ".. _nccom-test:\n\n======================\nNCCOM-TEST User Guide\n======================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nOverview\n--------\n\n**nccom-test** is a benchmarking tool for evaluating Collective Communication operations on AWS Trainium and Inferentia instances. It supports Trn1, Trn2, Trn3, and Inf2 instance types. The tool can assess performance across multiple instances or perform quick environment sanity checks before running more complex workloads. While single-instance benchmarking is supported for all compatible instance types, multi-instance benchmarking is limited to Trainium instances (Trn1, Trn2, and Trn3). \nTo execute collective operations, **nccom-test** will generate, and then execute, NEFFs (Neuron Executable File Format) containing several collective operation instructions.\n\n.. note::\n\n On Inf2 instances, only single-instance benchmarking is supported. Running a multi-node nccom-test benchmark\n will result in an error.\n\nUsing nccom-test\n----------------\n\nHere is a simple example which will run a 2 worker (ranks) all-reduce with a total size of 32MB:\n\n\n.. code-block::\n\n nccom-test -r 2 allr\n size(B) count(elems) type time(us) algbw(GB/s) busbw(GB/s)\n 33554432 33554432 uint8 768 40.69 40.69\n Avg bus bandwidth: 40.6901GB/s\n\n\nOutput description\n^^^^^^^^^^^^^^^^^^\n\nThe command will output a table containing several columns containing performance metrics.\nThere will be a line for every requested data size (by default the data size is 32MB as\nseen in the previous example).\n\n.. list-table::\n :widths: 40 260\n :header-rows: 1\n\n * - Column name\n - Description\n * - size(B)\n - Size in bytes for the data involved in this collective operation\n * - count(elems)\n - Number of elements in the data involved in this collective operation. For example, if **size(B)** is 4 and **type** is fp32,\n then **count** will be 1 since one single fp32 element has been processed.\n * - type\n - Data type for the processed data. Can be: **uint8**, **int8**, **uint16**, **int16**, **fp16**, **bf16**, **int32**, **uint32**, **fp32**\n * - time(us)\n - Time in microseconds representing the average of all durations for the Collective Communication operations executed during the benchmark.\n * - algbw(GB/s)\n - Algorithm bandwidth in gibibytes (1GiB = 1,073,741,824 bytes) per second which is calculated as **size(B)** / **time(us)**\n * - busbw(GB/s)\n - Bus bandwidth - bandwidth per data line in gibibytes per second - it provides a bandwidth number that is independent from the number of ranks (unlike **algbw**).\n For a more in-depth explanation on bus Bandwidth, please refer to `Bus Bandwidth Calculation`_\n * - algorithm (optional)\n - Algorithm used to execute this collective operation (e.g. Ring, Mesh, RDH)\n * - Avg bus bandwidth\n - Average of the values in the busbw column\n \n\n.. _Bus Bandwidth Calculation:\n**Bus Bandwidth Calculation:**\n\nThe purpose of bus bandwidth is to provide a number reflecting how optimally hardware is used, normalizing for different rank counts.\n\nGiven the following:\n\n- ``r`` as the number of ranks participating in a collective operation\n- ``s`` as the size of the collective operation\n- ``B`` as the bus bandwidth of a single rank\n- ``t`` latency of the operation\n\nLet's take an AllGather operation as an example. To complete an AllGather operation with ``r`` ranks, each rank must transfer ``r-1`` data chunks of size ``s/r``. Therefore, with a bandwidth of ``B``, the latency (``t``)\nof the operation would be:\n\n.. code-block::\n\n t = ((number of chunks to transfer) * (size of each chunk)) / (bandwidth of rank)\n t = ((r-1) * (s/r)) / B\n\nHowever, for a given collective operation result, we have the latency, but not the bandwidth of each rank. Rearranging to solve for bus bandwidth, we get:\n\n.. code-block::\n\n B = ((r-1) * (s/r)) / t\n\nwhich, given ``algbw = s / t``, can also be rewritten as:\n\n.. code-block::\n\n B = ((r-1) / r) * algbw\n\nUsing this formula, we can calculate the bus bandwidth, ``B``, for an AllGather collective operation among ``r`` ranks with size ``s`` that took ``t`` seconds.\n\nWe can now directly compare the calculated bus bandwidth to the actual hardware bandwidth to see how well the hardware is being utilized. For different operations that transfer a different\nnumber of chunks, the bandwidth calculation changes slightly, with our algbw factor ``(r-1) / r`` changing depending on the collective operation:\n\n.. list-table::\n :widths: 40 40\n :header-rows: 1\n\n * - Collective Operation\n - Bus Bandwidth Factor\n * - All-Reduce\n - ``(2 * (r-1)) / r``\n * - All-Gather\n - ``(r-1) / r``\n * - Reduce-Scatter\n - ``(r-1) / r``\n * - Send-Receive\n - 1\n * - All-to-All\n - ``(r-1) / r``\n * - Permute\n - 1\n * - All-to-Allv\n - ``(r-1) / r``\n\n\n\nCLI arguments\n^^^^^^^^^^^^^\n\nRequired Arguments:\n~~~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - \n - N/A, required argument\n - The type of Collective Communication operation to execute for this benchmark.\n Supported types:\n\n - ``all_reduce`` / ``allr``: All-Reduce\n - ``all_gather`` / ``allg``: All-Gather\n - ``reduce_scatter`` / ``redsct``: Reduce-Scatter\n - ``sendrecv``: Send-Receive\n - ``alltoall``: All-to-All\n - ``permute``: Permute\n - ``alltoallv``: All-to-Allv (Currently only supported for inter-node configurations)\n * - ``-r, --nworkers``\n - N/A, required argument\n - Total number of workers (ranks) to use\n\nBenchmark Configuration:\n~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``-N, --nnodes``\n - 1\n - Total number of nodes (instances) to use. The number of workers will be divided equally across all nodes.\n If this argument is greater than 1, `MPI Execution`_ or `Slurm Execution`_ will need to be used.\n * - ``-b, --minbytes``\n - 32M\n - The starting size for the benchmark\n * - ``-e, --maxbytes``\n - 32M\n - The end size for the benchmark. **nccom-test** will run benchmarks for all sizes between ``-b, --minbytes`` and\n ``-e, --maxbytes``, increasing the size by either ``-i, --stepbytes`` or ``--f, --stepfactor`` with every run.\n * - ``-i, --stepbytes``\n - (``--maxbytes`` - ``--minbytes``) / 10\n - Amount of bytes with which to increase the benchmark's size on every subsequent run.\n For example, for this combination of arguments: ``-b 8 -e 16 -i 4``, the benchmark will\n be ran for the following sizes: 8 bytes, 12 bytes, 16 bytes.\n * - ``-f, --stepfactor``\n - N/A\n - Factor with which to increase the benchmark's size on every subsequent run.\n For example, for this combination of argument values: ``-b 8 -e 32 -f 2``, the benchmark will\n be ran for the following sizes: 8 bytes, 16 bytes, 32 bytes.\n\n.. note::\n\n All arguments that take a size in bytes will also accept larger size units, for example:\n ``-f 2048`` can be written as ``-f 2kb`` or ``-f 1048576`` can be written as ``-f 1MB``.\n\nIteration Configuration:\n~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``-n, --iters``\n - 20\n - Number of Collective Communication operations to execute during the benchmark.\n * - ``-w, --warmup_iters``\n - 5\n - Number of Collective Communication operations to execute as warmup during the benchmark. \n The warmup operations will execute prior to any of the measured operations and their performance will be not be used calculate the reported statistics.\n * - ``-I, --neff_iters``\n - N/A\n - Number of times to execute the NEFF with Collective Communication operations during the benchmark. \n * - ``-W, --neff_warmup_iters``\n - N/A\n - Number of times to execute the NEFF with Collective Communication operations as warmup during the benchmark. All collective operations in a warmup NEFF execution will be ignored when calculating statistics.\n\nTo execute collective operations, ``nccom-test`` will generate, and then execute, NEFFs (Neuron Executable File Format) containing several collective operation instructions.\nThe above flags control how many collective operations are generated, run, and measured.\n\nThere are two primary modes for controlling the number of collective operations run:\n\n1. If neither the ``neff_iters`` nor the ``neff_warmup_iters`` flag is supplied, ``iters + warmup_iters`` will be treated as the desired total number of\n operations to be run. If necessary, ``nccom-test`` will spread this total number of operations out across several NEFFs.\n\n2. If the user desires more control over how collectives operation execution should be organized, they should use the ``neff_iters`` and ``neff_warmup_iters``\n flags. When these flags are used, the ``iters`` and the ``warmup_iters`` flags now represent the number of operations in a single NEFF. The NEFF itself will be repeatedly run\n ``neff_iters + neff_warmup_iters`` times.\n\nExamples:\n\n- ``-n 15``, ``-w 5``, ``-I 10``, would result in 200 Collective Communication operations being run with 150 being measured:\n The generated NEFF will have 20 (15 measured, 5 warmup) ops and the NEFF will be run 10 times.\n- ``-n 15``, ``-w 5``, ``-I 10``, ``-W 5``, would result in 300 Collective Communication operations being run with 150 being measured:\n The generated NEFF will have 20 (15 measured, 5 warmup) ops and the NEFF will be run 15 (10 measured, 5 warmup) times\n \n\nInput/Output Data:\n~~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``-d, --datatype``\n - ``uint8``\n - Data type for the data used by the benchmark. Supported types: ``uint8``, ``int8``, ``uint16``, ``int16``,\n ``fp16``, ``bf16``, ``uint32``, ``int32``, ``fp32``. Input data will be zero filled, unless ``--check`` is\n provided in which case it will be filled with either pseudo-random data or ones.\n * - ``-c, --check``\n - N/A\n - If provided, validates correctness of the operations. Can additionally specify options: ``random`` (default) or ``all_ones``.\n For an explanation of these options, see `Data Integrity`_.\n This will not impact device execution time and collective operation performance (time, algbw, and busbw),\n but will slightly increase the overall execution time.\n * - ``--seed``\n - N/A\n - Seed to use while generating pseudo-random data for ``random`` correctness check with ``--check`` flag\n * - ``--unique-buff``\n - false\n - Use a unique buffer for the input and output of every collective operation. When using this flag, each collective operation in a NEFF will use a\n different in-memory input/output buffer than every other operation. For All-Gather operations run with certain algorithms (e.g. Mesh, RDH),\n there is additional handshaking for output buffers, and using unique buffers may improve collective operation performance.\n * - ``--coalesced-cc-size-ratio``\n - N/A\n - List representing the ratio with which to split the input tensor into multiple tensors for coalesced, collective operations. Given a size of ``4MB`` and a ``coalesced-cc-size-ratio`` of ``[1,2,1]``, each collective\n operation would actually consist of 3 parallel, coalesced operations of sizes: ``1MB``, ``2MB``, and ``1MB``.\n * - ``--shared-output-buff``\n - false\n - For the CC operation, use a single, shared, HBM output buffer between 2 neuron cores in the same HBM domain.\n * - ``--alltoallv-metadata``\n - N/A\n - For ``alltoallv`` collective operation, a ``json`` file containing send counts, send displacements, receive counts, and receive displacements for the collective operation. \n Counts specify number of elements to send/receive between ranks, displacements specify where in buffer to send/receive data.\n Length of count and displacement arrays should equal size of replica group over which ``alltoallv`` collective operation is performed. \n If one metadata entry is provided, it applies to all ranks, otherwise, specify one entry per rank. `AlltoAllV Example`_.\n\n.. _Data Integrity:\n\nData Integrity:\n\nIf the ``--check`` flag is provided when running ``nccom-test``, the correctness of the CC operations will be verified. There are currently two modes for verification: ``random`` (the default used when only ``--check`` is provided)\nand ``all_ones``. \n\n1. The ``random`` mode will fill each input tensor with pseudo-random data and then, on the CPU, calculate a expected golden output. After collective operation execution,\n the output tensor of the operation will be compared against the calculated golden tensor. For non-integral types (e.g. ``fp16``, ``fp32``), golden comparison will use tolerances.\n For operations in which all participating ranks should finish with identical outputs (e.g ``allr``, ``allg``), there will also be a check between ranks to ensure this.\n If the ``random`` check fails, input, output, and golden tensors will be saved to disk for further investigation. The ``--seed`` flag can be used to set the seed for the\n pseudo-random input tensor generation. Otherwise, the seed value will be based on the current time and logged.\n\n2. The ``all_ones`` mode will fill each input tensor with the value ``1``. A single, golden value\\: ``G``, will be calculated based on the operation. For example, the golden value\\: ``G``\n for an All-Reduce with 16 ranks will be ``16``. After operation execution, ``nccom-test`` will verify each output tensor is filled with ``G``.\n\n``random`` mode should be preferred for more rigorous verification. However, for quicker, more easily understood verification, ``all_ones`` should be preferred.\n\n.. _MPI Execution:\nMPI Execution:\n~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``-s, --hosts``\n - N/A\n - Hosts on which to run execution.\n * - ``--hosts-file``\n - N/A\n - File containing hosts on which to run execution. One host specified per line.\n * - ``--mpi-log-dir``\n - N/A\n - If specified, logs from each node in ``mpi`` multi-node benchmark will be saved to a unique file within the specified directory\n\nTo use ``mpi`` mode, provide all hosts for your invocation, either with the ``--hosts`` flag or a ``~/hosts`` file, and set the ``NEURON_RT_ROOT_COMM_ID`` environment variable to the IP address of the first host listed and any free port.\nDepending on your environment, ``mpi`` may require passwordless SSH access to each host in your invocation. See the `Open MPI SSH documentation `_ for details.\n\nExample:\n\n``NEURON_RT_ROOT_COMM_ID=10.1.4.145:45654 nccom-test -r 64 -N 2 -d fp32 allr --hosts 10.1.4.145 10.1.4.138``\n\nThe above command will invoke a ``neuron-bench`` process on both hosts listed, to execute the collective operations, using 32 ranks from each host.\nLatency data will be reported back from each host and collected on the host on which the ``nccom-test`` command was invoked. \nThe host on which the ``nccom-test`` command is invoked should usually be one of the provided hosts, but it can be another unrelated host, as long as it can invoke MPI processes\non the provided hosts.\n\n\n.. _Slurm Execution:\nSlurm Execution:\n~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``-S, --slurm-mode``\n - false\n - Use ``srun`` to run benchmark on ``slurm``-based cluster\n * - ``-u, --slurm-vcpus-per-node``\n - Minimum CPU count amongst all nodes\n - Number of vCPUs available per node in ``slurm`` allocation\n * - ``--slurm-setup-script``\n - N/A\n - Script to run on each node in ``slurm`` allocation before executing benchmark. Can use ``default`` to run \n a default script installing the latest Neuron software.\n * - ``--slurm-job-id``\n - alloc\n - Specify jobId for ``slurm`` allocation to execute benchmark on. By default, will create a new allocation to execute benchmark on.\n * - ``--slurm-use-head-node-neuron-bench``\n - false\n - Copy ``neuron-bench`` binary from head node to all nodes in allocation\n\nTo use ``slurm`` mode, specify the ``--slurm-mode`` flag. When using slurm mode, ``nccom-test`` invocations should be run from the head node of the slurm cluster. \nUsers can either use an existing slurm job by providing a job id, or have ``nccom-test`` allocate one for you. \nAdditionally, users can provide a path to a setup script to run on each slurm node before execution. Users can alternatively specify ``default`` to use a supplied default setup script.\n\nExamples:\n\n``nccom-test -r 64 -N 2 allr --slurm-mode --slurm-setup-script path/to/my/custom-setup-script.sh``\n\nThe above command will execute collective operation across two nodes using slurm. Slurm will allocate a job with two nodes before beginning execution and will run the ``custom-setup-script.sh``\non each node before executing any collective operations.\n\n\n``nccom-test -r 64 -N 2 allr --slurm-mode --slurm-job-id 12345``\n\nThe above command will use an existing slurm allocation (``jobId: 12345``) with no setup.\n\n\nOutput:\n~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``--non-interactive``\n - false\n - Do not display any animation or progress indicator.\n * - ``--report-to-json-file``\n - N/A\n - Persist config and results to specified JSON file if a filepath is provided.\n * - ``-t, --stats``\n - avg\n - Latency (time) statistics to display in the final output. Currently supports ``avg`` and any percentile (e.g ``p15``, ``p50``, ``p90``).\n * - ``--show-algorithm``\n - false\n - Show which algorithm (e.g. Ring, Mesh, RDH) was used to execute the collective operation in ``nccom-test`` output.\n Currently, any hierarchical algorithms used will be displayed as ``hier``, and will not include any sub-algorithms.\n * - ``--show-input-output-size``\n - false\n - Print or save to JSON per rank input and output sizes in B.\n * - ``--debug``\n - false\n - Show debug logs from execution of ``nccom-test`` and ``neuron-bench`` in realtime. Enables ``non-interactive`` mode implicitly.\n\n\nSBUF Collectives:\n~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``--sb2sb``\n - false\n - Indicates whether to allocate input, output, and scratch-buffer on SBUF (rather than HBM). This may result in improved performance.\n * - ``--input-shape``\n - N/A\n - Provide input tensor dimensions in format: ``[step0,step1][num_elem0,num_elem1]``. ``step0/num_elem0`` correspond to the free dimension of the SBUF, while ``step1/num_elem1`` correspond to the partition dimension of the SBUF.\n * - ``--output-shape``\n - N/A\n - Provide output tensor dimensions in format: ``[step0,step1][num_elem0,num_elem1]``. ``step0/num_elem0`` correspond to the free dimension of the SBUF, while ``step1/num_elem1`` correspond to the partition dimension of the SBUF.\n * - ``--cc-dim``\n - 1\n - Control dimensions of tensor concatenation. Either concatenate tensor in free dimension (``cc-dim = 0``) or concatenate in partition dimension first and wrap around in free dimension second (``cc-dim = 1``)\n\n\nReplica Group:\n~~~~~~~~~~~~~~\n\nFlags to control which subset of ranks a collective operation will be executed on.\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n * - ``--data-parallel-dimension``\n - N/A\n - Run the given collective operation in parallel across multiple sub-groups of size ``data-parallel-dimension``. For 128 ranks and data parallel dimension of 2, \n there would be 64 parallel collective operations happening at the same time, each with 2 ranks. Primarily intended for multi-node executions with one-rank-per-node\n replica groups.\n * - ``--custom-replica-group``\n - N/A\n - Provide the JSON file for custom-defined replica groups.\n * - ``--custom-src-target-pairs``\n - N/A\n - Provide the JSON file for custom-defined source_target_pairs for the collective permute operation.\n\nAdditional Flags:\n~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Argument\n - Default value\n - Description\n \n * - ``--vcpu-pin-mode``\n - false\n - Pin CPU thread for each rank to a given CPU. \n * - ``--data-collector-port``\n - 60006\n - If running ``nccom-test`` in multi-node mode or on another node, a data collector is used to gather latencies from all nodes in benchmark.\n Port to use for data collector.\n * - ``--data-collector-host``\n - current host\n - Hostname or IP address of node to use as data collector, all latencies from other nodes will be sent to this host\n\nEnvironment Variables\n^^^^^^^^^^^^^^^^^^^^^\nIn addition to CLI arguments, there are also several environment variables which can be used to alter how collectives run inside ``nccom-test``\n\n.. list-table::\n :widths: 40 80 260\n :header-rows: 1\n\n * - Environment Variable\n - Default value\n - Description\n * - ``NEURON_LOGICAL_NC_CONFIG``\n - 2 for ``trn2`` and ``trn3``. 1 for ``inf2`` and ``trn1``\n - Controls how many physical NeuronCores are grouped to make up a logical NeuronCore.\n\nUsers may also find certain Neuron Runtime environment variables useful with ``nccom-test`` executions. See :ref:`nrt-configuration`\n\nExamples\n^^^^^^^^\n\n.. note::\n\n Performance data shown in these examples should not be considered up-to-date. For the latest performance\n data, please refer to the performance section.\n\n\nSingle Instance Examples\n~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Quick environment validation\n\n .. code-block::\n\n nccom-test -r 2 allr\n size(B) count(elems) type time(us) algbw(GB/s) busbw(GB/s)\n 33554432 33554432 uint8 768 40.69 40.69\n Avg bus bandwidth: 40.6901GB/s\n\n\n If a problem was found, it can be reported in two possible ways:\n\n - Immediately:\n\n .. code-block::\n\n nccom-test -r 2 allr\n Neuron DKMS Driver is not running! Read the troubleshooting guide at: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-troubleshoot.html#neuron-driver-installation-fails\n\n\n - After a benchmark attempt:\n\n .. code-block::\n\n nccom-test -r 2 allr\n size(B) count(elems) type time(us) algbw(GB/s) busbw(GB/s)\n 33554432 Failure running neuron-bench - log file /tmp/nccom_test_log_7pqpdfjf.log\n 1 errors found - test failed\n\n\n In this case, further information about the error can be found in the ``neuron-bench`` log file.\n\n- 2 rank all-reduce on a single instance for sizes ranging from 1MiB to 1GiB with a step of 4x\n\n .. code-block::\n\n nccom-test -r 2 --minbytes 1kb --maxbytes 1gb --stepfactor 4 --datatype fp32 allr\n size(B) count(elems) type time(us) algbw(GB/s) busbw(GB/s)\n 1024 256 fp32 58 0.02 0.02\n 4096 1024 fp32 58 0.07 0.07\n 16384 4096 fp32 58 0.26 0.26\n 65536 16384 fp32 58 1.05 1.05\n 262144 65536 fp32 60 4.07 4.07\n 1048576 262144 fp32 68 14.36 14.36\n 4194304 1048576 fp32 107 36.51 36.51\n 16777216 4194304 fp32 332 47.06 47.06\n 67108864 16777216 fp32 1214 51.48 51.48\n 268435456 67108864 fp32 4750 52.63 52.63\n 1073741824 268435456 fp32 18930 52.83 52.83\n Avg bus bandwidth: 23.6671GB/s\n\n\n- 32 rank all-gather on a single instance for sizes ranging from 1KiB to 1MiB with a step of 8x, with correctness checking\n\n\n.. code-block::\n\n nccom-test -r 32 --minbytes 1kb --maxbytes 1mb --stepfactor 8 --datatype fp32 --check allg\n size(B) count(elems) type time(us) algbw(GB/s) busbw(GB/s)\n 1024 256 fp32 151 0.01 0.01\n 8192 2048 fp32 149 0.05 0.05\n 65536 16384 fp32 150 0.41 0.39\n 524288 131072 fp32 179 2.73 2.64\n Avg bus bandwidth: 0.7731GB/s\n\n- Specify the custom source target pairs as a JSON file for the collective permute operator ``--custom-src-target-pairs``.\n\n.. code-block::\n\n nccom-test -r 8 --custom-src-target-pairs pairs.json permute\n size(B) count(elems) type time:avg(us) algbw(GB/s) busbw(GB/s)\n 33554432 33554432 uint8 894.24 37.52 37.52\n Avg bus bandwidth:\t37.5230GB/s\n\n cat pairs.json\n {\n \"src_target_pairs\": [\n [\n [0, 1],\n [1, 0],\n [2, 3],\n [3, 2],\n [4, 4],\n [5, 5],\n [6, 6],\n [7, 7]\n ]\n ]\n }\n\n\n- Reporting the input and output size explicitly with ``--show-input-output-size``.\n\n.. code-block::\n\n nccom-test -r 32 --minbytes 1kb --maxbytes 1mb --stepfactor 8 --datatype fp32 --check allg --show-input-output-size\n size(B) count(elems) total_input_size(B) total_output_size(B) type time:avg(us) algbw(GB/s) busbw(GB/s)\n 1024 256 32 1024 fp32 6.16 0.17 0.16\n 8192 2048 256 8192 fp32 6.48 1.26 1.23\n 65536 16384 2048 65536 fp32 8.17 8.02 7.77\n 524288 131072 16384 524288 fp32 23.16 22.64 21.93\n Avg bus bandwidth: 7.7715GB/s\n\n- Getting percentile latency results with ``--stats``\n\n.. code-block::\n\n nccom-test -r 8 --minbytes 1kb --maxbytes 1mb --stepfactor 8 --datatype fp32 --stats avg p25 p50 p90 p99 --iters 1000 allg\n size(B) count(elems) type time:avg(us) time:p25(us) time:p50(us) time:p90(us) time:p99(us) algbw(GB/s) busbw(GB/s)\n 1024 256 fp32 10.0 10 10 11 12 0.10 0.09\n 8192 2048 fp32 10.22 10 10 11 12 0.80 0.70\n 65536 16384 fp32 11.31 11 11 13 13 5.80 5.07\n 524288 131072 fp32 14.83 14 15 16 17 35.34 30.92\n Avg bus bandwidth:\t9.1966GB/s\n\n- Example results as JSON with ``--report-to-json-file``\n\n.. code-block::\n\n nccom-test -r 32 --minbytes 1kb --maxbytes 1mb --stepfactor 8 --datatype fp32 --check allg --report-to-json-file nccom-results.json\n size(B) count(elems) type time:avg(us) algbw(GB/s) busbw(GB/s)\n 1024 256 fp32 6.19 0.17 0.16\n 8192 2048 fp32 6.55 1.25 1.21\n 65536 16384 fp32 8.18 8.01 7.76\n 524288 131072 fp32 23.11 22.69 21.98\n Avg bus bandwidth: 7.7775GB/s\n\n python3 -m json.tool nccom-results.json\n {\n \"results\": [\n {\n \"size(B)\": 1024,\n \"count(elems)\": 256,\n \"type\": \"fp32\",\n \"algbw(GB/s)\": 0.16553675170497603,\n \"busbw(GB/s)\": 0.16036372821419553,\n \"time:avg(us)\": 6.19\n },\n {\n \"size(B)\": 8192,\n \"count(elems)\": 2048,\n \"type\": \"fp32\",\n \"algbw(GB/s)\": 1.2500906056270864,\n \"busbw(GB/s)\": 1.21102527420124,\n \"time:avg(us)\": 6.55\n },\n {\n \"size(B)\": 65536,\n \"count(elems)\": 16384,\n \"type\": \"fp32\",\n \"algbw(GB/s)\": 8.008982241741455,\n \"busbw(GB/s)\": 7.758701546687035,\n \"time:avg(us)\": 8.18\n },\n {\n \"size(B)\": 524288,\n \"count(elems)\": 131072,\n \"type\": \"fp32\",\n \"algbw(GB/s)\": 22.688776793562784,\n \"busbw(GB/s)\": 21.97975251876395,\n \"time:avg(us)\": 23.11\n }\n ]\n }\n\n- Example results with ``--show-algorithm`` flag\n\n.. code-block::\n\n nccom-test -r 16 allr -b 4 -e 1gb -f 16 -d fp32 --show-algorithm\n size(B) count(elems) type time:avg(us) algbw(GB/s) busbw(GB/s) algorithm\n 4 1 fp32 299.91 0.00 0.00 mesh\n 32 8 fp32 299.69 0.00 0.00 mesh\n 512 128 fp32 299.82 0.00 0.00 mesh\n 8192 2048 fp32 299.74 0.03 0.05 mesh\n 131072 32768 fp32 574.15 0.23 0.43 mesh\n 2097152 524288 fp32 686.32 3.06 5.73 rdh\n 33554432 8388608 fp32 2754.15 12.18 22.84 kangaring\n 536870912 134217728 fp32 9689.51 55.41 103.89 kangaring\n Avg bus bandwidth:\t16.6181GB/s\n\n\nMultiple Instances Example\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- 64 rank all-reduce on two instances for sizes ranging from 8 bytes to 1GiB with a step of 2x, running 50 ops\n\n .. code-block::\n\n NEURON_RT_ROOT_COMM_ID=10.1.4.145:45654 nccom-test -r 64 -N 2 -b 8 -e 1GB -f 2 -n 50 -w 5 -d fp32 allr --hosts 127.0.0.1 10.1.4.138\n size(B) count(elems) type time(us) algbw(GB/s) busbw(GB/s)\n 8 2 fp32 520 0.00 0.00\n 16 4 fp32 520 0.00 0.00\n 32 8 fp32 523 0.00 0.00\n 64 16 fp32 525 0.00 0.00\n 128 32 fp32 553 0.00 0.00\n 256 64 fp32 709 0.00 0.00\n 512 128 fp32 782 0.00 0.00\n 1024 256 fp32 840 0.00 0.00\n 2048 512 fp32 881 0.00 0.00\n 4096 1024 fp32 916 0.00 0.01\n 8192 2048 fp32 1013 0.01 0.01\n 16384 4096 fp32 1031 0.01 0.03\n 32768 8192 fp32 1174 0.03 0.05\n 65536 16384 fp32 1315 0.05 0.09\n 131072 32768 fp32 1315 0.09 0.18\n 262144 65536 fp32 1311 0.19 0.37\n 524288 131072 fp32 1312 0.37 0.73\n 1048576 262144 fp32 1328 0.74 1.45\n 2097152 524288 fp32 1329 1.47 2.89\n 4194304 1048576 fp32 1378 2.83 5.58\n 8388608 2097152 fp32 1419 5.51 10.84\n 16777216 4194304 fp32 2138 7.31 14.39\n 33554432 8388608 fp32 2711 11.53 22.69\n 67108864 16777216 fp32 3963 15.77 31.05\n 134217728 33554432 fp32 6279 19.91 39.19\n 268435456 67108864 fp32 11954 20.91 41.17\n 536870912 134217728 fp32 21803 22.93 45.15\n 1073741824 268435456 fp32 41806 23.92 47.09\n Avg bus bandwidth: 9.3924GB/s\n\n\n.. _AlltoAllV Example:\n- Specify alltoallv-metadata as JSON for ``alltoallv`` operation ``--alltoallv-metadata``.\n.. code-block::\n\n NEURON_RT_ROOT_COMM_ID=172.32.137.79:44444 nccom-test -r 2 -N 2 -d fp32 alltoallv -b 1MB -e 1MB --hosts 127.0.0.1 172.32.253.16 --alltoallv-metadata alltoallv_metadata.json\n size(B) count(elems) type time:avg(us) algbw(GB/s) busbw(GB/s)\n 1048608 262152 fp32 955.05 1.10 0.55\n Avg bus bandwidth:\t0.5490GB/s\n\n cat alltoallv_metadata.json\n {\n \"alltoallv_metadata\": [\n {\n \"send_counts\": [512, 1024],\n \"send_displs\": [0, 512],\n \"recv_counts\": [256, 768],\n \"recv_displs\": [0, 256]\n }\n ]\n }\n" }, { "path": "tools/neuron-sys-tools/neuron-ls.rst", "content": ".. _neuron-ls-ug:\n\nNeuron LS User Guide\n---------------------\n\nThe neuron-ls command is a tool for managing Neuron devices in your instance.\nThis command serves two key purposes: it identifies all Neuron devices present in the current instance \nand provides information about the processes running on each device along with the command that launched that process.\nTo use this command, simply type ``neuron-ls`` in your terminal.\n\n.. rubric:: neuron-ls CLI\n\n.. code-block:: text\n\n neuron-ls [options]\n\n**Options**\n\n``--wide, -w``\n Displays the table in a wider format.\n\n``--show-all-procs, -a``\n Show all processes using the Neuron Devices, including processes that aren't using\n Neuron Runtime 2.x such as ``neuron-monitor`` or ``neuron-ls`` itself.\n\n``--topology, -t``\n Display topology information about the system's Neuron Devices.\n\n``--json-output, -j``\n Output in JSON format.\n\n.. note::\n\n ``neuron-ls`` fully supports the newly launched Trn2 instances.\n\nExamples\n^^^^^^^^\n\n``neuron-ls`` is compatible with all Neuron instance types: inf1, inf2, trn1 and trn2.\nThese are a few examples on running the tool on a trn2n.48xlarge:\n\n::\n\n $ neuron-ls\n instance-type: trn2n.48xlarge\n instance-id: i-aabbccdd123456789\n logical-neuroncore-config: 2\n +--------+--------+----------+--------+---------------+--------------+---------------+------+\n | NEURON | NEURON | NEURON | NEURON | CONNECTED | PCI | CPU | NUMA |\n | DEVICE | CORES | CORE IDS | MEMORY | DEVICES | BDF | AFFINITY | NODE |\n +--------+--------+----------+--------+---------------+--------------+---------------+------+\n | 0 | 4 | 0-3 | 96 GB | 12, 3, 4, 1 | 0000:cc:00.0 | 48-95,144-191 | 1 |\n | 1 | 4 | 4-7 | 96 GB | 13, 0, 5, 2 | 0000:b5:00.0 | 48-95,144-191 | 1 |\n | 2 | 4 | 8-11 | 96 GB | 14, 1, 6, 3 | 0000:b6:00.0 | 48-95,144-191 | 1 |\n | 3 | 4 | 12-15 | 96 GB | 15, 2, 7, 0 | 0000:cb:00.0 | 48-95,144-191 | 1 |\n | 4 | 4 | 16-19 | 96 GB | 0, 7, 8, 5 | 0000:6f:00.0 | 0-47,96-143 | 0 |\n | 5 | 4 | 20-23 | 96 GB | 1, 4, 9, 6 | 0000:58:00.0 | 0-47,96-143 | 0 |\n | 6 | 4 | 24-27 | 96 GB | 2, 5, 10, 7 | 0000:59:00.0 | 0-47,96-143 | 0 |\n | 7 | 4 | 28-31 | 96 GB | 3, 6, 11, 4 | 0000:6e:00.0 | 0-47,96-143 | 0 |\n | 8 | 4 | 32-35 | 96 GB | 4, 11, 12, 9 | 0000:9b:00.0 | 0-47,96-143 | 0 |\n | 9 | 4 | 36-39 | 96 GB | 5, 8, 13, 10 | 0000:84:00.0 | 0-47,96-143 | 0 |\n | 10 | 4 | 40-43 | 96 GB | 6, 9, 14, 11 | 0000:85:00.0 | 0-47,96-143 | 0 |\n | 11 | 4 | 44-47 | 96 GB | 7, 10, 15, 8 | 0000:9a:00.0 | 0-47,96-143 | 0 |\n | 12 | 4 | 48-51 | 96 GB | 8, 15, 0, 13 | 0000:f8:00.0 | 48-95,144-191 | 1 |\n | 13 | 4 | 52-55 | 96 GB | 9, 12, 1, 14 | 0000:e1:00.0 | 48-95,144-191 | 1 |\n | 14 | 4 | 56-59 | 96 GB | 10, 13, 2, 15 | 0000:e2:00.0 | 48-95,144-191 | 1 |\n | 15 | 4 | 60-63 | 96 GB | 11, 14, 3, 12 | 0000:f7:00.0 | 48-95,144-191 | 1 |\n +--------+--------+----------+--------+---------------+--------------+---------------+------+\n\n::\n\n $ neuron-ls --wide\n instance-type: trn2n.48xlarge\n instance-id: i-aabbccdd123456789\n logical-neuroncore-config: 2\n +--------+--------+--------+---------------+---------+--------+----------------------------------------------------------------------------------+---------+\n | NEURON | NEURON | NEURON | CONNECTED | PCI | PID | COMMAND | RUNTIME |\n | DEVICE | CORES | MEMORY | DEVICES | BDF | | | VERSION |\n +--------+--------+--------+---------------+---------+--------+----------------------------------------------------------------------------------+---------+\n | 0 | 4 | 96 GB | 12, 3, 4, 1 | cc:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 1 | 4 | 96 GB | 13, 0, 5, 2 | b5:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 2 | 4 | 96 GB | 14, 1, 6, 3 | b6:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 3 | 4 | 96 GB | 15, 2, 7, 0 | cb:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 4 | 4 | 96 GB | 0, 7, 8, 5 | 6f:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 5 | 4 | 96 GB | 1, 4, 9, 6 | 58:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 6 | 4 | 96 GB | 2, 5, 10, 7 | 59:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 7 | 4 | 96 GB | 3, 6, 11, 4 | 6e:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 8 | 4 | 96 GB | 4, 11, 12, 9 | 9b:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 9 | 4 | 96 GB | 5, 8, 13, 10 | 84:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 10 | 4 | 96 GB | 6, 9, 14, 11 | 85:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 11 | 4 | 96 GB | 7, 10, 15, 8 | 9a:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 12 | 4 | 96 GB | 8, 15, 0, 13 | f8:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 13 | 4 | 96 GB | 9, 12, 1, 14 | e1:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 14 | 4 | 96 GB | 10, 13, 2, 15 | e2:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n | 15 | 4 | 96 GB | 11, 14, 3, 12 | f7:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --warmup none --fixed-instance-count 64 --... | 2.0.0 |\n +--------+--------+--------+---------------+---------+--------+----------------------------------------------------------------------------------+---------+\n\n::\n\n $ neuron-ls --show-all-procs\n instance-type: trn2n.48xlarge\n instance-id: i-aabbccdd123456789\n logical-neuroncore-config: 2\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | NEURON | NEURON | NEURON | CONNECTED | PCI | PID | COMMAND | RUNTIME |\n | DEVICE | CORES | MEMORY | DEVICES | BDF | | | VERSION |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 0 | 4 | 96 GB | 12, 3, 4, 1 | cc:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 1 | 4 | 96 GB | 13, 0, 5, 2 | b5:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 2 | 4 | 96 GB | 14, 1, 6, 3 | b6:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 3 | 4 | 96 GB | 15, 2, 7, 0 | cb:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 4 | 4 | 96 GB | 0, 7, 8, 5 | 6f:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 5 | 4 | 96 GB | 1, 4, 9, 6 | 58:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 6 | 4 | 96 GB | 2, 5, 10, 7 | 59:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 7 | 4 | 96 GB | 3, 6, 11, 4 | 6e:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 8 | 4 | 96 GB | 4, 11, 12, 9 | 9b:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 9 | 4 | 96 GB | 5, 8, 13, 10 | 84:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 10 | 4 | 96 GB | 6, 9, 14, 11 | 85:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 11 | 4 | 96 GB | 7, 10, 15, 8 | 9a:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 12 | 4 | 96 GB | 8, 15, 0, 13 | f8:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 13 | 4 | 96 GB | 9, 12, 1, 14 | e1:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 14 | 4 | 96 GB | 10, 13, 2, 15 | e2:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n | 15 | 4 | 96 GB | 11, 14, 3, 12 | f7:00.0 | 268911 | neuron-bench exec --run-as-cc-neff --... | 2.0.0 |\n | | | | | | 269192 | neuron-ls --show-all-procs | NA |\n +--------+--------+--------+---------------+---------+--------+------------------------------------------+---------+\n\n::\n\n $ neuron-ls --topology\n instance-type: trn2n.48xlarge\n instance-id: i-aabbccdd123456789\n logical-neuroncore-config: 2\n +--------+--------+--------+---------------+---------+\n | NEURON | NEURON | NEURON | CONNECTED | PCI |\n | DEVICE | CORES | MEMORY | DEVICES | BDF |\n +--------+--------+--------+---------------+---------+\n | 0 | 4 | 96 GB | 12, 3, 4, 1 | cc:00.0 |\n | 1 | 4 | 96 GB | 13, 0, 5, 2 | b5:00.0 |\n | 2 | 4 | 96 GB | 14, 1, 6, 3 | b6:00.0 |\n | 3 | 4 | 96 GB | 15, 2, 7, 0 | cb:00.0 |\n | 4 | 4 | 96 GB | 0, 7, 8, 5 | 6f:00.0 |\n | 5 | 4 | 96 GB | 1, 4, 9, 6 | 58:00.0 |\n | 6 | 4 | 96 GB | 2, 5, 10, 7 | 59:00.0 |\n | 7 | 4 | 96 GB | 3, 6, 11, 4 | 6e:00.0 |\n | 8 | 4 | 96 GB | 4, 11, 12, 9 | 9b:00.0 |\n | 9 | 4 | 96 GB | 5, 8, 13, 10 | 84:00.0 |\n | 10 | 4 | 96 GB | 6, 9, 14, 11 | 85:00.0 |\n | 11 | 4 | 96 GB | 7, 10, 15, 8 | 9a:00.0 |\n | 12 | 4 | 96 GB | 8, 15, 0, 13 | f8:00.0 |\n | 13 | 4 | 96 GB | 9, 12, 1, 14 | e1:00.0 |\n | 14 | 4 | 96 GB | 10, 13, 2, 15 | e2:00.0 |\n | 15 | 4 | 96 GB | 11, 14, 3, 12 | f7:00.0 |\n +--------+--------+--------+---------------+---------+\n\n\n Neuron Device Topology\n * * * * \n │ │ │ │ \n ▼ ▼ ▼ ▼ \n *––►[ 0 ]◄––►[ 1 ]◄––►[ 2 ]◄––►[ 3 ]◄––*\n ▲ ▲ ▲ ▲ \n │ │ │ │ \n ▼ ▼ ▼ ▼ \n *––►[ 4 ]◄––►[ 5 ]◄––►[ 6 ]◄––►[ 7 ]◄––*\n ▲ ▲ ▲ ▲ \n │ │ │ │ \n ▼ ▼ ▼ ▼ \n *––►[ 8 ]◄––►[ 9 ]◄––►[10 ]◄––►[11 ]◄––*\n ▲ ▲ ▲ ▲ \n │ │ │ │ \n ▼ ▼ ▼ ▼ \n *––►[12 ]◄––►[13 ]◄––►[14 ]◄––►[15 ]◄––*\n ▲ ▲ ▲ ▲ \n │ │ │ │ \n * * * * \n\n Legend:\n\n *––► = Wrap-around link\n\n::\n\n $ neuron-ls -j\n [\n {\n \"neuron_device\": 0,\n \"bdf\": \"cc:00.0\",\n \"cpu_affinity\": \"48-95,144-191\",\n \"numa_node\": \"1\",\n \"connected_to\": [\n 12,\n 3,\n 4,\n 1\n ],\n \"nc_count\": 4,\n \"logical_neuroncore_config\": 2,\n \"memory_size\": 103079215104,\n \"neuroncore_ids\": [\n 0,\n 1,\n 2,\n 3\n ],\n \"neuron_processes\": [\n {\n \"pid\": 113985,\n \"command\": \"neuron-bench exec --run-as-cc-neff --...\",\n \"neuron_runtime_version\": \"2.0.0\"\n }\n ]\n },\n ...\n {\n \"neuron_device\": 15,\n \"bdf\": \"f7:00.0\",\n \"cpu_affinity\": \"48-95,144-191\",\n \"numa_node\": \"1\",\n \"connected_to\": [\n 11,\n 14,\n 3,\n 12\n ],\n \"nc_count\": 4,\n \"logical_neuroncore_config\": 2,\n \"memory_size\": 103079215104,\n \"neuroncore_ids\": [\n 60,\n 61,\n 62,\n 63\n ],\n \"neuron_processes\": [\n {\n \"pid\": 113985,\n \"command\": \"neuron-bench exec --run-as-cc-neff --...\",\n \"neuron_runtime_version\": \"2.0.0\"\n }\n ]\n }\n ]\n\nField Definitions\n^^^^^^^^^^^^^^^^^\n\n- instance-type: Type of instance on which neuron-ls is running.\n- instance-id: EC2 ID of the instance on which neuron-ls is running.\n- logical-neuroncore-config: (only available on trn2 instances) the current logical NeuronCore configuration; for more information refer to :ref:`logical-neuroncore-config`\n- NEURON DEVICE / neuron_device: Logical ID assigned to the Neuron Device.\n- NEURON CORES / nc_count: Number of NeuronCores present in the Neuron Device.\n- NEURON CORE IDS / neuroncore_ids: Range or list of individual NeuronCore IDs belonging to the device, used with ``NEURON_RT_VISIBLE_CORES`` for selective core usage.\n- NEURON MEMORY / memory_size: Amount DRAM memory in Neuron Device.\n- CONNECTED DEVICES / connected_to: Logical ID of Neuron Devices connected to this\n Neuron Device.\n- PCI BDF / bdf: PCI Bus Device Function (BDF) ID of the device.\n- CPU AFFINITY / cpu_affinity: CPU cores that per NeuronCore proxy threads are pinned to\n- NUMA NODE / numa_node: NUMA (Non-Uniform Memory Access) node associated with the Neuron Device\n- PID / pid: ID of the process using this Neuron Device.\n- COMMAND / command: Command used to launch the process using this\n Neuron Device.\n- RUNTIME VERSION / neuron_runtime_version: Version of Neuron Runtime (if applicable) for\n the application using this Neuron Device.\n" }, { "path": "tools/neuron-sys-tools/neuron-monitor-user-guide.rst", "content": ".. _neuron-monitor-ug:\n\nNeuron Monitor User Guide\n=========================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nOverview\n--------\n\n**neuron-monitor** collects metrics and stats from the Neuron\nApplications running on the system and streams the collected data to\n``stdout`` in ``JSON`` format. It is provided as part of the\n``aws-neuron-tools`` package.\n\nThese metrics and stats are organized into **metric groups** which can\nbe configured by providing a configuration file as described in :ref:`using-neuron-monitor`\n\nWhen running, **neuron-monitor** will:\n\n- Collect the data for the metric groups which, based on the elapsed\n time since their last update, need to be updated\n- Take the newly collected data and consolidate it into a large report\n- Serialize that report to JSON and stream it to stdout from where it\n can be consumed by other tools - such as the sample\n :ref:`neuron-monitor-cloudwatch.py ` and\n :ref:`neuron-monitor-prometheus.py `\n scripts.\n- Wait until at least one **metric group** needs to be collected and\n repeat this flow\n\n.. note::\n\n ``neuron-monitor`` fully supports the newly launched Trn2 instances.\n\n.. _using-neuron-monitor:\n\nUsing neuron-monitor\n--------------------\n\n.. _monitor_cli:\n\n.. rubric:: neuron-monitor CLI\n\n.. program:: neuron-monitor\n\n.. option:: neuron-monitor [parameters]\n\n neuron-monitor accepts the following optional parameters:\n\n - ``--verbose`` (int) default=0: Can be 0 to 4, and controls the amount of\n debugging and verbose information sent to stderr; **0: no output**,\n **4: maximum verbosity**\n\n - ``-c, --config-file`` (string): Allows specifying a valid path to a\n neuron-monitor JSON configuration file\n\n\n**Example:**\n\n.. code-block::\n\n neuron-monitor -c monitor.conf\n\n\nNot specifying any configuration file will enable collecting all the metric groups\nwith a period of 5 seconds for all currently running Neuron applications.\n\nConfiguration file example\n~~~~~~~~~~~~~~~~~~~~~~~~~~\nExample of a configuration file which enables all available **metric\ngroups** for every running Neuron application, with a global update period of 1\nsecond and sets an update period of 2 seconds for the ``\"neuron_hw_counters\"``\nmetric group:\n\n::\n\n {\n \"period\": \"1s\",\n \"neuron_runtimes\": [\n {\n \"tag_filter\": \".*\",\n \"metrics\": [\n {\n \"type\": \"neuroncore_counters\"\n },\n {\n \"type\": \"memory_used\"\n },\n {\n \"type\": \"neuron_runtime_vcpu_usage\"\n },\n {\n \"type\": \"execution_stats\"\n }\n ]\n }\n ],\n \"system_metrics\": [\n {\n \"type\": \"vcpu_usage\"\n },\n {\n \"type\": \"memory_info\"\n },\n {\n \"period\": \"2s\",\n \"type\": \"neuron_hw_counters\"\n }\n ]\n }\n\nNeuron applications tagging\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\nIn order to make application monitoring easier, Neuron applications can be tagged with a 255 character\nstring which identifies that app. Tagging is done using the ``NEURON_PROCESS_TAG`` environment variable.\n\nFor example:\n``NEURON_PROCESS_TAG=my_app_1 python training.py`` will associate the ``my_app_1`` tag with that Python application.\nIf ``NEURON_PROCESS_TAG`` is not specified, the application's PID will be used as a TAG.\n\nThis tag will be used by neuron-monitor to filter Neuron applications.\n\nJSON objects and fields in the configuration file\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- ``\"neuron_runtimes\"`` - array of objects specifying which Neuron\n Applications to monitor and what metric groups are enabled for each\n of them\n\n - ``\"tag_filter\"`` - a regex which will be used to filter Neuron applications tags\n in order to determine if they will be monitored (optional)\n - ``\"metrics\"`` - array of objects specifying which metric groups to\n capture for this Neuron application\n\n - ``\"type\"`` - type of metric group\n\n- ``\"period\"`` - this field applies to **metric group** objects and\n sets the amount of time between two updates for that metric group\n\n - if can be specified as part of the **root** and/or\n **neuron_runtime** objects where it applies to all their children,\n and/or as part of a **metric group** object\n - if there's no period specified, a default value of **5 seconds**\n will be used\n\n- ``\"system_metrics\"`` - array of objects specifying which system level\n metric groups are enabled\n\nNeuron Runtime-level metric groups\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- :ref:`neuron-monitor-nc-counters` - NeuronCore related metrics\n- :ref:`neuron-monitor-memory-used` - data on the amount of memory used\n by the Neuron application\n- :ref:`neuron-monitor-vcpu-usage` - Neuron application vCPU\n utilization data\n- :ref:`neuron-monitor-execution-stats` - Neuron application execution\n stats, including error count and latency\n\nSystem-wide metric groups\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- :ref:`neuron-monitor-vcpu-usage` - system-wide vCPU usage\n- :ref:`neuron-monitor-memory-info` - system-wide memory usage\n- :ref:`neuron-monitor-hw-counters` - counters for correctable and\n uncorrectable memory ecc events\n\n\nExecution model\n---------------\n\n|image|\n\nneuron-monitor waits for one or more **metric groups** to be up for\nupdate, then collects the corresponding data, consolidates it into a\nreport which is streamed to stdout as a JSON and goes back to waiting.\n\nThe JSON output format\n----------------------\n\nWhenever the report gets updated, a complete JSON is written to stdout.\nThis is its structure:\n\n::\n\n {\n \"neuron_runtime_data\": [\n {\n \"pid\": 0,\n \"address\": \"\",\n \"neuron_runtime_tag\", \"my_app_1\",\n \"error\": \"\",\n \"report\": {\n \"neuroncore_counters\": {\n [...]\n },\n \"execution_stats\": {\n [...]\n },\n \"memory_used\": {\n [...]\n },\n \"neuron_runtime_vcpu_usage\": {\n [...]\n }\n }\n }\n ],\n \"system_data\": {\n \"neuron_hw_counters\": {\n [...]\n },\n \"vcpu_usage\": {\n [...]\n },\n \"memory_info\": {\n [...]\n }\n },\n \"instance_info\": {\n [...]\n },\n \"neuron_hardware_info\": {\n [...]\n },\n \"neuron_k8s_info\": {\n [...]\n }\n }\n\n- ``\"neuron_runtime_data\"`` is an array containing one entry per each\n Neuron application which passes the filter specified in the settings file\n\n - ``\"pid\"`` is the pid of this Neuron application\n - ``\"neuron_runtime_tag\"`` is the configured tag for the Neuron application\n - ``\"error\"`` specifies any error that occurred when collecting data\n from this Neuron application\n - ``\"report\"`` will contain the results for the Neuron application-level\n metric groups; their formats are described below\n\n- ``\"system_data\"`` has a similar structure to ``\"neuron_runtime_data\"``‘s\n ``\"report\"`` but only contains system-level metric groups (not\n associated to any Neuron application)\n\nRegardless of the configuration, the following two JSON objects are always present\nin the output:\n\n.. _neuron-monitor-instance-info:\n\ninstance_info\n~~~~~~~~~~~~~\n\nContains information about the instance on which neuron-monitor is running.\n::\n\n \"instance_info\": {\n \"instance_name\": \"My_Instance\",\n \"instance_id\": \"i-0011223344556677a\",\n \"instance_type\": \"trn2n.48xlarge\",\n \"instance_availability_zone\": \"us-west-2b\",\n \"instance_availability_zone_id\": \"usw2-az2\",\n \"instance_region\": \"us-west-2\",\n \"ami_id\": \"ami-0011223344556677b\",\n \"subnet_id\": \"subnet-112233ee\",\n \"error\": \"\"\n }\n\nDepending on when the instance was launched, the following fields might\nnot be available:\n\n- ``instance_availability_zone_id`` : available only for instances\n launched in 2020-08-24 and later\n- ``instance_region`` : available only for instances launched on\n 2020-08-24 and later\n- ``instance_name`` : available only if ``instance_region`` is set and\n aws-cli tools are installed\n\n``error`` will contain an error string if getting one of the fields,\n**except those mentioned above**, resulted in an error.\n\n.. _neuron-monitor-hardware-info:\n\nneuron_hardware_info\n~~~~~~~~~~~~~~~~~~~~\n\nContains basic information about the Neuron hardware.\n::\n\n \"neuron_hardware_info\": {\n \"neuron_device_type\": \"trainium2\",\n \"neuron_device_version\": \"v4\",\n \"neuroncore_version\": \"v3d\",\n \"neuron_device_count\": 16,\n \"neuron_device_memory_size\": 103079215104,\n \"neuroncore_per_device_count\": 4,\n \"logical_neuroncore_config\": 2,\n \"error\": \"\"\n }\n\n- ``neuron_device_type``: type of the Neuron Devices on the instance\n- ``neuroncore_version``: version of the NeuronCores on the instance\n- ``neuron_device_count`` : number of available Neuron Devices\n- ``neuron_device_memory_size``: total memory available on each Neuron Device\n- ``neuroncore_per_device_count`` : number of NeuronCores present on each Neuron Device\n- ``logical_neuroncore_config`` : the current Logical NeuronCore configuration\n- ``error`` : will contain an error string if any occurred when getting this information\n (usually due to the Neuron Driver not being installed or not running).\n\nThe following JSON object is disabled by default, but can be made available if \"k8s_info\" is enabled:\n\n.. _neuron-monitor-k8s-info:\n\nneuron_k8s_info\n~~~~~~~~~~~~~~~\n\nContains information about what Kubernetes pods/containers are using Neuron resources\n::\n\n \"neuron_k8s_info\": {\n \"period\": 15.030359284,\n \"neuroncores_k8s_info\": {\n \"0\": {\n \"pod_name\": \"p0\",\n \"namespace\": \"n0\",\n \"container_name\": [\"c0\"]\n },\n \"1\": {\n \"pod_name\": \"p0\",\n \"namespace\": \"n0\",\n \"container_name\": [\"c0\"]\n },\n ...\n \"neurondevices_k8s_info\": {\n \"0\": {\n \"pod_name\": \"p0\",\n \"namespace\": \"n0\",\n \"container_name\": [\"c0\"]\n },\n ...\n }\n \"error\": \"\"\n },\n\n- ``\"neuroncores_k8s_info\"`` - object containing information on which\n Neuron cores are being used by Kubernetes pod/containers, indexed by\n Neuron core index: ``\"neuroncore_index\": { neuroncore_k8s_data }``\n\n - ``\"pod_name\"`` - name of pod using Neuron core\n - ``\"namespace\"`` - namespace of pod using Neuron core\n - ``\"container_name\"`` - names of containers using Neuron core\n\n- ``\"neurondevices_k8s_info\"`` - object containing information on which\n Neuron devices are being used by Kubernetes pod/containers, indexed by\n Neuron device index: ``\"neurondevice_index\": { neurondevice_k8s_data }``\n\n - ``\"pod_name\"`` - name of pod using Neuron device\n - ``\"namespace\"`` - namespace of pod using Neuron device\n - ``\"container_name\"`` - names of containers using Neuron device\n\n- ``\"error\"`` - will contain an error string if any occurred when getting this information\n\nFor more information on how to enable K8s information, see :ref:`neuron-monitor-k8s-infopy`.\n\n.. _neuron-metric-groups:\n\nMetric Groups\n~~~~~~~~~~~~~\n\nEach **metric group** requested in the settings file will get an entry\nin the resulting output. The general format for such an entry is:\n\n::\n\n \"metric_group\": {\n \"period\": 1.015, // Actual captured period, in seconds\n \"error\": \"\", // Error, if any occurred, otherwise an empty string\n [...] // Metric group specific data\n }\n\n.. _runtime-level-metric-groups-1:\n\nNeuron application level metric groups\n--------------------------------------\n\n.. _neuron-monitor-nc-counters:\n\nneuroncore_counters\n~~~~~~~~~~~~~~~~~~~\n\n::\n\n \"neuroncore_counters\": {\n \"period\": 1.000113182,\n \"neuroncores_in_use\": {\n \"0\": {\n \"neuroncore_utilization\": 42.01,\n \"flops\": 1234567891011,\n \"v3d\": {\n \"nc_v3.0\": {\n \"neuroncore_utilization\": 21.01\n },\n \"nc_v3.1\": {\n \"neuroncore_utilization\": 63.01\n }\n }\n },\n \"1\": {\n \"neuroncore_utilization\": 42.02,\n \"flops\": 1234567891021,\n \"v3d\": {\n \"nc_v3.2\": {\n \"neuroncore_utilization\": 21.02\n },\n \"nc_v3.3\": {\n \"neuroncore_utilization\": 63.02\n }\n }\n },\n [...]\n },\n \"error\": \"\"\n }\n\n- ``\"neuroncores_in_use\"`` is an object containing data for all the\n NeuronCores that were active when the data was captured, indexed by\n NeuronCore index: ``\"neuroncore_index\": { neuroncore_data }``\n\n - ``\"neuroncore_utilization\"`` - NeuronCore utilization, in percent,\n during the captured period\n - ``\"flops\"`` - number of floating point operations per second during\n the captured period\n - ``\"v3d\"`` - only available on Trn2 - contains the utilization for every\n physical NeuronCore that makes up the current NeuronCore\n\n- ``\"error\"`` - string containing any error that occurred when\n collecting the data\n\n.. _neuron-monitor-execution-stats:\n\nexecution_stats\n~~~~~~~~~~~~~~~\n\n::\n\n \"execution_stats\": {\n \"period\": 1.030613214,\n \"error_summary\": {\n \"generic\": 0,\n \"numerical\": 0,\n \"transient\": 0,\n \"model\": 0,\n \"runtime\": 0,\n \"hardware\": 0\n },\n \"execution_summary\": {\n \"completed\": 123,\n \"completed_with_err\": 0,\n \"completed_with_num_err\": 0,\n \"timed_out\": 0,\n \"incorrect_input\": 0,\n \"failed_to_queue\": 0\n },\n \"latency_stats\": {\n \"total_latency\": {\n \"p0\": 0.01100001,\n \"p1\": 0.01100002,\n \"p25\": 0.01100004,\n \"p50\": 0.01100008,\n \"p75\": 0.01100010,\n \"p99\": 0.01100012,\n \"p100\": 0.01100013\n },\n \"device_latency\": {\n \"p0\": 0.01000001,\n \"p1\": 0.01000002,\n \"p25\": 0.01000004,\n \"p50\": 0.01000008,\n \"p75\": 0.01000010,\n \"p99\": 0.01000012,\n \"p100\": 0.01000013\n }\n },\n \"error\": \"\"\n },\n\n- ``\"error_summary\"`` is an object containing the error counts for the\n captured period indexed by their type\n\n - ``\"generic\"`` - generic execution errors\n - ``\"numeric\"`` - NAN errors encountered during execution\n - ``\"transient\"`` - recoverable errors, such as ECC corrections\n - ``\"model\"`` - model-related errors\n - ``\"runtime\"`` - Neuron Runtime errors\n - ``\"hardware\"`` - hardware errors such as uncorrectable ECC issues\n\n- ``\"execution_summary\"`` is an object containing all execution outcome\n counts for the captured period indexed by their type\n\n - ``\"completed\"`` - executions completed successfully\n - ``\"completed_with_err\"`` - executions that ended in an error other\n than a numeric error\n - ``\"completed_with_num_err\"`` - executions that ended in a numeric\n error\n - ``\"timed_out\"`` - executions that took longer than the Neuron\n Runtime configured timeout value\n - ``\"incorrect_input\"`` - executions that failed to start due to\n incorrect input being provided\n - ``\"failed_to_queue\"`` - execution requests that were rejected due\n to Neuron Runtime not being able to queue them\n\n- ``\"latency_stats\"`` contains two objects containing latency\n percentiles, in seconds, for the data captured for the model\n executed during the captured period. If there are no models being\n executed during this time, the two objects will be ``null`` (i.e.\n ``\"total_latency\": null``)\n\n - ``\"total_latency\"`` - percentiles, in seconds, representing latency for an execution as measured by the Neuron Runtime\n - ``\"device_latency\"`` - percentiles, in seconds, representing execution time exclusively on the Neuron Device\n\n- ``\"error\"`` - string containing any error that occurred when\n collecting the data\n\n\n.. _neuron-monitor-memory-used:\n\nmemory_used\n~~~~~~~~~~~\n\n::\n\n \"memory_used\": {\n \"period\": 1.00001,\n \"neuron_runtime_used_bytes\": {\n \"host\": 6997643264,\n \"neuron_device\": 12519788544,\n \"usage_breakdown\": {\n \"host\": {\n \"application_memory\": 6996594688,\n \"constants\": 0,\n \"dma_buffers\": 1048576,\n \"tensors\": 0\n },\n \"neuroncore_memory_usage\": {\n \"0\": {\n \"constants\": 193986816,\n \"model_code\": 176285056,\n \"model_shared_scratchpad\": 0,\n \"runtime_memory\": 0,\n \"tensors\": 20971520\n },\n \"1\": {\n \"constants\": 193986816,\n \"model_code\": 176285056,\n \"model_shared_scratchpad\": 0,\n \"runtime_memory\": 0,\n \"tensors\": 20971520\n },\n ...\n }\n }\n \"loaded_models\": [\n {\n \"name\": \"neff\",\n \"uuid\": \"91f2f66e83ea419dace1da07617ad39f\",\n \"model_id\": 10005,\n \"is_running\": false,\n \"subgraphs\": {\n \"sg_00\": {\n \"memory_used_bytes\": {\n \"host\": 20480,\n \"neuron_device\": 21001024,\n \"usage_breakdown\": {\n \"host\": {\n \"application_memory\": 20480,\n \"constants\": 0,\n \"dma_buffers\": 0,\n \"tensors\": 0\n },\n \"neuron_device\": {\n \"constants\": 20971520,\n \"model_code\": 29504,\n \"runtime_memory\": 0,\n \"tensors\": 0\n }\n }\n },\n \"neuroncore_index\": 0,\n \"neuron_device_index\": 12\n }\n }\n },\n ...\n ],\n \"error\": \"\"\n }\n\n\n- ``\"memory_used\"`` summarizes the amount of memory used by the\n Neuron application\n\n - ``\"neuron_runtime_used_bytes\"`` - current amount of memory used by\n the Neuron application\n\n - ``\"host\"`` - total host DRAM usage in bytes\n - ``\"neuron_device\"`` - total Neuron device memory usage in bytes\n - ``\"usage_breakdown\"`` - a breakdown of the total memory usage in the other two fields\n\n - ``\"host\"`` - breakdown of the host memory usage\n\n - ``\"application_memory\"`` - amount of host memory used by the application - this includes all allocations that are not included\n in the next categories\n - ``\"constants\"`` - amount of host memory used for constants during training (or weights during inference)\n - ``\"dma_buffers\"`` - amount of host memory used for DMA transfers\n - ``\"tensors\"`` - amount of host memory used for tensors\n\n - ``\"neuroncore_memory_usage\"`` - a breakdown of memory allocated on the Neuron Devices and the NeuronCores for which it was allocated\n\n - ``\"0\"`` - ``\"64\"`` (for trn2-48xlarge) - NeuronCores for which the memory was allocated\n - ``\"constants\"`` - amount of device memory used for constants during training (or weights during inference)\n - ``\"model_code\"`` - amount of device memory used for models' executable code\n - ``\"model_shared_scratchpad\"`` - amount of device memory used for the scratchpad shared by the models - a memory region reserved for the models' internal variables and auxiliary buffers\n - ``\"runtime_memory\"`` - amount of device memory used by the Neuron Runtime\n - ``\"tensors\"`` - amount of device memory used for tensors\n\n- ``\"loaded_models\"`` - array containing objects representing loaded models\n\n - ``\"name\"`` - name of the model\n - ``\"uuid\"`` - unique id for the model\n - ``\"model_id\"`` - Neuron application-assigned ID for this model\n - ``\"is_running\"`` - true if this model is currently started, false otherwise\n - \"``subgraphs\"`` - object containing all the subgraphs for the model, indexed by their name: ``\"subgraph_name\": { subgraph_data }``\n\n - ``\"memory_used_bytes\"`` - memory usage for this subgraph\n\n - ``\"host\"`` - total host DRAM usage in bytes\n - ``\"neuron_device\"`` - total Neuron device DRAM usage in bytes\n - ``\"usage_breakdown\"`` - a breakdown of memory allocated at load time for this model\n\n - ``\"host\"`` - breakdown of host memory allocated for this model\n\n - ``\"application_memory\"`` - amount of host memory allocated for this model by the Neuron Runtime which doesn't fall in any\n of the next categories\n - ``\"constants\"`` - amount of host memory used for constants during training (or weights during inference)\n - ``\"dma_buffers\"`` - host memory allocated for DMA transfers for this model\n - ``\"tensors\"`` - amount of device memory used for tensors at model load time\n\n - ``\"neuron_device\"`` - a breakdown of device memory allocated for this model\n\n - ``\"constants\"`` - amount of device memory used for constants during training (or weights during inference)\n - ``\"model_code\"`` - amount of device memory used for the model's executable code\n - ``\"runtime_memory\"`` - amount of device memory used by the Neuron Runtime for this model\n - ``\"tensors\"`` - amount of device memory allocated for tensors at this model's load time\n\n - ``\"neuroncore_index\"`` - NeuronCore index on which the subgraph is loaded\n - ``\"neuron_device_index\"`` - Neuron device index on which the subgraph is loaded\n\n\n- ``\"error\"`` - string containing any error that occurred when\n collecting the data\n\n\nneuron_runtime_vcpu_usage\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n \"neuron_runtime_vcpu_usage\": {\n \"period\": 1.030604818,\n \"vcpu_usage\": {\n \"user\": 42.01,\n \"system\": 12.34\n },\n \"error\": \"\"\n }\n\n- ``\"vcpu_usage\"`` - object showing vCPU usage in percentages for the\n Neuron application during the captured period\n\n - ``\"user\"`` - percentage of time spent in user code by this Neuron\n Application\n - ``\"system\"`` - percentage of time spent in kernel code by this\n Neuron application\n\n- ``\"error\"`` - string containing any error that occurred when\n collecting the data\n\nSystem level metric groups\n--------------------------\n\n.. _neuron-monitor-hw-counters:\n\nneuron_hw_counters\n~~~~~~~~~~~~~~~~~~\n\n::\n\n \"neuron_hw_counters\": {\n \"period\": 1.030359284,\n \"neuron_devices\": [\n {\n \"neuron_device_index\": 0,\n \"mem_ecc_corrected\": 0,\n \"mem_ecc_uncorrected\": 0,\n \"sram_ecc_uncorrected\": 0,\n \"sram_ecc_corrected\": 0\n }\n ],\n \"error\": \"\"\n },\n\n- ``\"neuron_devices\"`` - array containing ECC data for all Neuron devices\n\n - ``\"neuron_device_index\"`` - Neuron device index\n - ``\"mem_ecc_corrected\"`` - number of corrected ECC events in the\n Neuron device’s DRAM\n - ``\"mem_ecc_uncorrected\"`` - number of uncorrected ECC events in\n the Neuron device’s DRAM\n - ``\"sram_ecc_uncorrected\"`` - number of uncorrected ECC events in\n the Neuron device’s SRAM\n - ``\"sram_ecc_corrected\"`` - number of corrected ECC events in\n the Neuron device’s SRAM\n\n- ``\"error\"`` - string containing any error that occurred when\n collecting the data\n\n.. _neuron-monitor-vcpu-usage:\n\nvcpu_usage\n~~~~~~~~~~~~\n\n::\n\n \"vcpu_usage\": {\n \"period\": 0.999974868,\n \"average_usage\": {\n \"user\": 32.77,\n \"nice\": 0,\n \"system\": 22.87,\n \"idle\": 39.36,\n \"io_wait\": 0,\n \"irq\": 0,\n \"soft_irq\": 0\n },\n \"usage_data\": {\n \"0\": {\n \"user\": 34.41,\n \"nice\": 0,\n \"system\": 27.96,\n \"idle\": 37.63,\n \"io_wait\": 0,\n \"irq\": 0,\n \"soft_irq\": 0\n },\n \"1\": {\n \"user\": 56.84,\n \"nice\": 0,\n \"system\": 28.42,\n \"idle\": 14.74,\n \"io_wait\": 0,\n \"irq\": 0,\n \"soft_irq\": 0\n },\n [...]\n },\n \"context_switch_count\": 123456,\n \"error\": \"\"\n }\n\n- each vCPU usage object contains the following fields:\n\n - ``\"user\"`` - percentage of time spent in user code\n - ``\"nice\"`` - percentage of time spent executing niced user code\n - ``\"system\"`` - percentage of time spent executing kernel code\n - ``\"idle\"`` - percentage of time spent idle\n - ``\"io_wait\"`` - percentage of time spent waiting for IO operations\n - ``\"irq\"`` - percentage of time spent servicing hardware interrupts\n - ``\"soft_irq\"`` - percentage of time spent servicing software\n interrupts\n\n- ``\"average_usage\"`` - contains the average usage across all vCPUs\n during the captured period\n- ``\"usage_data\"`` - contains per vCPU usage during the captured period\n- ``\"context_switch_count\"`` - contains the number of vCPU context\n switches during the captured period\n- ``\"error\"`` - string containing any error that occurred when\n collecting the data\n\n.. _neuron-monitor-memory-info:\n\nmemory_info\n~~~~~~~~~~~\n\n::\n\n \"memory_info\": {\n \"period\": 5.346411129,\n \"memory_total_bytes\": 49345835008,\n \"memory_used_bytes\": 16042344448,\n \"swap_total_bytes\": 0,\n \"swap_used_bytes\": 0,\n \"error\": \"\"\n }\n\n- ``\"memory_total_bytes\"`` - total size of the host memory, in bytes\n\n- ``\"memory_used_bytes\"`` - amount of host memory in use, in bytes\n\n- ``\"swap_total_bytes\"`` - total size of the host swap file, in bytes\n\n- ``\"swap_used_bytes\"`` - amount of swap memory in use, in bytes\n\n\n.. _neuron-monitor-companion-scripts:\n\nCompanion scripts\n-----------------\n\nneuron-monitor is installed with three Python companion scripts:\n:ref:`neuron-monitor-cloudwatchpy`, :ref:`neuron-monitor-prometheuspy`, and\n:ref:`neuron-monitor-k8s-infopy`\n\n.. _neuron-monitor-cloudwatchpy:\n\nneuron-monitor-cloudwatch.py\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIt requires Python3 and the `boto3 Python\nmodule `__.\nIt is installed to:\n``/opt/aws/neuron/bin/neuron-monitor-cloudwatch.py``.\n\n.. _using-neuron-monitor-cloudwatchpy:\n\nUsing neuron-monitor-cloudwatch.py\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n::\n\n neuron-monitor | neuron-monitor-cloudwatch.py --namespace --region \n\nFor example:\n\n::\n\n neuron-monitor | neuron-monitor-cloudwatch.py --namespace neuron_monitor_test --region us-west-2\n\n.. _neuron-monitor-prometheuspy:\n\nneuron-monitor-prometheus.py\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIt requires Python3 and the `Prometheus client Python\nmodule `__. It is installed\nto: ``/opt/aws/neuron/bin/neuron-monitor-prometheus.py``.\n\n.. _using-neuron-monitor-prometheuspy:\n\nUsing neuron-monitor-prometheus.py\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n::\n\n neuron-monitor | neuron-monitor-prometheus.py --port \n\nFor example:\n\n::\n\n neuron-monitor | neuron-monitor-prometheus.py --port 8008\n\nThe default value for ``--port`` is ``8000``.\n\nIf your data visualization framework is Grafana, we provided a :download:`Grafana dashboard `\nwhich integrates with Prometheus and this script.\n\n.. |image| image:: ../../images/nm-img2.png\n\n.. _neuron-monitor-k8s-infopy:\n\nneuron-monitor-k8s-info.py (Beta)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIt requires Python3 and the `gRPC Python\npackage `__. It is installed\nto: ``/opt/aws/neuron/bin/neuron-monitor-k8s-info.py``.\n\n.. important::\n\n This companion script is in Beta and is disabled by default.\n\n It only works on EKS, and is currently not supported with EKS auto mode.\n\n.. _using-neuron-monitor-k8s-infopy:\n\nUsing neuron-monitor-k8s-info.py\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n::\n\n neuron-monitor | neuron-monitor-prometheus.py --port --enable-k8s-info | neuron-monitor-k8s-info.py --period \n\nFor example:\n\n::\n\n neuron-monitor | neuron-monitor-prometheus.py --port 8008 --enable-k8s-info | neuron-monitor-k8s-info.py --period 30\n\nThe default value for ``--period`` is ``15``.\n\nRunning neuron monitor in Kubernetes environment\n-------------------------------------------------\n\nFor running neuron monitor in Kubernetes environment, please refer to instructions `here `_.\n" }, { "path": "tools/neuron-sys-tools/neuron-sysfs-user-guide.rst", "content": ".. _neuron-sysfs-ug:\n\nNeuron Sysfs User Guide\n=======================\n\n.. contents:: Table of contents\n :local:\n :depth: 3\n\nIntroduction\n------------\nThe kernel provides a few ways in which userspace programs can get system information from the kernel space. Sysfs is one common way to do so. It is a virtual filesystem typically mounted on the ``/sys`` directory and contains information about hardware devices attached to the system and about drivers handling those devices. By navigating the hierarchical structure of the sysfs filesystem and viewing the information provided by its files and directories, you can gather valuable information that can help diagnose and resolve a wide range of hardware and system issues.\n\nThus a sysfs filesystem is set up per Neuron Device under ``/sys/devices/virtual/neuron_device`` to give you an insight into the Neuron Driver and Runtime at system level. By performing several simple CLIs such as reading or writing to a sysfs file, you can get information such as Runtime status, memory usage, Driver info etc. You can even create your own shell scripts to query Runtime and Driver statistics from sysfs and generate customized reports.\n\nThis user guide will first explain the Neuron sysfs structure and then introduce many ways where you can perform diagnostic works with Neuron sysfs.\n\n\nNeuron Sysfs Filesystem Structure\n---------------------------------\nHigh Level Overview\n^^^^^^^^^^^^^^^^^^^\n\nHere is the high level structure of the Neuron sysfs filesystem, where the total and present counters are not shown:\n\n.. code-block:: bash\n\n /sys/devices/virtual/neuron_device/\n ├── neuron0/\n │ ├── subsystem\n │ ├── uevent\n │ ├── connected_devices\n │ ├── core_count\n │ ├── reset\n │ ├── power/\n │ │ ├── async\n │ │ ├── control\n │ │ ├── runtime_active_time\n │ │ ├── runtime_active_kids\n │ │ └── ...\n │ ├── info/\n │ │ ├── notify_delay\n │ │ ├── serial_number\n │ │ └── architecture/\n │ │ ├── arch_type\n │ │ ├── device_name\n │ │ └── instance_type\n ├── stats\n │ ├── hardware\n │ │ ├── mem_ecc_uncorrected\n │ │ ├── mem_ecc_repairable_uncorrected\n │ │ └── sram_ecc_uncorrected\n │ ├── memory_usage\n │ │ └── host_mem\n │ │ ├── application_memory\n │ │ ├── constants\n │ │ ├── dma_buffers\n │ │ ├── dma_rings\n │ │ ├── driver_memory\n │ │ ├── notifications\n │ │ ├── tensors\n │ │ └── uncategorized\n │ └── power\n │ └── utilization\n ├── neuron_core0/\n │ ├── info/\n │ │ └── architecture/\n │ │ └── arch_type\n │ ├── stats/\n │ │ ├── status/\n │ │ │ ├── exec_bad_input\n │ │ │ ├── hw_error\n │ │ │ ├── infer_failed_to_queue\n │ │ │ ├── resource_nc_error\n │ │ │ ├── unsupported_neff_version\n │ │ │ ├── failure\n │ │ │ ├── infer_completed_with_error\n │ │ │ ├── invalid_error\n │ │ │ ├── oob_error\n │ │ │ ├── success\n │ │ │ ├── generic_error\n │ │ │ ├── infer_completed_with_num_error\n │ │ │ ├── resource_error\n │ │ │ └── timeout\n │ │ ├── memory_usage/\n │ │ │ ├── device_mem/\n │ │ │ │ ├── collectives\n │ │ │ │ ├── constants\n │ │ │ │ ├── dma_rings\n │ │ │ │ ├── driver_memory\n │ │ │ │ ├── model_code\n │ │ │ │ ├── model_shared_scratchpad\n │ │ │ │ ├── nonshared_scratchpad\n │ │ │ │ ├── notifications\n │ │ │ │ ├── runtime_memory\n │ │ │ │ ├── tensors\n │ │ | │ └── uncategorized\n │ │ │ └── host_mem\n │ │ └── other_info/\n │ │ ├── flop_count\n │ │ ├── inference_count\n │ │ ├── model_load_count\n │ │ ├── reset_fail_count\n │ │ ├── reset_req_count\n │ │ └── nc_time_in_use\n │ └── ...\n │── neuron_core1/\n │ │ ├── info/\n │ │ │ └── ...\n │ │ └── stats/\n │ │ └── ...\n │ └── ...\n ├── neuron1\n ├── neuron2\n ├── neuron3\n └── ...\n\n\nEach Neuron Device is represented as a directory under ``/sys/devices/virtual/neuron_device/``, where ``neuron0/`` represents the Neuron Device 0, ``neuron1/`` represents the Neuron Device 1, etc. Each NeuronCore is represented as a directory under a Neuron Device directory, represented as ``neuron_core{0,1,2,...}``. Metrics such as Runtime and Driver info and statistics are collected as per NeuronCore in two directories under the NeuronCore directory, i.e. ``info/`` and ``stats/``.\n\nMost of the metrics belong to a category called “counter.” \nEach counter is represented as a directory, which holds two numerical values as two files: total and present. Each memory usage counter has an additional value called peak.\nThe total value starts accumulating metrics when the Driver is loaded. The present value records the last changed metric value. The peak value records the max value so far.\nEach counter has the same filesystem structure like this:\n\n.. code-block:: dash\n\n /sys/devices/virtual/neuron_device/neuron0/neuron_core0/status/\n ├── exec_bad_input/\n │ ├── total\n │ └── present\n ├── hw_error/\n │ ├── total\n │ └── present\n ├── infer_failed_to_queue/\n │ ├── total\n │ └── present\n └── ...\n\n\n\nDescription for Each Field\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n``info/``: This directory stores general information about hardware and software. None of them are counter types.\n\n* ``notify_delay``: The delay between notifications from the Neuron Device. Current settings are on (``0``) or off (``-1``). Off by default. \n\n* ``serial_number``: The unique device identifier.\n\n* ``architecture/``: This directory stores hardware architecture information.\n\n * ``arch_type``: The architecture type of the Neuron Device. Sample architecture types are v1, v2, and v3. You can only read the value. You cannot change it.\n\n * ``instance_type``: The instance type of the Neuron Device. Sample instance types are Inf1, Inf2, and Trn1. You can only read the value. You cannot change it.\n\n * ``device_type``: The Neuron Device type. Sample Neuron Device types are Inferentia, Inferentia2, and Trainium1. You can only read the value. You cannot change it.\n\n\n``stats/``: This directory stores Neuron Runtime and Driver statistics. It contains three subdirectories: ``status/``, ``memory_usage/``, and ``other_info/``.\n\n* ``status/``: This directory stores the number of each return status of API calls. As explained in :ref:`The LIBNRT API Return Codes `, every API call returns an NRT_STATUS value, which represents the return status of that API call. Our sysfs filesystem stores all ``NRT_STATUS`` as subdirectories under the ``status/`` directory. They all have the counter structure. Thus each ``NRT_STATUS`` subdirectory holds two values (total and present) and records the number of times you receive a certain ``NRT_STATUS``. The following is description for each ``NRT_STATUS`` subdirectory. You should see the description align with what is described in :ref:`The LIBNRT API Return Codes `.\n\n* ``memory_usage/``: This directory contains memory usage statistics for both device and host, represented as counters. In this directory, the total counters indicate the current memory usage, present counters represent the memory allocation or deallocation amount in the previous operation, and peak counters indicate the maximum memory usage observed. Additionally, this directory provides detailed breakdown statistics for device and host memory usage. These memory breakdown details correspond to the :ref:`Memory Usage Summary ` section displayed on in Neuron Monitor.\n\n * ``device_mem/``: The amount of memory that Neuron Runtime uses for weights, instructions and DMA rings.\n\n * This device memory per NeuronCore is further categorized into five types: ``collectives/``, ``constants/``, ``dma_rings/``, ``driver_memory/``, ``model_code/``, ``model_shared_scratchpad/``, ``nonshared_scratchpad/``, ``notifications/``, ``runtime_memory/``, ``tensors/``, and ``uncategorized/``. Each of these categories has total, present, and peak.\n * ``collectives`` - amount of device memory used for collective communication between workers\n * ``constants`` - amount of device memory used for constants (for applications running training) or weights (for applications running inferences)\n * ``dma_rings`` - amount of device memory used for storing model executable code used for data movements\n * ``driver_memory`` - amount of device memory used by the Neuron Driver\n * ``model_code`` - amount of device memory used for storing model executable code\n * ``model_shared_scratchpad`` - amount of device memory used for the shared model scratchpad, a buffer shared between models on the same Neuron Core used for internal model variables and other auxiliary buffers\n * ``nonshared_scratchpad`` - amount of device memory used for non-shared model scratchpad, a buffer used by a single model for internal model variables and other auxiliary buffers\n * ``notifications`` - amount of device memory used to store instruction level trace information used to profile workloads ran on the device\n * ``runtime_memory`` - amount of device memory used by the Neuron Runtime (outside of the previous categories)\n * ``tensors`` - amount of device memory used for tensors\n * ``uncategorized`` - amount of device memory that does not belong in any other catagory in this list\n \n * ``host_mem/``: The amount of memory that Neuron Runtime uses for input and output tensors.\n\n * The host memory per Neuron Device is further categorized into four types: ``application_memory/``, ``constants/``, ``dma_buffers/``, ``dma_rings/``, ``driver_memory/``, ``notifications/``, ``tensors/``, ``uncategorized/``. These categories provide more granular host memory classification compared to :ref:`Host Used Memory ` section. Each of these categories has total, present, and peak\n\n * ``hardware/``: Hardware statistics.\n\n * ``mem_ecc_uncorrected``: The number of unrepairable uncorrected ECC events in the Neuron device's DRAM.\n\n * ``mem_ecc_repairable_uncorrected``: The number of repairable uncorrected ECC events in the Neuron device's DRAM.\n\n * ``sram_ecc_uncorrected``: The number of uncorrected ECC events in the Neuron device's SRAM.\n * ``power/``: Power statistics.\n\n * ``utilization``: Reports per-minute power usage statistics as a percentage of max power in the following format:\n\n ,,,,\n\n **Field descriptions:**\n\n status\n Indicates the sampling state in a string. Valid values are:\n\n ``POWER_STATUS_VALID`` - Sampling successful\n\n ``POWER_STATUS_NO_DATA`` - No samples available\n\n ``POWER_STATUS_INVALID`` - An internal sampling error occurred\n\n timestamp\n Time when the sample was collected in Unix epoch seconds (integer)\n\n min_power\n Minimum power utilization during the sampling period (0.00-100.00%)\n\n max_power\n Maximum power utilization during the sampling period (0.00-100.00%)\n\n avg_power\n Average power utilization during the sampling period (0.00-100.00%)\n\n The interface updates these statistics every minute based on continuous power sampling.\n* ``other_info/``: This directory contains statistics that are not included by ``status/`` and ``memory_usage/``. None of them are counter types.\n\n * ``flop_count``: The number of flops. You can use it to calculate the TFLOP/s by ``flop_count`` / time interval\n\n * ``inference_count``: The number of successful inferences\n\n * ``model_load_count``: The number of successful model loads\n\n * ``reset_fail_count``: The number of failed device resets\n\n * ``reset_req_count``: The number of device resets requests\n\n * ``nc_time_in_use``: The time interval in microseconds between the start and the end of the current execution on hardware\n\nOther fields:\n\n* ``connected_devices``: The list of connected devices' ids. You should see the same output as neuron-ls's CONNECTED DEVICES.\n\n* ``reset``: write to this file resets corresponding the Neuron Device.\n\n\nRead and Write to Sysfs\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nReading a sysfs file gives the value for the corresponding metric. You can use the cat command to view the contents of the sysfs files.: \n\n.. code-block:: bash\n\n ubuntu@ip-xxx-xx-xx-xxx:~$ sudo cat /sys/devices/virtual/neuron_device/neuron0/neuron_core0/stats/status/failure/total \n 0\n ubuntu@ip-xxx-xx-xx-xxx:~$ sudo cat /sys/devices/virtual/neuron_device/neuron0/neuron_core0/info/architecture/arch_type \n NCv2\n\nSysfs metrics of counter type are write to clear. You can write any value to the file, and the metric will be set to 0:\n\n.. code-block:: bash\n\n ubuntu@ip-xxx-xx-xx-xxx:~$ echo 1 | sudo tee /sys/devices/virtual/neuron_device/neuron0/neuron_core0/stats/status/failure/total \n 1\n\n\nWriting to ``reset`` resets the corresponding Neuron Device. E.g. the below resets Neuron Device 0:\n\n.. code-block:: bash\n\n ubuntu@ip-xxx-xx-xx-xxx:~$ echo 1 | sudo tee /sys/devices/virtual/neuron_device/neuron0/reset\n 1\n\nNote\n^^^^\n\nAll files under ``/sys/devices/virtual/neuron_device/neuron0/power`` such as ``runtime_active_kids`` or ``runtime_status`` are related to generic device power management. They are not created or controlled by our sysfs metrics. The word ``runtime`` in these files does not refer to Neuron Runtime.\n\n.. _troubleshoot_via_sysfs:\n\nHow to Troubleshoot via Sysfs\n-----------------------------\nYou can perform simple and easy tasks to troubleshoot your ML jobs with one or a few CLIs to read or write the sysfs filesystem.\nYou can do aggregations across all the NeuronCores and all the Neuron Device to get a summarized view using your scripts.\n\n\nYou can also use the Sysfs notification feature to wait passively (without wasting CPU cycles) for changes to the values of Sysfs files. To use this feature, you need to implement a user-space program that calls the poll() function on the Sysfs file that you want to wait on. \nThe poll() function has the following signature: ``unsigned int (*poll) (struct file *, struct poll_table_struct *)``.\nBy default, the Sysfs notification feature is turned off when the driver is loaded. To enable notifications, you can set the value of ``/sys/devices/virtual/neuron_device/neuron0/info/notify_delay`` to 0. To disable notifications, you can set it to -1. Please note that enabling this feature can impact performance.\n\nHere is a sample user space program using poll():\n\n.. code-block:: dash\n\n\t#include \n\t#include \n\t#include \n\t#include \n\t#include \n\n\tint main(int argc, char * argv[])\n\t{\n\t\tchar readbuf[128];\n\t\tint attr_fd = -1; \n\t\tstruct pollfd pfd;\n\t\tint retval = 0;\n\t\tssize_t read_bytes;\n\n\t\tif (argc < 2) {\n\t\t\tfprintf(stderr, \"Error: Please specify sysfs file path\\n\");\n\t\t\texit(1);\n\t\t} \n\t\tattr_fd = open(argv[1], O_RDONLY, 0); \n\t\tif (attr_fd < 0) {\n\t\t\tperror(argv[1]);\n\t\t\texit(2);\n\t\t} \n\n\t\tread_bytes = read(attr_fd, readbuf, sizeof(readbuf));\n\t\tif (read_bytes < 0) {\n\t\t\tperror(argv[1]);\n\t\t\texit(3);\n\t\t} \n\t\tprintf(\"%.*s\", (int)read_bytes, readbuf);\n\n\t\tpfd.fd = attr_fd;\n\t\tpfd.events = POLLERR | POLLPRI;\n\t\tpfd.revents = 0;\n\t\twhile ((retval = poll(&pfd, 1, 100)) >= 0) {\n\t\t\tif (pfd.revents & (POLLERR | POLLPRI)) {\n\t\t\t\tpfd.revents = 0;\n\n\t\t\t\tlseek(attr_fd, 0, SEEK_SET);\n\t\t\t\tread_bytes = read(attr_fd, readbuf, sizeof(readbuf));\n\t\t\t\tif (read_bytes < 0) {\n\t\t\t\t\tperror(argv[1]);\n\t\t\t\t\texit(4);\n\t\t\t\t}\n\t\t\t\tprintf(\"%.*s\", (int)read_bytes, readbuf);\n\t\t\t}\n\t\t}\n\t\treturn 0;\n\t}\n\n\n" }, { "path": "tools/neuron-sys-tools/neuron-top-user-guide.rst", "content": ".. _neuron-top-ug:\n\nNeuron Top User Guide\n=====================\n\n.. contents:: Table of contents\n :local:\n :depth: 2\n\nOverview\n--------\n``neuron-top`` provides useful information about NeuronCore and vCPU utilization, memory usage,\nloaded models, and Neuron applications.\n\n.. note::\n\n ``neuron-top`` fully supports the newly launched trn2 instances.\n\n.. note::\n\n If you are parsing ``neuron-top`` output in your automation environment, you can now replace it with ``neuron-monitor``\n (:ref:`neuron-monitor-ug`) which outputs data in a standardized, easier to parse JSON format.\n\nUsing neuron-top\n----------------\n\nCommand line arguments\n~~~~~~~~~~~~~~~~~~~~~~\nLaunch ``neuron-top`` by simply typing its name in the shell: ``neuron-top``.\n\nUser interface\n~~~~~~~~~~~~~~\n\nThe title section of the user interface shows the application's version number,\nEC2 instance ID, and the instance type on which it is running:\n\n|titleimg|\n\nThe rest of the user interface is divided in 4 sections. The data shown in these\nsections applies to the currently selected tab - which can be the 'all' tab,\nwhich aggregates data from all running Neuron processes, or a tab representing\na single Neuron process:\n\n|overview|\n\n* The ``NeuronCore Utilization`` section shows the NeuronCore utilization for the\n currently selected tab. ```` is the version of the NeuronCores on the instance (for example,\n ``v2`` for trn1 instances and inf2 instances, ``v3`` for trn2 instances with ``LNC=1``, ``v3d`` for trn2\n instances with ``LNC=2``) \n\n Pressing the 'F' key will toggle between displaying utilization percentages - as seen in the previous image -\n and teraflops (trillion floating point operations per second), as seen in the image below:\n\n|flops|\n\n* The ``VCPU Utilization`` section shows:\n\n * ``System vCPU usage`` - the two percentages are user% and system%\n * ``Runtime vCPU usage`` - same breakdown\n\n.. _neuron_top_mem_usage:\n\n* The ``Memory Usage Summary`` section provides a breakdown of the total memory usage on the Neuron Device as well\n as on the host:\n\n .. _neuron_top_host_mem_usage:\n\n * ``Host Used Memory`` - amount of host memory used by the selected application (or an aggregate of all applications if 'All' is selected)\n\n * ``Total`` - total amount of host memory used\n * ``Tensors`` - amount of host memory used for tensors\n * ``Constants`` - amount of host memory used for constants (for applications running training) or weights (for applications running inferences)\n * ``DMA Buffers`` - amount of host memory used for DMA transfers\n * ``App. Memory`` - amount of host memory used by the application that doesn't fall in any of the previous categories\n\n .. _neuron_top_device_mem_usage:\n\n * ``Device Used Memory`` - amount of device memory used by the selected application (or an aggregate of all applications if 'All' is selected)\n\n * ``Total`` - total amount of device memory used\n * ``Tensors`` - amount of device memory used for tensors\n * ``Constants`` - amount of device memory used for constants (for applications running training) or weights (for applications running inferences)\n * ``Model Code`` - amount of device memory used for storing model executable code\n * ``Runtime Memory`` - amount of device memory used by the Neuron Runtime (outside of the previous categories)\n * ``Model Scratchpad`` - amount of device memory used for the shared model scratchpad, a shared buffer used for internal model variables and other\n auxiliary buffers\n\n* ``Memory Usage Details`` contains memory usage data organized as a tree which can be expanded/collapsed. The columns are:\n\n * ``Model ID`` - the Neuron Runtime identifier for this model instance\n * ``Host Memory`` - amount of host memory used\n * ``Device Memory`` - amount of device memory used\n\nThe tree view shows the amount of memory used for the same categories shown in the ``Memory Usage Summary`` but in this section\nthey are attached to either a model (if the memory has been allocated at model load time for that model), or to a NeuronCore (if\nthe memory can't be associated with a model, but has been allocated for that NeuronCore).\nThe 'parent' shows the total amount of memory used - the sum of its children.\n\n.. note::\n The up/down/left/right keys can be used to navigate the tree view. The 'x' key expands/collapses the\n entire tree.\n\nThe bottom bar shows which Neuron process' data is currently displayed by highlighting\nits tag using a green font and marking it using a pair of '>', '<' characters. The 'all'\ntab shows an aggregated view of all the Neuron processes currently running on the instance.\n\n|tabbar|\n\n.. note::\n\n The '1'-'9' keys select the current tab. 'a'/'d' selects the previous/next\n tab on the bar.\n\n.. |titleimg| image:: ../../images/trn2-neuron-top-header.png\n.. |overview| image:: ../../images/trn2-neuron-top.png\n.. |flops| image:: ../../images/trn2-neuron-top-nc.png\n.. |tabbar| image:: ../../images/nt-2.png\n" }, { "path": "tools/profiler/neuron-profile-user-guide.rst", "content": ".. _neuron-profile-ug:\n\nNeuron Profiler User Guide\n============================\n\nThe Neuron Profiler, ``neuron-profile``, is a tool to profile and analyze performance of a ML model compiled with the Neuron compiler and run on NeuronDevices.\n\n.. important::\n\n The Neuron Profiler will be replaced by the new Neuron Explorer in a future release. For more details and migration guidance, see :ref:`neuron-explorer-faq`.\n\n``neuron-profile`` helps developers identify performance bottlenecks and optimize their workloads for NeuronDevices. neuron-profile provides insights into NeuronDevice activity including the instructions executed on each compute engine (ex. Tensor engine, Vector engine, etc.), DMA data movement activity, and performance metrics such as engine utilization, DMA throughput, memory usage, and more. NeuronDevice activity is collected by the ``neuron-profile capture`` command which runs the model with tracing enabled. Profiling typically has near zero overhead because NeuronDevices have dedicated on-chip hardware profiling.\n\nAdditionally, ``neuron-profile`` supports Neuron Kernel Interface (NKI) developers in profiling their kernels. For more information, please refer to :ref:`use-neuron-profile`\n\n.. _neuron-profiler-installation:\n\nInstallation\n------------\n\n``neuron-profile`` comes as part of the ``aws-neuronx-tools`` package, and will be installed to ``/opt/aws/neuron/bin``.\n\n.. note::\n\n ``neuron-profile`` requires Ubuntu 22.04 or newer, or Amazon Linux 2023 or newer.\n Capturing profiles requires an Inferentia or Trainium instance, but processing profiles \n can be done on any instance type.\n\nThe Neuron web profile viewer utilizes InfluxDB OSS 2.x to store time series data for the profiled workloads after post processing.\nPlease follow the instructions provided at https://portal.influxdata.com/downloads/ for the correct OS. A sample installation\nof Neuron Profile and InfluxDB is provided below.\n\nUbuntu\n~~~~~~\n\n.. code-block:: bash\n\n # Install Neuron Profile\n . /etc/os-release\n sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null < /dev/null\n echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' | sudo tee /etc/apt/sources.list.d/influxdata.list\n\n sudo apt-get update && sudo apt-get install influxdb2 influxdb2-cli -y\n sudo systemctl start influxdb\n influx setup\n # Fill in the information to finish the setup\n\n\n\nCapturing a profile\n-------------------\n\nThe ``neuron-profile`` tool can both capture and post-process profiling information. ``neuron-profile`` takes a compiled model (a NEFF), executes it, and saves the profile results to a NTFF (``profile.ntff`` by default).\nFor this example, we assume a NEFF is already available as ``file.neff``\n\n::\n\n $ neuron-profile capture -n file.neff -s profile.ntff\n\nCapturing profiles for multi-worker jobs\n----------------------------------------\n\n``neuron-profile`` can capture profiles for collectives-enabled NEFFs running across multiple NeuronCores, NeuronDevices, or even nodes. \nThis is useful for understanding performance and communication overheads when deploying larger distributed models.\n\nThe following example, performs a distributed run across all NeuronDevices and NeuronCores on an inf2.24xlarge instances, capturing profiles for all 12 workers (one for each NeuronCore).\n\n::\n\n $ neuron-profile capture -n file.neff --collectives-workers-per-node 12 -s output/profile.ntff\n\nA profile is saved for each worker in the output directory.\n\n:: \n\n $ ls output\n profile_rank_0.ntff profile_rank_2.ntff profile_rank_6.ntff profile_rank_1.ntff profile_rank_3.ntff profile_rank_7.ntff\n profile_rank_10.ntff profile_rank_4.ntff profile_rank_8.ntff profile_rank_11.ntff profile_rank_5.ntff profile_rank_9.ntff\n\nIt is also possible to run a distributed job while only capturing a profile for a specific worker instead of all workers. To do that, use the ``--collectives-profile-id`` option.\n\n::\n\n $ neuron-profile capture -n file.neff --collectives-profile-id 5 --collectives-workers-per-node 12 -s output/profile.ntff\n $ ls output\n profile_rank_5.ntff\n\n\nProviding per-worker inputs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBy default, ``neuron-profile capture`` uses all-zero inputs or a single set of inputs specified via positional arguments. For multi-worker jobs where each worker needs different inputs, use the ``--multi-input`` (``-m``) option to specify a file that maps inputs to each worker.\n\nEach line in the multi-input file corresponds to one worker and follows the same format as the positional ``inputs`` argument (`` `` pairs separated by spaces). For example, for a 2-worker job:\n\n::\n\n # inputs.txt\n IN1 worker0_x.npy IN2 worker0_y.npy\n IN1 worker1_x.npy IN2 worker1_y.npy\n\nThen capture the profile with:\n\n::\n\n $ neuron-profile capture -n file.neff -m inputs.txt --collectives-workers-per-node 2 -s output/profile.ntff\n\n.. note::\n\n The ``--multi-input`` option cannot be used together with the positional ``inputs`` argument.\n\n\nCapturing profiles for multi-node jobs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nFor multi-node jobs, ``neuron-profile`` must be invoked on each node using the ``collectives-worker-start-id`` to specify the global index of the first worker on the given\nnode. For example, for a two node job with a total of four workers and two workers per node, the following commands are run on each node.\n\n::\n\n # on node 0\n $ neuron-profile capture -n file.neff --collectives-worker-start-id 0 --collectives-workers-per-node 2 --collectives-worker-count 4\n # on node 1\n $ neuron-profile capture -n file.neff --collectives-worker-start-id 2 --collectives-workers-per-node 2 --collectives-worker-count 4\n\n``neuron-profile`` saves the profile for a worker on the node where that worker was launched. So in the case above, ``profile_rank_0.ntff`` and ``profile_rank_1.ntff``\nare saved to node 0, and ``profile_rank_2.ntff`` and ``profile_rank_3.ntff`` are saved to node 1.\n\n\n\nProcessing and viewing the profile results\n------------------------------------------\n\nTo analyze and view the collected profiling data, use the ``view`` subcommand of ``neuron-profile``. This command performs two main functions: it post-processes the profiling data and starts up an HTTP server. Once the server is running, you can access the profiling results through your web browser. Please note: Chrome is the officially supported browser for viewing profiling results\n\n\n.. note::\n Profiles can be processed and viewed on another machine without Neuron devices. The ``aws-neuronx-tools`` package\n needs to be installed so that you can run ``neuron-profile view``. To process the profile on another\n instance, you need to copy the NEFF and NTFF files from your Inf or Trn instance to that instance.\n\nViewing a single profile\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe first way to invoke ``neuron-profile view`` is to pass both the NEFF and the NTFF to this command.\nIt will post-process these artifacts and print out a direct link to the profile view.\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff\n View profile at http://localhost:3001/profile/n_fdc71a0b582ee3009711a96e59958af921243921\n ctrl-c to exit\n\n\nViewing profiles for multi-worker jobs\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nProfiles from multi-worker jobs (i.e. more than one NeuronCore) can either be viewed individually or in a combined collectives view.\nSince profile data is often similar between workers and processing profile data for all workers can be time-consuming, it is recommended to first \nexplore the profile for a single worker or small subset of workers. Viewing the profile for a specific worker is the same as for single-worker profiles.\n\n::\n\n $ neuron-profile view -n file.neff -s output/profile_rank_5.ntff\n View profile at http://localhost:3001/profile/n_fdc71a0b582ee3009711a96e59958af921243921\n\n\nTo view the profile for multiple workers, pass the directory containing all worker profiles to ``neuron-profile``.\n\n::\n\n $ neuron-profile view -n file.neff -d output\n View profile at http://localhost:3001/profile_cc/p_9a69d907e1350100c9b03745eaa67aa7422842ed\n\n|neuron-profile-multiworker-timeline|\n\nWhen viewing profiles with the combined collectives view you can easily switch between the timelines of different workers by clicking\nthe \"Rank \" tabs.\n\nNote: the \"CC Aggregated View\" currently shows no data. This will be populated in an upcoming release. \n\n\nViewing multiple profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAlternatively, when post-processing multiple profiles, it may be desirable to have a persistent server running while processing results in the background.\nIn this case, we can skip passing arguments to the command, which will direct users to the main page listing all available profiles.\n\n::\n\n $ neuron-profile view\n View a list of profiles at http://localhost:3001/\n\nIn a separate window, we can kick off the post-processing without launching another server by passing the ``--ingest-only`` flag.\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff --ingest-only\n Profile \"n_47cf9972d42798d236caa68952d0d29a76d8bd66\" is ready to view\n\n``n_47cf9972d42798d236caa68952d0d29a76d8bd66`` is the bucket where the data is stored. We can find this profile at ``localhost:3001/profile/``.\n\nAccessing the profiles\n~~~~~~~~~~~~~~~~~~~~~~\n\nIf ``neuron-profile view`` is run on a remote instance, you may need to use port forwarding to access the profiles.\n\nFrom the local machine, SSH to the remote instance and forward ports 3001 (the default ``neuron-profile`` HTTP server port) and 8086 (the default\nInfluxDB port). Then in the browser, go to ``localhost:3001`` to view the profiles.\n\n::\n\n $ ssh @ -L 3001:localhost:3001 -L 8086:localhost:8086\n\n\n.. _neuron-profile-ug-alternative-outputs:\n\nAlternative output formats\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBesides the web view mentioned above, ``neuron-profile`` also supports other output formats such as ``summary-text`` and ``summary-json`` for viewing overall metrics of the profile,\nas well as ``json`` for a parsable alternative.\n\nProfile summary\n^^^^^^^^^^^^^^^\n\nYou can see a summary of each profile using the command ``neuron-profile view --output-format summary-text -n file.neff -s output/profile_rank_.ntff``. This output\nincludes summary metrics and fields for the NeuronCore (``nc_idx``) and NeuronDevice (``nd_idx``) on which the worker was run. For example, the following shows worker 5 used core 1 on\ndevice 3 and took 0.017 seconds (17 ms) to run the model.\n\n::\n\n $ neuron-profile view --output-format summary-text -n file.neff -s output/profile_rank_5.ntff | grep -e \"nd_idx\" -e \"nc_idx\" -e \"total_time\"\n nc_idx 1\n nd_idx 2\n total_time 0.017\n\nThis summary is also available as JSON using ``--output-format summary-json``.\n\nJSON\n^^^^\n\nYou can also view the profile summary and all post-processed profiler events together as a single JSON. To do that, use the ``--output-format json`` option.\n\n::\n\n $ neuron-profile view --output-format json --output-file profile.json -n file.neff -s output/profile_rank_5.ntff\n $ cat profile.json\n {\n \"summary\": [\n {\n \"total_time\": 0.017,\n \"event_count\": 11215\n [...]\n }\n ],\n \"instruction\": [\n {\n \"timestamp\": 10261883214,\n \"duration\": 148,\n \"label\": \"TensorMatrix\",\n \"hlo_name\": \"%add.1 = add(%dot, %custom-call.44)\",\n \"opcode\": \"MATMUL\",\n \"operands\": \"S[5] (Tensor)++@complete acc_flags=3 row_grp=q0 src=fp16@0x5600[1,0,0][3,1,1] dst=0x2000000[1,0,0][3,1,1] 3*128 \"\n },\n [...]\n ]\n }\n\nUnderstanding a Neuron profile\n------------------------------\n\nThe section provides a quick overview on what features and information are available through the Neuron web profile viewer.\n\nFor more information on terms used, please check out the :ref:`neuron_hw_glossary`.\n\nTimeline\n~~~~~~~~\n\n|neuron-profile-web-timeline|\n\nThe execution timeline is plotted based on the elapsed nanoseconds since the start of execution.\n\nStarting from the bottom, the ``TensorMatrix Utilization`` shows the efficiency of the TensorEngine, and\nthe ``Pending DMA Count`` and ``DMA Throughput`` rows show the DMA activity. In general, we want these to be as high\nas possible, and in some cases may help give clues as to whether the workload is memory or compute bound.\n\nNext are the individual NeuronCore engine executions. These rows show the start and end times for instructions executed by each\nengine, and clicking on one of these bars will show more detailed information, as well as any dependencies that were found.\nFor models involving collective compute operations, you will additionally see rows labeled with ``CC-core``, which are used to synchronize\nthe CC operations.\n\nTowards the top is the DMA activity. These can include the transfers of input and output tensors, intermediate tensors, and any\nadditional spilling or loading to and from the on-chip SRAM memory.\n\n\n.. _neuron-profile-ug-features:\n\nFeatures\n~~~~~~~~\n\nThe following are some useful features that may help with navigating a profile:\n\n- Dragging your cursor across a portion of the timeline will zoom in to the selected window, providing a more in depth view of the execution during that time period.\n- Hovering over a point will reveal a subset of information associated with it.\n- Clicking a point will open a text box below the timeline with all the information associated with it.\n- Right-clicking a point will drop a marker at a certain location. This marker will persist when zooming in and out.\n\n - All marker information can be found by clicking the ``Annotations`` button.\n - Markers can be saved and loaded by using a provided name for the marker set.\n - Individual markers can be renamed or deleted in this menu as well.\n - Time span between markers will automatically be shown, and users can change the marker name next to ``diff vs`` to calculate time between other markers.\n\n|neuron-profile-annotation-menu|\n\n- The \"Search\" tab can be used to find and highlight specific points in the profile related to the queried field(s).\n- Click on the \"Box Select\" button in the top-right corner of the timeline and then click and drag on any region of the plot to select all events in that region and get summary statistics such as total duration and breakdowns of opcodes, transfer_sizes, and more.\n\nView Settings\n^^^^^^^^^^^^^\n\nOptions within the ``View Settings`` tab can be used to further customize the timeline view. Editing any settings will update the URL accordingly, which can be used to re-visit the current view at a later time.\nTo speed up initial load times, the default will be a ``Minimal View`` which only shows the instructions executed and the model FLOPs utilization (MFU) over time. Changing between the minimal and full views can also be done through the ``Reset to Full View`` or ``Reset to Minimal View`` buttons.\n\n- ``DMA color group`` will recolor DMAs based on the selected grouping. For example, \"Engine\" will re-color the DMAs based on the associated engine.\n- ``Instruction color group`` will recolor instructions based on the selected grouping. For example, \"Layer\" will re-color the timeline based on the associated framework layer name.\n- ``Layer group depth`` will group and color instructions at the selected layer depth. It will apply when ``Instruction color group`` is set to \"Layer\".\n\n **Example:**\n When ``Layer group depth`` is 2, instructions with layers `model/layer1/op1` and `model/layer1/op2` will be set to the same color.\n- ``Semaphore IDs`` allows for the selection of multiple semaphore values to show at once within the timeline\n \n\n|neuron-profile-view-settings|\n\nAdditionally, there are various summary tabs that can be clicked to provide more information on the model/NEFFs.\n\n- ``Layer Summary`` shows timing information, FLOPs and instructions counts per layer.\n- ``Selection Summary`` shows summarized information for all data points in the selected window when using the \"Box Select\" mode.\n- ``NEFF Header`` shows details on the profiled NEFF, such as the number of NeuronCores required to execute.\n- ``NEFF Nodes`` shows input, output, and weight tensor information, including name, size, and shape.\n- ``Model Info`` shows a summary of the NTFF, such as the NeuronCore the model was executed on, number of notifications, and hardware execution time.\n- ``DMA Queues Info`` shows more information on the queues used for data movement.\n- ``NC Memory Usage Info`` shows a snapshot of the device memory usage breakdown before profiling was started.\n- ``Terminology`` shows a description of metrics provided in the summary table.\n\n|neuron-profile-web-summaries|\n\nPerformance Warnings\n~~~~~~~~~~~~~~~~~~~~\n\nFurthermore, ``neuron-profile`` will automatically highlight some potential performance issues with warning annotations. For example if a tensor has been loaded more than 2 times a warning annotation (seen below as an orange box) will be drawn, encircling the dma instructions where the tensor was loaded many times.\nHover on the annotation to see more details about loading the tensor. Another kind of warning annotation will highlight areas of high throttling. This provides the user a potential reason for slow down (thermal protection). Specific throttling details are shown when hovering the annotation.\n\n|neuron-profile-tensor-reload-annotation|\n\n.. _neuron-profile-collectives-barrier:\n\nCollectives\n~~~~~~~~~~~\n\nFor models involving collective operations, the timeline will show a box around all data points related to each operation. Hovering the top left of the box will reveal more information associated with the operation.\n\n.. note::\n this feature requires profiles to be captured with Neuron Runtime 2.20 or higher.\n\n|neuron-profile-cc-op-annotation|\n\nAdditionally, for any on-device collectives synchronization barrier, a similar box will be display indicating a barrier instead of an actual collectives operation.\n\n|neuron-profile-cc-op-barrier|\n\nEvent Details\n~~~~~~~~~~~~~\n\nThe information when a point is clicked is grouped by categories such as `Timing` or `IDs` for convenience.\nEach row will also include a tool tip on the right side, which can be hovered for an explanation on what the field represents.\nFor instruction `Operands` specifically, clicking on the tooltip will reveal a breakdown of fields that compose an operand, as well as a generic example for reference. The examples may not apply directly to the currently viewed profile.\n\n|neuron-profile-click-tooltip|\n\n\n.. _neuron-profile-framework-stack-trace:\n\nFramework Stack Trace\n----------------------------\n\nThe Framework Stack Trace feature shows up in the Event Details when an instruction on the device profile is clicked. This can we used to map the device instructions back to framework level code in JAX or PyTorch to better understand what part of the application code resulted in a particular device instruction.\n\n|neuron-profile-stack-trace-event-details|\n\nTo enable tracking of the stack trace information, you need to set environment variables before compiling your NEFF:\n\n::\n\n export XLA_IR_DEBUG=1\n export XLA_HLO_DEBUG=1\n\nOnce you have the NEFF, you can simply capture the profile as usual. While viewing the profile use the ``--framework-source-root`` to pass the path to framework source files. This is optional and is only needed if you want to view your code along side the profile.\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff --framework-source-root /path/to/framework/source/files\n\n|neuron-profile-stack-trace-viewer|\n\nSearching Profiles\n~~~~~~~~~~~~~~~~~~\n\nSearching helps identify specific data points that may be worth investigating, such as all instructions related to a specific layer or operation.\nIn the \"Search\" tab, select the corresponding field of interest and enter the value to search for. Multiple fields can be searched together. Please refer to the tooltip within the tab for more help on the query syntax.\nThe search results will also include a summary of all data points found within the current time range.\n\n|neuron-profile-search-summary|\n\n\nHardware Errors\n~~~~~~~~~~~~~~~\n\nInvalid code can lead to errors on Neuron hardware. These errors will be displayed in Neuron Profile's Custom Notification timeline, as shown below. For example an Out of Bounds (OOB) error is displayed as:\n\n|neuron-profile-oob-error|\n\nUsers can correlate the error to the time it occurred and view nearby events to help debug.\n\n\n.. _neuron-profile-scratchpad-mem-usage:\n\nView Scratchpad Usage With Memory Tracker\n------------------------------------------\n\nThe Memory Tracker feature in Neuron Profiler provides detailed insights into scratchpad memory usage over time, showing how memory is allocated and utilized by different tensors during model execution. This is particularly useful for understanding memory bottlenecks and optimizing memory usage patterns.\n\nTo enable Memory Tracker, you need to set environment variables before compiling your NEFF:\n\n::\n\n export XLA_IR_DEBUG=1\n export XLA_HLO_DEBUG=1\n\nThen compile your model with these debug flags enabled. After compilation, capture the profile with the ``--enable-dge-notifs`` flag or set ``NEURON_RT_ENABLE_DGE_NOTIFICATIONS=1``:\n\n::\n\n $ neuron-profile capture -n file.neff --enable-dge-notifs\n\nFinally, view the profile with Memory Tracker enabled:\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff --enable-memory-tracker\n\nThe Memory Tracker displays a timeline showing scratchpad memory usage over time, with a detailed breakdown of which tensors are consuming memory at any given point. This visualization helps identify:\n\n- Peak scratchpad memory usage\n- Memory allocation patterns\n- Tensor-specific memory consumption\n- Potential memory optimization opportunities\n\n|neuron-profiler-memory-tracker|\n\nYou can interact with the Memory Tracker timeline similar to other profile views - clicking on memory usage bars will show detailed information about the tensors using memory at that time, and you can zoom in to specific time ranges to get a more detailed view of memory allocation patterns.\n\n\nViewing Profiles with Perfetto\n------------------------------\n\nPerfetto is an open-source trace analysis toolkit with a powerful UI for visualizing and analyzing trace data.\nUsers of Neuron Profiler have the option of viewing their profiles in the Perfetto UI.\n\nTo process your profile and generate a Perfetto trace file that can be viewed in the Perfetto UI run the following command:\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff --output-format perfetto\n\nThis will generate a ntff.pftrace file. Go to https://ui.perfetto.dev/ in your browser and open the ntff.pftrace file to view your profile in Perfetto.\n\n.. note::\n When loading trace files in the Perfetto UI, your data is processed locally and not uploaded to Perfetto’s servers.\n\n\n|neuron-profile-perfetto-device|\n\n.. _neuron-profile-large-perfetto-profiles:\n\nViewing Large Profiles In Perfetto\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYour browser may run out of memory when viewing ``ntff.pftrace`` (Perfetto trace) files that are more than a few hundred MB.\nTo get around this problem you can use the trace processor script by running the following command on your local system where you wish to view the profile\n\n::\n\n curl -LO https://get.perfetto.dev/trace_processor\n chmod +x ./trace_processor\n ./trace_processor --httpd ntff.pftrace\n\nNow go to https://ui.perfetto.dev/ in your browser and in the dialog box that pops up click the “YES, use loaded trace” button.\n\nFor more information on using the trace processor script and viewing large traces, please refer to the \nPerfetto documentation at https://perfetto.dev/docs/visualization/large-traces.\n\nShowing Dependencies In Perfetto\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBy default Neuron Profiler does not process dependencies for profiles to be viewed in Perfetto because Perfetto renders \nthe full dependency chain which can be visually overwhelming. To include dependencies that can be viewed when clicking \ninstructions and DMAs in the Perfetto UI, use the ``--show-perfetto-flows`` flag when processing your profile.\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff --output-format perfetto --show-perfetto-flows\n\n\nCLI reference\n-------------\n\n.. rubric:: neuron-profile capture\n\n.. rubric:: neuron-profile capture\n\n.. code-block:: text\n\n neuron-profile capture [parameters] [inputs...]\n\nTakes a given compiled NEFF, executes it, and collects the profile results.\nWhen no inputs are provided, all-zero inputs are used, which may result in inf or NaNs.\nIt is recommended to use ``--ignore-exec-errors``.\n\n**Parameters**\n\n``-n, --neff`` (string)\n The compiled NEFF to profile.\n\n``-s, --session-file`` (string)\n The file to store profile session information in.\n\n``--ignore-exec-errors``\n Ignore errors during execution.\n\n``inputs`` (positional args)\n List of inputs in the form of `` `` separated by space. For example: ``IN1 x.npy IN2 y.npy``.\n\nThe following ``neuron-profile capture`` arguments are only relevant for multi-worker jobs:\n\n``-m, --multi-input`` (string)\n Path to a file that describes the input list for each requested worker. Each line in the file should correspond to one worker and follow the same format as the ``inputs`` positional argument (i.e. `` `` pairs separated by spaces). Cannot be used together with the ``inputs`` positional argument. If ``inputs`` is used instead, all workers will use the same inputs.\n\n``--collectives-profile-id`` (string)\n Worker id which will be profiled. Passing ``all`` profiles all workers. (default: ``all``)\n\n``-r, --collectives-workers-per-node`` (int)\n The number of workers on the current node. The global worker id (rank) of worker n on current node is ``collectives-worker-start-id+n``.\n\n``--collectives-worker-count`` (int)\n Total number of Neuron workers across all nodes for this collectives run.\n\n``--collectives-worker-start-id`` (int)\n The rank offset for the first worker on the current node. For example, if node 0 has workers 0,1 and node 1 has workers 2,3 then ``collectives-worker-start-id`` for node 0 and 1 will be 0 and 2, respectively. (default: ``0``)\n\n.. rubric:: neuron-profile view\n\n.. code-block:: text\n\n neuron-profile view [parameters]\n\n**Parameters**\n\n``-n, --neff-path`` (string)\n The compiled NEFF file location.\n\n``-s, --session-file`` (string)\n The profile results NTFF file location.\n\n``-d, --session-dir`` (string)\n Directory containing profile files for multi-worker runs.\n\n``--output-format`` (string)\n How the processed profile should be presented. The default ``db`` writes processed data to the database. ``summary-text`` and ``summary-json`` print the summary data as a table or json, respectively, without writing to the database. The ``perfetto`` option writes processed data to Perfetto's native protobuf based tracing format, and can be visualized in the Perfetto UI. The ``JSON`` option writes processed data to human-readable JSON. (default: ``db``)\n\n``--output-file`` (string)\n File path to write results to, if applicable for the given output format.\n\n``--db-endpoint`` (string)\n The endpoint of InfluxDB. (default: ``http://localhost:8086``)\n\n``--db-org`` (string)\n The org name of InfluxDB.\n\n``--db-bucket`` (string)\n Name of the InfluxDB bucket where ingested profile data is stored. Also used in the URL for viewing the profile. (Optional)\n\n``--port`` (int)\n The port number of the http server. (default: ``3001``)\n\n``--force``\n Force overwrite an existing profile in the database.\n\n``--terminology``\n Print a helpful table of terminology used by the profiler.\n\n``--enable-memory-tracker``\n Enable Memory Tracker to view scratchpad usage over time with a breakdown of usage per tensor. This requires having set ``XLA_IR_DEBUG=1`` and ``XLA_HLO_DEBUG=1`` before NEFF compilation and passing ``--enable-dge-notifs`` when capturing the profile.\n\n\nFAQ\n---\n\nDifference between TensorE and TensorMatrixE Rows in Timeline\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- TensorE includes instruction trace for LoadStationary (LoadWeight)\n- TensorMatrixE includes instruction trace for MultiplyMoving (Matmul)\n- Both instruction traces happen on the same TensorE engine, but we separate them into two rows to de-clutter the timeline due to the background load stationary feature (loading stationary matrix for the next matmul in parallel to current matmul). See more info in :ref:`NKI architecture guide `. \n\nOut of memory (OOM) when capturing a profile\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf ``neuron-profile capture`` fails due to device out-of-memory (OOM), you can increase available memory using using the single-IO mode:\n\nSingle-IO creates one shared I/O buffer on the device equal to the size of the largest I/O tensor. All inputs and outputs then point to slices of this shared buffer instead of allocating separate tensors. This significantly lowers the device memory needed during capture at the cost of producing incorrect outputs.\n\nExample usage:\n\n::\n\n neuron-profile capture --single-io -n file.neff -s profile.ntff\n\nImportant: with ``--single-io``, the profiled performance characteristics (e.g., timing, utilization, bandwidth) are representative, but the model outputs are intentionally not correct. Use this option only to get accurate performance measurements when device memory is tight; do not use it for correctness/accuracy validation.\n\nIf you are able to make changes to your model itself to reduce memory usage, consider the following:\n- Reduce batch size\n- Lower numerical precision\n- Reduce number of layers\n\nIn some cases, a full device profile isn’t necessary to understand performance at a high level. You can instead capture a system profile, which shows overall model execution time and a runtime API trace across all workers and does not require extra device memory. See :ref:`System Profiles overview `.\n\nTroubleshooting\n---------------\n\n\nOutputting to Unsupported NumPy Data Type\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen running ``neuron-profile capture --save-output-npy``, you may encounter an error if the output tensor uses a data type that NumPy doesn't natively support:\n\n::\n\n failed to save output output_hbm to file: unsupported type for npy output: bfloat16\n\nTo work around this, use ``--save-output`` instead to save the output as raw binary, then convert it to the desired data type using NumPy and the ``ml_dtypes`` library.\nThis preserves the precision of the output since it is written to binary instead of casting to a supported data type.\n\n::\n\n # Capture with raw binary output\n neuron-profile capture --save-output -n file.neff\n\n # Convert from raw binary to bfloat16\n import numpy as np\n import ml_dtypes\n \n output = np.fromfile('output0.npy', dtype=np.uint16)\n output = output.view(ml_dtypes.bfloat16)\n\nInfluxDB not installed\n~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff\n ERRO[0001] To install influxdb, go to https://portal.influxdata.com/downloads/ and follow the instructions there\n influxdb not setup correctly: exec: \"influx\": executable file not found in $PATH\n\n::\n\n $ neuron-profile view -n file.neff -s profile.ntff\n ERRO[0000] \n influxdb token not setup correctly: exit status 1\n Try executing \"systemctl start influxdb\" and \"influx setup\"\n\nRunning ``neuron-profile view`` without InfluxDB installed will result in an error and a pointer to the InfluxDB installation instructions.\nPlease follow the provided instructions and retry.\n\nToo many open files\n~~~~~~~~~~~~~~~~~~~\n\n::\n\n influxdb2client E! Write error: internal error: unexpected error writing points to database: [shard 10677] open /home/ubuntu/.influxdbv2/engine/data/7caae65aaa48380d/autogen/10677/index/0/MANIFEST: too many open files\n\nInfluxDB will encounter \"too many open files\" and out of memory errors after a few hundred buckets have been created.\nTwo ways to solve this are to delete unused buckets or increase the system file descriptor limit.\n\nTo increase the file descriptor limit, add the following lines to ``/etc/security/limits.d/efa.conf`` and ``/etc/security/limits.conf``:\n\n::\n\n * soft nofile 1048576\n * hard nofile 1048576\n\nAdd the following lines to /etc/sysctl.conf\n\n::\n\n fs.file-max = 197341270\n vm.max_map_count=1048576\n\nCommit changes by running ``sudo sysctl -p``.\n\n.. |neuron-profile-web-timeline| image:: /images/neuron-profile-web-timeline_2-11.png\n.. |neuron-profile-annotation-menu| image:: /images/neuron-profile-annotation-menu_2-21.png\n.. |neuron-profile-view-settings| image:: /images/neuron-profile-view-settings_2-26.png\n.. |neuron-profile-web-summaries| image:: /images/neuron-profile-web-summaries_2-21.png\n.. |neuron-profile-tensor-reload-annotation| image:: /images/neuron-profile-tensor-reload-annotation.png\n.. |neuron-profile-multiworker-timeline| image:: /images/neuron-profile-multiworker-timelime_2-16.png\n.. |neuron-profile-cc-op-annotation| image:: /images/neuron-profile-cc-op-annotation.png\n.. |neuron-profile-cc-op-barrier| image:: /images/neuron-profile-cc-op-barrier.png\n.. |neuron-profile-click-tooltip| image:: /images/neuron-profile-click-tooltip.png\n.. |neuron-profile-oob-error| image:: /images/neuron-profile-oob-error.png\n.. |neuron-profile-search-summary| image:: /images/neuron-profile-search-summary.png\n.. |neuron-profiler-memory-tracker| image:: /images/neuron-profiler-memory-tracker.png\n.. |neuron-profile-stack-trace-event-details| image:: /images/neuron-profile-stack-trace-event-details.png\n.. |neuron-profile-stack-trace-viewer| image:: /images/neuron-profile-stack-trace-viewer.png\n.. |neuron-profile-perfetto-device| image:: /images/neuron-profiler2-perfetto-device.png\n\nWhen viewing UI \"FATAL - Failed metadata query\"\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf you are SSH port forwarding the web UI from a remote machine to your local desktop you will need to port forward both the web UI (3001) and the database (8086) like so:\n\n::\n\n ssh -L 3001:localhost:3001 -L 8086:localhost:8086 remote_machine\n\nVisual Artifacts when viewing profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSome users have reported visual artifacts when viewing certain profiles in browsers other than Chrome. If you encounter this issue, please try using Chrome. \nFor more details, refer to the GitHub issue: https://github.com/aws-neuron/aws-neuron-sdk/issues/1033\n" }, { "path": "tools/profiler/neuron-profiler-2-0-beta-user-guide.rst", "content": ".. _neuron-profiler-2-0-guide:\n\nNeuron Profiler 2.0 (Beta) User Guide\n=====================================\n\nOverview\n--------\n\nNeuron Profiler 2.0 offers a user-friendly experience for capturing and analyzing application performance \nthrough both high-level system profiles and detailed device-level profiles. Users can profile their workloads \nusing framework-specific APIs within their application code or by setting an environment variable before \nexecution. This tool supports profiling for both single-node and distributed workloads, integrating with \nenvironments such as ParallelCluster and EKS. Once captured, profile results can be explored through multiple \ninterfaces: the Neuron Profiler UI, the open-source trace viewer `Perfetto `_, \nor by exporting to a human-readable JSON format. This flexibility in data capture and visualization enables \nusers to gain comprehensive insights into their application's performance across various scenarios and scales.\n\n.. important::\n\n The Neuron Profiler will be replaced by the new Neuron Explorer in a future release. For more details and migration guidance, see :ref:`neuron-explorer-faq`.\n\n.. note::\n Neuron Profiler 2.0 is a set of new features currently in beta that enhance and simplify the experience of \n capturing and viewing profiles. It is not a replacement of :ref:`Neuron Profiler `, \n which is the existing feature set specifically for capturing and viewing device profiles.\n\n.. _system-profiles-overview:\n\nKey benefits\n~~~~~~~~~~~~\n\n- End-to-end timing of model execution and a Neuron Runtime API trace across all workers, helping identify scheduling gaps, synchronization, and host/runtime overheads.\n- No extra device memory usage by default, making system profiles ideal when device memory is limited or when only high-level insights are needed.\n- Option to capture device profiles for individual models during your workload. \n- Flexible capture and viewing: enable via environment variables or framework APIs; view in the Neuron Profiler UI, in Perfetto, or export as JSON.\n\nCapturing profiles\n------------------\n\nNeuron Profiler 2.0 offers several flexible options for capturing profiles. Users can either set an environment \nvariable ``NEURON_RT_INSPECT_ENABLE`` or use the PyTorch or JAX profiling APIs from their application code for \nfine-grained control over which sections of their code are profiled. PyTorch and JAX users who prefer not to \nmodify their application code can still enable profiling by setting the environment variable before running \ntheir application.\n\nJAX User Experience\n-------------------\n\nJAX Setup\n~~~~~~~~~~~~\n\nFollow the :ref:`JAX Setup ` instructions to install the required\nJAX Neuron Plugin and the latest Neuron Driver, Runtime and Tools packages.\n\n\nJAX Profiler\n~~~~~~~~~~~~\n\nThe JAX context-managed profiling API allows you to profile blocks of code. This will capture a system profile \nincluding a Neuron Runtime API trace and Python trace for your application code in the captured block. This \nwill also capture device profiles for any compiled graphs (NEFFs) executed on NeuronCores within this block. To use \nthe profiler, import the ``jax`` package.\n\n.. code-block:: python\n\n import jax\n\nProfiling is enabled for all code enclosed in the context when using \n``with jax.profiler.trace(os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"]):``\n\n.. note::\n It is important to pass the output directory ``os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"]`` to \n ``with jax.profiler.trace`` and run ``export NEURON_RT_INSPECT_OUTPUT_DIR=`` \n before enabling profiling. This ensures all captured profile data is saved to the correct output directory.\n\nCustom Annotations in JAX\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo add custom annotations to blocks of code in your profile, you can use ``jax.profiler.TraceAnnotation``. \nAnnotation names can be created at runtime, such as in the :ref:`example here ` \nusing ``with jax.profiler.TraceAnnotation(\"my_label\"+str(i)):``. For more information on TraceAnnotations, \nsee the official `JAX documentation `_.\n\nJAX Profiling using environment variable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nInstead of using the jax.profiler context manager, you can enable profiling for your entire application using \nan environment variable. This is desirable if you want to capture a profile without modifying your application \ncode. To enable profiling with the environment variable ``NEURON_RT_INSPECT_ENABLE=1`` and \n``NEURON_RT_INSPECT_OUTPUT_DIR=./output`` before running your application.\n\nFor example:\n\n.. code-block:: shell\n\n # make sure to remove call to with jax.profiler.trace from python script\n NEURON_RT_INSPECT_ENABLE=1 NEURON_RT_INSPECT_OUTPUT_DIR=./output python jax_script.py\n\nWhen using the ``NEURON_RT_INSPECT_ENABLE`` environment variable instead of ``jax.profiler``, system profiles \nwill not contain a framework and application code trace, only Neuron Runtime API trace.\n\nDo not set the ``NEURON_RT_INSPECT_ENABLE`` environment variable and use the ``jax.profiler`` within your \napplication code at the same time. Use one or the other.\n\nFor more profiling options that can be set through environment variables, see the section :ref:`Profile Capture Environment Variables `.\n\n.. _neuron-profile-full-jax-example:\n\nFull JAX Example\n~~~~~~~~~~~~~~~~\n\nCreate a file ``jax_script.py`` which performs repeated matrix multiplications distributed across Neuron devices.\n\n.. code-block:: python\n\n from functools import partial\n import os\n import jax\n import jax.numpy as jnp\n\n from jax.sharding import Mesh, NamedSharding, PartitionSpec as P\n from jax.experimental.shard_map import shard_map\n from time import sleep\n\n os.environ[\"XLA_FLAGS\"] = \"--xla_dump_hlo_snapshots --xla_dump_to=./dump\"\n\n jax.config.update(\"jax_default_prng_impl\", \"rbg\")\n\n mesh = Mesh(jax.devices(), ('i',))\n\n def device_put(x, pspec):\n return jax.device_put(x, NamedSharding(mesh, pspec))\n\n lhs_spec = P('i', None)\n lhs = device_put(jax.random.normal(jax.random.key(0), (128, 128)), lhs_spec)\n\n rhs_spec = P('i', None)\n rhs = device_put(jax.random.normal(jax.random.key(1), (128, 16)), rhs_spec)\n\n @jax.jit\n @partial(shard_map, mesh=mesh, in_specs=(lhs_spec, rhs_spec), out_specs=rhs_spec)\n def matmul_allgather(lhs_block, rhs_block):\n rhs = jax.lax.all_gather(rhs_block, 'i', tiled=True)\n return lhs_block @ rhs\n\n with jax.profiler.trace(os.environ[\"NEURON_RT_INSPECT_OUTPUT_DIR\"]):\n out = matmul_allgather(lhs, rhs)\n for i in range(10):\n with jax.profiler.TraceAnnotation(\"my_label\"+str(i)):\n out = matmul_allgather(lhs, rhs)\n sleep(0.001)\n\n expected = lhs @ rhs\n with jax.default_device(jax.devices('cpu')[0]):\n equal = jnp.allclose(jax.device_get(out), jax.device_get(expected), atol=1e-3, rtol=1e-3)\n print(\"Tensors are the same\") if equal else print(\"Tensors are different\")\n\nSet your profile output directory and run the script:\n\n.. code-block:: shell\n\n export NEURON_RT_INSPECT_OUTPUT_DIR=./output\n python jax_script.py\n\nPyTorch User Experience\n-----------------------\n\nPyTorch Setup\n~~~~~~~~~~~~~\n\nFollow the :ref:`PyTorch Setup ` instructions to install the required PyTorch Neuron packages \nas well as the latest Neuron Driver, Runtime and Tools. \n\nPyTorch Profiler\n~~~~~~~~~~~~~~~~\n\nThe PyTorch context-managed profiling API allows you to profile blocks of code. This will capture a system \nprofile including a Neuron Runtime API trace and Python trace for your application code in the captured block. \nThis will also capture device profiles for any compiled graphs executed on NeuronCores within this block. To \nuse the profiler, import it in your application:\n\n.. code-block:: python\n\n from torch_neuronx.experimental import profiler\n\nThen profile a block of code using:\n\n.. code-block:: python\n\n with torch_neuronx.experimental.profiler.profile(\n port=9012,\n profile_type='system',\n target='neuron_profile_perfetto',\n output_dir=os.environ['NEURON_RT_INSPECT_OUTPUT_DIR'],\n ms_duration=30000) as profiler:\n\nAfter modifying your code to call the profiler, run your application as you normally would \nbut set the environment variable ``NEURON_RT_INSPECT_OUTPUT_DIR`` to specify the output directory.\n\n.. code-block:: shell\n\n NEURON_RT_INSPECT_OUTPUT_DIR=./output python application.py\n\n.. note::\n it is essential to set ``output_dir=os.environ['NEURON_RT_INSPECT_OUTPUT_DIR']`` when starting the profiler from your application code. \n This ensures that all profile data sources dump to the same output directory. \n\nPyTorch Profiling using Environment Variable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nInstead of using the ``torch_neuronx.experimental.profiler.profile`` context manager, you can enable profiling \nfor your entire application using environment variable. This is desirable if you want to capture a profile without modifying your application code. To enable profiling \nwith environment variable ``NEURON_RT_INSPECT_ENABLE=1`` and ``NEURON_RT_INSPECT_OUTPUT_DIR=./output`` before running your application.\n\nFor example\n\n.. code-block:: shell\n\n # make sure to remove call to with torch_neuronx.experimental.profiler.profile from python script\n NEURON_RT_INSPECT_ENABLE=1 NEURON_RT_INSPECT_OUTPUT_DIR=./output python pytorch_script.py\n\nWhen using the ``NEURON_RT_INSPECT_ENABLE`` environment variable instead of ``torch_neuronx.experimental.profiler.profile`` system profiles will not contain a framework and application code trace, only Neuron Runtime API trace.\n\nDo not set the ``NEURON_RT_INSPECT_ENABLE`` environment variable and use the ``torch_neuronx.experimental.profiler.profile`` within your application code at the same time. Use one or the other. \n\nFor more profiling options that can be set through environment variables, see the section :ref:`Profile Capture Environment Variables `.\n\n\nFull PyTorch Example\n~~~~~~~~~~~~~~~~~~~~\n\nCreate a file ``train_torchrun_context.py`` with the following contents\n\n.. code-block:: python\n\n import os\n\n import torch\n import torch.nn as nn\n import torch.nn.functional as F\n\n # XLA imports\n import torch_xla\n import torch_xla.core.xla_model as xm\n import torch_xla.debug.profiler as xp\n\n import torch_neuronx\n from torch_neuronx.experimental import profiler\n\n os.environ[\"NEURON_CC_FLAGS\"] = \"--cache_dir=./compiler_cache\"\n\n # Global constants\n EPOCHS = 2\n\n # Declare 3-layer MLP Model\n class MLP(nn.Module):\n def __init__(self, input_size=10, output_size=2, layers=[5, 5]):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_size, layers[0])\n self.fc2 = nn.Linear(layers[0], layers[1])\n self.fc3 = nn.Linear(layers[1], output_size)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return F.log_softmax(x, dim=1)\n\n def main():\n # Fix the random number generator seeds for reproducibility\n torch.manual_seed(0)\n\n # XLA: Specify XLA device (defaults to a NeuronCore on Trn1 instance)\n device = xm.xla_device()\n\n # Start the profiler context-manager\n with torch_neuronx.experimental.profiler.profile(\n port=9012,\n profile_type='system',\n target='neuron_profile_perfetto',\n output_dir=os.environ['NEURON_RT_INSPECT_OUTPUT_DIR'],\n ms_duration=30000) as profiler:\n\n # IMPORTANT: the model has to be transferred to XLA within\n # the context manager, otherwise profiling won't work\n model = MLP().to(device)\n optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n loss_fn = torch.nn.NLLLoss()\n\n # start training loop\n print('----------Training ---------------')\n model.train()\n for epoch in range(EPOCHS):\n optimizer.zero_grad()\n train_x = torch.randn(1, 10).to(device)\n train_label = torch.tensor([1]).to(device)\n\n # forward\n loss = loss_fn(model(train_x), train_label)\n\n # back\n loss.backward()\n optimizer.step()\n\n # XLA: collect ops and run them in XLA runtime\n xm.mark_step()\n\n print('----------End Training ---------------')\n\n if __name__ == '__main__':\n main()\n\nRun this workload with the following command:\n\n.. code-block:: shell\n\n NEURON_RT_INSPECT_OUTPUT_DIR=\"output\" python simple_demo.py\n\n.. _neuron-profiler-non-framework-user-experience:\n\nNon-framework Specific User Experience\n--------------------------------------\n\nYou can also control profiling with environment variables. This is useful when you can’t easily change your \napplication code, such as when running an executable which calls the Neuron Runtime or in a containerized \nenvironment where the application code is built into the container image.\n\n.. _neuron-profiler-capture-environment-variables:\n\nProfile Capture Environment Variables\n--------------------------------------\n\n.. _core-control-variables:\n\nCore control variables\n~~~~~~~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Variable\n - Description\n - Default behavior\n * - ``NEURON_RT_INSPECT_ENABLE``\n - Set to ``1`` to enable profiling\n - Enables system profiling and disables device profiling. To control which profile types are captured, see :ref:`Profile type selection `\n * - ``NEURON_RT_INSPECT_OUTPUT_DIR``\n - Directory for profile data output\n - Default directory for captured profile data is ``./output``\n\n.. _profile-type-selection:\n\nProfile type selection\n~~~~~~~~~~~~~~~~~~~~~~~\n\n.. note:: \n \n When ``NEURON_RT_INSPECT_ENABLE`` set to ``1``, ``NEURON_RT_INSPECT_SYSTEM_PROFILE`` is enabled by default (set to 1) and ``NEURON_RT_INSPECT_DEVICE_PROFILE`` is disabled by default (set to ``0``).\n\nWhen ``NEURON_RT_INSPECT_ENABLE`` = 1, two different profile types are available:\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Variable\n - Profile type\n - Description\n - Enable capture\n - Disable capture\n * - ``NEURON_RT_INSPECT_SYSTEM_PROFILE``\n - System-level\n - Captures runtime system events and operations\n - Set to ``1``\n - Set to ``0``\n * - ``NEURON_RT_INSPECT_DEVICE_PROFILE``\n - Device-level\n - Captures detailed NeuronCore hardware metrics\n - Set to ``1``\n - Set to ``0``\n\n.. note::\n\n These variables have no effect if ``NEURON_RT_INSPECT_ENABLE`` is not set to ``1``.\n\n.. _advanced-config-vars:\n \nAdvanced configuration\n~~~~~~~~~~~~~~~~~~~~~~~\n\n.. list-table::\n :widths: auto\n :header-rows: 1\n :align: left\n\n * - Variable\n - Profile type\n - Description\n - Default behavior\n * - ``NEURON_RT_INSPECT_SYS_TRACE_MAX_EVENTS_PER_NC``\n - System-level\n - Maximum trace events per NeuronCore before oldest events are overwritten\n - 1,000,000\n\n.. note:: \n \n Increasing the event limit will consume more host memory.\n\nExample Capturing Profile of Application Using Environment Variables\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nInstead of using the PyTorch or JAX profilers you can profile your Python application (or any application calling the Neuron Runtime API) using environment variables.\n\n.. code-block:: shell\n\n NEURON_RT_INSPECT_ENABLE=1 NEURON_RT_INSPECT_OUTPUT_DIR=./output python app.py\n\nSee :ref:`Profile Capture Environment Variables ` for other profiling options that can be set via environment variable.\n\nExample Capturing Profile of nccom-test Using Environment Variables\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nProfiling can be enabled using environment variables. For simplicity, we have a quick way to generate a Neuron workload through using :ref:`nccom-test `. nccom-test is a benchmarking tool which is already available with Neuron AMI.\n\n.. code-block:: shell\n\n export NEURON_RT_INSPECT_ENABLE=1\n export NEURON_RT_INSPECT_OUTPUT_DIR=./output\n nccom-test allr allg -b 512kb -e 512kb -r 32 -n 10 -d fp32 -w 1 -f 512\n\n.. note::\n If you have problems with nccom-test add the --debug flag.\n If using a trn1.2xlarge instance, change -r 32 to -r 2 to use fewer neuron cores.\n\nTo understand the profiling output see this section: :ref:`Inspect Output `\n\nCLI reference for System Profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn addition to controlling profiling with environment variables, you can use the ``neuron-profile inspect`` command line interface \nfor profiling applications. This provides the same functionality as environment variables but helps you avoid typos, invalid arguments, \nand provides a useful ``--help`` command to explain available options.\n\n.. code-block:: shell\n\n Usage:\n neuron-profile [OPTIONS] inspect [inspect-OPTIONS] [userscript...]\n\n Application Options:\n -v, --version Show version and exit\n\n Help Options:\n -h, --help Show this help message\n\n [inspect command options]\n -o, --output-dir= Output directory for the captured profile data, including system and device profiles (default: ./output)\n -n, --num-trace-events= Maximum number of trace events to capture when profiling. Once hitting this limit, no new events are recorded\n --capture-system-profiles Disable capture of system profile data. Can reduce output size.\n --capture-device-profiles Disable capture of device profile data. Can reduce output size.\n\n [inspect command arguments]\n userscript: Run command/script that launches a Neuron workload. E.g. 'python app.py' or './runscript.sh'\n\n\nExample of using System Profiles CLI\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUser can provide any type of their own script to generate a Neuron workload such as Pytorch to the System Profiles CLI. \nFor simplicity, we have a quick way to generate a Neuron workload \nthrough using ``nccom-test``. ``nccom-test`` is a benchmarking tool which is already available with Neuron AMI and ``aws-neuronx-tools`` package.\n\n.. code-block:: shell\n\n ubuntu@ip-172-31-63-210:~$ neuron-profile inspect -o inspect-output-nccom-test nccom-test allg -b 512kb -e 512kb -r 32 -n 10 -d fp32 -w 1 -f 512\n INFO[0000] Running command \"nccom-test allg -b 512kb -e 512kb -r 32 -n 10 -d fp32 -w 1 -f 512\" with profiling enabled\n size(B) count(elems) type time:avg(us) algbw(GB/s) busbw(GB/s)\n 524288 131072 fp32 24.15 21.71 21.03\n Avg bus bandwidth: 21.0339GB/s\n\n.. note::\n If you have problems with nccom-test add the --debug flag.\n If using a trn1.2xlarge instance, change -r 32 to -r 2 to use fewer neuron cores.\n\n.. _neuron-profiler-inspect-output:\n\n``neuron-profile inspect`` Output\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe above command shows a Neuron workload execution is being traced and output to ``inspect-output-nccom-test`` directory. \nYou will see the output directory contains a single NEFF file and a device profile (NTFF) for all Neuron Cores which executed that NEFF. \nYou will also see ``ntrace.pb`` and ``trace_info.pb`` files storing the system profile data.\nBelow showing what the outputs will look like:\n\n.. code-block:: shell\n\n ubuntu@ip-172-31-63-210:~$ tree inspect-output-nccom-test\n inspect-output-nccom-test\n ├── i-012590440bb9fd263_pid_98399\n │ ├── 14382885777943380728_instid_0_vnc_0.ntff\n │ ├── 14382885777943380728_instid_0_vnc_1.ntff\n │ ├── 14382885777943380728_instid_0_vnc_10.ntff\n │ ├── 14382885777943380728_instid_0_vnc_11.ntff\n ...\n │ ├── 14382885777943380728_instid_0_vnc_8.ntff\n │ ├── 14382885777943380728_instid_0_vnc_9.ntff\n │ ├── cpu_util.pb\n │ ├── host_mem.pb\n │ ├── neff_14382885777943380728.neff\n │ ├── ntrace.pb\n │ └── trace_info.pb\n └──\n\n 2 directories, 74 files\n\n\nTo view a summary of the captured profile data run the command\n\n.. code-block:: shell\n\n neuron-profile view -d inspect-output-nccom-test --output-format summary-text\n\n\nEKS User Experience\n-------------------\n\nCapturing a profile on EKS is most easily done through setting of environment variables as described in the section \n:ref:`Non-framework specific User Experience `. By using environment \nvariables, users do not need to change application code in their container image or modify their run commands. \n\nUpdate the deployment yaml to include the ``NEURON_RT_INSPECT_ENABLE`` and ``NEURON_RT_INSPECT_OUTPUT_DIR`` \nenvironment variables. For distributed workloads, it’s important that ``NEURON_RT_INSPECT_OUTPUT_DIR`` points to a \ndirectory on a shared volume which all workers have access to.\n\n.. code-block:: yaml\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: trn1-mlp\n spec:\n restartPolicy: Never\n schedulerName: default-scheduler\n nodeSelector:\n beta.kubernetes.io/instance-type: trn1.32xlarge\n containers:\n - name: trn1-mlp\n env:\n - name: NEURON_RT_INSPECT_ENABLE\n value: \"1\"\n - name: NEURON_RT_INSPECT_OUTPUT_DIR\n value: \"/shared/output\"\n command: ['torchrun']\n args:\n - '--nnodes=1'\n - '--nproc_per_node=32'\n - 'train_torchrun.py'\n image: ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:mlp\n imagePullPolicy: IfNotPresent\n resources:\n limits: \n aws.amazon.com/neuron: 16\n\n\n.. note::\n\n EKS users running PyTorch and JAX applications are still free to change their application code \n and use the PyTorch or JAX Python profiling APIs if they want finer-grained control over profiling. \n However, using the environment variables conveniently allows profiling without modifying the \n container image or application code.\n\nProcessing and Viewing Profiles\n-------------------------------\n\nUsers have three output options for interacting with their captured profiles\n\n* Neuron Profiler UI - Neuron’s custom UI which allows easily drilling down to detailed device profiles from high level system profiles\n* Perfetto - Allows sharing profiles as a single file and viewing your profiles in the Perfetto UI at https://ui.perfetto.dev/\n* JSON - human-readable text output that enables simple scripting \n\nNeuron Profiler UI\n~~~~~~~~~~~~~~~~~~\n\nTo view a profile in the Neuron Profiler UI run the following command to process a profile and launch the UI\n\n.. code-block:: shell\n\n neuron-profile view -d ./output\n\nTo view profiles with the Neuron Profiler UI running locally you will need to have InfluxDB installed on your system. \nTo install and setup InfluxDB follow the :ref:`directions in the official Neuron Profile documentation `.\n\n\nNeuron Profiler System Profile UI\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe system profile timeline shows a trace of Neuron Runtime API calls, ML framework function calls, CPU utilization, and memory usage on each of the instances in your workload. \nThe Neuron Runtime API trace is grouped by NeuronCore IDX and ec2 instance ID. For example, all events in the row \nlabeled nrt-nc-003-i-0f207fb2a99bd2d08 are associated with NeuronCore 3 and instance i-0f207fb2a99bd2d08.\n\nFramework function traces are grouped by thread id and ec2 instance id. For example, all events in \nthe row framework-3266405268-i-0f207fb2a99bd2d08 are framework or application function calls made on thread \n3266405268 running on instance i-0f207fb2a99bd2d08.\n\n\n|neuron-profiler2-annotate-system-ui|\n\nClicking on trace events in the timeline shows a “Event attributes” view with a list of attributes associated with that event. \nFor example, clicking on an nrt_execute event (the Neuron Runtime API call for executing a compiled model on a NeuronCore) \nwill show events such as Flop count (the number of floating point operations for a single execution of the model), \nthe model name, and the NeuronCore idx and ec2 instance id associated with the function call. \n\n|neuron-profiler2-attributes-window|\n\nNeuron Profiler 2.0 allows users to drill-down from a system timeline to a device profile timeline in order to see a detailed view \nof hardware activity during the execution of a graph. To do this, select an nrt_execute event in the timeline and in the \n“Event attributes” view select the \"Open device profile\" button under the Model Name attribute. \nThis will open a new window with a device profile. For help understanding a device profile see the section documentation section \"Understanding a Neuron Profile\"\n\n|neuron-profiler2-drilldown-device|\n\nTo see a list of all device profiles that were captured during your workload press the “Device Profiles” button at the bottom of the timeline. From this list you can \nsee all unique compiled graphs (NEFFs) that were executed on NeuronCores during your workload. For each graph there is a link to a device \nprofile that will show a detailed view of hardware activity on the NeuronCore during execution of this graph. \n\n|neuron-profiler2-device-profile-list|\n\n\nViewing Profiles with Perfetto\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPerfetto is an open-source trace analysis toolkit with a powerful UI for visualizing and analyzing trace data.\nUsers of Neuron Profiler have the option of viewing their profiles in the Perfetto UI.\n\nThe ``--output-format perfetto`` option writes processed data to Perfetto's native protobuf-based tracing format which can be visualized in the Perfetto UI at https://ui.perfetto.dev/.\n\nExample:\n\n.. code-block:: shell\n\n neuron-profile view -d ./output --output-format perfetto\n\nThis will generate a ``system_profile.pftrace`` file for the system profile and a ``device_profile_model_.pftrace`` file for each unique compiled model that was executed on a Neuron Device.\n\nTo view the system profile, go to https://ui.perfetto.dev/ and open the ``system_profile.pftrace`` file.\n\n.. note::\n When loading trace files in the Perfetto UI, your data is processed locally and not uploaded to Perfetto’s servers.\n\n|neuron-profiler2-perfetto-timeline|\n\nTo view a device profile go to https://ui.perfetto.dev/ and open the ``device_profile_model_.pftrace`` file. This will show a detailed view of hardware activity on the NeuronCore during execution of this graph.\n\n|neuron-profiler2-perfetto-device-timeline|\n\n.. note::\n Your browser may run out of memory when viewing ``*.pftrace`` (Perfetto trace) files that are more than a few hundred MB. See the section :ref:`Viewing Large Profiles in Perfetto ` for directions on how to view large traces using the trace processor.\n\n\nPerfetto Output View Options\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen outputting to Perfetto it is possible to group your traces by different attributes. This is useful for\nlarger profiles involving many NeuronCores and instances. The following options are available:\n\n.. list-table:: Perfetto output view options\n :header-rows: 1\n :widths: 30 70\n\n * - CLI option\n - Description\n * - ``--system-trace-primary-group``\n - First-order grouping of trace events (maps to a Perfetto process / process group of rows). Provide a comma-delimited list of field names. Allowed fields: ``instance_id``, ``thread_id``, ``lnc_idx``, ``process_id``. Default: ``instance_id,process_id``.\n * - ``--system-trace-secondary-group``\n - Second-order grouping of trace events (maps to a Perfetto thread / single row). Provide a comma-delimited list of field names. Allowed fields: ``instance_id``, ``worker_gid``, ``thread_id``, ``lnc_idx``, ``process_id``. Default: ``worker_gid,lnc_idx, thread_id``.\n\n\nFor example, the following profile uses ``neuron-profile view --output-format=perfetto --system-trace-primary-group=instance_id,process_id --system-trace-secondary-group=lnc_idx,thread_id`` to group the system profile first by unique combinations\nof instance_id and process_id, and then in each of those groups there are rows of events with unique combinations of lnc_idx and thread_id.\n\n|neuron-profiler2-perfetto-grouping|\n\nGrouping By Global Worker ID\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBy default, Perfetto traces are grouped by ``worker_gid`` which is a unique global identifier for each NeuronCore across all instances in a distributed workload.\nWhen clicking on an event in the trace you will see fields for both ``lnc_idx`` (local NeuronCore index on that process) and ``worker_gid`` (global NeuronCore index across all instances).\nIt is possible for ``lnc_idx`` to be the same for different processes on the same instance or across different instances in a distributed workload. However, ``worker_gid`` is unique for each NeuronCore across all instances.\nThe image below shows how to correlate the naming of tracks (rows) in the Perfetto UI to both ``lnc_idx`` and ``worker_gid``.\n\n|neuron-profiler2-perfetto-gid|\n\n\n\nGenerating JSON Output From Profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe ``--output-format`` json option writes processed profile data to human-readable JSON that can be used for scripting and manual inspection.\n\n.. code-block:: shell\n\n neuron-profile view -d ./output --output-format json\n\nThis will generate a ``system_profile.json`` file containing the system profile data and a ``device_profile_model_.json`` file for each unique compiled model that was executed on a Neuron Device. \n\nThe system_profile.json JSON contains the following data types:\n\n* ``trace_events``: Neuron Runtime API trace events and Framework/Application trace events containing timestamps, durations, names, and the ec2 instance-id to differentiate between events from different compute nodes in a distributed workload.\n\n.. code-block:: json\n\n {\n \"Neuron_Runtime_API_Event\": {\n \"duration\": 27094,\n \"group\": \"nrt-nc-000\",\n \"id\": 1,\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"lnc_idx\": \"0\",\n \"name\": \"nrt_tensor_write\",\n \"parent_id\": 0,\n \"process_id\": \"1627711\",\n \"size\": \"4\",\n \"tensor_id\": \"4900392441224765051\",\n \"tensor_name\": \"_unknown_\",\n \"thread_id\": 1627711,\n \"timestamp\": 1729888371056597613,\n \"type\": 11\n },\n \"Framework_Event\": {\n \"duration\": 3758079,\n \"group\": \"framework-80375131\",\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"name\": \"PjitFunction(matmul_allgather)\",\n \"process_id\": \"701\",\n \"thread_id\": 80375131,\n \"timestamp\": 1729888382798557372,\n \"type\": 99999\n }\n }\n\n* ``mem_usage``: sampled host memory usage \n\n.. code-block:: json\n\n {\n \"duration\": 1,\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"percent_usage\": 9.728179797845964,\n \"timestamp\": 1729888369286687792,\n \"usage\": 51805806592\n }\n\n* ``cpu_util``: sampled CPU utilization. Results are provided per core and per ec2 instance involved in a distributed workload\n\n.. code-block:: json\n\n {\n \"cpu_id\": \"47\",\n \"duration\": 1,\n \"instance_id\": \"i-0f207fb2a99bd2d08\",\n \"timestamp\": 1729888371287337243,\n \"util\": 2.3255813\n },\n\n\nProcessing only system or device profiles\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo reduce processing times it is possible to skip processing of system or device profiles. Sometimes users may only be interested in one or want to start with a limited set of profiling data before exploring the full profile. \n\nTo skip processing of device profiles use the ``--ignore-device-profile`` option. To skip processing of system profiles use the ``--ignore-system-profile`` option. These options can be used with the ``--output-format`` values ``db`` (default), ``perfetto``, or ``json``.\n\nFor example:\n\n.. code-block:: shell\n\n neuron-profile view -d ./output --ignore-device-profile --output-format perfetto\n\n.. _neuron-profiler-filtering-system-profiles:\n\nFiltering System Profiles\n--------------------------\n\nThis guide explains how to filter system trace events to optimize memory usage, reduce output size, and speed up trace processing. **Capture-time filtering** reduces memory usage and trace file size by only collecting specific events, but filtered data cannot be recovered later. **Processing-time filtering** preserves the complete trace and allows flexible analysis with different filters, but requires more memory and storage during capture.\n\nCapture-Time Filtering\n~~~~~~~~~~~~~~~~~~~~~~\n\nConfigure filters before trace capture using environment variables or API functions. \nYou can use NeuronCore filters to only capture events for specific NeuronCores (for example only events associated with NeuronCore 0 or all the NeuronCores on a specific NeuronDevice). \nYou can use event type filters to only capture specific events (for example model execute or collectives events). \nIt is possible to combine both NeuronCore and event type filters.\n\nFiltering by NeuronCore\n^^^^^^^^^^^^^^^^^^^^^^^\n\nIf capture is enabled for a NeuronCore then a ring buffer will be allocated in host memory for storing those core's events. Thus filtering by NeuronCore decreases host memory usage during capture.\n\nDefault Behavior\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nBy default, all visible NeuronCores are enabled for capture. \n\nUsing Environment Variables\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. code-block:: shell\n\n # Filter to capture events only from NeuronCore 0\n export NEURON_RT_INSPECT_EVENT_FILTER_NC=0\n\n # Filter to capture events from NeuronCores 0, 2, and 4\n export NEURON_RT_INSPECT_EVENT_FILTER_NC=0,2,4\n\n # Filter to capture events from a range of NeuronCores (0 through 3)\n export NEURON_RT_INSPECT_EVENT_FILTER_NC=0-3\n\n # Reset to default behavior\n unset NEURON_RT_INSPECT_EVENT_FILTER_NC # Back to capturing all visible cores\n\nUsing API Functions\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. code-block:: c\n\n #include \n\n // Allocate and configure trace options\n nrt_sys_trace_config_t *config;\n nrt_sys_trace_config_allocate(&config);\n nrt_sys_trace_config_set_defaults(config);\n\n // Enable capture only for specific NeuronCores\n\n // Disable all cores since by default they are all enabled\n int num_cores = 128;\n for (int i=0; i\n\n // Get all available event types\n const char **event_types = nullptr;\n size_t count = 0;\n NRT_STATUS status = nrt_sys_trace_get_event_types(&event_types, &count);\n\n if (status == NRT_SUCCESS) {\n printf(\"Available event types:\\n\");\n for (size_t i = 0; i < count; ++i) {\n printf(\" %s\\n\", event_types[i]);\n }\n \n // Free the event types array\n for (size_t i = 0; i < count; ++i) {\n free((void*)event_types[i]);\n }\n free((void*)event_types);\n }\n\nUsing Environment Variables\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe ``NEURON_RT_INSPECT_EVENT_FILTER_TYPE`` environment variable supports:\n\n* **Default**: If not set, all event types are captured\n* **Specific event types**: Use exact event names from ``nrt_sys_trace_get_event_types()``\n* **Event categories**: Use ``hardware`` or ``software`` to filter by category\n* **Exclusion**: Use ``^`` prefix to exclude specific events from a category\n\n.. code-block:: shell\n\n # Filter to capture only specific event types\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=nrt_load,nrt_execute,nc_exec_running\n\n # Filter to capture all hardware events\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=hardware\n\n # Filter to capture all software events\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=software\n\n # Filter to capture all hardware events EXCEPT cc_exec\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=hardware,^cc_running\n\n # Filter to capture all software events EXCEPT nrt_load\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=software,^nrt_load\n\n # Mix categories and specific events\n export NEURON_RT_INSPECT_EVENT_FILTER_TYPE=hardware,nrt_tensor_write,nrt_tensor_read\n\n # Reset to default behavior\n unset NEURON_RT_INSPECT_EVENT_FILTER_TYPE # Back to capturing all event types\n\nThe ``hardware`` group contains events that are executed on the NeuronCore. \nThese are ``nc_exec_running``, ``cc_running``, ``cc_exec_barrier``, ``numerical_err``, ``nrt_model_switch``, ``timestamp_sync_point``, ``hw_notify``.\nThe ``software`` group contains all other events.\n\nUsing API Functions\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nUse the ``nrt_sys_trace_config_set_capture_enabled_for_event_type`` API to filter by event type.\n\n.. code-block:: c\n\n #include \n\n // Configure trace options\n nrt_sys_trace_config_t *config;\n nrt_sys_trace_config_allocate(&config);\n nrt_sys_trace_config_set_defaults(config); // By default, all event types are enabled\n\n // Disable specific event types (others remain enabled)\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, \"device_exec\", false);\n\n // Or disable all first, then enable only specific ones\n const char **all_event_types = nullptr;\n size_t all_count = 0;\n nrt_sys_trace_get_event_types(&all_event_types, &all_count);\n\n // Disable all event types first\n for (size_t i = 0; i < all_count; ++i) {\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, all_event_types[i], false);\n }\n\n // Enable only specific event types\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, \"model_load\", true);\n nrt_sys_trace_config_set_capture_enabled_for_event_type(config, \"nrt_execute\", true);\n\n // Verify which event types are enabled\n const char **enabled_types = nullptr;\n size_t enabled_count = 0;\n nrt_sys_trace_config_get_enabled_event_types(config, &enabled_types, &enabled_count);\n printf(\"Enabled event types: %zu\\n\", enabled_count);\n for (size_t i = 0; i < enabled_count; ++i) {\n printf(\" %s\\n\", enabled_types[i]);\n }\n\n // Clean up memory (caller is responsible)\n for (size_t i = 0; i < enabled_count; ++i) {\n free((void*)enabled_types[i]);\n }\n free((void*)enabled_types);\n\n for (size_t i = 0; i < all_count; ++i) {\n free((void*)all_event_types[i]);\n }\n free((void*)all_event_types);\n\n // Start tracing\n nrt_sys_trace_start(config);\n\n // Your application code here...\n\n // Cleanup\n nrt_sys_trace_stop();\n nrt_sys_trace_config_free(config);\n\n.. _neuron-profile-system-timestamp-adjustment:\n\nAdjusting Hardware Timestamps\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nHardware events executed on the NeuronCore use device-specific timestamps that are in a different time domain than CPU timestamps. To enable accurate correlation between hardware and software events in the JSON system trace output, the runtime automatically adjusts hardware event timestamps to the CPU time domain using synchronization point events.\n\nHow Timestamp Adjustment Works\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nSystem trace events are generated from multiple independent time domains: the CPU host and each ML accelerator devices operating with their own clocks. To align events from different domains, the runtime performs software-based time synchronization after event collection.\n\n**Sync Point Events**: After each execution, a special ``timestamp_sync_point`` event captures nearly simultaneous timestamps from both the host CPU (``cpu_timestamp_ns``) and the device (``nc_timestamp_ns``). These sync events are used to adjust the timestamps of hardware events to the CPU domain. \nThese synchronization events are included in the returned event trace and serve as reference points for timestamp adjustment. Users can see the sync point used for aligning hardware events in the timeline.\n\n**Adjustment Algorithm**: For each hardware event, the runtime:\n\n- Uses the sync point with matching exec_id for that NeuronCore\n- Calculates the time difference between the hardware event and the sync point (in device time)\n- Applies that same time difference to the sync point's CPU timestamp\n- Formula: ``adjusted_timestamp = sync_cpu_timestamp + (event_device_timestamp - sync_device_timestamp)``\n\nIllustration::\n\n Sync_Point HW_Event\n │ │\n ▼ ▼\n Device Time ─●────────────────●───>\n |-------Δt------>| - sync_device_timestamp and sync_cpu_timestamp occur ~simultaneously, though their clocks differ\n CPU Time ────●────────────────●───> - Calc Δt = event_device_timestamp - sync_device_timestamp (elapsed time since sync point on device)\n |-------Δt------>| - Add Δt to sync_cpu_timestamp to get adjusted_timestamp\n\n|neuron-profiler2-syncpoint-timeline|\n\n**Hardware Events**: Hardware events that require timestamp adjustment include:\n\n- ``nc_exec_running`` (NeuronCore execution start/stop)\n- ``cc_running`` (collective communication execution)\n- ``cc_exec_barrier`` (collective communication barriers)\n- ``numerical_err`` (numerical errors)\n- ``nc_model_switch`` (NeuronCore model switching)\n\nTips\n^^^^\n\n1. **Memory Optimization**: Use NeuronCore filtering to avoid allocating ring buffers for unused cores and decrease host memory usage. Use both event type or NeuronCore to decrease output trace sizes.\n2. **Event Type Discovery**: Use ``nrt_sys_trace_get_event_types()`` to discover available event types\n3. **Category Filtering**: Use ``hardware``/``software`` categories for broad filtering\n4. **Exclusion Filtering**: Use ``^`` prefix to exclude specific events from categories\n5. **Combine Filters**: Use both NeuronCore and event type filters together for maximum optimization\n\nProcessing-Time Filtering\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nApply filters when viewing or processing already captured profiles. This approach allows you to \nanalyze the same trace data in different ways without recapturing. The filters can be used for any \n``neuron-profile`` output format including ``--output-format json`` and ``--output-format perfetto``.\n\nFiltering by NeuronCore\n^^^^^^^^^^^^^^^^^^^^^^^\n\nUse the ``--system-trace-filter-neuron-core`` to only process events for specific NeuronCores. The IDs are local to the instance and not global IDs. \n\nIf the ``--system-trace-filter-neuron-core`` argument is not set then events from all NeuronCores will be included in the processed trace.\n\n.. code-block:: shell\n\n # Filter by single neuron core\n neuron-profile view -d ./output --system-trace-filter-neuron-core \"0\" --output-format perfetto\n\n # Filter by multiple neuron cores\n neuron-profile view -d ./output --system-trace-filter-neuron-core \"0,1,2,3\" --output-format perfetto\n\nFiltering by Event Type\n^^^^^^^^^^^^^^^^^^^^^^^\n\nUse the ``--system-trace-filter-event-type`` to only process specific trace events types.\n\nIf the ``--system-trace-filter-event-type`` argument is not set then all event types will be included in the processed trace.\n\n.. code-block:: shell\n\n # Filter by single event type\n neuron-profile view -d ./output --system-trace-filter-event-type \"nrt_execute\" --output-format perfetto\n\n # Filter by multiple event types\n neuron-profile view -d ./output --system-trace-filter-event-type \"nrt_execute,nrt_load\" --output-format perfetto\n\nFiltering by Instance ID\n^^^^^^^^^^^^^^^^^^^^^^^^\n\nUse the ``--system-trace-filter-instance-id`` to only process events for specific ec2 instances.\n\nIf the ``--system-trace-filter-instance-id`` argument is not set then events from all instances will be included in the processed trace.\n\n.. code-block:: shell\n\n # Filter by single instance\n neuron-profile view -d ./output --system-trace-filter-instance-id \"i-abc123\" --output-format perfetto\n\n # Filter by multiple instances (comma-separated)\n neuron-profile view -d ./output --system-trace-filter-instance-id \"i-abc123,i-def456,i-ghi789\" --output-format perfetto\n\nTroubleshooting\n---------------\n\nIncomplete JAX Profiles\n~~~~~~~~~~~~~~~~~~~~~~~\n\nIf your JAX profile has fewer events than expected or lacks the Runtime API trace, check whether \n``jax.profiler.stop_trace`` is being called inside a ``with jax.profiler.trace`` context block. \nThis can prematurely stop tracing. Use ``jax.profiler.stop_trace`` only when profiling was started \nwith ``jax.profiler.start_trace``, not when using the context-managed ``with jax.profiler.trace`` API.\n\nAlso when using ``jax.profiler`` within your script ensure that the \nenvironment variable ``NEURON_RT_INSPECT_ENABLE`` is not set to 1. \nAdditionally, ensure that ``NEURON_RT_INSPECT_OUTPUT_DIR`` is set to \nthe correct output directory and this is the output directory passed to \n``with jax.profiler.trace``.\n\nDropped Events in System Profile\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen processing a system profile, you may see a warning indicating that some trace events were dropped during capture.\n\n.. code-block:: shell\n\n WARN[0000] Warning: 1001 trace events were dropped during capture (stored 530560 out of 531561 total events). Consider increasing buffer size, reducing trace duration, or filtering events.\n\nThis means during capture the trace event buffers filled and oldest events were overwritten. If you need to avoid dropping events for the full duration of your workload consider the following adjustments:\n\n* Increase buffer size by setting ``NEURON_RT_INSPECT_SYS_TRACE_MAX_EVENTS_PER_NC`` (see :ref:`Profile Capture Environment Variables `). This will increase host memory usage.\n* Apply capture-time filters (NeuronCores / event types) (see :ref:`Filtering System Profiles `.)\n* Shorten profiled region: limit the code span under the profiling context / runtime.\n\n\n.. |neuron-profiler2-annotate-system-ui| image:: /images/neuron-profiler2-annotate-system-ui.png\n.. |neuron-profiler2-attributes-window| image:: /images/neuron-profiler2-attributes-window.png\n.. |neuron-profiler2-device-profile-list| image:: /images/neuron-profiler2-device-profile-list.png\n.. |neuron-profiler2-drilldown-device| image:: /images/neuron-profiler2-drilldown-device.png\n.. |neuron-profiler2-perfetto-timeline| image:: /images/neuron-profiler2-perfetto-timeline.png\n.. |neuron-profiler2-perfetto-device-timeline| image:: /images/neuron-profiler2-perfetto-device-timeline.png\n.. |neuron-profiler2-perfetto-grouping| image:: /images/neuron-profiler2-perfetto-grouping.png\n.. |neuron-profiler2-syncpoint-timeline| image:: /images/neuron-profiler2-syncpoint-timeline.png\n.. |neuron-profiler2-perfetto-gid| image:: /images/neuron-profiler2-perfetto-gid.png\n" }, { "path": "tools/tensorboard/getting-started-tensorboard-neuronx-plugin.rst", "content": ".. _neuronx-plugin-tensorboard:\n\nNeuronX Plugin for TensorBoard (Trn1)\n======================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\n\nOverview\n--------\n\nThis guide is for developers who want to better understand how their\nmodel is executed using Neuron SDK through TensorBoard.\n\nThe Neuron plugin for TensorBoard provides metrics to the performance of machine learning tasks accelerated using the Neuron SDK. It is\ncompatible with TensorBoard versions 1.15 and higher. It provides visualizations and profiling results for graphs executed on NeuronCores.\n\n.. note::\n\n The following information is compatible with Neuron SDK for Trn1. For a walkthrough on Inf1, please check out the guide\n :ref:`neuron-plugin-tensorboard`.\n\n\nEnable profiling on Trn1\n------------------------\n\n.. note::\n\n Profiling is currently only supported with PyTorch Neuron (``torch-neuronx``).\n\nPlease refer to the following guides:\n\n- PyTorch-Neuron\n - :ref:`torch-neuronx-profiling-with-tb`\n\n\nLaunch TensorBoard\n------------------\n\nIn this step, we will process the Neuron profile data and launch TensorBoard.\n\n1. Install the Neuron plugin for Tensorboard on your EC2 instance.\n\n.. code:: bash\n\n\n pip install tensorboard-plugin-neuronx --extra-index-url https://pip.repos.neuron.amazonaws.com\n\n.. note::\n\n If using TensorBoard >= 2.5, please use the ``--load_fast=false`` option when launching.\n ``tensorboard --logdir results --load_fast=false``\n\n2. After you see the following message, TensorBoard is ready to use. By default,\nTensorBoard will be launched at ``localhost:6006``.\n\n::\n\n ...\n Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n TensorBoard 2.4.1 at http://localhost:6006/ (Press CTRL+C to quit)\n\n\nView results in TensorBoard\n---------------------------\n\nIn this step, we will view the Neuron plugin for TensorBoard from a browser on your local\ndevelopment machine.\n\n1. Connect to the EC2 instance where TensorBoard is running while enabling port forwarding.\nIn this example, we assume TensorBoard has been launched using the default address ``localhost:6006``.\n\n.. code:: bash\n\n # if Ubuntu-based AMI\n ssh -i ubuntu@ -L 6006:localhost:6006\n\n # if AL2-based AMI\n ssh -i ec2-user@ -L 6006:localhost:6006\n\n2. In a browser, visit |tensorboard_address|.\n\n3. In the top navigation bar, switch from ``Graphs`` to ``Neuron``. If it does not show up,\nplease wait a while and refresh the page while the plugin loads. If the issue persists, check\nthe ``Inactive`` dropdown list on the right and check for ``Neuron``.\n\n|image1|\n\n4. If TensorBoard failed to find the generated logs, you will see the following message:\n\n|image2|\n\n\nIn this case, please make sure the version of the ``aws-neuronx-tools``\npackage and the Neuron framework package is from Neuron release 2.6 or newer.\n\n\nNeuron Trace View\n-----------------\n\n|image3|\n\nThe trace view gives a high level timeline of execution by aligning Neuron events, such as Neuron Device execution,\ndata transfers, and Collective Compute synchronization (if applicable), with other events from the XLA profiler.\n\nUse this view to better understand bottlenecks during the run, and potentially experiment with how execution changes\nby moving the ``mark_step()`` call which will execute the graph.\n\n\nNeuron Operator View\n--------------------\n\n|image4|\n\nThe operator view can show timing information for both the framework operators and HLO operators by selecting\nthe ``operator-framework`` and ``operator-hlo`` tools respectively. The pie charts show breakdowns of the time taken\nby device, as well as per operator on a single device. The table below lists out the operators and can be sorted by clicking\non the columnn headers. For fused operations, hover over the ``?`` to see which operators are being executed.\n\nFor a quick glance at the most time consuming operators, click the ``Time %`` column in the table to sort by the relative\ntime spent on this type of operation compared to the rest of the model.\n\n\nNeuron Operator Timeline View\n-----------------------------\n\n|image5|\n\nThe operator timeline view is a detailed look into a single execution with Neuron. A high level overview at the top breaks\ndown the execution into categories, including Neuron Runtime setup time, as well as NeuronCore compute engine and DMA engine busyness.\nActivity on the compute and DMA engines are further categorized as compute, control, and data transfer intervals which are\nshown as separate processes, with each showing a hierarchical view of the framework operators and their corresponding\nHLO operation. The fused operations can be a result of compiler optimizations or are operations that are running in\nparallel on the device. Each bar can be clicked to show information regarding which operators are overlapped.\n\nThis view can give better insight into how operators translate to Neuron, as well as how certain Neuron compiler options\nmay improve performance.\n\n\nTroubleshooting\n---------------\n\nTensorBoard launch fails\n~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n ImportError: cannot import name 'Mapping' from 'collections'\n\nThis is an issue with Python 3.10 and a dependency of an old tensorboard version. To workaround this error, please run\n``pip install --upgrade tensorboard``. For more information, see https://github.com/tensorflow/tensorboard/pull/5490.\n\n.. |image1| image:: /images/Neuron_Profiler_Tensorboard_Dropdown.jpg\n.. |image2| image:: /images/tb-plugin-img12.png\n :height: 2914\n :width: 5344\n :scale: 10%\n.. |image3| image:: /images/Neuron_Profiler_Runtime_Trace_Original.jpg\n.. |image4| image:: /images/Neuron_Profiler_T1_Op_Framework_View.png\n.. |image5| image:: /images/TB_Operator_Timeline_2-10.png\n.. |tensorboard_address| raw:: html\n\n localhost:6006\n" }, { "path": "tools/tensorboard/index.rst", "content": ".. _tensorboard-neuron:\n\nTensorBoard\n===========\n\nTensorBoard integration with AWS Neuron provides powerful visualization and debugging capabilities for machine learning workloads. The Neuron TensorBoard plugins enable developers to monitor training progress, analyze model performance, and debug compilation issues through familiar TensorBoard interfaces.\n\n.. toctree::\n :maxdepth: 1\n :hidden:\n\n TensorBoard for NeuronX \n\nTensorBoard for Trn1\n--------------------\n\n.. grid:: 1\n :gutter: 3\n\n .. grid-item-card:: TensorBoard Plugin for NeuronX (Trn1)\n :link: /tools/tensorboard/getting-started-tensorboard-neuronx-plugin\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Comprehensive guide for using the TensorBoard Neuron plugin on Trn1 instances, including installation, configuration, and advanced visualization features.\n\n .. grid-item-card:: Profiling PyTorch NeuronX (``torch-neuronx``) with TensorBoard\n :link: /tools/tutorials/torch-neuronx-profiling-with-tb\n :link-type: doc\n :class-header: sd-bg-primary sd-text-white\n\n Step-by-step tutorial for monitoring PyTorch training progress on Trn1 instances using TensorBoard scalars, metrics visualization, and performance tracking.\n\n" }, { "path": "tools/third-party-solutions.rst", "content": ".. _third-party-tool-solutions:\n\nThird-party solutions\n======================\n\nAWS Neuron integrates with multiple third-party partner solutions that alow you to run deep learning workloads on Amazon EC2 \ninstances powered by AWS Trainium and AWS Inferentia chips. The following list gives an overview of third-party solutions \nthat work with AWS Neuron.\n\nDatadog\n\"\"\"\"\"\"\"\nDatadog, an observability and security platform, provides real-time monitoring for cloud infrastructure and ML operations. Datadog is \nexcited to launch its AWS Neuron integration, which pulls metrics collected by Neuron SDK’s Neuron Monitor tool into Datadog, \nenabling users to track the performance of their Trainium and Inferentia-based instances. By providing real-time visibility into \nmodel performance and hardware usage, Datadog helps customers ensure efficient training and inference, optimized resource \nutilization, and the prevention of service slowdowns.\n\n`Datadog documentation `_\n\n" }, { "path": "tools/tutorials/index.rst", "content": ".. _neuron-tools-tutorials:\n\nTutorials\n============\n\n.. toctree::\n :hidden:\n :maxdepth: 1\n\n performance-profiling-vllm\n torch-neuronx-profiling-with-tb\n tutorial-tensorboard-scalars-mnist\n tutorial-neuron-monitor-mnist\n\n.. grid:: 1 2 2 2\n :gutter: 3\n\n .. grid-item-card:: Profiling a vLLM Inference Workload\n :link: /tools/tutorials/performance-profiling-vllm\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to capture and analyze device-level and system-level profiles for vLLM inference workloads on AWS Trainium. \n\n .. grid-item-card:: Profiling a NKI Kernel\n :link: /nki/guides/use-neuron-profile\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to profile a NKI kernel with Neuron Explorer.\n\n .. grid-item-card:: Profiling PyTorch Neuron with TensorBoard\n :link: tutorial-tensorboard-scalars-mnist\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to use Neuron's plugin for TensorBoard that allows users to measure and visualize performance on a torch runtime level or an operator level.\n\n .. grid-item-card:: Track System Resource Utilization during Training with Neuron Monitor\n :link: tutorial-neuron-monitor-mnist\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to monitor resource utilization using neuron-monitor, Prometheus and Grafana while running a multi-layer perceptron MNIST model on Trainium using PyTorch Neuron.\n\n .. grid-item-card:: Track Training Progress in TensorBoard using PyTorch Neuron\n :link: torch-neuronx-profiling-with-tb\n :link-type: doc\n :class-card: sd-border-1\n\n Learn how to track training progress in TensorBoard while running a multi-layer perceptron MNIST model on Trainium using PyTorch Neuron.\n" }, { "path": "tools/tutorials/performance-profiling-vllm.rst", "content": ".. meta::\n :description: Learn how to use Neuron Explorer to capture and analyze system-level and device-level profiles for vLLM inference workloads on AWS Trainium\n :date-modified: 12/02/2025\n\nProfiling a vLLM Inference Workload on AWS Trainium\n==========================================================================\n\nThis tutorial outlines the steps involved in using Neuron Explorer to capture and view system-level and device-level profiles for a vLLM-hosted inference workload on AWS Trainium.\n\nOverview\n--------\n\nBy following this tutorial you will learn how to:\n\n* Launch a vLLM-hosted inference workload on AWS Trainium with system and device-level profiling enabled\n* View the system-level profile using Perfetto\n* Identify regions within the system profile that show LLM context-encoding (prefill) and token generation (decode) running on the NeuronDevices, along with the names of the associated compute graphs\n* View the device-level profiles for context-encoding & token generation compute graphs in the Neuron Explorer UI\n\nPrepare your environment\n------------------------\n\nThe following steps show how to launch a Trainium EC2 instance using the latest Neuron Deep Learning AMI (DLAMI) and then install vLLM so that an example vLLM-hosted model can be profiled using the Neuron Explorer. If you would prefer to use a containerized environment (Docker, EKS), please refer to the Neuron documentation to get started with a Neuron Deep Learning Container (DLC) image that has vLLM pre-installed.\n\n1. Launch a Trainium instance (trn1.32xlarge, trn2.3xlarge, trn2.48xlarge)\n 1. Option 1: Launch the instance using the latest AWS Deep Learning AMI (DLAMI), which includes the Neuron SDK preinstalled. Once the instance is launched, please SSH into it and use the virtual environment for neuronx-distributed-inference by following this command -\n 1. ``source /opt/aws_neuronx_venv_pytorch_2_9_nxd_inference/bin/activate``\n 2. Option 2: If using a fresh Linux instance, manually install the latest Neuron packages by following the AWS Neuron installation guide.\n2. Install vLLM\n 1. Refer to the Neuron documentation which outlines how to install the Neuron vLLM fork from source.\n\nStep 1: Save a smaller version of your model\n--------------------------------------------\n\nWhen profiling LLMs it is usually desirable to use only a subset of the model's layers in order to understand model performance and to identify possible bottlenecks. Capturing traces for the entire model could lead to an excessive volume of profiling data, making analysis cumbersome. To address this, the following script takes the Qwen3-8B-base model, truncates it to the first 4 layers, and saves the resulting smaller model for profiling purposes.\n\n.. code-block:: python\n\n import transformers\n\n model_id = \"Qwen/Qwen3-8B-Base\"\n config = transformers.AutoConfig.from_pretrained(model_id)\n config.num_hidden_layers = 4\n config.layer_types = [\"full_attention\"] * 4\n tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)\n output_dir = \"4layer_qwen3\"\n\n model = transformers.AutoModelForCausalLM.from_pretrained(model_id, config=config)\n model.save_pretrained(output_dir)\n tokenizer.save_pretrained(output_dir)\n\nSave the above python script as ``save_4layer_qwen.py`` and then run it using the python interpreter:\n\n.. code-block:: bash\n\n python3 ./save_4layer_qwen.py\n\nOnce the script has completed, you should see the new ``4layer_qwen`` directory which contains the truncated model.\n\nStep 2: Run a vLLM offline inference workload with profiling enabled\n--------------------------------------------------------------------\n\nIn this step, you will run a small vLLM offline inference script that will compile, run, and profile your 4-layer Qwen3 model on the Trainium chips.\n\nBegin by saving the following python script as ``qwen3_offline_inference.py``:\n\n.. code-block:: python\n\n import os\n os.environ['VLLM_NEURON_FRAMEWORK'] = \"neuronx-distributed-inference\"\n\n # Enable Neuron profiling via environment variables\n os.environ['XLA_IR_DEBUG'] = \"1\"\n os.environ['XLA_HLO_DEBUG'] = \"1\"\n os.environ['NEURON_FRAMEWORK_DEBUG'] = \"1\"\n os.environ['NEURON_RT_INSPECT_ENABLE'] = \"1\"\n os.environ['NEURON_RT_INSPECT_SYSTEM_PROFILE'] = \"1\"\n os.environ['NEURON_RT_INSPECT_DEVICE_PROFILE'] = \"1\"\n os.environ['NEURON_RT_INSPECT_OUTPUT_DIR'] = \"./neuron_profiles\"\n\n from vllm import LLM, SamplingParams\n\n # Sample prompts.\n prompts = [\n \"The president of the United States is\",\n \"The capital of France is\",\n \"The future of AI is\",\n ]\n # Create a sampling params object.\n sampling_params = SamplingParams(top_k=1)\n\n # Create an LLM instance using the 4-layer Qwen3 model\n llm = LLM(\n model=\"4layer_qwen3\",\n max_num_seqs=4,\n max_model_len=128,\n additional_config={\n \"override_neuron_config\": {\n \"enable_bucketing\":False,\n },\n },\n enable_prefix_caching=False,\n tensor_parallel_size=8)\n\n # Run inference using the sample prompts\n outputs = llm.generate(prompts, sampling_params)\n\nNext, run the offline inference script with a Python interpreter:\n\n.. code-block:: bash\n\n python3 ./qwen3_offline_inference.py\n\nAfter ~60s the script should complete, and you will see a new ``neuron_profiles`` directory which contains both system-level and device-level profile traces for this example inference workload.\n\nStep 3: Visualize the system profile for your model\n---------------------------------------------------\n\n.. note::\n System profiles are currently viewed using the open-source Perfetto tool. Viewing of system profiles will be natively supported by the Neuron Explorer UI in an upcoming release.\n\nRun the following command to generate a Perfetto compatible file from the system profile traces that you previously captured:\n\n.. code-block:: bash\n\n neuron-explorer view -d ./neuron_profiles --ignore-device-profile \\\n --output-format perfetto\n\nThe above command generates a file called ``system_profile.pftrace`` in your working directory.\n\nCopy the ``system_profile.pftrace`` file to your local machine and open up the Perfetto UI in your local web browser.\n\nIn the left-hand menu, choose \"Open trace file\" and select your ``system_profile.pftrace`` file to view the system profile. Expand the first row under Default Workspace and you will see a timeline view similar to the following:\n\n.. image:: /tools/profiler/images/perf-profiling-1.png\n\nThe system profile shows a high-level chronological view of the various Neuron Runtime API calls that took place during your example inference workload. If you hover the mouse cursor over the various pink/green bars you can see which specific API call occurred at each time point, such as ``nrt_tensor_read``, ``nrt_tensor_write``, ``nrt_execute``, and ``nrt_load_collectives``.\n\nLook for the **nrt_execute** bar identified below and select it. This will open an information dialog providing details of the specific ``nrt_execute`` call:\n\n.. image:: /tools/profiler/images/perf-profiling-2.png\n\n.. image:: /tools/profiler/images/perf-profiling-3.png\n\nIn the Arguments pane you will find useful information such as the following:\n\n* device_profile - the unique name of the device profile associated with this event\n* nc_idx - the index of the NeuronCore that is associated with this API call\n* model_name - path to the compiled Neuron Executable File Format (NEFF) compute graph associated with this event\n\nIn the above screenshot, notice that the model_name field provides additional information about what is happening during this part of the model execution:\n\n.. code-block:: text\n\n tmp/nxd_model/context_encoding_model/_tp0_bk0/model.MODULE_6d1668c2294e2409dd72+ad9e832d.neff\n\n* ``context_encoding_model`` - indicates that this is handling context-encoding (prefill) during vLLM inference (other model names will alternatively include token_generation_model to indicate the token-generation / decode phase of inference).\n* ``tp0`` - indicates that this profile is associated with the rank0 of the tensor-parallel (TP) replica group\n* ``bk0`` - indicates that this profile is associated with the first sequence bucket as configured in Neuronx Distributed Inference (NxDI) NeuronConfig.\n\nStep 4: Visualize device profiles in Neuron Explorer\n----------------------------------------------------\n\nIn this step, you will view a device profile for your model in Neuron Explorer UI.\n\nIf you look inside the ``neuron_profiles`` directory that was created during Step 2, you will see many Neuron Executable File Format (NEFF) and their associated Neuron Trace File Format (NTFF) files. For each pair of NEFF/NTFF files, the NEFF represents the Neuron-compiled compute graph for a portion of your model, and the NTFF represents the device-level profile trace for that specific compute graph.\n\nWhile you are free to view any of the device-level profiles using the Neuron Explorer UI, it is often more useful to start from the system-level profile and identify a specific device-level profile of interest. Let's refer back to the nrt_execute region of the system-level profile that was covered in the previous section. Please find and left-click this region to bring up the information dialog at the bottom of Perfetto:\n\n.. image:: /tools/profiler/images/perf-profiling-4.png\n\n.. image:: /tools/profiler/images/perf-profiling-5.png\n\nIn the device_profile field, note that numerical ID that is included at the end of the device profile name, in this case 2120860766. This ID is what you will use to locate the NEFF/NTFF pair associated with this specific nrt_execute API call.\n\nUse the following find command (substituting-in your device profile ID) to locate the NEFF/NTFF files associated with your identified ID:\n\n.. code-block:: bash\n\n find ./neuron_profiles -name \\*2120860766\\* | sort\n\n.. image:: /tools/profiler/images/perf-profiling-6.png\n\nIn the above output you can see that there is a single NEFF file ``neff_2120860766.neff``, and multiple NTFF files ``2120860766_instid_0_vnc_0.ntff`` ... ``2120860766_instid_0_vnc_7.ntff`` each representing the profile trace for one of the 8 NeuronCores that participated in this inference request.\n\nThese are the files you will open in the Neuron profiler UI to inspect the device-level execution.\n\nPlease copy the NEFF and one of the NTFF files to your local machine, as you will need to upload the files to the Neuron Explorer UI using your web browser.\n\nTo view the Neuron Profile Web UI, execute the ``view`` command to start the Neuron Explorer web UI:\n\n.. code-block:: bash\n\n $ neuron-explorer view --data-path ./ --output-format parquet\n\n```` is a path that neuron-explorer will use for storing and managing profiles.\n\nThe above command also prints a URL that you can click to open the web UI:\n\n.. code-block:: text\n\n View a list of profiles at http://localhost:3001/\n\nIf ``neuron-explorer view`` is run on a remote instance, you may need to use port forwarding to access the web UI. By default, ``neuron-explorer`` creates a web server on port 3001 and the API server on port 3002. To enable connection to your browser on your local computer, you must to establish an SSH tunnel to both ports 3001 and 3002.\n\nFor example:\n\n.. code-block:: bash\n\n ssh -L 3001:localhost:3001 -L 3002:localhost:3002 @ -fN\n\nIf you created an EC2 instance with PEM credentials, include them in the SSH tunnel as seen below:\n\n.. code-block:: bash\n\n ssh -i ~/my-ec2.pem -L 3001:localhost:3001 -L 3002:localhost:3002 ubuntu@[PUBLIC_IP_ADDRESS] -fN\n\nOnce the SSH tunnel is setup, you can now open a browser and navigate to http://localhost:3001.\n\nWith the Neuron Explorer UI open, go to \"Profile Manager\", and click \"Upload Profile\" at the top-right of the screen. Give your profile an appropriate name, and upload the NEFF and NTFF files that you previously identified:\n\n.. image:: /tools/profiler/images/perf-profiling-7.png\n\nAfter a few seconds, you should receive a message indicating that NEFF/NTFF were uploaded successfully:\n\n.. image:: /tools/profiler/images/perf-profiling-8.png\n\nWithin the Neuron Explorer UI, go tot he Profile Manager screen and look for your newly uploaded profile.\n\n.. image:: /tools/profiler/images/perf-profiling-9.png\n\nDepending on the size of your profile, it could take a few minutes before the Status field shows \"PROCESSED\". Once processing is complete, click the profile name to open the profile:\n\n.. image:: /tools/profiler/images/perf-profiling-10.png\n\nConfirmation\n------------\n\nCongratulations, you have now successfully generated both system-level and device-level profiles for a vLLM inference workload using Neuron Explorer and learned how to visualize them. This knowledge will enable you to effectively analyze the performance characteristics of your workload and identify potential optimization opportunities.\n\nClean up\n--------\n\nAfter completing your profiling experiments, remember to terminate the instance you launched to avoid unnecessary costs.\n\nNext steps\n----------\n\nNow that you've completed this tutorial, try profiling your own model to analyze its workload. Identify performance gaps, apply optimizations, and profile again to measure the improvements. For a deeper dive into performance analysis, check out Neuron's blog series on profiling." }, { "path": "tools/tutorials/torch-neuronx-profiling-with-tb.rst", "content": ".. _torch-neuronx-profiling-with-tb:\n\nProfiling PyTorch NeuronX with TensorBoard\n==============================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nIntroduction\n------------\n\nNeuron provides a plugin for TensorBoard that allows users to measure and visualize\nperformance on a torch runtime level or an operator\nlevel. With this information, it becomes quicker to identify any\nperformance bottleneck allowing for quicker addressing of that issue.\n\nFor more information on the Neuron plugin for TensorBoard, see :ref:`neuronx-plugin-tensorboard`.\n\nSetup\n-----\n\nPrerequisites\n~~~~~~~~~~~~~\n\n1. Initial `Trn1 setup for PyTorch\n (torch-neuronx) `__\n has been done\n\nEnvironment\n~~~~~~~~~~~\n\n::\n\n #activate python virtual environment and install tensorboard_plugin_neuron\n source ~/aws_neuron_venv_pytorch_p38/bin/activate\n pip install tensorboard_plugin_neuronx\n\n #create work directory for the Neuron Profiling tutorials\n mkdir -p ~/neuron_profiling_tensorboard_examples\n cd ~/neuron_profiling_tensorboard_examples\n\n\n\nPart 1: Operator Level Trace for ``xm.markstep()`` workflow\n-------------------------------------------------------------\n\nGoal\n~~~~\n\nAfter completing this tutorial, the user should be able to understand\nthe features of the Operator Level Trace. The user should also be able\nto form a narrative/surface level analysis from what is being presented\nin the Operator Level Trace.\n\nSet Up\n~~~~~~\n\nLet’s set up a directory containing the material for this demo\n\n::\n\n cd ~/neuron_profiling_tensorboard_examples\n mkdir tutorial_1\n cd tutorial_1\n\n # this is where our code will be written\n touch run.py\n\nHere is the code for ``run.py``:\n\n::\n\n import os\n import torch\n import torch_neuronx\n from torch_neuronx.experimental import profiler\n import torch_xla.core.xla_model as xm\n\n os.environ[\"NEURON_CC_FLAGS\"] = \"--cache_dir=./compiler_cache\"\n\n device = xm.xla_device()\n\n class NN(torch.nn.Module):\n def __init__(self):\n super().__init__()\n\n self.layer1 = torch.nn.Linear(4,4)\n self.nl1 = torch.nn.ReLU()\n self.layer2 = torch.nn.Linear(4,2)\n self.nl2 = torch.nn.Tanh()\n\n def forward(self, x):\n x = self.nl1(self.layer1(x))\n return self.nl2(self.layer2(x))\n\n with torch.no_grad():\n\n model = NN()\n\n inp = torch.rand(4,4)\n output = model(inp)\n\n with torch_neuronx.experimental.profiler.profile(\n port=9012,\n profile_type='operator',\n ms_duration=10000 ):\n \n \n # IMPORTANT: the model has to be transferred to XLA within\n # the context manager, otherwise profiling won't work\n neuron_model = model.to(device)\n neuron_inp = inp.to(device)\n \n output_neuron = neuron_model(neuron_inp)\n xm.mark_step() \n\n print(\"==CPU OUTPUT==\")\n print(output)\n print()\n print(\"==TRN1 OUTPUT==\")\n print(output_neuron)\n\n\nUnderstanding the Code\n~~~~~~~~~~~~~~~~~~~~~~\n\nFor this first tutorial, we’ll be using a simple Feed forward NN model.\nHowever, once the TensorBoard dashboard is up, we’ll see some\ninteresting and unexpected things. A simple model is helpful since it is\neasy to reference back to.\n\nAnother important part is the “operator” profiling type we specified in the context manager.\n\n**Low Level:** The “operator“ dashboard is the dashboard that contains\nthe Operator Level Trace This view also only zooms in on the\nNeuronDevice, while the ”trace“ dashboard shows processes from all\ndevices. The Operator Level Trace View is organized by levels of\nabstraction, with the top level showing the model class. The next lower\ntier shows model components, and the lowest tier shows specific\noperators that occur for a specific model component. This view is useful\nfor identifying model bottlenecks at the operator level.\n\nWe also print out the outputs from the CPU model and the TRN1 model to note\nthe small differences in output.\n\nRunning The Profiler\n~~~~~~~~~~~~~~~~~~~~\n\n::\n\n python run.py\n\n**Output:**\n\nInitial Output & Compilation Success\n\n::\n\n 0% 10 20 30 40 50 60 70 80 90 100%\n |----|----|----|----|----|----|----|----|----|----|\n ***************************************************\n Analyzing dependencies of Block1\n 0% 10 20 30 40 50 60 70 80 90 100%\n |----|----|----|----|----|----|----|----|----|----|\n ***************************************************\n Analyzing dependencies of Block1\n 0% 10 20 30 40 50 60 70 80 90 100%\n |----|----|----|----|----|----|----|----|----|----|\n ***************************************************\n Dependency reduction of sg0000\n 0% 10 20 30 40 50 60 70 80 90 100%\n |----|----|----|----|----|----|----|----|----|----|\n ***************************************************\n\nProcessing the Neuron Profiler Traces\n\n::\n\n torch_neuron: Waiting for XLA profile completion ...\n torch_neuron: translate_xplane: Processing plane: '/host:CPU'\n torch_neuron: XLA decode - Read filename 2023_04_28_00_54_04\n torch_neuron: XLA decode - Read date parts ['2023', '04', '28', '00', '54', '04']\n torch_neuron: XLA decode - Read start date 2023-04-28 00:54:04 from directory stamp\n torch_neuron: translate_xplane: Processing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_op_timeline_split.json'\n torch_neuron: translate_xplane: Writing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_op_timeline_split.json' to 'temp_profiler_logs/c1a992f0ea378f7a_1/neuron_op_timeline_split.json'\n torch_neuron: translate_xplane: Processing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_op_timeline.json'\n torch_neuron: translate_xplane: Writing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_op_timeline.json' to 'temp_profiler_logs/c1a992f0ea378f7a_1/neuron_op_timeline.json'\n torch_neuron: translate_xplane: Processing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_hlo_op.json'\n torch_neuron: translate_xplane: Writing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_hlo_op.json' to 'temp_profiler_logs/c1a992f0ea378f7a_1/neuron_hlo_op.json'\n torch_neuron: translate_xplane: Processing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_framework_op.json'\n torch_neuron: translate_xplane: Writing plane: '/host:Neuron-runtime:profile//c1a992f0ea378f7a_1/model10001/node5/plugins/neuron/1682643254/neuron_framework_op.json' to 'temp_profiler_logs/c1a992f0ea378f7a_1/neuron_framework_op.json'\n\nPrinting output from CPU model and Trn1 Model:\n\n::\n\n ==CPU OUTPUT==\n tensor([[-0.1396, -0.3266],\n [-0.0327, -0.3105],\n [-0.0073, -0.3268],\n [-0.1683, -0.3230]])\n\n ==TRN1 OUTPUT==\n tensor([[-0.1396, -0.3266],\n [-0.0328, -0.3106],\n [-0.0067, -0.3270],\n [-0.1684, -0.3229]], device='xla:1')\n\nLoading the Operators Level Trace in TensorBoard\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nRun ``tensorboard --load_fast=false --logdir logs/``\n\nTake note of the port (usually 6006) and enter ``localhost:`` into\nthe local browser (assuming port forwarding is set up properly)\n\n.. note::\n\n Check :ref:`Tensorboard Interface Overview` to understand TensorBoard interface\n\n\nThe Operator Level Trace views are the same format plus an id at the\nend; ``year_month_day_hour_minute_second_millisecond_id``. The Tool\ndropdown will have 3 options: operator-framework, operator-hlo, and\noperator-timeline.\n\nOperator Framework View\n~~~~~~~~~~~~~~~~~~~~~~~\n\n|tensorboard-operator-framework-view|\n\nThis view contains a pie-chart displaying the\nproportional execution time for each of the model operators on the framework level for a\nneuron device. The list of operators is shown in the bottom along with\nother details about number of occurrences, execution time and neuron\ndevice and core.\n\nOperator HLO View\n~~~~~~~~~~~~~~~~~\n\n|tensorboard-operator-hlo-view|\n\nThis view contains a pie-chart displaying the\nproportional execution time for each of the model operators on the hlo level for a\nNeuron device. The list of operators is shown in the bottom along with\nother details about number of occurrences, execution time and neuron\ndevice and core.\n\n.. note::\n\n For this simple model, the pie chart will be the same as the framework view. This won't be\n the case for larger and more complex models.\n\nOperator Trace View\n~~~~~~~~~~~~~~~~~~~\n\n|tensorboard-operator-trace-view|\n\n\n.. _trace_view_sections:\n\nTrace View Sections\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nNotice there are four sections: Process Overview, Control, Execution, and Data\nTransfer. In each section there are more subdivisions with each layer\nrepresenting a certain level of abstraction. Also important to note that\nthe timescale axis is aligned between the two sections. This is\nimportant to note as sometimes there are gaps in the process execution.\nMost of the time, there are data transfer operations happening in\nbetween the gaps.\n\nFusion Operators\n^^^^^^^^^^^^^^^^\n\n**Simple Case:** Zooming in on the operations, we can recognize some\noperations for a neural network, such as a dot product and transpose,\nbut sometimes there will be fused operators (fusion operators). To\nunderstand these operators, click on it, and on the bottom of the\ndashboard, some information will appear. \n\n|tensorboard-operator-trace-fusion-simple|\n\nNotice in the above example the fusion operator is fusing the operator before and\nafter itself on the timeline. More specifically, ``fused_3`` is a fusion\nof ``NN[model]/input`` and\n``NN[model]/ReLU[nl1]/Tensor_1/aten__relu_maximum``. These kinds of\nfusions occur when the ``neuronx-cc`` compiler has found an optimization\nrelating to the two operators. Most often this would be the execution of\nthe operators on separate compute engines or another form of parallelism.\n\n**Complex Case:** Most often, the order of fusion operators can get a\nlittle complicated or contain \"hidden\" information. For the first example,\nlet’s zoom into the data transfer section such that we see the timescale range \nfrom 6000 ns. to 6600 ns. It should look similar to below:\n\n|tensorboard-operator-trace-fusion-complex|\n\nLooking at ``fused_16`` (11452 ns) we see it's surrounded by other fused operators.\nFurthermore, the ``fused_16`` operator fuses more than two operators: ``NN[model]/Linear[layer1]/aten__addmm_add``,\n``NN[model]/input``, and ``NN[model]/Linear[layer1]/aten__addmm_dot``. These operators can be found in the timeline, but sometimes\nthe fused operators may not exist in the timeline due to it occurring within another operation. We go over an example of this case\nin Part 2.\n\n\nUnderstanding the Low Level Timeline\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLooking at the trace we can look behind the scenes at how the model is\nexecuted on neuron hardware. Before proceeding with the analysis, it is worth recalling the\nway we defined the model for this tutorial:\n\n.. code:: python\n\n class NN(torch.nn.Module):\n def __init__(self):\n super().__init__()\n\n self.layer1 = torch.nn.Linear(4,4)\n self.nl1 = torch.nn.ReLU()\n self.layer2 = torch.nn.Linear(4,2)\n self.nl2 = torch.nn.Tanh()\n\n def forward(self, x):\n x = self.nl1(self.layer1(x))\n return self.nl2(self.layer2(x))\n\nAnalysis\n^^^^^^^^\n**Input Operators:** We see input operators here. This is because in a markstep flow, we need to transfer inputs to the xla device. This is represented by the ``SyncTensorsGraph.53`` call.\n\n**ReLU at the beginning:** The first couple of blocks in the Process Data Transfer section initially appear to be confusing. There is an ``Input`` (0 ns.)\nblock followed by a ``ReLU`` (100 ns.) operator. Under the hood here, ``ReLU`` is rewritten as an ``elementwise_max(arr,0)``, \n(0 here means an array with zeros) but to create this operation, the zeros have to be set in memory, which is a data operation.\nA general rule is that if an operator appears this early in the data transfer section, it most likely means there is an operation\nlowering involving setting some values into memory for use later on.\n\n**Memory allocation for Linear[layer1]:** We resume with the data transfer operations. Here, memory is getting allocated for specific operators, and sometimes the allocated\ninputs get loaded onto operators while the rest of the input gets allocated. This can be seen at ``fused_18`` (11811 ns.) and ``fused_23`` (12181 ns.).\nEventually the input gets fully allocated, and other allocations occur for dot products, transpose, and broadcast operators for\n``Linear[layer1]`` and ``Linear[layer2]``.\n\nConclusion\n^^^^^^^^^^^\n\nThere are a few conclusions that can be determined from analyzing the timeline. We can see that we’ve been able to save a bit of time due to \nparallelism with fusion operations, and saving some compute time with preloading operations (ex. ``ReLU``). A clear trend is that a majority of the time is spent on data transfer operations.\nIt is also evident that even a simple Feed Forward NN becomes complicated when put under a microscope in the profiler. Facts such as the implementation of ``ReLU`` in the runtime/architecture, aren’t explicitly stated in the profiler, but do make\nthemselves known by the unusual ordering placement of the trace blocks and unusual fusion operators.\n\nIn terms of action items that can be taken based on our narrative, there\nreally isn’t any. This is a very very simple model that outputs after 8\nmicroseconds, and we chose it because it is simple to understand. In\nmore realistic examples we will aim to do more compute than data\ntransfer on the hardware, and where possible to overlap data transfer\nand compute between sequential operations.\n\nThe profiler revealed a lot of optimizations that were done, via fusion\noperators and parallelism. However, the end goal of this tool is to be\nable to improve performance by revealing the bottlenecks of the model.\n\n.. note::\n\n While we did explain some of the quirks visible in the profiler at a microscopic level, it isn’t necessary\n to do so for normal use. This tutorial introduced the microscopic explanation for these occurrences to show to the \n user that this is *indeed* what happens in the hardware when executing a simple FFNN.\n\nPart 2: Operator Level Trace with ``torch_neuronx.trace()`` workflow\n----------------------------------------------------------------------\n\nSet Up\n~~~~~~\n\nThe setup will be similar to Part 1.\n::\n\n cd ~/neuron_profiling_tensorboard_examples\n mkdir tutorial_2\n cd tutorial_2\n\n # this is where our code will be written\n touch run.py\n\nHere is the code for ``run.py``:\n\n::\n\n import os\n import time\n import torch\n import torch_neuronx\n from torch_neuronx.experimental import profiler\n\n class NN(torch.nn.Module):\n def __init__(self):\n super().__init__()\n\n self.layer1 = torch.nn.Linear(4,4)\n self.nl1 = torch.nn.ReLU()\n self.layer2 = torch.nn.Linear(4,2)\n self.nl2 = torch.nn.Tanh()\n\n def forward(self, x):\n x = self.nl1(self.layer1(x))\n return self.nl2(self.layer2(x))\n\n model = NN()\n model.eval()\n\n inp = torch.rand(4,4)\n\n output = model(inp)\n\n with torch_neuronx.experimental.profiler.profile(\n port=9012,\n profile_type='operator',\n ms_duration=10000,\n traced_only=True):\n\n neuron_model = torch_neuronx.trace(model,inp,compiler_workdir=\"./compiler_cache\")\n neuron_model(inp)\n\n print(\"==CPU OUTPUT==\")\n print(output)\n print()\n print(\"==INF2 OUTPUT==\")\n print(output_neuron)\n\nImportant code differences from Part 1\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1. ``import torch_xla.core.xla_model as xm`` is no longer necessary\n2. Set ``traced_only=True`` in ``torch_neuronx.experimental.profiler.profile()``. This option is necessary for traced models, otherwise the generated profile will not be accurate or not work.\n3. Tracing the model with ``torch_neuronx.trace()`` and removing ``xm.markstep()``.\n\nOtherwise, the code is the same as Part 1.\n\nRunning Part 2\n~~~~~~~~~~~~~~~~~\nTo Run:\n\n::\n\n python run.py\n\nThe output will look almost identical as Part 1\n\nLoading the Operators Level Trace in TensorBoard\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nRun ``tensorboard --load_fast=false --logdir logs/``, just like Part 1.\n\n.. note::\n\n Check :ref:`Tensorboard Interface Overview` to understand TensorBoard interface\n\nTimeline View:\n\n|tensorboard-operator-trace-view-traced|\n\nNotable Differences in Timeline View from Part 1\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**No Input Operators:** For a traced model, we do not transfer the input to an xla device, so these operations are not seen on the timeline. This also affects scheduling, which is why the time taken in\nthe profiling is less than the markstep one.\n\n**Combined Loading of Linear[layer1] and Tanh:** ``fused_19`` (5824 ns) contains a fusion between ``Linear[layer1]`` and ``Tanh[nl2]``. This might be a bit odd, but such data loading parallelism\ncan be understood by understanding how tanh is implemented. Typically, functions like tanh are implemented by lookup tables that require being pre-loaded onto memory, which is a data transfer operation.\nA bulk of data transfer operations are done in the beginning to optimize computations.\n\n.. note::\n Despite these differences, the big picture conclusion drawn from Part 1 still holds, as the two timelines are more similar than different. Some new insights drawn is that the traced model performs better than the markstep flow, since this was profiling a single forward pass.\n\n\n.. |tensorboard-url-image| image:: /images/Neuron_Profiler_Tensorboard_Url.jpg\n\n.. |tensorboard-NEURON-header| image:: /images/Neuron_Profiler_Tensorboard_Header.jpg\n\n.. |tensorboard-NEURON-dropdown| image:: /images/Neuron_Profiler_Tensorboard_Dropdown.jpg\n\n.. |tensorboard-run-tool-dropdowns| image:: /images/Neuron_Profiler_Tensorboard_Run_Tool_Dropdowns.jpg\n\n.. |tensorboard-run-trace-original| image:: /images/Neuron_Profiler_Runtime_Trace_Original.jpg\n\n.. |tensorboard-run-trace-selected-section| image:: /images/Neuron_Profiler_Runtime_Trace_Section_Selection.jpg\n\n.. |tensorboard-run-trace-selected-section-zoomed| image:: /images/Neuron_Profiler_Runtime_Trace_Section_Selection_Zoomed.jpg\n\n.. |tensorboard-run-trace-selected-section-zoomed-named-traces| image:: /images/Neuron_Profiler_Runtime_Trace_Section_Selection_Zoomed_Named_Traces.jpg\n\n.. |tensorboard-operator-framework-view| image:: /images/Neuron_Profiler_T1_Op_Framework_View.png\n\n.. |tensorboard-operator-hlo-view| image:: /images/Neuron_Profiler_T1_Op_HLO_View.png\n\n.. |tensorboard-operator-trace-view| image:: /images/Neuron_Profiler_T1_Op_Trace_View.png\n\n.. |tensorboard-operator-trace-view-traced| image:: /images/Neuron_Profiler_T1_Op_Trace_View_Traced.png\n\n.. |tensorboard-operator-trace-fusion-simple| image:: /images/Neuron_Profiler_T1_Op_Trace_Fusion_Simple.png\n\n.. |tensorboard-operator-trace-fusion-complex| image:: /images/Neuron_Profiler_T1_Op_Trace_Fusion_Complex.png" }, { "path": "tools/tutorials/tutorial-neuron-monitor-mnist.rst", "content": ".. _track-system-monitor:\n\nTrack System Resource Utilization during Training with neuron-monitor using PyTorch Neuron\n==========================================================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nThis tutorial explains how to monitor resource utilization using **neuron-monitor**, **Prometheus** and **Grafana** while running a multi-layer\nperceptron MNIST model on Trainium using PyTorch Neuron.\n\nMulti-layer Perceptron MNIST Model\n----------------------------------\n\nThis tutorial is based on the MNIST example for PyTorch Neuron on Trainium.\nFor the full tutorial, please see :ref:`Multi-Layer Perceptron Training Tutorial `.\n\nThe Training Job\n----------------\n\nFor this tutorial, we will make the original script do more work thus giving us more system utilization data to observe. The training\nloop is simply repeated 1000 times:\n\n.. code:: python\n\n for run in range(0, 1000):\n print(f'Run {run}')\n model.train()\n ...\n\nSave the following code as :download:`train_monitor.py ` and you can run it as\n``python3 train_monitor.py`` on a Trn1 instance.\n\n.. literalinclude:: /src/examples/pytorch/mnist_mlp/train_monitor.py\n :language: python\n\nSetting up **Prometheus** and **Grafana**\n-----------------------------------------\n\n.. note::\n The setup presented in the following paragraphs can be extended to monitor any number of instances running training jobs or\n inference workloads. For this tutorial, we will set everything up on a single Trn1 instance running Amazon Linux 2.\n\nSetting up **Prometheus**\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor a more detailed guide on how to install **Prometheus** visit their official guide at https://prometheus.io/docs/prometheus/latest/getting_started/.\n\nDownload and unzip a prebuilt **Prometheus** binary on your Trn1 instance:\n\n.. code:: bash\n\n wget https://github.com/prometheus/prometheus/releases/download/v2.38.0/prometheus-2.38.0.linux-amd64.tar.gz\n tar -xzvf prometheus-2.38.0.linux-amd64.tar.gz\n cd prometheus-2.38.0.linux-amd64/\n\nCreate a config and add a scrape target:\n\n.. code:: bash\n\n vim prometheus.yml\n\n.. code:: yml\n\n scrape_configs:\n - job_name: 'neuron'\n\n # Scrape target every 5 seconds.\n scrape_interval: 5s\n\n static_configs:\n - targets: ['localhost:8000']\n\nFinally, start **Prometheus**:\n\n.. code:: bash\n\n ./prometheus --config.file=prometheus.yml\n\nSetting up **Grafana**\n~~~~~~~~~~~~~~~~~~~~~~\n\nFor a more detailed guide on how to install **Grafana** visit their official guide at https://grafana.com/grafana/download.\n\nAdd the Grafana repo to dnf:\n\n.. code:: bash\n\n sudo vim /etc/yum.repos.d/grafana.repo\n\n [grafana]\n name=grafana\n baseurl=https://packages.grafana.com/oss/rpm\n repo_gpgcheck=1\n enabled=1\n gpgcheck=1\n gpgkey=https://packages.grafana.com/gpg.key\n sslverify=1\n sslcacert=/etc/pki/tls/certs/ca-bundle.crt\n\nInstall and start **Grafana**:\n\n.. code:: bash\n\n sudo dnf install -y grafana\n sudo /bin/systemctl start grafana-server.service\n\nBy default, **Grafana** will run a HTTP server on port 3000. If you need to change that, update its config and restart the service:\n\n.. code:: bash\n\n sudo vim /etc/grafana/grafana.ini\n ...\n sudo /bin/systemctl start grafana-server.service\n\nUsing your favorite web browser, access the Grafana webpage and add a new dashboard.\n\nThe default user and password are both 'admin':\n\n.. image:: tutorial_grafana_login.png\n :alt: Image: image.png\n\nNext, you'll add a Prometheus data source by going to ``Configuration`` -> ``Data Sources``:\n\n.. image:: tutorial_grafana_data_sources.png\n :alt: Image: image.png\n\n... and adding the local **Prometheus** server as a data source:\n\n.. image:: tutorial_grafana_add_prometheus.png\n :alt: Image: image.png\n\nFinally, upload the sample dashboard :download:`neuron-monitor-grafana.json `\nto **Grafana**:\n\n.. image:: tutorial_grafana_upload_dash.png\n :alt: Image: image.png\n\nMonitoring the Training Workload\n--------------------------------\n\nStart the training job which, due to the artificially added complexity, will take more than 15 minutes:\n\n.. code:: bash\n\n python train_monitor.py\n\nOn the same instance, start ``neuron-monitor`` and its companion script, ``neuron-monitor-prometheus.py``:\n\n.. code:: bash\n\n neuron-monitor | neuron-monitor-prometheus.py\n\nOnce they are running, you can use your web browser, access the **Grafana** server running on your Trn1 instance and\nview a timeline of the system utilization.\n\nThe upper part of the dashboard contains:\n - a list of the currently monitored instances (for this tutorial there is a single Trn1 instance)\n - aggregated metrics for stats such as NeuronCore utilization, NeuronCores in use, iteration success rates, error rates etc.\n - a timeline of execution status rates and execution latencies\n\n.. image:: tutorial_grafana_dash_1.png\n :alt: Image: image.png\n\nThe lower part of the dashboard contains:\n- one line of charts containing a timeline of Neuron resource utilization (NeuronCore, vCPU and memory utilization)\n- one line of charts containing a timeline of host resource utilization (vCPU and memory utilization)\n\n.. image:: tutorial_grafana_dash_2.png\n :alt: Image: image.png\n" }, { "path": "tools/tutorials/tutorial-tensorboard-scalars-mnist.rst", "content": ".. _tb_track_training_minst:\n\nTrack Training Progress in TensorBoard using PyTorch Neuron\n============================================================\n\n.. contents:: Table of Contents\n :local:\n :depth: 2\n\nThis tutorial explains how to track training progress in TensorBoard while running a multi-layer perceptron MNIST model on Trainium using PyTorch Neuron.\n\nMulti-layer perceptron MNIST model\n----------------------------------\n\nThis tutorial is based on the MNIST example for PyTorch Neuron on Trainium.\nFor the full tutorial, please see :ref:`Multi-Layer Perceptron Training Tutorial `.\n\nOutput TensorBoard logs\n-----------------------\n\nTo generate TensorBoard logs, we first modify the training script to use the ``SummaryWriter``:\n\n.. code:: python\n\n from torch.utils.tensorboard import SummaryWriter\n writer = SummaryWriter('./output')\n\nIn the training loop, we can then use the ``add_scalar`` API to log the loss per step.\n\n.. code:: python\n\n writer.add_scalar(\"step loss\", loss, idx)\n\nAt the end of the script, add ``writer.flush()`` to ensure all logs are written.\n\nSave the following code as :download:`train_tb.py ` and run it as ``python3 train_tb.py`` on a Trn1 instance.\nThe generated logs can be found in the ``./output`` directory that was passed to ``SummaryWriter``.\n\n.. literalinclude:: /src/examples/pytorch/mnist_mlp/train_tb.py\n :language: python\n\nView loss in TensorBoard\n------------------------\n\nIn order to view your training metrics, install TensorBoard in your Python environment:\n\n.. code:: bash\n\n pip install tensorboard\n\nThen, launch TensorBoard with the ``./output`` directory\n\n.. code:: bash\n\n tensorboard --logdir ./output\n\nOnce running, open a new SSH connection to the instance and port-forward\nTCP port 6006 (ex: -L 6006:127.0.0.1:6006). Once the tunnel is\nestablished, TensorBoard can then be accessed via web browser at the\nfollowing URL: `http://localhost:6006 `__.\nPlease note that you will not be able to access TensorBoard if you\ndisconnect your port-forwarding SSH session to the Trainium instance.\n\n.. image:: tb-scalars.png\n :alt: Image: image.png\n\nIn TensorBoard, you can now see the loss per step plotted.\nWhen capturing loss for multiple runs, you can plot them together on the same graph to compare runs.\nBe sure to change the output directory for different runs, for example ``./output/run1`` for the first, ``./output/run2`` for the second, etc.\n" } ]