[
  {
    "path": ".github/ISSUE_TEMPLATE/bug_template.md",
    "content": "---\nname: \"\\U0001F41B Bug report\"\nabout: Create a report to help us improve\n\n---\n\nYour issue may already be reported!\nPlease search on the [issue tracker](https://github.com/pytorch/examples/issues) before creating one.\n\n## Context\n<!--- How has this issue affected you? What are you trying to accomplish? -->\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\n* Pytorch version:\n* Operating System and version:\n\n## Your Environment\n<!--- Include as many relevant details about the environment you experienced the bug in -->\n* Installed using source? [yes/no]:\n* Are you planning to deploy it using docker container? [yes/no]:\n* Is it a CPU or GPU environment?:\n* Which example are you using:\n* Link to code or data to repro [if any]:\n\n## Expected Behavior\n<!--- If you're describing a bug, tell us what should happen -->\n\n## Current Behavior\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\n\n## Possible Solution\n<!--- Not obligatory, but suggest a fix/reason for the bug -->\n\n## Steps to Reproduce\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\n1.\n2.\n...\n\n## Failure Logs [if any]\n<!--- Provide any relevant log snippets or files here. -->\n\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/doc_template.md",
    "content": "---\nname: \"\\U0001F4DA Documentation\"\nabout: Report a documentation related issue\n\n---\n\n## 📚 Documentation\n\n<!-- A clear and concise description of what content in any of the README.md files is an issues\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature_template.md",
    "content": "---\nname: \"\\U0001F680 Feature request\"\nabout: Suggest a new example or an improvement to the repo\n\n---\n\n<!--\nThank you for suggesting an idea to improve pytorch/examples\n\nPlease fill in as much of the template below as you're able.\n-->\n\n## Is your feature request related to a problem? Please describe.\n<!-- Please describe the problem you are trying to solve. -->\n\n## Describe the solution\n<!-- Please describe the desired behavior. -->\n\n## Describe alternatives solution\n<!-- Please describe alternative solutions or features you have considered. -->\n"
  },
  {
    "path": ".github/PULL_REQUEST_TEMPLATE/pull_request_template.md",
    "content": "---\nname: \"\\U0001F41B Pull Request\"\nabout: Fix a bug or create new example\n\n---\n\n## Description\n\nPlease include a summary of the newly proposed example or issue being fixed. Please also include relevant motivation, context.\n\nIf this is a new example, how is your example different enough from the remaining examples in the repo. \n\nIf this is a bug fix please link the issue you are fixing. Fixes #(issue)\n\n## Type of change\n\nPlease delete options that are not relevant.\n\n- [ ] Bug fix (non-breaking change which fixes an issue)\n- [ ] New Example (new example contribution)\n- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)\n- [ ] This change requires a documentation update\n\n## Feature/Issue validation/testing\n\nPlease describe the tests [UT/IT] that you ran to verify your changes and relevant result summary. If this is a bug fix please run `run_python_examples.sh` before and after your change locally to make sure it works and add the logs here.\n\n- [ ] Logs before change\n- [ ] Logs after change\n\n- Logs\n\nIf this is a new example please add a corresponding test in `run_python_examples.sh`\n\n- [ ] Test Added\n\n## Checklist:\n\n- [ ] Have you added tests that prove your fix is effective or that this example works?\n- [ ] Has code been commented, particularly in hard-to-understand areas?\n- [ ] Have you made corresponding changes to the documentation?\n"
  },
  {
    "path": ".github/workflows/doc-build.yml",
    "content": "name: Doc Build\n\non:  \n  push:\n    branches:\n      - main\n  workflow_dispatch:\n\njobs:\n  build_docs_job:\n    runs-on: ubuntu-latest\n    # Grant write permission here so that the doc can be pushed to gh-pages branch\n    permissions:\n      contents: write\n    strategy:\n      matrix:\n        python-version: [3.9]\n    steps:\n    - name: Checkout\n      uses: actions/checkout@v4\n      id: build\n    - name: Build the docset\n      run: | \n        cd docs\n        pip install -r requirements.txt\n        make html \n    - name: Get output time\n      run: echo \"The time was ${{ steps.build.outputs.time }}\"      \n    - name: Deploy\n      uses: JamesIves/github-pages-deploy-action@releases/v3\n      with:\n          ACCESS_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n          BRANCH: gh-pages # The branch the action should deploy to.\n          FOLDER: ./docs/build/html # The folder the action should deploy. \n"
  },
  {
    "path": ".github/workflows/main_cpp.yml",
    "content": "name: Run CPP Examples\n\non:\n  push:\n    branches: [ main ]\n  pull_request:\n    branches: [ main ]\n  schedule:\n    # Every day at 3:00am\n    - cron: '0 3 * * *'\n\n\njobs:\n  test:\n\n    runs-on: ubuntu-latest\n\n    steps:\n    - uses: actions/checkout@v2\n    - name: Set up Python 3.11\n      uses: actions/setup-python@v2\n      with:\n        python-version: 3.11\n        \n    - name: Install Cmake, Make, g++, MKL\n      run: |\n        sudo apt update && sudo apt upgrade\n        sudo apt install cmake g++ make\n        sudo apt-get -y install intel-mkl\n    - name: Install OpenCV\n      run: |\n        sudo apt -y install libtbb-dev\n        sudo apt install libopencv-dev\n    - name: Install argparse\n      run: |\n        git clone https://github.com/p-ranav/argparse\n        cd argparse\n        mkdir build\n        cd build\n        cmake -DARGPARSE_BUILD_SAMPLES=off -DARGPARSE_BUILD_TESTS=off ..\n        sudo make install\n    # Alternatively, you can install OpenCV from source\n    # - name: Install OpenCV from source\n    #   run: |\n        # wget -O opencv.zip https://github.com/opencv/opencv/archive/4.x.zip\n        # unzip opencv.zip\n        # mkdir -p build && cd build\n        # cmake  ../opencv-4.x\n        # cmake --build .\n        # sudo make install    \n    \n    - name: Run Cpp Tests\n      run: |\n        chmod +x ./run_cpp_examples.sh\n        ./run_cpp_examples.sh \"get_libtorch,run_all,clean\"\n"
  },
  {
    "path": ".github/workflows/main_distributed.yaml",
    "content": "name: Run Distributed Examples\n\non:\n  push:\n    branches: [ main ]\n  pull_request:\n    branches: [ main ]\n  schedule:\n    # Every day at 3:00am\n    - cron: '0 3 * * *'\n\n\njobs:\n  test:\n\n    runs-on: 4-core-ubuntu-gpu-t4\n\n    steps:\n    - uses: actions/checkout@v4\n    - name: Set up Python 3.10\n      uses: actions/setup-python@v5\n      with:\n        python-version: '3.10'\n    - name: Install PyTorch\n      uses: astral-sh/setup-uv@v6\n    - name: Run Tests\n      env:\n        USE_CUDA: 'True'\n        VIRTUAL_ENV: '.venv'\n        PIP_INSTALL_ARGS: '--pre -f https://download.pytorch.org/whl/nightly/cu118/torch_nightly.html'\n      run: |\n        ./run_distributed_examples.sh\n    - name: Open issue on failure\n      if: ${{ failure() && github.event_name  == 'schedule' }}\n      uses: rishabhgupta/git-action-issue@v2\n      with:\n        token: ${{ secrets.GITHUB_TOKEN }}\n        title: Daily CI failed\n        body:  Commit ${{ github.sha }} daily scheduled [CI run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}) failed, please check why\n        assignees: ''\n"
  },
  {
    "path": ".github/workflows/main_python.yml",
    "content": "name: Run Python Examples\n\non:\n  push:\n    branches: [ main ]\n  pull_request:\n    branches: [ main ]\n  schedule:\n    # Every day at 3:00am\n    - cron: '0 3 * * *'\n\n\njobs:\n  test:\n\n    runs-on: ubuntu-latest\n\n    steps:\n    - uses: actions/checkout@v4\n    - name: Set up Python 3.10\n      uses: actions/setup-python@v5\n      with:\n        python-version: '3.10'\n    - name: Install uv\n      uses: astral-sh/setup-uv@v6\n    - name: Run Tests\n      env:\n        VIRTUAL_ENV: '.venv'\n        PIP_INSTAL_ARGS: '--pre -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html'\n      run: |\n        ./run_python_examples.sh\n    - name: Open issue on failure\n      if: ${{ failure() && github.event_name  == 'schedule' }}\n      uses: rishabhgupta/git-action-issue@v2\n      with:\n        token: ${{ secrets.GITHUB_TOKEN }}\n        title: Daily CI failed\n        body:  Commit ${{ github.sha }} daily scheduled [CI run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}) failed, please check why\n        assignees: ''\n"
  },
  {
    "path": ".gitignore",
    "content": "dcgan/data\ndata\n*.pyc\nOpenNMT/data\ncpp/mnist/build\ncpp/dcgan/build\ndcgan/*.png\ndcgan/*.pth\nsnli/.data\nsnli/.vector_cache\nsnli/results\nword_language_model/model.pt\nfast_neural_style/saved_models\nfast_neural_style/saved_models.zip\ngcn/cora/\ngat/cora/\ndocs/build\ndocs/venv\n\n# vi backups\n*~\n\n# development\n.vscode\n**/.DS_Store\n"
  },
  {
    "path": "CODEOWNERS",
    "content": "# This is a comment.\n# Each line is a file pattern followed by one or more owners.\n\n# Github Actions, tests and CI\n./github/ @msaroufim\nrun_python_examples.sh @msaroufim\n\n# Distributed examples \n# Can also add the distributed oncall\n./distributed/ @mrshenli @pritamdamania87 @rohan-varma @H-Huang\n./mnist_hogwild/ @mrshenli @pritamdamania87 @rohan-varma @H-Huang\n\n# FX examples \n./fx/ @jamesr66a @Chillee\n\n# Domain Examples \n./reinforcement_learning/ @msaroufim \n./word_language_model/ @msaroufim\n\n# Need an owner \n./regression/\n./mnist/\n./imagenet/\n./super_resolution/ \n./time_sequence_prediction/\n./vae/\n\n# Legacy examples\n./cpp/\n./legacy/snli/ \n"
  },
  {
    "path": "CODE_OF_CONDUCT.md",
    "content": "# Code of Conduct\n\n## Our Pledge\n\nIn the interest of fostering an open and welcoming environment, we as\ncontributors and maintainers pledge to make participation in our project and\nour community a harassment-free experience for everyone, regardless of age, body\nsize, disability, ethnicity, sex characteristics, gender identity and expression,\nlevel of experience, education, socio-economic status, nationality, personal\nappearance, race, religion, or sexual identity and orientation.\n\n## Our Standards\n\nExamples of behavior that contributes to creating a positive environment\ninclude:\n\n- Using welcoming and inclusive language\n- Being respectful of differing viewpoints and experiences\n- Gracefully accepting constructive criticism\n- Focusing on what is best for the community\n- Showing empathy towards other community members\n\nExamples of unacceptable behavior by participants include:\n\n- The use of sexualized language or imagery and unwelcome sexual attention or\n  advances\n- Trolling, insulting/derogatory comments, and personal or political attacks\n- Public or private harassment\n- Publishing other's private information, such as physical or electronic\n  address, without explicit permission\n- Other conduct which could reasonably be considered inappropriate in a\n  professional setting\n\n## Our Responsibilities\n\nProject maintainers are responsible for clarifying the standards of acceptable\nbehavior and are expected to take appropriate and fair corrective action in\nresponse to any instances of unacceptable behavior.\n\nProject maintainers have the right and responsibility to remove, edit, or\nreject comments, commits, code, wiki edits, issues, and other contributions\nthat are not aligned to this Code of Conduct, or to ban temporarily or\npermanently any contributor for other behaviors that they deem inappropriate,\nthreatening, offensive, or harmful.\n\n## Scope\n\nThis Code of Conduct applies within all project spaces, and it also applies when\nan individual is representing the project or its community in public spaces.\nExamples of representing a project or community include using an official\nproject e-mail address, posting via an official social media account, or acting\nas an appointed representative at an online or offline event. Representation of\na project may be further defined and clarified by project maintainers.\n\n## Enforcement\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\nreported by contacting the project team at <opensource-conduct@fb.com>. All\ncomplaints will be reviewed and investigated and will result in a response that\nis deemed necessary and appropriate to the circumstances. The project team is\nobligated to maintain confidentiality with regard to the reporter of an incident.\nFurther details of specific enforcement policies may be posted separately.\n\nProject maintainers who do not follow or enforce the Code of Conduct in good\nfaith may face temporary or permanent repercussions as determined by other\nmembers of the project's leadership.\n\n## Attribution\n\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,\navailable at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html\n\n[homepage]: https://www.contributor-covenant.org\n\nFor answers to common questions about this code of conduct, see\nhttps://www.contributor-covenant.org/faq\n"
  },
  {
    "path": "CONTRIBUTING.md",
    "content": "# Contributing to examples\n\nWe want to make contributing to this project as easy and transparent as\npossible.\n\n## Pull Requests\n\nWe actively welcome your pull requests.\n\nIf you're new, we encourage you to take a look at issues tagged with [good first issue](https://github.com/pytorch/examples/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)\n\n### For new examples\n\n0. Create a GitHub issue proposing a new example and make sure it's substantially different from an existing one.\n1. Fork the repo and create your branch from `main`.\n2. If you've added code that should be tested, add tests to `run_python_examples.sh`.\n3. Create a `README.md`.\n4. Add a card with a brief description of your example and link to the repo to\n   the `docs/source/index.rst` file and build the docs by running:\n\n   ```\n   cd docs\n   virtualenv venv\n   source venv/bin/activate\n   pip install -r requirements.txt\n   make html\n   ```\n\n   When done working with `virtualenv`, run `deactivate`.\n\n5. Verify that there are no issues in your doc build. You can check the preview locally\n   by installing [sphinx-serve](https://pypi.org/project/sphinx-serve/)\n   then running `sphinx-serve -b build`.\n6. Ensure your test passes locally.\n7. If you haven't already, complete the Contributor License Agreement (\"CLA\").\n8. Address any feedback in code review promptly.\n\n## For bug fixes\n\n1. Fork the repo and create your branch from `main`.\n2. Make sure you have a GPU-enabled machine, either locally or in the cloud. `g4dn.4xlarge` is a good starting point on AWS.\n3. Make your code change.\n4. Install `uv`.\n5. Then, make sure that `VIRTUAL_ENV=.venv ./run_python_examples.sh` passes locally by running the script end to end.\n6. If you haven't already, complete the Contributor License Agreement (\"CLA\").\n7. Address any feedback in code review promptly.\n\n## Contributor License Agreement (\"CLA\")\n\nTo accept your pull request, we need you to submit a CLA. You only need\nto do this once to work on any of Facebook's open source projects.\n\nComplete your CLA here: <https://code.facebook.com/cla>\n\n## Issues\n\nWe use GitHub issues to track public bugs. Please ensure your description is\nclear and has sufficient instructions to be able to reproduce the issue.\n\n## License\n\nBy contributing to examples, you agree that your contributions will be licensed\nunder the LICENSE file in the root directory of this source tree.\n"
  },
  {
    "path": "LICENSE",
    "content": "BSD 3-Clause License\n\nCopyright (c) 2017, Pytorch contributors\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\n  list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\n  this list of conditions and the following disclaimer in the documentation\n  and/or other materials provided with the distribution.\n\n* Neither the name of the copyright holder nor the names of its\n  contributors may be used to endorse or promote products derived from\n  this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n"
  },
  {
    "path": "README.md",
    "content": "# PyTorch Examples\n\nhttps://pytorch.org/examples/\n\n`pytorch/examples` is a repository showcasing examples of using [PyTorch](https://github.com/pytorch/pytorch). The goal is to have curated, short, few/no dependencies _high quality_ examples that are substantially different from each other that can be emulated in your existing work.\n\n- For tutorials: https://github.com/pytorch/tutorials\n- For changes to pytorch.org: https://github.com/pytorch/pytorch.github.io\n- For a general model hub: https://pytorch.org/hub/ or https://huggingface.co/models\n- For recipes on how to run PyTorch in production: https://github.com/facebookresearch/recipes\n- For general Q&A and support: https://discuss.pytorch.org/\n\n## Available models\n\n- [Image classification (MNIST) using Convnets](./mnist/README.md)\n- [Word-level Language Modeling using RNN and Transformer](./word_language_model/README.md)\n- [Training Imagenet Classifiers with Popular Networks](./imagenet/README.md)\n- [Generative Adversarial Networks (DCGAN)](./dcgan/README.md)\n- [Variational Auto-Encoders](./vae/README.md)\n- [Superresolution using an efficient sub-pixel convolutional neural network](./super_resolution/README.md)\n- [Hogwild training of shared ConvNets across multiple processes on MNIST](mnist_hogwild)\n- [Training a CartPole to balance with actor-critic](./reinforcement_learning/README.md)\n- [Natural Language Inference (SNLI) with GloVe vectors, LSTMs, and torchtext](snli)\n- [Time sequence prediction - use an LSTM to learn Sine waves](./time_sequence_prediction/README.md)\n- [Implement the Neural Style Transfer algorithm on images](./fast_neural_style/README.md)\n- [Reinforcement Learning with Actor Critic and REINFORCE algorithms on OpenAI gym](./reinforcement_learning/README.md)\n- [PyTorch Module Transformations using fx](./fx/README.md)\n- Distributed PyTorch examples with [Distributed Data Parallel](./distributed/ddp/README.md) and [RPC](./distributed/rpc)\n- [Several examples illustrating the C++ Frontend](cpp)\n- [Image Classification Using Forward-Forward](./mnist_forward_forward/README.md)\n- [Language Translation using Transformers](./language_translation/README.md)\n\nAdditionally, a list of good examples hosted in their own repositories:\n\n- [Neural Machine Translation using sequence-to-sequence RNN with attention (OpenNMT)](https://github.com/OpenNMT/OpenNMT-py)\n\n## Contributing\n\nIf you'd like to contribute your own example or fix a bug please make sure to take a look at [CONTRIBUTING.md](CONTRIBUTING.md).\n"
  },
  {
    "path": "cpp/.clang-format",
    "content": "---\nAccessModifierOffset: -1\nAlignAfterOpenBracket: AlwaysBreak\nAlignConsecutiveAssignments: false\nAlignConsecutiveDeclarations: false\nAlignEscapedNewlinesLeft: true\nAlignOperands:   false\nAlignTrailingComments: false\nAllowAllParametersOfDeclarationOnNextLine: false\nAllowShortBlocksOnASingleLine: false\nAllowShortCaseLabelsOnASingleLine: false\nAllowShortFunctionsOnASingleLine: Empty\nAllowShortIfStatementsOnASingleLine: false\nAllowShortLoopsOnASingleLine: false\nAlwaysBreakAfterReturnType: None\nAlwaysBreakBeforeMultilineStrings: true\nAlwaysBreakTemplateDeclarations: true\nBinPackArguments: false\nBinPackParameters: false\nBraceWrapping:\n  AfterClass:      false\n  AfterControlStatement: false\n  AfterEnum:       false\n  AfterFunction:   false\n  AfterNamespace:  false\n  AfterObjCDeclaration: false\n  AfterStruct:     false\n  AfterUnion:      false\n  BeforeCatch:     false\n  BeforeElse:      false\n  IndentBraces:    false\nBreakBeforeBinaryOperators: None\nBreakBeforeBraces: Attach\nBreakBeforeTernaryOperators: true\nBreakConstructorInitializersBeforeComma: false\nBreakAfterJavaFieldAnnotations: false\nBreakStringLiterals: false\nColumnLimit:     80\nCommentPragmas:  '^ IWYU pragma:'\nCompactNamespaces: false\nConstructorInitializerAllOnOneLineOrOnePerLine: true\nConstructorInitializerIndentWidth: 4\nContinuationIndentWidth: 4\nCpp11BracedListStyle: true\nDerivePointerAlignment: false\nDisableFormat:   false\nForEachMacros:   [ FOR_EACH_RANGE, FOR_EACH, ]\nIncludeCategories:\n  - Regex:           '^<.*\\.h(pp)?>'\n    Priority:        1\n  - Regex:           '^<.*'\n    Priority:        2\n  - Regex:           '.*'\n    Priority:        3\nIndentCaseLabels: true\nIndentWidth:     2\nIndentWrappedFunctionNames: false\nKeepEmptyLinesAtTheStartOfBlocks: false\nMacroBlockBegin: ''\nMacroBlockEnd:   ''\nMaxEmptyLinesToKeep: 1\nNamespaceIndentation: None\nObjCBlockIndentWidth: 2\nObjCSpaceAfterProperty: false\nObjCSpaceBeforeProtocolList: false\nPenaltyBreakBeforeFirstCallParameter: 1\nPenaltyBreakComment: 300\nPenaltyBreakFirstLessLess: 120\nPenaltyBreakString: 1000\nPenaltyExcessCharacter: 1000000\nPenaltyReturnTypeOnItsOwnLine: 2000000\nPointerAlignment: Left\nReflowComments:  true\nSortIncludes:    true\nSpaceAfterCStyleCast: false\nSpaceBeforeAssignmentOperators: true\nSpaceBeforeParens: ControlStatements\nSpaceInEmptyParentheses: false\nSpacesBeforeTrailingComments: 1\nSpacesInAngles:  false\nSpacesInContainerLiterals: true\nSpacesInCStyleCastParentheses: false\nSpacesInParentheses: false\nSpacesInSquareBrackets: false\nStandard:        Cpp11\nTabWidth:        8\nUseTab:          Never\n...\n"
  },
  {
    "path": "cpp/autograd/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\n\nproject(autograd)\nset(CMAKE_CXX_STANDARD 17)\n\nfind_package(Torch REQUIRED)\n\nadd_executable(${PROJECT_NAME} \"autograd.cpp\")\ntarget_link_libraries(${PROJECT_NAME} \"${TORCH_LIBRARIES}\")\n\n# The following code block is suggested to be used on Windows.\n# According to https://github.com/pytorch/pytorch/issues/25457,\n# the DLLs need to be copied to avoid memory errors.\nif (MSVC)\n  file(GLOB TORCH_DLLS \"${TORCH_INSTALL_PREFIX}/lib/*.dll\")\n  add_custom_command(TARGET ${PROJECT_NAME}\n                     POST_BUILD\n                     COMMAND ${CMAKE_COMMAND} -E copy_if_different\n                     ${TORCH_DLLS}\n                     $<TARGET_FILE_DIR:${PROJECT_NAME}>)\nendif (MSVC)\n"
  },
  {
    "path": "cpp/autograd/README.md",
    "content": "# C++ autograd example\n\n`autograd.cpp` contains several examples of doing autograd in PyTorch C++ frontend.\n\nTo build the code, run the following commands from your terminal:\n\n```shell\n$ cd autograd\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..\n$ make\n```\n\nwhere `/path/to/libtorch` should be the path to the unzipped _LibTorch_\ndistribution, which you can get from the [PyTorch\nhomepage](https://pytorch.org/get-started/locally/).\n\nExecute the compiled binary to run:\n\n```shell\n$ ./autograd\n====== Running: \"Basic autograd operations\" ======\n 1  1\n 1  1\n[ CPUFloatType{2,2} ]\n 3  3\n 3  3\n[ CPUFloatType{2,2} ]\nAddBackward1\n 27  27\n 27  27\n[ CPUFloatType{2,2} ]\nMulBackward1\n27\n[ CPUFloatType{} ]\nMeanBackward0\nfalse\ntrue\nSumBackward0\n 4.5000  4.5000\n 4.5000  4.5000\n[ CPUFloatType{2,2} ]\n  813.6625\n 1015.0142\n -664.8849\n[ CPUFloatType{3} ]\nMulBackward1\n  204.8000\n 2048.0000\n    0.2048\n[ CPUFloatType{3} ]\ntrue\ntrue\nfalse\ntrue\nfalse\ntrue\n\n====== Running \"Computing higher-order gradients in C++\" ======\n 0.0025  0.0946  0.1474  0.1387\n 0.0238 -0.0018  0.0259  0.0094\n 0.0513 -0.0549 -0.0604  0.0210\n[ CPUFloatType{3,4} ]\n\n====== Running \"Using custom autograd function in C++\" ======\n-3.5513  3.7160  3.6477\n-3.5513  3.7160  3.6477\n[ CPUFloatType{2,3} ]\n 0.3095  1.4035 -0.0349\n 0.3095  1.4035 -0.0349\n 0.3095  1.4035 -0.0349\n 0.3095  1.4035 -0.0349\n[ CPUFloatType{4,3} ]\n 5.5000\n 5.5000\n[ CPUFloatType{2} ]\n```\n"
  },
  {
    "path": "cpp/autograd/autograd.cpp",
    "content": "#include <torch/torch.h>\n#include <iostream>\n\nusing namespace torch::autograd;\n\nvoid basic_autograd_operations_example() {\n  std::cout << \"====== Running: \\\"Basic autograd operations\\\" ======\" << std::endl;\n\n  // Create a tensor and set ``torch::requires_grad()`` to track computation with it\n  auto x = torch::ones({2, 2}, torch::requires_grad());\n  std::cout << x << std::endl;\n\n  // Do a tensor operation:\n  auto y = x + 2;\n  std::cout << y << std::endl;\n\n  // ``y`` was created as a result of an operation, so it has a ``grad_fn``.\n  std::cout << y.grad_fn()->name() << std::endl;\n\n  // Do more operations on ``y``\n  auto z = y * y * 3;\n  auto out = z.mean();\n\n  std::cout << z << std::endl;\n  std::cout << z.grad_fn()->name() << std::endl;\n  std::cout << out << std::endl;\n  std::cout << out.grad_fn()->name() << std::endl;\n\n  // ``.requires_grad_( ... )`` changes an existing tensor's ``requires_grad`` flag in-place.\n  auto a = torch::randn({2, 2});\n  a = ((a * 3) / (a - 1));\n  std::cout << a.requires_grad() << std::endl;\n\n  a.requires_grad_(true);\n  std::cout << a.requires_grad() << std::endl;\n\n  auto b = (a * a).sum();\n  std::cout << b.grad_fn()->name() << std::endl;\n\n  // Let's backprop now. Because ``out`` contains a single scalar, ``out.backward()``\n  // is equivalent to ``out.backward(torch::tensor(1.))``.\n  out.backward();\n\n  // Print gradients d(out)/dx\n  std::cout << x.grad() << std::endl;\n\n  // Now let's take a look at an example of vector-Jacobian product:\n  x = torch::randn(3, torch::requires_grad());\n\n  y = x * 2;\n  while (y.norm().item<double>() < 1000) {\n    y = y * 2;\n  }\n\n  std::cout << y << std::endl;\n  std::cout << y.grad_fn()->name() << std::endl;\n\n  // If we want the vector-Jacobian product, pass the vector to ``backward`` as argument:\n  auto v = torch::tensor({0.1, 1.0, 0.0001}, torch::kFloat);\n  y.backward(v);\n\n  std::cout << x.grad() << std::endl;\n\n  // You can also stop autograd from tracking history on tensors that require gradients\n  // either by putting ``torch::NoGradGuard`` in a code block\n  std::cout << x.requires_grad() << std::endl;\n  std::cout << x.pow(2).requires_grad() << std::endl;\n\n  {\n    torch::NoGradGuard no_grad;\n    std::cout << x.pow(2).requires_grad() << std::endl;\n  }\n\n  // Or by using ``.detach()`` to get a new tensor with the same content but that does\n  // not require gradients:\n  std::cout << x.requires_grad() << std::endl;\n  y = x.detach();\n  std::cout << y.requires_grad() << std::endl;\n  std::cout << x.eq(y).all().item<bool>() << std::endl;\n}\n\nvoid compute_higher_order_gradients_example() {\n  std::cout << \"====== Running \\\"Computing higher-order gradients in C++\\\" ======\" << std::endl;\n\n  // One of the applications of higher-order gradients is calculating gradient penalty.\n  // Let's see an example of it using ``torch::autograd::grad``:\n\n  auto model = torch::nn::Linear(4, 3);\n\n  auto input = torch::randn({3, 4}).requires_grad_(true);\n  auto output = model(input);\n\n  // Calculate loss\n  auto target = torch::randn({3, 3});\n  auto loss = torch::nn::MSELoss()(output, target);\n\n  // Use norm of gradients as penalty\n  auto grad_output = torch::ones_like(output);\n  auto gradient = torch::autograd::grad({output}, {input}, /*grad_outputs=*/{grad_output}, /*create_graph=*/true)[0];\n  auto gradient_penalty = torch::pow((gradient.norm(2, /*dim=*/1) - 1), 2).mean();\n\n  // Add gradient penalty to loss\n  auto combined_loss = loss + gradient_penalty;\n  combined_loss.backward();\n\n  std::cout << input.grad() << std::endl;\n}\n\n// Inherit from Function\nclass LinearFunction : public Function<LinearFunction> {\n public:\n  // Note that both forward and backward are static functions\n\n  // bias is an optional argument\n  static torch::Tensor forward(\n      AutogradContext *ctx, torch::Tensor input, torch::Tensor weight, torch::Tensor bias = torch::Tensor()) {\n    ctx->save_for_backward({input, weight, bias});\n    auto output = input.mm(weight.t());\n    if (bias.defined()) {\n      output += bias.unsqueeze(0).expand_as(output);\n    }\n    return output;\n  }\n\n  static tensor_list backward(AutogradContext *ctx, tensor_list grad_outputs) {\n    auto saved = ctx->get_saved_variables();\n    auto input = saved[0];\n    auto weight = saved[1];\n    auto bias = saved[2];\n\n    auto grad_output = grad_outputs[0];\n    auto grad_input = grad_output.mm(weight);\n    auto grad_weight = grad_output.t().mm(input);\n    auto grad_bias = torch::Tensor();\n    if (bias.defined()) {\n      grad_bias = grad_output.sum(0);\n    }\n\n    return {grad_input, grad_weight, grad_bias};\n  }\n};\n\nclass MulConstant : public Function<MulConstant> {\n public:\n  static torch::Tensor forward(AutogradContext *ctx, torch::Tensor tensor, double constant) {\n    // ctx is a context object that can be used to stash information\n    // for backward computation\n    ctx->saved_data[\"constant\"] = constant;\n    return tensor * constant;\n  }\n\n  static tensor_list backward(AutogradContext *ctx, tensor_list grad_outputs) {\n    // We return as many input gradients as there were arguments.\n    // Gradients of non-tensor arguments to forward must be `torch::Tensor()`.\n    return {grad_outputs[0] * ctx->saved_data[\"constant\"].toDouble(), torch::Tensor()};\n  }\n};\n\nvoid custom_autograd_function_example() {\n  std::cout << \"====== Running \\\"Using custom autograd function in C++\\\" ======\" << std::endl;\n  {\n    auto x = torch::randn({2, 3}).requires_grad_();\n    auto weight = torch::randn({4, 3}).requires_grad_();\n    auto y = LinearFunction::apply(x, weight);\n    y.sum().backward();\n\n    std::cout << x.grad() << std::endl;\n    std::cout << weight.grad() << std::endl;\n  }\n  {\n    auto x = torch::randn({2}).requires_grad_();\n    auto y = MulConstant::apply(x, 5.5);\n    y.sum().backward();\n\n    std::cout << x.grad() << std::endl;\n  }\n}\n\nint main() {\n  std::cout << std::boolalpha;\n\n  basic_autograd_operations_example();\n\n  std::cout << \"\\n\";\n\n  compute_higher_order_gradients_example();\n\n  std::cout << \"\\n\";\n\n  custom_autograd_function_example();\n}\n"
  },
  {
    "path": "cpp/custom-dataset/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\n\nproject(custom-dataset)\nset(CMAKE_CXX_STANDARD 17)\n\nfind_package(Torch REQUIRED)\nfind_package(OpenCV REQUIRED COMPONENTS core imgproc imgcodecs)\n\nmessage(STATUS \"OpenCV include dirs: ${OpenCV_INCLUDE_DIRS}\")\nmessage(STATUS \"OpenCV libraries: ${OpenCV_LIBS}\")\n\n\ninclude_directories(${OpenCV_INCLUDE_DIRS})\nadd_executable(${PROJECT_NAME} \"custom-dataset.cpp\")\ntarget_link_libraries(${PROJECT_NAME} \"${OpenCV_LIBS}\")\ntarget_link_libraries(${PROJECT_NAME} \"${TORCH_LIBRARIES}\")\n\nconfigure_file(\"info.txt\" \"info.txt\" COPYONLY)\n\n# The following code block is suggested to be used on Windows.\n# According to https://github.com/pytorch/pytorch/issues/25457,\n# the DLLs need to be copied to avoid memory errors.\nif (MSVC)\n  file(GLOB TORCH_DLLS \"${TORCH_INSTALL_PREFIX}/lib/*.dll\")\n  add_custom_command(TARGET ${PROJECT_NAME}\n                     POST_BUILD\n                     COMMAND ${CMAKE_COMMAND} -E copy_if_different\n                     ${TORCH_DLLS}\n                     $<TARGET_FILE_DIR:${PROJECT_NAME}>)\nendif (MSVC)\n"
  },
  {
    "path": "cpp/custom-dataset/README.md",
    "content": "# Custom Dataset Example with the PyTorch C++ Frontend\n\nThis folder contains an example of loading a custom image dataset with OpenCV and training a model to label images, using the PyTorch C++ frontend.\n\nThe dataset used here is [Caltech 101](https://data.caltech.edu/records/mzrjq-6wc02) dataset.\n\nThe entire training code is contained in custom-data.cpp.\n\nYou can find instructions on how to install OpenCV [here](../tools/InstallingOpenCV.md).\n\nTo build the code, run the following commands from your terminal:\n\n```shell\n$ cd custom-dataset\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..\n$ make\n```\n\nwhere /path/to/libtorch should be the path to the unzipped LibTorch distribution, which you can get from the [PyTorch homepage](https://pytorch.org/get-started/locally/).\n\nIf you see an error like `undefined reference to cv::imread(std::string const&, int)` when running the `make` command, you should build LibTorch from source using the instructions [here](https://github.com/pytorch/pytorch#from-source), and then set `CMAKE_PREFIX_PATH` to that PyTorch source directory. An alternative solution is to use `libtorch-cxx11-abi-shared-with-deps` instead of `libtorch-shared-with-deps` as the latter is not compatible with openCV (reported [here](https://discuss.pytorch.org/t/library-conflict-between-libtorch-and-opencv/64489)).\n\nThe build directory should look like this:\n\n```\n.\n├── custom-dataset\n├── dataset\n│   ├── accordion\n│   │   ├── image_0001.jpg\n│   │   ├── ...\n│   ├── airplanes\n│   │   ├── ...\n│   ├── ...\n├── info.txt\n└── Makefile\n└── ...\n```\n\n`info.txt` file gets copied from source directory during build.\n\nExecute the compiled binary to train the model:\n\n```shell\n./custom-dataset\nRunning on: CUDA\nTrain Epoch: 1 16/7281\tLoss: 0.314655\tAcc: 0\nTrain Epoch: 1 176/7281\tLoss: 0.532111\tAcc: 0.0681818\nTrain Epoch: 1 336/7281\tLoss: 0.538482\tAcc: 0.0714286\nTrain Epoch: 1 496/7281\tLoss: 0.535302\tAcc: 0.0705645\nTrain Epoch: 1 656/7281\tLoss: 0.536113\tAcc: 0.0716463\nTrain Epoch: 1 816/7281\tLoss: 0.537626\tAcc: 0.0784314\nTrain Epoch: 1 976/7281\tLoss: 0.537055\tAcc: 0.079918\n...\n\n```"
  },
  {
    "path": "cpp/custom-dataset/custom-dataset.cpp",
    "content": "#include <opencv2/opencv.hpp>\n#include <stdint.h>\n#include <torch/torch.h>\n#include <fstream>\n#include <iostream>\n#include <string>\n#include <vector>\n#include <random>\n\nstruct Options {\n  int image_size = 224;\n  size_t train_batch_size = 8;\n  size_t test_batch_size = 200;\n  size_t iterations = 10;\n  size_t log_interval = 20;\n  // path must end in delimiter\n  std::string datasetPath = \"./dataset/\";\n  std::string infoFilePath = \"info.txt\";\n  torch::DeviceType device = torch::kCPU;\n};\n\nstatic Options options;\n\nusing Data = std::vector<std::pair<std::string, long>>;\n\nclass CustomDataset : public torch::data::datasets::Dataset<CustomDataset> {\n  using Example = torch::data::Example<>;\n\n  const Data data;\n\n public:\n  CustomDataset(const Data& data) : data(data) {}\n\n  Example get(size_t index) {\n    std::string path = options.datasetPath + data[index].first;\n    auto mat = cv::imread(path);\n    assert(!mat.empty());\n\n    cv::resize(mat, mat, cv::Size(options.image_size, options.image_size));\n    std::vector<cv::Mat> channels(3);\n    cv::split(mat, channels);\n\n    auto R = torch::from_blob(\n        channels[2].ptr(),\n        {options.image_size, options.image_size},\n        torch::kUInt8);\n    auto G = torch::from_blob(\n        channels[1].ptr(),\n        {options.image_size, options.image_size},\n        torch::kUInt8);\n    auto B = torch::from_blob(\n        channels[0].ptr(),\n        {options.image_size, options.image_size},\n        torch::kUInt8);\n\n    auto tdata = torch::cat({R, G, B})\n                     .view({3, options.image_size, options.image_size})\n                     .to(torch::kFloat);\n    auto tlabel = torch::tensor(data[index].second, torch::kLong);\n    return {tdata, tlabel};\n  }\n\n  torch::optional<size_t> size() const {\n    return data.size();\n  }\n};\n\nstd::pair<Data, Data> readInfo() {\n  std::random_device randomDevice;\n  std::mt19937 mersenneTwisterGenerator(randomDevice());\n  Data train, test;\n\n  std::ifstream stream(options.infoFilePath);\n  assert(stream.is_open());\n\n  long label;\n  std::string path, type;\n\n  while (true) {\n    stream >> path >> label >> type;\n\n    if (type == \"train\")\n      train.push_back(std::make_pair(path, label));\n    else if (type == \"test\")\n      test.push_back(std::make_pair(path, label));\n    else\n      assert(false);\n\n    if (stream.eof())\n      break;\n  }\n\n  std::shuffle(train.begin(), train.end(), mersenneTwisterGenerator);\n  std::shuffle(test.begin(), test.end(), mersenneTwisterGenerator);\n  return std::make_pair(train, test);\n}\n\nstruct NetworkImpl : torch::nn::SequentialImpl {\n  NetworkImpl() {\n    using namespace torch::nn;\n\n    auto stride = torch::ExpandingArray<2>({2, 2});\n    torch::ExpandingArray<2> shape({-1, 256 * 6 * 6});\n    push_back(Conv2d(Conv2dOptions(3, 64, 11).stride(4).padding(2)));\n    push_back(Functional(torch::relu));\n    push_back(Functional(torch::max_pool2d, 3, stride, 0, 1, false));\n    push_back(Conv2d(Conv2dOptions(64, 192, 5).padding(2)));\n    push_back(Functional(torch::relu));\n    push_back(Functional(torch::max_pool2d, 3, stride, 0, 1, false));\n    push_back(Conv2d(Conv2dOptions(192, 384, 3).padding(1)));\n    push_back(Functional(torch::relu));\n    push_back(Conv2d(Conv2dOptions(384, 256, 3).padding(1)));\n    push_back(Functional(torch::relu));\n    push_back(Conv2d(Conv2dOptions(256, 256, 3).padding(1)));\n    push_back(Functional(torch::relu));\n    push_back(Functional(torch::max_pool2d, 3, stride, 0, 1, false));\n    push_back(Functional(torch::reshape, shape));\n    push_back(Dropout());\n    push_back(Linear(256 * 6 * 6, 4096));\n    push_back(Functional(torch::relu));\n    push_back(Dropout());\n    push_back(Linear(4096, 4096));\n    push_back(Functional(torch::relu));\n    push_back(Linear(4096, 102));\n    push_back(Functional(\n        [](torch::Tensor input) { return torch::log_softmax(input, 1); }));\n  }\n};\nTORCH_MODULE(Network);\n\ntemplate <typename DataLoader>\nvoid train(\n    Network& network,\n    DataLoader& loader,\n    torch::optim::Optimizer& optimizer,\n    size_t epoch,\n    size_t data_size) {\n  size_t index = 0;\n  network->train();\n  float Loss = 0, Acc = 0;\n\n  for (auto& batch : loader) {\n    auto data = batch.data.to(options.device);\n    auto targets = batch.target.to(options.device).view({-1});\n\n    auto output = network->forward(data);\n    auto loss = torch::nll_loss(output, targets);\n    assert(!std::isnan(loss.template item<float>()));\n    auto acc = output.argmax(1).eq(targets).sum();\n\n    optimizer.zero_grad();\n    loss.backward();\n    optimizer.step();\n\n    Loss += loss.template item<float>();\n    Acc += acc.template item<float>();\n\n    if (index++ % options.log_interval == 0) {\n      auto end = std::min(data_size, (index + 1) * options.train_batch_size);\n\n      std::cout << \"Train Epoch: \" << epoch << \" \" << end << \"/\" << data_size\n                << \"\\tLoss: \" << Loss / end << \"\\tAcc: \" << Acc / end\n                << std::endl;\n    }\n  }\n}\n\ntemplate <typename DataLoader>\nvoid test(Network& network, DataLoader& loader, size_t data_size) {\n  size_t index = 0;\n  network->eval();\n  torch::NoGradGuard no_grad;\n  float Loss = 0, Acc = 0;\n\n  for (const auto& batch : loader) {\n    auto data = batch.data.to(options.device);\n    auto targets = batch.target.to(options.device).view({-1});\n\n    auto output = network->forward(data);\n    auto loss = torch::nll_loss(output, targets);\n    assert(!std::isnan(loss.template item<float>()));\n    auto acc = output.argmax(1).eq(targets).sum();\n\n    Loss += loss.template item<float>();\n    Acc += acc.template item<float>();\n  }\n\n  if (index++ % options.log_interval == 0)\n    std::cout << \"Test Loss: \" << Loss / data_size\n              << \"\\tAcc: \" << Acc / data_size << std::endl;\n}\n\nint main() {\n  torch::manual_seed(1);\n\n  if (torch::cuda::is_available())\n    options.device = torch::kCUDA;\n  std::cout << \"Running on: \"\n            << (options.device == torch::kCUDA ? \"CUDA\" : \"CPU\") << std::endl;\n\n  const auto data = readInfo();\n\n  auto train_set =\n      CustomDataset(data.first).map(torch::data::transforms::Stack<>());\n  auto train_size = train_set.size().value();\n  auto train_loader =\n      torch::data::make_data_loader<torch::data::samplers::SequentialSampler>(\n          std::move(train_set), options.train_batch_size);\n\n  auto test_set =\n      CustomDataset(data.second).map(torch::data::transforms::Stack<>());\n  auto test_size = test_set.size().value();\n  auto test_loader =\n      torch::data::make_data_loader<torch::data::samplers::SequentialSampler>(\n          std::move(test_set), options.test_batch_size);\n\n  Network network;\n  network->to(options.device);\n\n  torch::optim::SGD optimizer(\n      network->parameters(), torch::optim::SGDOptions(0.001).momentum(0.5));\n\n  for (size_t i = 0; i < options.iterations; ++i) {\n    train(network, *train_loader, optimizer, i + 1, train_size);\n    std::cout << std::endl;\n    test(network, *test_loader, test_size);\n    std::cout << std::endl;\n  }\n\n  return 0;\n}\n"
  },
  {
    "path": "cpp/custom-dataset/info.txt",
    "content": "airplanes/image_0316.jpg 0 train\nairplanes/image_0716.jpg 0 train\nairplanes/image_0258.jpg 0 train\nairplanes/image_0426.jpg 0 train\nairplanes/image_0511.jpg 0 train\nairplanes/image_0676.jpg 0 train\nairplanes/image_0528.jpg 0 train\nairplanes/image_0720.jpg 0 train\nairplanes/image_0556.jpg 0 train\nairplanes/image_0302.jpg 0 train\nairplanes/image_0171.jpg 0 train\nairplanes/image_0062.jpg 0 train\nairplanes/image_0504.jpg 0 train\nairplanes/image_0346.jpg 0 train\nairplanes/image_0546.jpg 0 train\nairplanes/image_0435.jpg 0 train\nairplanes/image_0341.jpg 0 train\nairplanes/image_0776.jpg 0 train\nairplanes/image_0340.jpg 0 train\nairplanes/image_0331.jpg 0 train\nairplanes/image_0037.jpg 0 train\nairplanes/image_0737.jpg 0 train\nairplanes/image_0786.jpg 0 train\nairplanes/image_0560.jpg 0 train\nairplanes/image_0472.jpg 0 train\nairplanes/image_0049.jpg 0 train\nairplanes/image_0337.jpg 0 train\nairplanes/image_0779.jpg 0 train\nairplanes/image_0317.jpg 0 train\nairplanes/image_0357.jpg 0 train\nairplanes/image_0083.jpg 0 train\nairplanes/image_0101.jpg 0 train\nairplanes/image_0051.jpg 0 train\nairplanes/image_0006.jpg 0 train\nairplanes/image_0142.jpg 0 train\nairplanes/image_0681.jpg 0 train\nairplanes/image_0250.jpg 0 train\nairplanes/image_0427.jpg 0 train\nairplanes/image_0486.jpg 0 train\nairplanes/image_0396.jpg 0 train\nairplanes/image_0733.jpg 0 train\nairplanes/image_0332.jpg 0 train\nairplanes/image_0448.jpg 0 train\nairplanes/image_0335.jpg 0 train\nairplanes/image_0752.jpg 0 train\nairplanes/image_0201.jpg 0 train\nairplanes/image_0505.jpg 0 train\nairplanes/image_0016.jpg 0 train\nairplanes/image_0373.jpg 0 train\nairplanes/image_0772.jpg 0 train\nairplanes/image_0565.jpg 0 train\nairplanes/image_0513.jpg 0 train\nairplanes/image_0709.jpg 0 train\nairplanes/image_0176.jpg 0 train\nairplanes/image_0411.jpg 0 train\nairplanes/image_0155.jpg 0 train\nairplanes/image_0030.jpg 0 train\nairplanes/image_0091.jpg 0 train\nairplanes/image_0664.jpg 0 train\nairplanes/image_0563.jpg 0 train\nairplanes/image_0397.jpg 0 train\nairplanes/image_0618.jpg 0 train\nairplanes/image_0012.jpg 0 train\nairplanes/image_0066.jpg 0 train\nairplanes/image_0657.jpg 0 train\nairplanes/image_0765.jpg 0 train\nairplanes/image_0585.jpg 0 train\nairplanes/image_0612.jpg 0 train\nairplanes/image_0544.jpg 0 train\nairplanes/image_0703.jpg 0 train\nairplanes/image_0465.jpg 0 train\nairplanes/image_0728.jpg 0 train\nairplanes/image_0464.jpg 0 train\nairplanes/image_0283.jpg 0 train\nairplanes/image_0551.jpg 0 train\nairplanes/image_0318.jpg 0 train\nairplanes/image_0224.jpg 0 train\nairplanes/image_0771.jpg 0 train\nairplanes/image_0148.jpg 0 train\nairplanes/image_0501.jpg 0 train\nairplanes/image_0496.jpg 0 train\nairplanes/image_0077.jpg 0 train\nairplanes/image_0129.jpg 0 train\nairplanes/image_0339.jpg 0 train\nairplanes/image_0554.jpg 0 train\nairplanes/image_0025.jpg 0 train\nairplanes/image_0328.jpg 0 train\nairplanes/image_0549.jpg 0 train\nairplanes/image_0659.jpg 0 train\nairplanes/image_0540.jpg 0 train\nairplanes/image_0518.jpg 0 train\nairplanes/image_0479.jpg 0 train\nairplanes/image_0539.jpg 0 train\nairplanes/image_0527.jpg 0 train\nairplanes/image_0783.jpg 0 train\nairplanes/image_0621.jpg 0 train\nairplanes/image_0476.jpg 0 train\nairplanes/image_0139.jpg 0 train\nairplanes/image_0519.jpg 0 train\nairplanes/image_0425.jpg 0 train\nairplanes/image_0292.jpg 0 train\nairplanes/image_0348.jpg 0 train\nairplanes/image_0499.jpg 0 train\nairplanes/image_0115.jpg 0 train\nairplanes/image_0123.jpg 0 train\nairplanes/image_0119.jpg 0 train\nairplanes/image_0670.jpg 0 train\nairplanes/image_0721.jpg 0 train\nairplanes/image_0639.jpg 0 train\nairplanes/image_0060.jpg 0 train\nairplanes/image_0269.jpg 0 train\nairplanes/image_0052.jpg 0 train\nairplanes/image_0108.jpg 0 train\nairplanes/image_0239.jpg 0 train\nairplanes/image_0442.jpg 0 train\nairplanes/image_0684.jpg 0 train\nairplanes/image_0370.jpg 0 train\nairplanes/image_0491.jpg 0 train\nairplanes/image_0469.jpg 0 train\nairplanes/image_0326.jpg 0 train\nairplanes/image_0223.jpg 0 train\nairplanes/image_0172.jpg 0 train\nairplanes/image_0372.jpg 0 train\nairplanes/image_0704.jpg 0 train\nairplanes/image_0344.jpg 0 train\nairplanes/image_0390.jpg 0 train\nairplanes/image_0007.jpg 0 train\nairplanes/image_0213.jpg 0 train\nairplanes/image_0739.jpg 0 train\nairplanes/image_0312.jpg 0 train\nairplanes/image_0463.jpg 0 train\nairplanes/image_0063.jpg 0 train\nairplanes/image_0019.jpg 0 train\nairplanes/image_0407.jpg 0 train\nairplanes/image_0192.jpg 0 train\nairplanes/image_0490.jpg 0 train\nairplanes/image_0533.jpg 0 train\nairplanes/image_0743.jpg 0 train\nairplanes/image_0384.jpg 0 train\nairplanes/image_0195.jpg 0 train\nairplanes/image_0113.jpg 0 train\nairplanes/image_0677.jpg 0 train\nairplanes/image_0127.jpg 0 train\nairplanes/image_0215.jpg 0 train\nairplanes/image_0389.jpg 0 train\nairplanes/image_0031.jpg 0 train\nairplanes/image_0349.jpg 0 train\nairplanes/image_0228.jpg 0 train\nairplanes/image_0380.jpg 0 train\nairplanes/image_0219.jpg 0 train\nairplanes/image_0796.jpg 0 train\nairplanes/image_0688.jpg 0 train\nairplanes/image_0597.jpg 0 train\nairplanes/image_0353.jpg 0 train\nairplanes/image_0011.jpg 0 train\nairplanes/image_0562.jpg 0 train\nairplanes/image_0233.jpg 0 train\nairplanes/image_0447.jpg 0 train\nairplanes/image_0747.jpg 0 train\nairplanes/image_0477.jpg 0 train\nairplanes/image_0680.jpg 0 train\nairplanes/image_0061.jpg 0 train\nairplanes/image_0444.jpg 0 train\nairplanes/image_0090.jpg 0 train\nairplanes/image_0745.jpg 0 train\nairplanes/image_0351.jpg 0 train\nairplanes/image_0261.jpg 0 train\nairplanes/image_0229.jpg 0 train\nairplanes/image_0126.jpg 0 train\nairplanes/image_0761.jpg 0 train\nairplanes/image_0775.jpg 0 train\nairplanes/image_0794.jpg 0 train\nairplanes/image_0240.jpg 0 train\nairplanes/image_0467.jpg 0 train\nairplanes/image_0768.jpg 0 train\nairplanes/image_0455.jpg 0 train\nairplanes/image_0441.jpg 0 train\nairplanes/image_0305.jpg 0 train\nairplanes/image_0572.jpg 0 train\nairplanes/image_0334.jpg 0 train\nairplanes/image_0602.jpg 0 train\nairplanes/image_0354.jpg 0 train\nairplanes/image_0542.jpg 0 train\nairplanes/image_0218.jpg 0 train\nairplanes/image_0573.jpg 0 train\nairplanes/image_0366.jpg 0 train\nairplanes/image_0288.jpg 0 train\nairplanes/image_0583.jpg 0 train\nairplanes/image_0216.jpg 0 train\nairplanes/image_0459.jpg 0 train\nairplanes/image_0429.jpg 0 train\nairplanes/image_0277.jpg 0 train\nairplanes/image_0623.jpg 0 train\nairplanes/image_0358.jpg 0 train\nairplanes/image_0588.jpg 0 train\nairplanes/image_0050.jpg 0 train\nairplanes/image_0246.jpg 0 train\nairplanes/image_0361.jpg 0 train\nairplanes/image_0177.jpg 0 train\nairplanes/image_0356.jpg 0 train\nairplanes/image_0648.jpg 0 train\nairplanes/image_0789.jpg 0 train\nairplanes/image_0196.jpg 0 train\nairplanes/image_0293.jpg 0 train\nairplanes/image_0406.jpg 0 train\nairplanes/image_0596.jpg 0 train\nairplanes/image_0640.jpg 0 train\nairplanes/image_0575.jpg 0 train\nairplanes/image_0625.jpg 0 train\nairplanes/image_0347.jpg 0 train\nairplanes/image_0521.jpg 0 train\nairplanes/image_0443.jpg 0 train\nairplanes/image_0187.jpg 0 train\nairplanes/image_0558.jpg 0 train\nairplanes/image_0234.jpg 0 train\nairplanes/image_0045.jpg 0 train\nairplanes/image_0303.jpg 0 train\nairplanes/image_0363.jpg 0 train\nairplanes/image_0679.jpg 0 train\nairplanes/image_0017.jpg 0 train\nairplanes/image_0131.jpg 0 train\nairplanes/image_0762.jpg 0 train\nairplanes/image_0492.jpg 0 train\nairplanes/image_0296.jpg 0 train\nairplanes/image_0555.jpg 0 train\nairplanes/image_0735.jpg 0 train\nairplanes/image_0150.jpg 0 train\nairplanes/image_0018.jpg 0 train\nairplanes/image_0350.jpg 0 train\nairplanes/image_0795.jpg 0 train\nairplanes/image_0209.jpg 0 train\nairplanes/image_0013.jpg 0 train\nairplanes/image_0570.jpg 0 train\nairplanes/image_0748.jpg 0 train\nairplanes/image_0569.jpg 0 train\nairplanes/image_0696.jpg 0 train\nairplanes/image_0614.jpg 0 train\nairplanes/image_0054.jpg 0 train\nairplanes/image_0629.jpg 0 train\nairplanes/image_0094.jpg 0 train\nairplanes/image_0613.jpg 0 train\nairplanes/image_0082.jpg 0 train\nairplanes/image_0097.jpg 0 train\nairplanes/image_0130.jpg 0 train\nairplanes/image_0457.jpg 0 train\nairplanes/image_0488.jpg 0 train\nairplanes/image_0382.jpg 0 train\nairplanes/image_0484.jpg 0 train\nairplanes/image_0141.jpg 0 train\nairplanes/image_0700.jpg 0 train\nairplanes/image_0144.jpg 0 train\nairplanes/image_0109.jpg 0 train\nairplanes/image_0645.jpg 0 train\nairplanes/image_0468.jpg 0 train\nairplanes/image_0515.jpg 0 train\nairplanes/image_0079.jpg 0 train\nairplanes/image_0306.jpg 0 train\nairplanes/image_0398.jpg 0 train\nairplanes/image_0055.jpg 0 train\nairplanes/image_0367.jpg 0 train\nairplanes/image_0365.jpg 0 train\nairplanes/image_0566.jpg 0 train\nairplanes/image_0217.jpg 0 train\nairplanes/image_0343.jpg 0 train\nairplanes/image_0432.jpg 0 train\nairplanes/image_0538.jpg 0 train\nairplanes/image_0600.jpg 0 train\nairplanes/image_0502.jpg 0 train\nairplanes/image_0661.jpg 0 train\nairplanes/image_0731.jpg 0 train\nairplanes/image_0071.jpg 0 train\nairplanes/image_0590.jpg 0 train\nairplanes/image_0282.jpg 0 train\nairplanes/image_0393.jpg 0 train\nairplanes/image_0287.jpg 0 train\nairplanes/image_0198.jpg 0 train\nairplanes/image_0262.jpg 0 train\nairplanes/image_0686.jpg 0 train\nairplanes/image_0462.jpg 0 train\nairplanes/image_0698.jpg 0 train\nairplanes/image_0168.jpg 0 train\nairplanes/image_0368.jpg 0 train\nairplanes/image_0420.jpg 0 train\nairplanes/image_0650.jpg 0 train\nairplanes/image_0763.jpg 0 train\nairplanes/image_0211.jpg 0 train\nairplanes/image_0537.jpg 0 train\nairplanes/image_0580.jpg 0 train\nairplanes/image_0627.jpg 0 train\nairplanes/image_0270.jpg 0 train\nairplanes/image_0594.jpg 0 train\nairplanes/image_0103.jpg 0 train\nairplanes/image_0643.jpg 0 train\nairplanes/image_0591.jpg 0 train\nairplanes/image_0024.jpg 0 train\nairplanes/image_0787.jpg 0 train\nairplanes/image_0297.jpg 0 train\nairplanes/image_0188.jpg 0 train\nairplanes/image_0792.jpg 0 train\nairplanes/image_0105.jpg 0 train\nairplanes/image_0074.jpg 0 train\nairplanes/image_0294.jpg 0 train\nairplanes/image_0440.jpg 0 train\nairplanes/image_0391.jpg 0 train\nairplanes/image_0189.jpg 0 train\nairplanes/image_0003.jpg 0 train\nairplanes/image_0009.jpg 0 train\nairplanes/image_0387.jpg 0 train\nairplanes/image_0275.jpg 0 train\nairplanes/image_0226.jpg 0 train\nairplanes/image_0628.jpg 0 train\nairplanes/image_0718.jpg 0 train\nairplanes/image_0714.jpg 0 train\nairplanes/image_0658.jpg 0 train\nairplanes/image_0601.jpg 0 train\nairplanes/image_0185.jpg 0 train\nairplanes/image_0790.jpg 0 train\nairplanes/image_0656.jpg 0 train\nairplanes/image_0567.jpg 0 train\nairplanes/image_0485.jpg 0 train\nairplanes/image_0541.jpg 0 train\nairplanes/image_0118.jpg 0 train\nairplanes/image_0652.jpg 0 train\nairplanes/image_0304.jpg 0 train\nairplanes/image_0278.jpg 0 train\nairplanes/image_0512.jpg 0 train\nairplanes/image_0230.jpg 0 train\nairplanes/image_0474.jpg 0 train\nairplanes/image_0553.jpg 0 train\nairplanes/image_0404.jpg 0 train\nairplanes/image_0487.jpg 0 train\nairplanes/image_0531.jpg 0 train\nairplanes/image_0638.jpg 0 train\nairplanes/image_0525.jpg 0 train\nairplanes/image_0482.jpg 0 train\nairplanes/image_0522.jpg 0 train\nairplanes/image_0460.jpg 0 train\nairplanes/image_0608.jpg 0 train\nairplanes/image_0134.jpg 0 train\nairplanes/image_0249.jpg 0 train\nairplanes/image_0042.jpg 0 train\nairplanes/image_0730.jpg 0 train\nairplanes/image_0352.jpg 0 train\nairplanes/image_0333.jpg 0 train\nairplanes/image_0221.jpg 0 train\nairplanes/image_0093.jpg 0 train\nairplanes/image_0266.jpg 0 train\nairplanes/image_0412.jpg 0 train\nairplanes/image_0759.jpg 0 train\nairplanes/image_0470.jpg 0 train\nairplanes/image_0497.jpg 0 train\nairplanes/image_0751.jpg 0 train\nairplanes/image_0301.jpg 0 train\nairplanes/image_0392.jpg 0 train\nairplanes/image_0473.jpg 0 train\nairplanes/image_0254.jpg 0 train\nairplanes/image_0561.jpg 0 train\nairplanes/image_0530.jpg 0 train\nairplanes/image_0321.jpg 0 train\nairplanes/image_0557.jpg 0 train\nairplanes/image_0386.jpg 0 train\nairplanes/image_0232.jpg 0 train\nairplanes/image_0324.jpg 0 train\nairplanes/image_0040.jpg 0 train\nairplanes/image_0067.jpg 0 train\nairplanes/image_0769.jpg 0 train\nairplanes/image_0310.jpg 0 train\nairplanes/image_0330.jpg 0 train\nairplanes/image_0548.jpg 0 train\nairplanes/image_0122.jpg 0 train\nairplanes/image_0671.jpg 0 train\nairplanes/image_0205.jpg 0 train\nairplanes/image_0161.jpg 0 train\nairplanes/image_0146.jpg 0 train\nairplanes/image_0300.jpg 0 train\nairplanes/image_0369.jpg 0 train\nairplanes/image_0451.jpg 0 train\nairplanes/image_0461.jpg 0 train\nairplanes/image_0204.jpg 0 train\nairplanes/image_0125.jpg 0 train\nairplanes/image_0047.jpg 0 train\nairplanes/image_0285.jpg 0 train\nairplanes/image_0160.jpg 0 train\nairplanes/image_0678.jpg 0 train\nairplanes/image_0422.jpg 0 train\nairplanes/image_0579.jpg 0 train\nairplanes/image_0064.jpg 0 train\nairplanes/image_0070.jpg 0 train\nairplanes/image_0552.jpg 0 train\nairplanes/image_0264.jpg 0 train\nairplanes/image_0112.jpg 0 train\nairplanes/image_0078.jpg 0 train\nairplanes/image_0637.jpg 0 train\nairplanes/image_0068.jpg 0 train\nairplanes/image_0506.jpg 0 train\nairplanes/image_0433.jpg 0 train\nairplanes/image_0547.jpg 0 train\nairplanes/image_0085.jpg 0 train\nairplanes/image_0410.jpg 0 train\nairplanes/image_0110.jpg 0 train\nairplanes/image_0722.jpg 0 train\nairplanes/image_0593.jpg 0 train\nairplanes/image_0259.jpg 0 train\nairplanes/image_0143.jpg 0 train\nairplanes/image_0043.jpg 0 train\nairplanes/image_0162.jpg 0 train\nairplanes/image_0132.jpg 0 train\nairplanes/image_0157.jpg 0 train\nairplanes/image_0475.jpg 0 train\nairplanes/image_0206.jpg 0 train\nairplanes/image_0376.jpg 0 train\nairplanes/image_0644.jpg 0 train\nairplanes/image_0319.jpg 0 train\nairplanes/image_0371.jpg 0 train\nairplanes/image_0044.jpg 0 train\nairplanes/image_0199.jpg 0 train\nairplanes/image_0729.jpg 0 train\nairplanes/image_0691.jpg 0 train\nairplanes/image_0033.jpg 0 train\nairplanes/image_0136.jpg 0 train\nairplanes/image_0180.jpg 0 train\nairplanes/image_0260.jpg 0 train\nairplanes/image_0604.jpg 0 train\nairplanes/image_0184.jpg 0 train\nairplanes/image_0719.jpg 0 train\nairplanes/image_0672.jpg 0 train\nairplanes/image_0784.jpg 0 train\nairplanes/image_0073.jpg 0 train\nairplanes/image_0395.jpg 0 train\nairplanes/image_0257.jpg 0 train\nairplanes/image_0364.jpg 0 train\nairplanes/image_0756.jpg 0 train\nairplanes/image_0001.jpg 0 train\nairplanes/image_0167.jpg 0 train\nairplanes/image_0284.jpg 0 train\nairplanes/image_0375.jpg 0 train\nairplanes/image_0080.jpg 0 train\nairplanes/image_0665.jpg 0 train\nairplanes/image_0156.jpg 0 train\nairplanes/image_0021.jpg 0 train\nairplanes/image_0571.jpg 0 train\nairplanes/image_0325.jpg 0 train\nairplanes/image_0517.jpg 0 train\nairplanes/image_0713.jpg 0 train\nairplanes/image_0516.jpg 0 train\nairplanes/image_0574.jpg 0 train\nairplanes/image_0471.jpg 0 train\nairplanes/image_0767.jpg 0 train\nairplanes/image_0581.jpg 0 train\nairplanes/image_0267.jpg 0 train\nairplanes/image_0299.jpg 0 train\nairplanes/image_0421.jpg 0 train\nairplanes/image_0495.jpg 0 train\nairplanes/image_0529.jpg 0 train\nairplanes/image_0436.jpg 0 train\nairplanes/image_0360.jpg 0 train\nairplanes/image_0649.jpg 0 train\nairplanes/image_0095.jpg 0 train\nairplanes/image_0057.jpg 0 train\nairplanes/image_0707.jpg 0 train\nairplanes/image_0797.jpg 0 train\nairplanes/image_0403.jpg 0 train\nairplanes/image_0114.jpg 0 train\nairplanes/image_0641.jpg 0 train\nairplanes/image_0595.jpg 0 train\nairplanes/image_0620.jpg 0 train\nairplanes/image_0153.jpg 0 train\nairplanes/image_0697.jpg 0 train\nairplanes/image_0654.jpg 0 train\nairplanes/image_0564.jpg 0 train\nairplanes/image_0416.jpg 0 train\nairplanes/image_0607.jpg 0 train\nairplanes/image_0098.jpg 0 train\nairplanes/image_0256.jpg 0 train\nairplanes/image_0402.jpg 0 train\nairplanes/image_0788.jpg 0 train\nairplanes/image_0203.jpg 0 train\nairplanes/image_0577.jpg 0 train\nairplanes/image_0764.jpg 0 train\nairplanes/image_0338.jpg 0 train\nairplanes/image_0582.jpg 0 train\nairplanes/image_0265.jpg 0 train\nairplanes/image_0147.jpg 0 train\nairplanes/image_0536.jpg 0 train\nairplanes/image_0008.jpg 0 train\nairplanes/image_0446.jpg 0 train\nairplanes/image_0514.jpg 0 train\nairplanes/image_0617.jpg 0 train\nairplanes/image_0619.jpg 0 train\nairplanes/image_0727.jpg 0 train\nairplanes/image_0589.jpg 0 train\nairplanes/image_0207.jpg 0 train\nairplanes/image_0208.jpg 0 train\nairplanes/image_0222.jpg 0 train\nairplanes/image_0320.jpg 0 train\nairplanes/image_0781.jpg 0 train\nairplanes/image_0532.jpg 0 train\nairplanes/image_0227.jpg 0 train\nairplanes/image_0651.jpg 0 train\nairplanes/image_0286.jpg 0 train\nairplanes/image_0035.jpg 0 train\nairplanes/image_0399.jpg 0 train\nairplanes/image_0166.jpg 0 train\nairplanes/image_0711.jpg 0 train\nairplanes/image_0669.jpg 0 train\nairplanes/image_0169.jpg 0 train\nairplanes/image_0191.jpg 0 train\nairplanes/image_0128.jpg 0 train\nairplanes/image_0493.jpg 0 train\nairplanes/image_0414.jpg 0 train\nairplanes/image_0606.jpg 0 train\nairplanes/image_0734.jpg 0 train\nairplanes/image_0683.jpg 0 train\nairplanes/image_0102.jpg 0 train\nairplanes/image_0523.jpg 0 train\nairplanes/image_0434.jpg 0 train\nairplanes/image_0699.jpg 0 train\nairplanes/image_0178.jpg 0 train\nairplanes/image_0431.jpg 0 train\nairplanes/image_0243.jpg 0 train\nairplanes/image_0005.jpg 0 train\nairplanes/image_0263.jpg 0 train\nairplanes/image_0780.jpg 0 train\nairplanes/image_0173.jpg 0 train\nairplanes/image_0622.jpg 0 train\nairplanes/image_0692.jpg 0 train\nairplanes/image_0087.jpg 0 train\nairplanes/image_0417.jpg 0 train\nairplanes/image_0089.jpg 0 train\nairplanes/image_0799.jpg 0 train\nairplanes/image_0059.jpg 0 train\nairplanes/image_0673.jpg 0 train\nairplanes/image_0253.jpg 0 train\nairplanes/image_0159.jpg 0 train\nairplanes/image_0225.jpg 0 train\nairplanes/image_0445.jpg 0 train\nairplanes/image_0186.jpg 0 train\nairplanes/image_0706.jpg 0 train\nairplanes/image_0388.jpg 0 train\nairplanes/image_0694.jpg 0 train\nairplanes/image_0069.jpg 0 train\nairplanes/image_0758.jpg 0 train\nairplanes/image_0315.jpg 0 train\nairplanes/image_0295.jpg 0 train\nairplanes/image_0746.jpg 0 train\nairplanes/image_0200.jpg 0 train\nairplanes/image_0099.jpg 0 train\nairplanes/image_0738.jpg 0 train\nairplanes/image_0336.jpg 0 train\nairplanes/image_0034.jpg 0 train\nairplanes/image_0740.jpg 0 train\nairplanes/image_0712.jpg 0 train\nairplanes/image_0559.jpg 0 train\nairplanes/image_0409.jpg 0 train\nairplanes/image_0791.jpg 0 train\nairplanes/image_0635.jpg 0 train\nairplanes/image_0668.jpg 0 train\nairplanes/image_0314.jpg 0 train\nairplanes/image_0276.jpg 0 train\nairplanes/image_0121.jpg 0 train\nairplanes/image_0438.jpg 0 train\nairplanes/image_0766.jpg 0 train\nairplanes/image_0428.jpg 0 train\nairplanes/image_0568.jpg 0 train\nairplanes/image_0489.jpg 0 train\nairplanes/image_0181.jpg 0 train\nairplanes/image_0742.jpg 0 train\nairplanes/image_0280.jpg 0 train\nairplanes/image_0183.jpg 0 train\nairplanes/image_0605.jpg 0 train\nairplanes/image_0383.jpg 0 train\nairplanes/image_0175.jpg 0 train\nairplanes/image_0002.jpg 0 train\nairplanes/image_0291.jpg 0 train\nairplanes/image_0646.jpg 0 train\nairplanes/image_0165.jpg 0 train\nairplanes/image_0154.jpg 0 train\nairplanes/image_0271.jpg 0 train\nairplanes/image_0193.jpg 0 train\nairplanes/image_0048.jpg 0 train\nairplanes/image_0437.jpg 0 train\nairplanes/image_0480.jpg 0 train\nairplanes/image_0723.jpg 0 train\nairplanes/image_0053.jpg 0 train\nairplanes/image_0255.jpg 0 train\nairplanes/image_0450.jpg 0 train\nairplanes/image_0736.jpg 0 train\nairplanes/image_0616.jpg 0 train\nairplanes/image_0084.jpg 0 train\nairplanes/image_0313.jpg 0 train\nairplanes/image_0757.jpg 0 train\nairplanes/image_0020.jpg 0 train\nairplanes/image_0586.jpg 0 train\nairplanes/image_0494.jpg 0 train\nairplanes/image_0633.jpg 0 train\nairplanes/image_0251.jpg 0 train\nairplanes/image_0631.jpg 0 train\nairplanes/image_0046.jpg 0 train\nairplanes/image_0041.jpg 0 train\nairplanes/image_0027.jpg 0 train\nairplanes/image_0655.jpg 0 train\nairplanes/image_0322.jpg 0 train\nairplanes/image_0458.jpg 0 train\nairplanes/image_0014.jpg 0 train\nairplanes/image_0732.jpg 0 train\nairplanes/image_0058.jpg 0 train\nairplanes/image_0717.jpg 0 train\nairplanes/image_0408.jpg 0 train\nairplanes/image_0329.jpg 0 train\nairplanes/image_0212.jpg 0 train\nairplanes/image_0483.jpg 0 train\nairplanes/image_0611.jpg 0 train\nairplanes/image_0615.jpg 0 train\nairplanes/image_0741.jpg 0 train\nairplanes/image_0163.jpg 0 train\nairplanes/image_0202.jpg 0 train\nairplanes/image_0690.jpg 0 train\nairplanes/image_0104.jpg 0 train\nairplanes/image_0509.jpg 0 train\nairplanes/image_0081.jpg 0 train\nairplanes/image_0272.jpg 0 train\nairplanes/image_0689.jpg 0 train\nairplanes/image_0534.jpg 0 train\nairplanes/image_0400.jpg 0 train\nairplanes/image_0237.jpg 0 train\nairplanes/image_0543.jpg 0 train\nairplanes/image_0667.jpg 0 train\nairplanes/image_0308.jpg 0 train\nairplanes/image_0149.jpg 0 train\nairplanes/image_0500.jpg 0 train\nairplanes/image_0088.jpg 0 train\nairplanes/image_0309.jpg 0 train\nairplanes/image_0026.jpg 0 train\nairplanes/image_0231.jpg 0 train\nairplanes/image_0004.jpg 0 train\nairplanes/image_0015.jpg 0 train\nairplanes/image_0634.jpg 0 train\nairplanes/image_0675.jpg 0 train\nairplanes/image_0770.jpg 0 train\nairplanes/image_0355.jpg 0 train\nant/image_0037.jpg 1 train\nant/image_0006.jpg 1 train\nant/image_0016.jpg 1 train\nant/image_0030.jpg 1 train\nant/image_0012.jpg 1 train\nant/image_0025.jpg 1 train\nant/image_0007.jpg 1 train\nant/image_0019.jpg 1 train\nant/image_0031.jpg 1 train\nant/image_0011.jpg 1 train\nant/image_0017.jpg 1 train\nant/image_0018.jpg 1 train\nant/image_0013.jpg 1 train\nant/image_0024.jpg 1 train\nant/image_0003.jpg 1 train\nant/image_0009.jpg 1 train\nant/image_0042.jpg 1 train\nant/image_0040.jpg 1 train\nant/image_0033.jpg 1 train\nant/image_0001.jpg 1 train\nant/image_0021.jpg 1 train\nant/image_0008.jpg 1 train\nant/image_0035.jpg 1 train\nant/image_0005.jpg 1 train\nant/image_0034.jpg 1 train\nant/image_0002.jpg 1 train\nant/image_0020.jpg 1 train\nant/image_0041.jpg 1 train\nant/image_0027.jpg 1 train\nant/image_0014.jpg 1 train\nant/image_0026.jpg 1 train\nant/image_0004.jpg 1 train\nant/image_0015.jpg 1 train\nFaces/image_0316.jpg 2 train\nFaces/image_0258.jpg 2 train\nFaces/image_0426.jpg 2 train\nFaces/image_0302.jpg 2 train\nFaces/image_0171.jpg 2 train\nFaces/image_0062.jpg 2 train\nFaces/image_0346.jpg 2 train\nFaces/image_0435.jpg 2 train\nFaces/image_0341.jpg 2 train\nFaces/image_0340.jpg 2 train\nFaces/image_0331.jpg 2 train\nFaces/image_0037.jpg 2 train\nFaces/image_0049.jpg 2 train\nFaces/image_0337.jpg 2 train\nFaces/image_0317.jpg 2 train\nFaces/image_0357.jpg 2 train\nFaces/image_0083.jpg 2 train\nFaces/image_0101.jpg 2 train\nFaces/image_0051.jpg 2 train\nFaces/image_0006.jpg 2 train\nFaces/image_0142.jpg 2 train\nFaces/image_0250.jpg 2 train\nFaces/image_0427.jpg 2 train\nFaces/image_0396.jpg 2 train\nFaces/image_0332.jpg 2 train\nFaces/image_0335.jpg 2 train\nFaces/image_0201.jpg 2 train\nFaces/image_0016.jpg 2 train\nFaces/image_0373.jpg 2 train\nFaces/image_0176.jpg 2 train\nFaces/image_0411.jpg 2 train\nFaces/image_0155.jpg 2 train\nFaces/image_0030.jpg 2 train\nFaces/image_0091.jpg 2 train\nFaces/image_0397.jpg 2 train\nFaces/image_0012.jpg 2 train\nFaces/image_0066.jpg 2 train\nFaces/image_0283.jpg 2 train\nFaces/image_0318.jpg 2 train\nFaces/image_0224.jpg 2 train\nFaces/image_0148.jpg 2 train\nFaces/image_0077.jpg 2 train\nFaces/image_0129.jpg 2 train\nFaces/image_0339.jpg 2 train\nFaces/image_0025.jpg 2 train\nFaces/image_0328.jpg 2 train\nFaces/image_0139.jpg 2 train\nFaces/image_0425.jpg 2 train\nFaces/image_0292.jpg 2 train\nFaces/image_0348.jpg 2 train\nFaces/image_0115.jpg 2 train\nFaces/image_0123.jpg 2 train\nFaces/image_0119.jpg 2 train\nFaces/image_0060.jpg 2 train\nFaces/image_0269.jpg 2 train\nFaces/image_0052.jpg 2 train\nFaces/image_0108.jpg 2 train\nFaces/image_0239.jpg 2 train\nFaces/image_0370.jpg 2 train\nFaces/image_0326.jpg 2 train\nFaces/image_0223.jpg 2 train\nFaces/image_0172.jpg 2 train\nFaces/image_0372.jpg 2 train\nFaces/image_0344.jpg 2 train\nFaces/image_0390.jpg 2 train\nFaces/image_0007.jpg 2 train\nFaces/image_0213.jpg 2 train\nFaces/image_0312.jpg 2 train\nFaces/image_0063.jpg 2 train\nFaces/image_0019.jpg 2 train\nFaces/image_0407.jpg 2 train\nFaces/image_0192.jpg 2 train\nFaces/image_0384.jpg 2 train\nFaces/image_0195.jpg 2 train\nFaces/image_0113.jpg 2 train\nFaces/image_0127.jpg 2 train\nFaces/image_0215.jpg 2 train\nFaces/image_0389.jpg 2 train\nFaces/image_0031.jpg 2 train\nFaces/image_0349.jpg 2 train\nFaces/image_0228.jpg 2 train\nFaces/image_0380.jpg 2 train\nFaces/image_0219.jpg 2 train\nFaces/image_0353.jpg 2 train\nFaces/image_0011.jpg 2 train\nFaces/image_0233.jpg 2 train\nFaces/image_0061.jpg 2 train\nFaces/image_0090.jpg 2 train\nFaces/image_0351.jpg 2 train\nFaces/image_0261.jpg 2 train\nFaces/image_0229.jpg 2 train\nFaces/image_0126.jpg 2 train\nFaces/image_0240.jpg 2 train\nFaces/image_0305.jpg 2 train\nFaces/image_0334.jpg 2 train\nFaces/image_0354.jpg 2 train\nFaces/image_0218.jpg 2 train\nFaces/image_0366.jpg 2 train\nFaces/image_0288.jpg 2 train\nFaces/image_0216.jpg 2 train\nFaces/image_0429.jpg 2 train\nFaces/image_0277.jpg 2 train\nFaces/image_0358.jpg 2 train\nFaces/image_0050.jpg 2 train\nFaces/image_0246.jpg 2 train\nFaces/image_0361.jpg 2 train\nFaces/image_0177.jpg 2 train\nFaces/image_0356.jpg 2 train\nFaces/image_0196.jpg 2 train\nFaces/image_0293.jpg 2 train\nFaces/image_0406.jpg 2 train\nFaces/image_0347.jpg 2 train\nFaces/image_0187.jpg 2 train\nFaces/image_0234.jpg 2 train\nFaces/image_0045.jpg 2 train\nFaces/image_0303.jpg 2 train\nFaces/image_0363.jpg 2 train\nFaces/image_0017.jpg 2 train\nFaces/image_0131.jpg 2 train\nFaces/image_0296.jpg 2 train\nFaces/image_0150.jpg 2 train\nFaces/image_0018.jpg 2 train\nFaces/image_0350.jpg 2 train\nFaces/image_0209.jpg 2 train\nFaces/image_0013.jpg 2 train\nFaces/image_0054.jpg 2 train\nFaces/image_0094.jpg 2 train\nFaces/image_0082.jpg 2 train\nFaces/image_0097.jpg 2 train\nFaces/image_0130.jpg 2 train\nFaces/image_0382.jpg 2 train\nFaces/image_0141.jpg 2 train\nFaces/image_0144.jpg 2 train\nFaces/image_0109.jpg 2 train\nFaces/image_0079.jpg 2 train\nFaces/image_0306.jpg 2 train\nFaces/image_0398.jpg 2 train\nFaces/image_0055.jpg 2 train\nFaces/image_0367.jpg 2 train\nFaces/image_0365.jpg 2 train\nFaces/image_0217.jpg 2 train\nFaces/image_0343.jpg 2 train\nFaces/image_0432.jpg 2 train\nFaces/image_0071.jpg 2 train\nFaces/image_0282.jpg 2 train\nFaces/image_0393.jpg 2 train\nFaces/image_0287.jpg 2 train\nFaces/image_0198.jpg 2 train\nFaces/image_0262.jpg 2 train\nFaces/image_0168.jpg 2 train\nFaces/image_0368.jpg 2 train\nFaces/image_0420.jpg 2 train\nFaces/image_0211.jpg 2 train\nFaces/image_0270.jpg 2 train\nFaces/image_0103.jpg 2 train\nFaces/image_0024.jpg 2 train\nFaces/image_0297.jpg 2 train\nFaces/image_0188.jpg 2 train\nFaces/image_0105.jpg 2 train\nFaces/image_0074.jpg 2 train\nFaces/image_0294.jpg 2 train\nFaces/image_0391.jpg 2 train\nFaces/image_0189.jpg 2 train\nFaces/image_0003.jpg 2 train\nFaces/image_0009.jpg 2 train\nFaces/image_0387.jpg 2 train\nFaces/image_0275.jpg 2 train\nFaces/image_0226.jpg 2 train\nFaces/image_0185.jpg 2 train\nFaces/image_0118.jpg 2 train\nFaces/image_0304.jpg 2 train\nFaces/image_0278.jpg 2 train\nFaces/image_0230.jpg 2 train\nFaces/image_0404.jpg 2 train\nFaces/image_0134.jpg 2 train\nFaces/image_0249.jpg 2 train\nFaces/image_0042.jpg 2 train\nFaces/image_0352.jpg 2 train\nFaces/image_0333.jpg 2 train\nFaces/image_0221.jpg 2 train\nFaces/image_0093.jpg 2 train\nFaces/image_0266.jpg 2 train\nFaces/image_0412.jpg 2 train\nFaces/image_0301.jpg 2 train\nFaces/image_0392.jpg 2 train\nFaces/image_0254.jpg 2 train\nFaces/image_0321.jpg 2 train\nFaces/image_0386.jpg 2 train\nFaces/image_0232.jpg 2 train\nFaces/image_0324.jpg 2 train\nFaces/image_0040.jpg 2 train\nFaces/image_0067.jpg 2 train\nFaces/image_0310.jpg 2 train\nFaces/image_0330.jpg 2 train\nFaces/image_0122.jpg 2 train\nFaces/image_0205.jpg 2 train\nFaces/image_0161.jpg 2 train\nFaces/image_0146.jpg 2 train\nFaces/image_0300.jpg 2 train\nFaces/image_0369.jpg 2 train\nFaces/image_0204.jpg 2 train\nFaces/image_0125.jpg 2 train\nFaces/image_0047.jpg 2 train\nFaces/image_0285.jpg 2 train\nFaces/image_0160.jpg 2 train\nFaces/image_0422.jpg 2 train\nFaces/image_0064.jpg 2 train\nFaces/image_0070.jpg 2 train\nFaces/image_0264.jpg 2 train\nFaces/image_0112.jpg 2 train\nFaces/image_0078.jpg 2 train\nFaces/image_0068.jpg 2 train\nFaces/image_0433.jpg 2 train\nFaces/image_0085.jpg 2 train\nFaces/image_0410.jpg 2 train\nFaces/image_0110.jpg 2 train\nFaces/image_0259.jpg 2 train\nFaces/image_0143.jpg 2 train\nFaces/image_0043.jpg 2 train\nFaces/image_0162.jpg 2 train\nFaces/image_0132.jpg 2 train\nFaces/image_0157.jpg 2 train\nFaces/image_0206.jpg 2 train\nFaces/image_0376.jpg 2 train\nFaces/image_0319.jpg 2 train\nFaces/image_0371.jpg 2 train\nFaces/image_0044.jpg 2 train\nFaces/image_0199.jpg 2 train\nFaces/image_0033.jpg 2 train\nFaces/image_0136.jpg 2 train\nFaces/image_0180.jpg 2 train\nFaces/image_0260.jpg 2 train\nFaces/image_0184.jpg 2 train\nFaces/image_0073.jpg 2 train\nFaces/image_0395.jpg 2 train\nFaces/image_0257.jpg 2 train\nFaces/image_0364.jpg 2 train\nFaces/image_0001.jpg 2 train\nFaces/image_0167.jpg 2 train\nFaces/image_0284.jpg 2 train\nFaces/image_0375.jpg 2 train\nFaces/image_0080.jpg 2 train\nFaces/image_0156.jpg 2 train\nFaces/image_0021.jpg 2 train\nFaces/image_0325.jpg 2 train\nFaces/image_0267.jpg 2 train\nFaces/image_0299.jpg 2 train\nFaces/image_0421.jpg 2 train\nFaces/image_0360.jpg 2 train\nFaces/image_0095.jpg 2 train\nFaces/image_0057.jpg 2 train\nFaces/image_0403.jpg 2 train\nFaces/image_0114.jpg 2 train\nFaces/image_0153.jpg 2 train\nFaces/image_0416.jpg 2 train\nFaces/image_0098.jpg 2 train\nFaces/image_0256.jpg 2 train\nFaces/image_0402.jpg 2 train\nFaces/image_0203.jpg 2 train\nFaces/image_0338.jpg 2 train\nFaces/image_0265.jpg 2 train\nFaces/image_0147.jpg 2 train\nFaces/image_0008.jpg 2 train\nFaces/image_0207.jpg 2 train\nFaces/image_0208.jpg 2 train\nFaces/image_0222.jpg 2 train\nFaces/image_0320.jpg 2 train\nFaces/image_0227.jpg 2 train\nFaces/image_0286.jpg 2 train\nFaces/image_0035.jpg 2 train\nFaces/image_0399.jpg 2 train\nFaces/image_0166.jpg 2 train\nFaces/image_0169.jpg 2 train\nFaces/image_0191.jpg 2 train\nFaces/image_0128.jpg 2 train\nFaces/image_0414.jpg 2 train\nFaces/image_0102.jpg 2 train\nFaces/image_0434.jpg 2 train\nFaces/image_0178.jpg 2 train\nFaces/image_0431.jpg 2 train\nFaces/image_0243.jpg 2 train\nFaces/image_0005.jpg 2 train\nFaces/image_0263.jpg 2 train\nFaces/image_0173.jpg 2 train\nFaces/image_0087.jpg 2 train\nFaces/image_0417.jpg 2 train\nFaces/image_0089.jpg 2 train\nFaces/image_0059.jpg 2 train\nFaces/image_0253.jpg 2 train\nFaces/image_0159.jpg 2 train\nFaces/image_0225.jpg 2 train\nFaces/image_0186.jpg 2 train\nFaces/image_0388.jpg 2 train\nFaces/image_0069.jpg 2 train\nFaces/image_0315.jpg 2 train\nFaces/image_0295.jpg 2 train\nFaces/image_0200.jpg 2 train\nFaces/image_0099.jpg 2 train\nFaces/image_0336.jpg 2 train\nFaces/image_0034.jpg 2 train\nFaces/image_0409.jpg 2 train\nFaces/image_0314.jpg 2 train\nFaces/image_0276.jpg 2 train\nFaces/image_0121.jpg 2 train\nFaces/image_0428.jpg 2 train\nFaces/image_0181.jpg 2 train\nFaces/image_0280.jpg 2 train\nFaces/image_0183.jpg 2 train\nFaces/image_0383.jpg 2 train\nFaces/image_0175.jpg 2 train\nFaces/image_0002.jpg 2 train\nFaces/image_0291.jpg 2 train\nFaces/image_0165.jpg 2 train\nFaces/image_0154.jpg 2 train\nFaces/image_0271.jpg 2 train\nFaces/image_0193.jpg 2 train\nFaces/image_0048.jpg 2 train\nFaces/image_0053.jpg 2 train\nFaces/image_0255.jpg 2 train\nFaces/image_0084.jpg 2 train\nFaces/image_0313.jpg 2 train\nFaces/image_0020.jpg 2 train\nFaces/image_0251.jpg 2 train\nFaces/image_0046.jpg 2 train\nFaces/image_0041.jpg 2 train\nFaces/image_0027.jpg 2 train\nFaces/image_0322.jpg 2 train\nFaces/image_0014.jpg 2 train\nFaces/image_0058.jpg 2 train\nFaces/image_0408.jpg 2 train\nFaces/image_0329.jpg 2 train\nFaces/image_0212.jpg 2 train\nFaces/image_0163.jpg 2 train\nFaces/image_0202.jpg 2 train\nFaces/image_0104.jpg 2 train\nFaces/image_0081.jpg 2 train\nFaces/image_0272.jpg 2 train\nFaces/image_0400.jpg 2 train\nFaces/image_0237.jpg 2 train\nFaces/image_0308.jpg 2 train\nFaces/image_0149.jpg 2 train\nFaces/image_0088.jpg 2 train\nFaces/image_0309.jpg 2 train\nFaces/image_0026.jpg 2 train\nFaces/image_0231.jpg 2 train\nFaces/image_0004.jpg 2 train\nFaces/image_0015.jpg 2 train\nFaces/image_0355.jpg 2 train\nrooster/image_0037.jpg 3 train\nrooster/image_0049.jpg 3 train\nrooster/image_0006.jpg 3 train\nrooster/image_0016.jpg 3 train\nrooster/image_0030.jpg 3 train\nrooster/image_0012.jpg 3 train\nrooster/image_0025.jpg 3 train\nrooster/image_0007.jpg 3 train\nrooster/image_0019.jpg 3 train\nrooster/image_0031.jpg 3 train\nrooster/image_0011.jpg 3 train\nrooster/image_0045.jpg 3 train\nrooster/image_0017.jpg 3 train\nrooster/image_0018.jpg 3 train\nrooster/image_0013.jpg 3 train\nrooster/image_0024.jpg 3 train\nrooster/image_0003.jpg 3 train\nrooster/image_0009.jpg 3 train\nrooster/image_0042.jpg 3 train\nrooster/image_0040.jpg 3 train\nrooster/image_0047.jpg 3 train\nrooster/image_0043.jpg 3 train\nrooster/image_0044.jpg 3 train\nrooster/image_0033.jpg 3 train\nrooster/image_0001.jpg 3 train\nrooster/image_0021.jpg 3 train\nrooster/image_0008.jpg 3 train\nrooster/image_0035.jpg 3 train\nrooster/image_0005.jpg 3 train\nrooster/image_0034.jpg 3 train\nrooster/image_0002.jpg 3 train\nrooster/image_0048.jpg 3 train\nrooster/image_0020.jpg 3 train\nrooster/image_0046.jpg 3 train\nrooster/image_0041.jpg 3 train\nrooster/image_0027.jpg 3 train\nrooster/image_0014.jpg 3 train\nrooster/image_0026.jpg 3 train\nrooster/image_0004.jpg 3 train\nwater_lilly/image_0037.jpg 4 train\nwater_lilly/image_0006.jpg 4 train\nwater_lilly/image_0016.jpg 4 train\nwater_lilly/image_0030.jpg 4 train\nwater_lilly/image_0012.jpg 4 train\nwater_lilly/image_0025.jpg 4 train\nwater_lilly/image_0007.jpg 4 train\nwater_lilly/image_0019.jpg 4 train\nwater_lilly/image_0031.jpg 4 train\nwater_lilly/image_0011.jpg 4 train\nwater_lilly/image_0017.jpg 4 train\nwater_lilly/image_0018.jpg 4 train\nwater_lilly/image_0013.jpg 4 train\nwater_lilly/image_0024.jpg 4 train\nwater_lilly/image_0003.jpg 4 train\nwater_lilly/image_0009.jpg 4 train\nwater_lilly/image_0033.jpg 4 train\nwater_lilly/image_0001.jpg 4 train\nwater_lilly/image_0021.jpg 4 train\nwater_lilly/image_0008.jpg 4 train\nwater_lilly/image_0035.jpg 4 train\nwater_lilly/image_0005.jpg 4 train\nwater_lilly/image_0034.jpg 4 train\nwater_lilly/image_0002.jpg 4 train\nwater_lilly/image_0020.jpg 4 train\nwater_lilly/image_0027.jpg 4 train\nwater_lilly/image_0014.jpg 4 train\nwater_lilly/image_0026.jpg 4 train\nwater_lilly/image_0004.jpg 4 train\nelephant/image_0062.jpg 5 train\nelephant/image_0037.jpg 5 train\nelephant/image_0049.jpg 5 train\nelephant/image_0051.jpg 5 train\nelephant/image_0006.jpg 5 train\nelephant/image_0016.jpg 5 train\nelephant/image_0030.jpg 5 train\nelephant/image_0012.jpg 5 train\nelephant/image_0025.jpg 5 train\nelephant/image_0060.jpg 5 train\nelephant/image_0052.jpg 5 train\nelephant/image_0007.jpg 5 train\nelephant/image_0063.jpg 5 train\nelephant/image_0019.jpg 5 train\nelephant/image_0031.jpg 5 train\nelephant/image_0011.jpg 5 train\nelephant/image_0061.jpg 5 train\nelephant/image_0050.jpg 5 train\nelephant/image_0045.jpg 5 train\nelephant/image_0017.jpg 5 train\nelephant/image_0018.jpg 5 train\nelephant/image_0013.jpg 5 train\nelephant/image_0054.jpg 5 train\nelephant/image_0055.jpg 5 train\nelephant/image_0024.jpg 5 train\nelephant/image_0003.jpg 5 train\nelephant/image_0009.jpg 5 train\nelephant/image_0042.jpg 5 train\nelephant/image_0040.jpg 5 train\nelephant/image_0047.jpg 5 train\nelephant/image_0064.jpg 5 train\nelephant/image_0043.jpg 5 train\nelephant/image_0044.jpg 5 train\nelephant/image_0033.jpg 5 train\nelephant/image_0001.jpg 5 train\nelephant/image_0021.jpg 5 train\nelephant/image_0057.jpg 5 train\nelephant/image_0008.jpg 5 train\nelephant/image_0035.jpg 5 train\nelephant/image_0005.jpg 5 train\nelephant/image_0059.jpg 5 train\nelephant/image_0034.jpg 5 train\nelephant/image_0002.jpg 5 train\nelephant/image_0048.jpg 5 train\nelephant/image_0053.jpg 5 train\nelephant/image_0020.jpg 5 train\nelephant/image_0046.jpg 5 train\nelephant/image_0041.jpg 5 train\nelephant/image_0027.jpg 5 train\nelephant/image_0014.jpg 5 train\nelephant/image_0058.jpg 5 train\numbrella/image_0062.jpg 6 train\numbrella/image_0037.jpg 6 train\numbrella/image_0049.jpg 6 train\numbrella/image_0051.jpg 6 train\numbrella/image_0006.jpg 6 train\numbrella/image_0016.jpg 6 train\numbrella/image_0030.jpg 6 train\numbrella/image_0012.jpg 6 train\numbrella/image_0066.jpg 6 train\numbrella/image_0025.jpg 6 train\numbrella/image_0060.jpg 6 train\numbrella/image_0052.jpg 6 train\numbrella/image_0007.jpg 6 train\numbrella/image_0063.jpg 6 train\numbrella/image_0019.jpg 6 train\numbrella/image_0031.jpg 6 train\numbrella/image_0011.jpg 6 train\numbrella/image_0061.jpg 6 train\numbrella/image_0050.jpg 6 train\numbrella/image_0045.jpg 6 train\numbrella/image_0017.jpg 6 train\numbrella/image_0018.jpg 6 train\numbrella/image_0013.jpg 6 train\numbrella/image_0054.jpg 6 train\numbrella/image_0055.jpg 6 train\numbrella/image_0071.jpg 6 train\numbrella/image_0024.jpg 6 train\numbrella/image_0074.jpg 6 train\numbrella/image_0003.jpg 6 train\numbrella/image_0009.jpg 6 train\numbrella/image_0042.jpg 6 train\numbrella/image_0040.jpg 6 train\numbrella/image_0067.jpg 6 train\numbrella/image_0047.jpg 6 train\numbrella/image_0064.jpg 6 train\numbrella/image_0070.jpg 6 train\numbrella/image_0068.jpg 6 train\numbrella/image_0043.jpg 6 train\numbrella/image_0044.jpg 6 train\numbrella/image_0033.jpg 6 train\numbrella/image_0073.jpg 6 train\numbrella/image_0001.jpg 6 train\numbrella/image_0021.jpg 6 train\numbrella/image_0057.jpg 6 train\numbrella/image_0008.jpg 6 train\numbrella/image_0035.jpg 6 train\numbrella/image_0005.jpg 6 train\numbrella/image_0059.jpg 6 train\numbrella/image_0069.jpg 6 train\numbrella/image_0034.jpg 6 train\numbrella/image_0002.jpg 6 train\numbrella/image_0048.jpg 6 train\numbrella/image_0053.jpg 6 train\numbrella/image_0020.jpg 6 train\numbrella/image_0046.jpg 6 train\numbrella/image_0041.jpg 6 train\numbrella/image_0027.jpg 6 train\numbrella/image_0014.jpg 6 train\numbrella/image_0058.jpg 6 train\numbrella/image_0026.jpg 6 train\ndolphin/image_0062.jpg 7 train\ndolphin/image_0037.jpg 7 train\ndolphin/image_0049.jpg 7 train\ndolphin/image_0051.jpg 7 train\ndolphin/image_0006.jpg 7 train\ndolphin/image_0016.jpg 7 train\ndolphin/image_0030.jpg 7 train\ndolphin/image_0012.jpg 7 train\ndolphin/image_0025.jpg 7 train\ndolphin/image_0060.jpg 7 train\ndolphin/image_0052.jpg 7 train\ndolphin/image_0007.jpg 7 train\ndolphin/image_0063.jpg 7 train\ndolphin/image_0019.jpg 7 train\ndolphin/image_0031.jpg 7 train\ndolphin/image_0011.jpg 7 train\ndolphin/image_0061.jpg 7 train\ndolphin/image_0050.jpg 7 train\ndolphin/image_0045.jpg 7 train\ndolphin/image_0017.jpg 7 train\ndolphin/image_0018.jpg 7 train\ndolphin/image_0013.jpg 7 train\ndolphin/image_0054.jpg 7 train\ndolphin/image_0055.jpg 7 train\ndolphin/image_0024.jpg 7 train\ndolphin/image_0003.jpg 7 train\ndolphin/image_0009.jpg 7 train\ndolphin/image_0042.jpg 7 train\ndolphin/image_0040.jpg 7 train\ndolphin/image_0047.jpg 7 train\ndolphin/image_0064.jpg 7 train\ndolphin/image_0043.jpg 7 train\ndolphin/image_0044.jpg 7 train\ndolphin/image_0033.jpg 7 train\ndolphin/image_0001.jpg 7 train\ndolphin/image_0021.jpg 7 train\ndolphin/image_0057.jpg 7 train\ndolphin/image_0008.jpg 7 train\ndolphin/image_0035.jpg 7 train\ndolphin/image_0005.jpg 7 train\ndolphin/image_0059.jpg 7 train\ndolphin/image_0034.jpg 7 train\ndolphin/image_0002.jpg 7 train\ndolphin/image_0048.jpg 7 train\ndolphin/image_0053.jpg 7 train\ndolphin/image_0020.jpg 7 train\ndolphin/image_0046.jpg 7 train\ndolphin/image_0041.jpg 7 train\ndolphin/image_0027.jpg 7 train\ndolphin/image_0014.jpg 7 train\ndolphin/image_0058.jpg 7 train\ndolphin/image_0026.jpg 7 train\ngerenuk/image_0006.jpg 8 train\ngerenuk/image_0016.jpg 8 train\ngerenuk/image_0030.jpg 8 train\ngerenuk/image_0012.jpg 8 train\ngerenuk/image_0025.jpg 8 train\ngerenuk/image_0007.jpg 8 train\ngerenuk/image_0019.jpg 8 train\ngerenuk/image_0031.jpg 8 train\ngerenuk/image_0011.jpg 8 train\ngerenuk/image_0017.jpg 8 train\ngerenuk/image_0018.jpg 8 train\ngerenuk/image_0013.jpg 8 train\ngerenuk/image_0024.jpg 8 train\ngerenuk/image_0003.jpg 8 train\ngerenuk/image_0009.jpg 8 train\ngerenuk/image_0033.jpg 8 train\ngerenuk/image_0001.jpg 8 train\ngerenuk/image_0021.jpg 8 train\ngerenuk/image_0008.jpg 8 train\ngerenuk/image_0005.jpg 8 train\ngerenuk/image_0034.jpg 8 train\ngerenuk/image_0002.jpg 8 train\ngerenuk/image_0020.jpg 8 train\ngerenuk/image_0027.jpg 8 train\ngerenuk/image_0014.jpg 8 train\ngerenuk/image_0026.jpg 8 train\ngerenuk/image_0004.jpg 8 train\ndragonfly/image_0062.jpg 9 train\ndragonfly/image_0037.jpg 9 train\ndragonfly/image_0049.jpg 9 train\ndragonfly/image_0051.jpg 9 train\ndragonfly/image_0006.jpg 9 train\ndragonfly/image_0016.jpg 9 train\ndragonfly/image_0030.jpg 9 train\ndragonfly/image_0012.jpg 9 train\ndragonfly/image_0066.jpg 9 train\ndragonfly/image_0025.jpg 9 train\ndragonfly/image_0060.jpg 9 train\ndragonfly/image_0052.jpg 9 train\ndragonfly/image_0007.jpg 9 train\ndragonfly/image_0063.jpg 9 train\ndragonfly/image_0019.jpg 9 train\ndragonfly/image_0031.jpg 9 train\ndragonfly/image_0011.jpg 9 train\ndragonfly/image_0061.jpg 9 train\ndragonfly/image_0050.jpg 9 train\ndragonfly/image_0045.jpg 9 train\ndragonfly/image_0017.jpg 9 train\ndragonfly/image_0018.jpg 9 train\ndragonfly/image_0013.jpg 9 train\ndragonfly/image_0054.jpg 9 train\ndragonfly/image_0055.jpg 9 train\ndragonfly/image_0024.jpg 9 train\ndragonfly/image_0003.jpg 9 train\ndragonfly/image_0009.jpg 9 train\ndragonfly/image_0042.jpg 9 train\ndragonfly/image_0040.jpg 9 train\ndragonfly/image_0067.jpg 9 train\ndragonfly/image_0047.jpg 9 train\ndragonfly/image_0064.jpg 9 train\ndragonfly/image_0068.jpg 9 train\ndragonfly/image_0043.jpg 9 train\ndragonfly/image_0044.jpg 9 train\ndragonfly/image_0033.jpg 9 train\ndragonfly/image_0001.jpg 9 train\ndragonfly/image_0021.jpg 9 train\ndragonfly/image_0057.jpg 9 train\ndragonfly/image_0008.jpg 9 train\ndragonfly/image_0035.jpg 9 train\ndragonfly/image_0005.jpg 9 train\ndragonfly/image_0059.jpg 9 train\ndragonfly/image_0034.jpg 9 train\ndragonfly/image_0002.jpg 9 train\ndragonfly/image_0048.jpg 9 train\ndragonfly/image_0053.jpg 9 train\ndragonfly/image_0020.jpg 9 train\ndragonfly/image_0046.jpg 9 train\ndragonfly/image_0041.jpg 9 train\ndragonfly/image_0027.jpg 9 train\ndragonfly/image_0014.jpg 9 train\ndragonfly/image_0058.jpg 9 train\nyin_yang/image_0037.jpg 10 train\nyin_yang/image_0049.jpg 10 train\nyin_yang/image_0051.jpg 10 train\nyin_yang/image_0006.jpg 10 train\nyin_yang/image_0016.jpg 10 train\nyin_yang/image_0030.jpg 10 train\nyin_yang/image_0012.jpg 10 train\nyin_yang/image_0025.jpg 10 train\nyin_yang/image_0060.jpg 10 train\nyin_yang/image_0052.jpg 10 train\nyin_yang/image_0007.jpg 10 train\nyin_yang/image_0019.jpg 10 train\nyin_yang/image_0031.jpg 10 train\nyin_yang/image_0011.jpg 10 train\nyin_yang/image_0050.jpg 10 train\nyin_yang/image_0045.jpg 10 train\nyin_yang/image_0017.jpg 10 train\nyin_yang/image_0018.jpg 10 train\nyin_yang/image_0013.jpg 10 train\nyin_yang/image_0054.jpg 10 train\nyin_yang/image_0055.jpg 10 train\nyin_yang/image_0024.jpg 10 train\nyin_yang/image_0003.jpg 10 train\nyin_yang/image_0009.jpg 10 train\nyin_yang/image_0042.jpg 10 train\nyin_yang/image_0040.jpg 10 train\nyin_yang/image_0047.jpg 10 train\nyin_yang/image_0043.jpg 10 train\nyin_yang/image_0044.jpg 10 train\nyin_yang/image_0033.jpg 10 train\nyin_yang/image_0001.jpg 10 train\nyin_yang/image_0021.jpg 10 train\nyin_yang/image_0057.jpg 10 train\nyin_yang/image_0008.jpg 10 train\nyin_yang/image_0035.jpg 10 train\nyin_yang/image_0005.jpg 10 train\nyin_yang/image_0059.jpg 10 train\nyin_yang/image_0034.jpg 10 train\nyin_yang/image_0002.jpg 10 train\nyin_yang/image_0048.jpg 10 train\nyin_yang/image_0053.jpg 10 train\nyin_yang/image_0020.jpg 10 train\nyin_yang/image_0046.jpg 10 train\nyin_yang/image_0041.jpg 10 train\nyin_yang/image_0027.jpg 10 train\nyin_yang/image_0014.jpg 10 train\nyin_yang/image_0058.jpg 10 train\nyin_yang/image_0026.jpg 10 train\nstarfish/image_0062.jpg 11 train\nstarfish/image_0037.jpg 11 train\nstarfish/image_0049.jpg 11 train\nstarfish/image_0083.jpg 11 train\nstarfish/image_0051.jpg 11 train\nstarfish/image_0006.jpg 11 train\nstarfish/image_0016.jpg 11 train\nstarfish/image_0030.jpg 11 train\nstarfish/image_0012.jpg 11 train\nstarfish/image_0066.jpg 11 train\nstarfish/image_0077.jpg 11 train\nstarfish/image_0025.jpg 11 train\nstarfish/image_0060.jpg 11 train\nstarfish/image_0052.jpg 11 train\nstarfish/image_0007.jpg 11 train\nstarfish/image_0063.jpg 11 train\nstarfish/image_0019.jpg 11 train\nstarfish/image_0031.jpg 11 train\nstarfish/image_0011.jpg 11 train\nstarfish/image_0061.jpg 11 train\nstarfish/image_0050.jpg 11 train\nstarfish/image_0045.jpg 11 train\nstarfish/image_0017.jpg 11 train\nstarfish/image_0018.jpg 11 train\nstarfish/image_0013.jpg 11 train\nstarfish/image_0054.jpg 11 train\nstarfish/image_0082.jpg 11 train\nstarfish/image_0079.jpg 11 train\nstarfish/image_0055.jpg 11 train\nstarfish/image_0071.jpg 11 train\nstarfish/image_0024.jpg 11 train\nstarfish/image_0074.jpg 11 train\nstarfish/image_0003.jpg 11 train\nstarfish/image_0009.jpg 11 train\nstarfish/image_0042.jpg 11 train\nstarfish/image_0040.jpg 11 train\nstarfish/image_0067.jpg 11 train\nstarfish/image_0047.jpg 11 train\nstarfish/image_0064.jpg 11 train\nstarfish/image_0070.jpg 11 train\nstarfish/image_0078.jpg 11 train\nstarfish/image_0068.jpg 11 train\nstarfish/image_0085.jpg 11 train\nstarfish/image_0043.jpg 11 train\nstarfish/image_0044.jpg 11 train\nstarfish/image_0033.jpg 11 train\nstarfish/image_0073.jpg 11 train\nstarfish/image_0001.jpg 11 train\nstarfish/image_0080.jpg 11 train\nstarfish/image_0021.jpg 11 train\nstarfish/image_0057.jpg 11 train\nstarfish/image_0008.jpg 11 train\nstarfish/image_0035.jpg 11 train\nstarfish/image_0005.jpg 11 train\nstarfish/image_0059.jpg 11 train\nstarfish/image_0069.jpg 11 train\nstarfish/image_0034.jpg 11 train\nstarfish/image_0002.jpg 11 train\nstarfish/image_0048.jpg 11 train\nstarfish/image_0053.jpg 11 train\nstarfish/image_0084.jpg 11 train\nstarfish/image_0020.jpg 11 train\nstarfish/image_0046.jpg 11 train\nstarfish/image_0041.jpg 11 train\nstarfish/image_0027.jpg 11 train\nstarfish/image_0014.jpg 11 train\nstarfish/image_0058.jpg 11 train\nstarfish/image_0081.jpg 11 train\nceiling_fan/image_0037.jpg 12 train\nceiling_fan/image_0006.jpg 12 train\nceiling_fan/image_0016.jpg 12 train\nceiling_fan/image_0030.jpg 12 train\nceiling_fan/image_0012.jpg 12 train\nceiling_fan/image_0025.jpg 12 train\nceiling_fan/image_0007.jpg 12 train\nceiling_fan/image_0019.jpg 12 train\nceiling_fan/image_0031.jpg 12 train\nceiling_fan/image_0011.jpg 12 train\nceiling_fan/image_0045.jpg 12 train\nceiling_fan/image_0017.jpg 12 train\nceiling_fan/image_0018.jpg 12 train\nceiling_fan/image_0013.jpg 12 train\nceiling_fan/image_0024.jpg 12 train\nceiling_fan/image_0003.jpg 12 train\nceiling_fan/image_0009.jpg 12 train\nceiling_fan/image_0042.jpg 12 train\nceiling_fan/image_0040.jpg 12 train\nceiling_fan/image_0047.jpg 12 train\nceiling_fan/image_0043.jpg 12 train\nceiling_fan/image_0044.jpg 12 train\nceiling_fan/image_0033.jpg 12 train\nceiling_fan/image_0001.jpg 12 train\nceiling_fan/image_0021.jpg 12 train\nceiling_fan/image_0008.jpg 12 train\nceiling_fan/image_0035.jpg 12 train\nceiling_fan/image_0005.jpg 12 train\nceiling_fan/image_0034.jpg 12 train\nceiling_fan/image_0002.jpg 12 train\nceiling_fan/image_0020.jpg 12 train\nceiling_fan/image_0046.jpg 12 train\nceiling_fan/image_0041.jpg 12 train\nceiling_fan/image_0027.jpg 12 train\nceiling_fan/image_0014.jpg 12 train\nceiling_fan/image_0026.jpg 12 train\nceiling_fan/image_0004.jpg 12 train\nsoccer_ball/image_0062.jpg 13 train\nsoccer_ball/image_0037.jpg 13 train\nsoccer_ball/image_0049.jpg 13 train\nsoccer_ball/image_0051.jpg 13 train\nsoccer_ball/image_0006.jpg 13 train\nsoccer_ball/image_0016.jpg 13 train\nsoccer_ball/image_0030.jpg 13 train\nsoccer_ball/image_0012.jpg 13 train\nsoccer_ball/image_0025.jpg 13 train\nsoccer_ball/image_0060.jpg 13 train\nsoccer_ball/image_0052.jpg 13 train\nsoccer_ball/image_0007.jpg 13 train\nsoccer_ball/image_0063.jpg 13 train\nsoccer_ball/image_0019.jpg 13 train\nsoccer_ball/image_0031.jpg 13 train\nsoccer_ball/image_0011.jpg 13 train\nsoccer_ball/image_0061.jpg 13 train\nsoccer_ball/image_0050.jpg 13 train\nsoccer_ball/image_0045.jpg 13 train\nsoccer_ball/image_0017.jpg 13 train\nsoccer_ball/image_0018.jpg 13 train\nsoccer_ball/image_0013.jpg 13 train\nsoccer_ball/image_0054.jpg 13 train\nsoccer_ball/image_0055.jpg 13 train\nsoccer_ball/image_0024.jpg 13 train\nsoccer_ball/image_0003.jpg 13 train\nsoccer_ball/image_0009.jpg 13 train\nsoccer_ball/image_0042.jpg 13 train\nsoccer_ball/image_0040.jpg 13 train\nsoccer_ball/image_0047.jpg 13 train\nsoccer_ball/image_0064.jpg 13 train\nsoccer_ball/image_0043.jpg 13 train\nsoccer_ball/image_0044.jpg 13 train\nsoccer_ball/image_0033.jpg 13 train\nsoccer_ball/image_0001.jpg 13 train\nsoccer_ball/image_0021.jpg 13 train\nsoccer_ball/image_0057.jpg 13 train\nsoccer_ball/image_0008.jpg 13 train\nsoccer_ball/image_0035.jpg 13 train\nsoccer_ball/image_0005.jpg 13 train\nsoccer_ball/image_0059.jpg 13 train\nsoccer_ball/image_0034.jpg 13 train\nsoccer_ball/image_0002.jpg 13 train\nsoccer_ball/image_0048.jpg 13 train\nsoccer_ball/image_0053.jpg 13 train\nsoccer_ball/image_0020.jpg 13 train\nsoccer_ball/image_0046.jpg 13 train\nsoccer_ball/image_0041.jpg 13 train\nsoccer_ball/image_0027.jpg 13 train\nsoccer_ball/image_0014.jpg 13 train\nsoccer_ball/image_0058.jpg 13 train\nLeopards/image_0171.jpg 14 train\nLeopards/image_0062.jpg 14 train\nLeopards/image_0037.jpg 14 train\nLeopards/image_0049.jpg 14 train\nLeopards/image_0083.jpg 14 train\nLeopards/image_0101.jpg 14 train\nLeopards/image_0051.jpg 14 train\nLeopards/image_0006.jpg 14 train\nLeopards/image_0142.jpg 14 train\nLeopards/image_0016.jpg 14 train\nLeopards/image_0176.jpg 14 train\nLeopards/image_0155.jpg 14 train\nLeopards/image_0030.jpg 14 train\nLeopards/image_0091.jpg 14 train\nLeopards/image_0012.jpg 14 train\nLeopards/image_0066.jpg 14 train\nLeopards/image_0148.jpg 14 train\nLeopards/image_0077.jpg 14 train\nLeopards/image_0129.jpg 14 train\nLeopards/image_0025.jpg 14 train\nLeopards/image_0139.jpg 14 train\nLeopards/image_0115.jpg 14 train\nLeopards/image_0123.jpg 14 train\nLeopards/image_0119.jpg 14 train\nLeopards/image_0060.jpg 14 train\nLeopards/image_0052.jpg 14 train\nLeopards/image_0108.jpg 14 train\nLeopards/image_0172.jpg 14 train\nLeopards/image_0007.jpg 14 train\nLeopards/image_0063.jpg 14 train\nLeopards/image_0019.jpg 14 train\nLeopards/image_0192.jpg 14 train\nLeopards/image_0195.jpg 14 train\nLeopards/image_0113.jpg 14 train\nLeopards/image_0127.jpg 14 train\nLeopards/image_0031.jpg 14 train\nLeopards/image_0011.jpg 14 train\nLeopards/image_0061.jpg 14 train\nLeopards/image_0090.jpg 14 train\nLeopards/image_0126.jpg 14 train\nLeopards/image_0050.jpg 14 train\nLeopards/image_0177.jpg 14 train\nLeopards/image_0196.jpg 14 train\nLeopards/image_0187.jpg 14 train\nLeopards/image_0045.jpg 14 train\nLeopards/image_0017.jpg 14 train\nLeopards/image_0131.jpg 14 train\nLeopards/image_0150.jpg 14 train\nLeopards/image_0018.jpg 14 train\nLeopards/image_0013.jpg 14 train\nLeopards/image_0054.jpg 14 train\nLeopards/image_0094.jpg 14 train\nLeopards/image_0082.jpg 14 train\nLeopards/image_0097.jpg 14 train\nLeopards/image_0130.jpg 14 train\nLeopards/image_0141.jpg 14 train\nLeopards/image_0144.jpg 14 train\nLeopards/image_0109.jpg 14 train\nLeopards/image_0079.jpg 14 train\nLeopards/image_0055.jpg 14 train\nLeopards/image_0071.jpg 14 train\nLeopards/image_0198.jpg 14 train\nLeopards/image_0168.jpg 14 train\nLeopards/image_0103.jpg 14 train\nLeopards/image_0024.jpg 14 train\nLeopards/image_0188.jpg 14 train\nLeopards/image_0105.jpg 14 train\nLeopards/image_0074.jpg 14 train\nLeopards/image_0189.jpg 14 train\nLeopards/image_0003.jpg 14 train\nLeopards/image_0009.jpg 14 train\nLeopards/image_0185.jpg 14 train\nLeopards/image_0118.jpg 14 train\nLeopards/image_0134.jpg 14 train\nLeopards/image_0042.jpg 14 train\nLeopards/image_0093.jpg 14 train\nLeopards/image_0040.jpg 14 train\nLeopards/image_0067.jpg 14 train\nLeopards/image_0122.jpg 14 train\nLeopards/image_0161.jpg 14 train\nLeopards/image_0146.jpg 14 train\nLeopards/image_0125.jpg 14 train\nLeopards/image_0047.jpg 14 train\nLeopards/image_0160.jpg 14 train\nLeopards/image_0064.jpg 14 train\nLeopards/image_0070.jpg 14 train\nLeopards/image_0112.jpg 14 train\nLeopards/image_0078.jpg 14 train\nLeopards/image_0068.jpg 14 train\nLeopards/image_0085.jpg 14 train\nLeopards/image_0110.jpg 14 train\nLeopards/image_0143.jpg 14 train\nLeopards/image_0043.jpg 14 train\nLeopards/image_0162.jpg 14 train\nLeopards/image_0132.jpg 14 train\nLeopards/image_0157.jpg 14 train\nLeopards/image_0044.jpg 14 train\nLeopards/image_0199.jpg 14 train\nLeopards/image_0033.jpg 14 train\nLeopards/image_0136.jpg 14 train\nLeopards/image_0180.jpg 14 train\nLeopards/image_0184.jpg 14 train\nLeopards/image_0073.jpg 14 train\nLeopards/image_0001.jpg 14 train\nLeopards/image_0167.jpg 14 train\nLeopards/image_0080.jpg 14 train\nLeopards/image_0156.jpg 14 train\nLeopards/image_0021.jpg 14 train\nLeopards/image_0095.jpg 14 train\nLeopards/image_0057.jpg 14 train\nLeopards/image_0114.jpg 14 train\nLeopards/image_0153.jpg 14 train\nLeopards/image_0098.jpg 14 train\nLeopards/image_0147.jpg 14 train\nLeopards/image_0008.jpg 14 train\nLeopards/image_0035.jpg 14 train\nLeopards/image_0166.jpg 14 train\nLeopards/image_0169.jpg 14 train\nLeopards/image_0191.jpg 14 train\nLeopards/image_0128.jpg 14 train\nLeopards/image_0102.jpg 14 train\nLeopards/image_0178.jpg 14 train\nLeopards/image_0005.jpg 14 train\nLeopards/image_0173.jpg 14 train\nLeopards/image_0087.jpg 14 train\nLeopards/image_0089.jpg 14 train\nLeopards/image_0059.jpg 14 train\nLeopards/image_0159.jpg 14 train\nLeopards/image_0186.jpg 14 train\nLeopards/image_0069.jpg 14 train\nLeopards/image_0200.jpg 14 train\nLeopards/image_0099.jpg 14 train\nLeopards/image_0034.jpg 14 train\nLeopards/image_0121.jpg 14 train\nLeopards/image_0181.jpg 14 train\nLeopards/image_0183.jpg 14 train\nLeopards/image_0175.jpg 14 train\nLeopards/image_0002.jpg 14 train\nLeopards/image_0165.jpg 14 train\nLeopards/image_0154.jpg 14 train\nLeopards/image_0193.jpg 14 train\nLeopards/image_0048.jpg 14 train\nLeopards/image_0053.jpg 14 train\nLeopards/image_0084.jpg 14 train\nLeopards/image_0020.jpg 14 train\nLeopards/image_0046.jpg 14 train\nLeopards/image_0041.jpg 14 train\nLeopards/image_0027.jpg 14 train\nLeopards/image_0014.jpg 14 train\nLeopards/image_0058.jpg 14 train\nLeopards/image_0163.jpg 14 train\nLeopards/image_0104.jpg 14 train\nLeopards/image_0081.jpg 14 train\nLeopards/image_0149.jpg 14 train\nLeopards/image_0088.jpg 14 train\nLeopards/image_0026.jpg 14 train\nLeopards/image_0004.jpg 14 train\nLeopards/image_0015.jpg 14 train\nLeopards/image_0029.jpg 14 train\nLeopards/image_0120.jpg 14 train\nscorpion/image_0062.jpg 15 train\nscorpion/image_0037.jpg 15 train\nscorpion/image_0049.jpg 15 train\nscorpion/image_0083.jpg 15 train\nscorpion/image_0051.jpg 15 train\nscorpion/image_0006.jpg 15 train\nscorpion/image_0016.jpg 15 train\nscorpion/image_0030.jpg 15 train\nscorpion/image_0012.jpg 15 train\nscorpion/image_0066.jpg 15 train\nscorpion/image_0077.jpg 15 train\nscorpion/image_0025.jpg 15 train\nscorpion/image_0060.jpg 15 train\nscorpion/image_0052.jpg 15 train\nscorpion/image_0007.jpg 15 train\nscorpion/image_0063.jpg 15 train\nscorpion/image_0019.jpg 15 train\nscorpion/image_0031.jpg 15 train\nscorpion/image_0011.jpg 15 train\nscorpion/image_0061.jpg 15 train\nscorpion/image_0050.jpg 15 train\nscorpion/image_0045.jpg 15 train\nscorpion/image_0017.jpg 15 train\nscorpion/image_0018.jpg 15 train\nscorpion/image_0013.jpg 15 train\nscorpion/image_0054.jpg 15 train\nscorpion/image_0082.jpg 15 train\nscorpion/image_0079.jpg 15 train\nscorpion/image_0055.jpg 15 train\nscorpion/image_0071.jpg 15 train\nscorpion/image_0024.jpg 15 train\nscorpion/image_0074.jpg 15 train\nscorpion/image_0003.jpg 15 train\nscorpion/image_0009.jpg 15 train\nscorpion/image_0042.jpg 15 train\nscorpion/image_0040.jpg 15 train\nscorpion/image_0067.jpg 15 train\nscorpion/image_0047.jpg 15 train\nscorpion/image_0064.jpg 15 train\nscorpion/image_0070.jpg 15 train\nscorpion/image_0078.jpg 15 train\nscorpion/image_0068.jpg 15 train\nscorpion/image_0043.jpg 15 train\nscorpion/image_0044.jpg 15 train\nscorpion/image_0033.jpg 15 train\nscorpion/image_0073.jpg 15 train\nscorpion/image_0001.jpg 15 train\nscorpion/image_0080.jpg 15 train\nscorpion/image_0021.jpg 15 train\nscorpion/image_0057.jpg 15 train\nscorpion/image_0008.jpg 15 train\nscorpion/image_0035.jpg 15 train\nscorpion/image_0005.jpg 15 train\nscorpion/image_0059.jpg 15 train\nscorpion/image_0069.jpg 15 train\nscorpion/image_0034.jpg 15 train\nscorpion/image_0002.jpg 15 train\nscorpion/image_0048.jpg 15 train\nscorpion/image_0053.jpg 15 train\nscorpion/image_0084.jpg 15 train\nscorpion/image_0020.jpg 15 train\nscorpion/image_0046.jpg 15 train\nscorpion/image_0041.jpg 15 train\nscorpion/image_0027.jpg 15 train\nscorpion/image_0014.jpg 15 train\nscorpion/image_0058.jpg 15 train\nscorpion/image_0081.jpg 15 train\nllama/image_0062.jpg 16 train\nllama/image_0037.jpg 16 train\nllama/image_0049.jpg 16 train\nllama/image_0051.jpg 16 train\nllama/image_0006.jpg 16 train\nllama/image_0016.jpg 16 train\nllama/image_0030.jpg 16 train\nllama/image_0012.jpg 16 train\nllama/image_0066.jpg 16 train\nllama/image_0077.jpg 16 train\nllama/image_0025.jpg 16 train\nllama/image_0060.jpg 16 train\nllama/image_0052.jpg 16 train\nllama/image_0007.jpg 16 train\nllama/image_0063.jpg 16 train\nllama/image_0019.jpg 16 train\nllama/image_0031.jpg 16 train\nllama/image_0011.jpg 16 train\nllama/image_0061.jpg 16 train\nllama/image_0050.jpg 16 train\nllama/image_0045.jpg 16 train\nllama/image_0017.jpg 16 train\nllama/image_0018.jpg 16 train\nllama/image_0013.jpg 16 train\nllama/image_0054.jpg 16 train\nllama/image_0055.jpg 16 train\nllama/image_0071.jpg 16 train\nllama/image_0024.jpg 16 train\nllama/image_0074.jpg 16 train\nllama/image_0003.jpg 16 train\nllama/image_0009.jpg 16 train\nllama/image_0042.jpg 16 train\nllama/image_0040.jpg 16 train\nllama/image_0067.jpg 16 train\nllama/image_0047.jpg 16 train\nllama/image_0064.jpg 16 train\nllama/image_0070.jpg 16 train\nllama/image_0078.jpg 16 train\nllama/image_0068.jpg 16 train\nllama/image_0043.jpg 16 train\nllama/image_0044.jpg 16 train\nllama/image_0033.jpg 16 train\nllama/image_0073.jpg 16 train\nllama/image_0001.jpg 16 train\nllama/image_0021.jpg 16 train\nllama/image_0057.jpg 16 train\nllama/image_0008.jpg 16 train\nllama/image_0035.jpg 16 train\nllama/image_0005.jpg 16 train\nllama/image_0059.jpg 16 train\nllama/image_0069.jpg 16 train\nllama/image_0034.jpg 16 train\nllama/image_0002.jpg 16 train\nllama/image_0048.jpg 16 train\nllama/image_0053.jpg 16 train\nllama/image_0020.jpg 16 train\nllama/image_0046.jpg 16 train\nllama/image_0041.jpg 16 train\nllama/image_0027.jpg 16 train\nllama/image_0014.jpg 16 train\nllama/image_0058.jpg 16 train\nllama/image_0026.jpg 16 train\nwild_cat/image_0006.jpg 17 train\nwild_cat/image_0016.jpg 17 train\nwild_cat/image_0030.jpg 17 train\nwild_cat/image_0012.jpg 17 train\nwild_cat/image_0025.jpg 17 train\nwild_cat/image_0007.jpg 17 train\nwild_cat/image_0019.jpg 17 train\nwild_cat/image_0031.jpg 17 train\nwild_cat/image_0011.jpg 17 train\nwild_cat/image_0017.jpg 17 train\nwild_cat/image_0018.jpg 17 train\nwild_cat/image_0013.jpg 17 train\nwild_cat/image_0024.jpg 17 train\nwild_cat/image_0003.jpg 17 train\nwild_cat/image_0009.jpg 17 train\nwild_cat/image_0033.jpg 17 train\nwild_cat/image_0001.jpg 17 train\nwild_cat/image_0021.jpg 17 train\nwild_cat/image_0008.jpg 17 train\nwild_cat/image_0005.jpg 17 train\nwild_cat/image_0034.jpg 17 train\nwild_cat/image_0002.jpg 17 train\nwild_cat/image_0020.jpg 17 train\nwild_cat/image_0027.jpg 17 train\nwild_cat/image_0014.jpg 17 train\nwild_cat/image_0026.jpg 17 train\nwild_cat/image_0004.jpg 17 train\nlamp/image_0037.jpg 18 train\nlamp/image_0049.jpg 18 train\nlamp/image_0051.jpg 18 train\nlamp/image_0006.jpg 18 train\nlamp/image_0016.jpg 18 train\nlamp/image_0030.jpg 18 train\nlamp/image_0012.jpg 18 train\nlamp/image_0025.jpg 18 train\nlamp/image_0060.jpg 18 train\nlamp/image_0052.jpg 18 train\nlamp/image_0007.jpg 18 train\nlamp/image_0019.jpg 18 train\nlamp/image_0031.jpg 18 train\nlamp/image_0011.jpg 18 train\nlamp/image_0061.jpg 18 train\nlamp/image_0050.jpg 18 train\nlamp/image_0045.jpg 18 train\nlamp/image_0017.jpg 18 train\nlamp/image_0018.jpg 18 train\nlamp/image_0013.jpg 18 train\nlamp/image_0054.jpg 18 train\nlamp/image_0055.jpg 18 train\nlamp/image_0024.jpg 18 train\nlamp/image_0003.jpg 18 train\nlamp/image_0009.jpg 18 train\nlamp/image_0042.jpg 18 train\nlamp/image_0040.jpg 18 train\nlamp/image_0047.jpg 18 train\nlamp/image_0043.jpg 18 train\nlamp/image_0044.jpg 18 train\nlamp/image_0033.jpg 18 train\nlamp/image_0001.jpg 18 train\nlamp/image_0021.jpg 18 train\nlamp/image_0057.jpg 18 train\nlamp/image_0008.jpg 18 train\nlamp/image_0035.jpg 18 train\nlamp/image_0005.jpg 18 train\nlamp/image_0059.jpg 18 train\nlamp/image_0034.jpg 18 train\nlamp/image_0002.jpg 18 train\nlamp/image_0048.jpg 18 train\nlamp/image_0053.jpg 18 train\nlamp/image_0020.jpg 18 train\nlamp/image_0046.jpg 18 train\nlamp/image_0041.jpg 18 train\nlamp/image_0027.jpg 18 train\nlamp/image_0014.jpg 18 train\nlamp/image_0058.jpg 18 train\nlotus/image_0062.jpg 19 train\nlotus/image_0037.jpg 19 train\nlotus/image_0049.jpg 19 train\nlotus/image_0051.jpg 19 train\nlotus/image_0006.jpg 19 train\nlotus/image_0016.jpg 19 train\nlotus/image_0030.jpg 19 train\nlotus/image_0012.jpg 19 train\nlotus/image_0066.jpg 19 train\nlotus/image_0025.jpg 19 train\nlotus/image_0060.jpg 19 train\nlotus/image_0052.jpg 19 train\nlotus/image_0007.jpg 19 train\nlotus/image_0063.jpg 19 train\nlotus/image_0019.jpg 19 train\nlotus/image_0031.jpg 19 train\nlotus/image_0011.jpg 19 train\nlotus/image_0061.jpg 19 train\nlotus/image_0050.jpg 19 train\nlotus/image_0045.jpg 19 train\nlotus/image_0017.jpg 19 train\nlotus/image_0018.jpg 19 train\nlotus/image_0013.jpg 19 train\nlotus/image_0054.jpg 19 train\nlotus/image_0055.jpg 19 train\nlotus/image_0024.jpg 19 train\nlotus/image_0003.jpg 19 train\nlotus/image_0009.jpg 19 train\nlotus/image_0042.jpg 19 train\nlotus/image_0040.jpg 19 train\nlotus/image_0047.jpg 19 train\nlotus/image_0064.jpg 19 train\nlotus/image_0043.jpg 19 train\nlotus/image_0044.jpg 19 train\nlotus/image_0033.jpg 19 train\nlotus/image_0001.jpg 19 train\nlotus/image_0021.jpg 19 train\nlotus/image_0057.jpg 19 train\nlotus/image_0008.jpg 19 train\nlotus/image_0035.jpg 19 train\nlotus/image_0005.jpg 19 train\nlotus/image_0059.jpg 19 train\nlotus/image_0034.jpg 19 train\nlotus/image_0002.jpg 19 train\nlotus/image_0048.jpg 19 train\nlotus/image_0053.jpg 19 train\nlotus/image_0020.jpg 19 train\nlotus/image_0046.jpg 19 train\nlotus/image_0041.jpg 19 train\nlotus/image_0027.jpg 19 train\nlotus/image_0014.jpg 19 train\nlotus/image_0058.jpg 19 train\ncrocodile_head/image_0037.jpg 20 train\ncrocodile_head/image_0049.jpg 20 train\ncrocodile_head/image_0051.jpg 20 train\ncrocodile_head/image_0006.jpg 20 train\ncrocodile_head/image_0016.jpg 20 train\ncrocodile_head/image_0030.jpg 20 train\ncrocodile_head/image_0012.jpg 20 train\ncrocodile_head/image_0025.jpg 20 train\ncrocodile_head/image_0007.jpg 20 train\ncrocodile_head/image_0019.jpg 20 train\ncrocodile_head/image_0031.jpg 20 train\ncrocodile_head/image_0011.jpg 20 train\ncrocodile_head/image_0050.jpg 20 train\ncrocodile_head/image_0045.jpg 20 train\ncrocodile_head/image_0017.jpg 20 train\ncrocodile_head/image_0018.jpg 20 train\ncrocodile_head/image_0013.jpg 20 train\ncrocodile_head/image_0024.jpg 20 train\ncrocodile_head/image_0003.jpg 20 train\ncrocodile_head/image_0009.jpg 20 train\ncrocodile_head/image_0042.jpg 20 train\ncrocodile_head/image_0040.jpg 20 train\ncrocodile_head/image_0047.jpg 20 train\ncrocodile_head/image_0043.jpg 20 train\ncrocodile_head/image_0044.jpg 20 train\ncrocodile_head/image_0033.jpg 20 train\ncrocodile_head/image_0001.jpg 20 train\ncrocodile_head/image_0021.jpg 20 train\ncrocodile_head/image_0008.jpg 20 train\ncrocodile_head/image_0035.jpg 20 train\ncrocodile_head/image_0005.jpg 20 train\ncrocodile_head/image_0034.jpg 20 train\ncrocodile_head/image_0002.jpg 20 train\ncrocodile_head/image_0048.jpg 20 train\ncrocodile_head/image_0020.jpg 20 train\ncrocodile_head/image_0046.jpg 20 train\ncrocodile_head/image_0041.jpg 20 train\ncrocodile_head/image_0027.jpg 20 train\ncrocodile_head/image_0014.jpg 20 train\ncrocodile_head/image_0026.jpg 20 train\nbutterfly/image_0062.jpg 21 train\nbutterfly/image_0037.jpg 21 train\nbutterfly/image_0049.jpg 21 train\nbutterfly/image_0083.jpg 21 train\nbutterfly/image_0051.jpg 21 train\nbutterfly/image_0006.jpg 21 train\nbutterfly/image_0016.jpg 21 train\nbutterfly/image_0030.jpg 21 train\nbutterfly/image_0091.jpg 21 train\nbutterfly/image_0012.jpg 21 train\nbutterfly/image_0066.jpg 21 train\nbutterfly/image_0077.jpg 21 train\nbutterfly/image_0025.jpg 21 train\nbutterfly/image_0060.jpg 21 train\nbutterfly/image_0052.jpg 21 train\nbutterfly/image_0007.jpg 21 train\nbutterfly/image_0063.jpg 21 train\nbutterfly/image_0019.jpg 21 train\nbutterfly/image_0031.jpg 21 train\nbutterfly/image_0011.jpg 21 train\nbutterfly/image_0061.jpg 21 train\nbutterfly/image_0090.jpg 21 train\nbutterfly/image_0050.jpg 21 train\nbutterfly/image_0045.jpg 21 train\nbutterfly/image_0017.jpg 21 train\nbutterfly/image_0018.jpg 21 train\nbutterfly/image_0013.jpg 21 train\nbutterfly/image_0054.jpg 21 train\nbutterfly/image_0082.jpg 21 train\nbutterfly/image_0079.jpg 21 train\nbutterfly/image_0055.jpg 21 train\nbutterfly/image_0071.jpg 21 train\nbutterfly/image_0024.jpg 21 train\nbutterfly/image_0074.jpg 21 train\nbutterfly/image_0003.jpg 21 train\nbutterfly/image_0009.jpg 21 train\nbutterfly/image_0042.jpg 21 train\nbutterfly/image_0040.jpg 21 train\nbutterfly/image_0067.jpg 21 train\nbutterfly/image_0047.jpg 21 train\nbutterfly/image_0064.jpg 21 train\nbutterfly/image_0070.jpg 21 train\nbutterfly/image_0078.jpg 21 train\nbutterfly/image_0068.jpg 21 train\nbutterfly/image_0085.jpg 21 train\nbutterfly/image_0043.jpg 21 train\nbutterfly/image_0044.jpg 21 train\nbutterfly/image_0033.jpg 21 train\nbutterfly/image_0073.jpg 21 train\nbutterfly/image_0001.jpg 21 train\nbutterfly/image_0080.jpg 21 train\nbutterfly/image_0021.jpg 21 train\nbutterfly/image_0057.jpg 21 train\nbutterfly/image_0008.jpg 21 train\nbutterfly/image_0035.jpg 21 train\nbutterfly/image_0005.jpg 21 train\nbutterfly/image_0087.jpg 21 train\nbutterfly/image_0089.jpg 21 train\nbutterfly/image_0059.jpg 21 train\nbutterfly/image_0069.jpg 21 train\nbutterfly/image_0034.jpg 21 train\nbutterfly/image_0002.jpg 21 train\nbutterfly/image_0048.jpg 21 train\nbutterfly/image_0053.jpg 21 train\nbutterfly/image_0084.jpg 21 train\nbutterfly/image_0020.jpg 21 train\nbutterfly/image_0046.jpg 21 train\nbutterfly/image_0041.jpg 21 train\nbutterfly/image_0027.jpg 21 train\nbutterfly/image_0014.jpg 21 train\nbutterfly/image_0058.jpg 21 train\nbutterfly/image_0081.jpg 21 train\nwindsor_chair/image_0037.jpg 22 train\nwindsor_chair/image_0049.jpg 22 train\nwindsor_chair/image_0051.jpg 22 train\nwindsor_chair/image_0006.jpg 22 train\nwindsor_chair/image_0016.jpg 22 train\nwindsor_chair/image_0030.jpg 22 train\nwindsor_chair/image_0012.jpg 22 train\nwindsor_chair/image_0025.jpg 22 train\nwindsor_chair/image_0052.jpg 22 train\nwindsor_chair/image_0007.jpg 22 train\nwindsor_chair/image_0019.jpg 22 train\nwindsor_chair/image_0031.jpg 22 train\nwindsor_chair/image_0011.jpg 22 train\nwindsor_chair/image_0050.jpg 22 train\nwindsor_chair/image_0045.jpg 22 train\nwindsor_chair/image_0017.jpg 22 train\nwindsor_chair/image_0018.jpg 22 train\nwindsor_chair/image_0013.jpg 22 train\nwindsor_chair/image_0054.jpg 22 train\nwindsor_chair/image_0055.jpg 22 train\nwindsor_chair/image_0024.jpg 22 train\nwindsor_chair/image_0003.jpg 22 train\nwindsor_chair/image_0009.jpg 22 train\nwindsor_chair/image_0042.jpg 22 train\nwindsor_chair/image_0040.jpg 22 train\nwindsor_chair/image_0047.jpg 22 train\nwindsor_chair/image_0043.jpg 22 train\nwindsor_chair/image_0044.jpg 22 train\nwindsor_chair/image_0033.jpg 22 train\nwindsor_chair/image_0001.jpg 22 train\nwindsor_chair/image_0021.jpg 22 train\nwindsor_chair/image_0008.jpg 22 train\nwindsor_chair/image_0035.jpg 22 train\nwindsor_chair/image_0005.jpg 22 train\nwindsor_chair/image_0034.jpg 22 train\nwindsor_chair/image_0002.jpg 22 train\nwindsor_chair/image_0048.jpg 22 train\nwindsor_chair/image_0053.jpg 22 train\nwindsor_chair/image_0020.jpg 22 train\nwindsor_chair/image_0046.jpg 22 train\nwindsor_chair/image_0041.jpg 22 train\nwindsor_chair/image_0027.jpg 22 train\nwindsor_chair/image_0014.jpg 22 train\nwindsor_chair/image_0026.jpg 22 train\nmetronome/image_0006.jpg 23 train\nmetronome/image_0016.jpg 23 train\nmetronome/image_0030.jpg 23 train\nmetronome/image_0012.jpg 23 train\nmetronome/image_0025.jpg 23 train\nmetronome/image_0007.jpg 23 train\nmetronome/image_0019.jpg 23 train\nmetronome/image_0031.jpg 23 train\nmetronome/image_0011.jpg 23 train\nmetronome/image_0017.jpg 23 train\nmetronome/image_0018.jpg 23 train\nmetronome/image_0013.jpg 23 train\nmetronome/image_0024.jpg 23 train\nmetronome/image_0003.jpg 23 train\nmetronome/image_0009.jpg 23 train\nmetronome/image_0001.jpg 23 train\nmetronome/image_0021.jpg 23 train\nmetronome/image_0008.jpg 23 train\nmetronome/image_0005.jpg 23 train\nmetronome/image_0002.jpg 23 train\nmetronome/image_0020.jpg 23 train\nmetronome/image_0027.jpg 23 train\nmetronome/image_0014.jpg 23 train\nmetronome/image_0026.jpg 23 train\nmetronome/image_0004.jpg 23 train\nrevolver/image_0062.jpg 24 train\nrevolver/image_0037.jpg 24 train\nrevolver/image_0049.jpg 24 train\nrevolver/image_0051.jpg 24 train\nrevolver/image_0006.jpg 24 train\nrevolver/image_0016.jpg 24 train\nrevolver/image_0030.jpg 24 train\nrevolver/image_0012.jpg 24 train\nrevolver/image_0066.jpg 24 train\nrevolver/image_0077.jpg 24 train\nrevolver/image_0025.jpg 24 train\nrevolver/image_0060.jpg 24 train\nrevolver/image_0052.jpg 24 train\nrevolver/image_0007.jpg 24 train\nrevolver/image_0063.jpg 24 train\nrevolver/image_0019.jpg 24 train\nrevolver/image_0031.jpg 24 train\nrevolver/image_0011.jpg 24 train\nrevolver/image_0061.jpg 24 train\nrevolver/image_0050.jpg 24 train\nrevolver/image_0045.jpg 24 train\nrevolver/image_0017.jpg 24 train\nrevolver/image_0018.jpg 24 train\nrevolver/image_0013.jpg 24 train\nrevolver/image_0054.jpg 24 train\nrevolver/image_0082.jpg 24 train\nrevolver/image_0079.jpg 24 train\nrevolver/image_0055.jpg 24 train\nrevolver/image_0071.jpg 24 train\nrevolver/image_0024.jpg 24 train\nrevolver/image_0074.jpg 24 train\nrevolver/image_0003.jpg 24 train\nrevolver/image_0009.jpg 24 train\nrevolver/image_0042.jpg 24 train\nrevolver/image_0040.jpg 24 train\nrevolver/image_0067.jpg 24 train\nrevolver/image_0047.jpg 24 train\nrevolver/image_0064.jpg 24 train\nrevolver/image_0070.jpg 24 train\nrevolver/image_0078.jpg 24 train\nrevolver/image_0068.jpg 24 train\nrevolver/image_0043.jpg 24 train\nrevolver/image_0044.jpg 24 train\nrevolver/image_0033.jpg 24 train\nrevolver/image_0073.jpg 24 train\nrevolver/image_0001.jpg 24 train\nrevolver/image_0080.jpg 24 train\nrevolver/image_0021.jpg 24 train\nrevolver/image_0057.jpg 24 train\nrevolver/image_0008.jpg 24 train\nrevolver/image_0035.jpg 24 train\nrevolver/image_0005.jpg 24 train\nrevolver/image_0059.jpg 24 train\nrevolver/image_0069.jpg 24 train\nrevolver/image_0034.jpg 24 train\nrevolver/image_0002.jpg 24 train\nrevolver/image_0048.jpg 24 train\nrevolver/image_0053.jpg 24 train\nrevolver/image_0020.jpg 24 train\nrevolver/image_0046.jpg 24 train\nrevolver/image_0041.jpg 24 train\nrevolver/image_0027.jpg 24 train\nrevolver/image_0014.jpg 24 train\nrevolver/image_0058.jpg 24 train\nrevolver/image_0081.jpg 24 train\nschooner/image_0062.jpg 25 train\nschooner/image_0037.jpg 25 train\nschooner/image_0049.jpg 25 train\nschooner/image_0051.jpg 25 train\nschooner/image_0006.jpg 25 train\nschooner/image_0016.jpg 25 train\nschooner/image_0030.jpg 25 train\nschooner/image_0012.jpg 25 train\nschooner/image_0025.jpg 25 train\nschooner/image_0060.jpg 25 train\nschooner/image_0052.jpg 25 train\nschooner/image_0007.jpg 25 train\nschooner/image_0063.jpg 25 train\nschooner/image_0019.jpg 25 train\nschooner/image_0031.jpg 25 train\nschooner/image_0011.jpg 25 train\nschooner/image_0061.jpg 25 train\nschooner/image_0050.jpg 25 train\nschooner/image_0045.jpg 25 train\nschooner/image_0017.jpg 25 train\nschooner/image_0018.jpg 25 train\nschooner/image_0013.jpg 25 train\nschooner/image_0054.jpg 25 train\nschooner/image_0055.jpg 25 train\nschooner/image_0024.jpg 25 train\nschooner/image_0003.jpg 25 train\nschooner/image_0009.jpg 25 train\nschooner/image_0042.jpg 25 train\nschooner/image_0040.jpg 25 train\nschooner/image_0047.jpg 25 train\nschooner/image_0043.jpg 25 train\nschooner/image_0044.jpg 25 train\nschooner/image_0033.jpg 25 train\nschooner/image_0001.jpg 25 train\nschooner/image_0021.jpg 25 train\nschooner/image_0057.jpg 25 train\nschooner/image_0008.jpg 25 train\nschooner/image_0035.jpg 25 train\nschooner/image_0005.jpg 25 train\nschooner/image_0059.jpg 25 train\nschooner/image_0034.jpg 25 train\nschooner/image_0002.jpg 25 train\nschooner/image_0048.jpg 25 train\nschooner/image_0053.jpg 25 train\nschooner/image_0020.jpg 25 train\nschooner/image_0046.jpg 25 train\nschooner/image_0041.jpg 25 train\nschooner/image_0027.jpg 25 train\nschooner/image_0014.jpg 25 train\nschooner/image_0058.jpg 25 train\nminaret/image_0062.jpg 26 train\nminaret/image_0037.jpg 26 train\nminaret/image_0049.jpg 26 train\nminaret/image_0051.jpg 26 train\nminaret/image_0006.jpg 26 train\nminaret/image_0016.jpg 26 train\nminaret/image_0030.jpg 26 train\nminaret/image_0012.jpg 26 train\nminaret/image_0066.jpg 26 train\nminaret/image_0025.jpg 26 train\nminaret/image_0060.jpg 26 train\nminaret/image_0052.jpg 26 train\nminaret/image_0007.jpg 26 train\nminaret/image_0063.jpg 26 train\nminaret/image_0019.jpg 26 train\nminaret/image_0031.jpg 26 train\nminaret/image_0011.jpg 26 train\nminaret/image_0061.jpg 26 train\nminaret/image_0050.jpg 26 train\nminaret/image_0045.jpg 26 train\nminaret/image_0017.jpg 26 train\nminaret/image_0018.jpg 26 train\nminaret/image_0013.jpg 26 train\nminaret/image_0054.jpg 26 train\nminaret/image_0055.jpg 26 train\nminaret/image_0071.jpg 26 train\nminaret/image_0024.jpg 26 train\nminaret/image_0074.jpg 26 train\nminaret/image_0003.jpg 26 train\nminaret/image_0009.jpg 26 train\nminaret/image_0042.jpg 26 train\nminaret/image_0040.jpg 26 train\nminaret/image_0067.jpg 26 train\nminaret/image_0047.jpg 26 train\nminaret/image_0064.jpg 26 train\nminaret/image_0070.jpg 26 train\nminaret/image_0068.jpg 26 train\nminaret/image_0043.jpg 26 train\nminaret/image_0044.jpg 26 train\nminaret/image_0033.jpg 26 train\nminaret/image_0073.jpg 26 train\nminaret/image_0001.jpg 26 train\nminaret/image_0021.jpg 26 train\nminaret/image_0057.jpg 26 train\nminaret/image_0008.jpg 26 train\nminaret/image_0035.jpg 26 train\nminaret/image_0005.jpg 26 train\nminaret/image_0059.jpg 26 train\nminaret/image_0069.jpg 26 train\nminaret/image_0034.jpg 26 train\nminaret/image_0002.jpg 26 train\nminaret/image_0048.jpg 26 train\nminaret/image_0053.jpg 26 train\nminaret/image_0020.jpg 26 train\nminaret/image_0046.jpg 26 train\nminaret/image_0041.jpg 26 train\nminaret/image_0027.jpg 26 train\nminaret/image_0014.jpg 26 train\nminaret/image_0058.jpg 26 train\nminaret/image_0026.jpg 26 train\nstop_sign/image_0062.jpg 27 train\nstop_sign/image_0037.jpg 27 train\nstop_sign/image_0049.jpg 27 train\nstop_sign/image_0051.jpg 27 train\nstop_sign/image_0006.jpg 27 train\nstop_sign/image_0016.jpg 27 train\nstop_sign/image_0030.jpg 27 train\nstop_sign/image_0012.jpg 27 train\nstop_sign/image_0025.jpg 27 train\nstop_sign/image_0060.jpg 27 train\nstop_sign/image_0052.jpg 27 train\nstop_sign/image_0007.jpg 27 train\nstop_sign/image_0063.jpg 27 train\nstop_sign/image_0019.jpg 27 train\nstop_sign/image_0031.jpg 27 train\nstop_sign/image_0011.jpg 27 train\nstop_sign/image_0061.jpg 27 train\nstop_sign/image_0050.jpg 27 train\nstop_sign/image_0045.jpg 27 train\nstop_sign/image_0017.jpg 27 train\nstop_sign/image_0018.jpg 27 train\nstop_sign/image_0013.jpg 27 train\nstop_sign/image_0054.jpg 27 train\nstop_sign/image_0055.jpg 27 train\nstop_sign/image_0024.jpg 27 train\nstop_sign/image_0003.jpg 27 train\nstop_sign/image_0009.jpg 27 train\nstop_sign/image_0042.jpg 27 train\nstop_sign/image_0040.jpg 27 train\nstop_sign/image_0047.jpg 27 train\nstop_sign/image_0064.jpg 27 train\nstop_sign/image_0043.jpg 27 train\nstop_sign/image_0044.jpg 27 train\nstop_sign/image_0033.jpg 27 train\nstop_sign/image_0001.jpg 27 train\nstop_sign/image_0021.jpg 27 train\nstop_sign/image_0057.jpg 27 train\nstop_sign/image_0008.jpg 27 train\nstop_sign/image_0035.jpg 27 train\nstop_sign/image_0005.jpg 27 train\nstop_sign/image_0059.jpg 27 train\nstop_sign/image_0034.jpg 27 train\nstop_sign/image_0002.jpg 27 train\nstop_sign/image_0048.jpg 27 train\nstop_sign/image_0053.jpg 27 train\nstop_sign/image_0020.jpg 27 train\nstop_sign/image_0046.jpg 27 train\nstop_sign/image_0041.jpg 27 train\nstop_sign/image_0027.jpg 27 train\nstop_sign/image_0014.jpg 27 train\nstop_sign/image_0058.jpg 27 train\nbeaver/image_0037.jpg 28 train\nbeaver/image_0006.jpg 28 train\nbeaver/image_0016.jpg 28 train\nbeaver/image_0030.jpg 28 train\nbeaver/image_0012.jpg 28 train\nbeaver/image_0025.jpg 28 train\nbeaver/image_0007.jpg 28 train\nbeaver/image_0019.jpg 28 train\nbeaver/image_0031.jpg 28 train\nbeaver/image_0011.jpg 28 train\nbeaver/image_0045.jpg 28 train\nbeaver/image_0017.jpg 28 train\nbeaver/image_0018.jpg 28 train\nbeaver/image_0013.jpg 28 train\nbeaver/image_0024.jpg 28 train\nbeaver/image_0003.jpg 28 train\nbeaver/image_0009.jpg 28 train\nbeaver/image_0042.jpg 28 train\nbeaver/image_0040.jpg 28 train\nbeaver/image_0043.jpg 28 train\nbeaver/image_0044.jpg 28 train\nbeaver/image_0033.jpg 28 train\nbeaver/image_0001.jpg 28 train\nbeaver/image_0021.jpg 28 train\nbeaver/image_0008.jpg 28 train\nbeaver/image_0035.jpg 28 train\nbeaver/image_0005.jpg 28 train\nbeaver/image_0034.jpg 28 train\nbeaver/image_0002.jpg 28 train\nbeaver/image_0020.jpg 28 train\nbeaver/image_0046.jpg 28 train\nbeaver/image_0041.jpg 28 train\nbeaver/image_0027.jpg 28 train\nbeaver/image_0014.jpg 28 train\nbeaver/image_0026.jpg 28 train\nbeaver/image_0004.jpg 28 train\nlaptop/image_0062.jpg 29 train\nlaptop/image_0037.jpg 29 train\nlaptop/image_0049.jpg 29 train\nlaptop/image_0051.jpg 29 train\nlaptop/image_0006.jpg 29 train\nlaptop/image_0016.jpg 29 train\nlaptop/image_0030.jpg 29 train\nlaptop/image_0012.jpg 29 train\nlaptop/image_0066.jpg 29 train\nlaptop/image_0077.jpg 29 train\nlaptop/image_0025.jpg 29 train\nlaptop/image_0060.jpg 29 train\nlaptop/image_0052.jpg 29 train\nlaptop/image_0007.jpg 29 train\nlaptop/image_0063.jpg 29 train\nlaptop/image_0019.jpg 29 train\nlaptop/image_0031.jpg 29 train\nlaptop/image_0011.jpg 29 train\nlaptop/image_0061.jpg 29 train\nlaptop/image_0050.jpg 29 train\nlaptop/image_0045.jpg 29 train\nlaptop/image_0017.jpg 29 train\nlaptop/image_0018.jpg 29 train\nlaptop/image_0013.jpg 29 train\nlaptop/image_0054.jpg 29 train\nlaptop/image_0079.jpg 29 train\nlaptop/image_0055.jpg 29 train\nlaptop/image_0071.jpg 29 train\nlaptop/image_0024.jpg 29 train\nlaptop/image_0074.jpg 29 train\nlaptop/image_0003.jpg 29 train\nlaptop/image_0009.jpg 29 train\nlaptop/image_0042.jpg 29 train\nlaptop/image_0040.jpg 29 train\nlaptop/image_0067.jpg 29 train\nlaptop/image_0047.jpg 29 train\nlaptop/image_0064.jpg 29 train\nlaptop/image_0070.jpg 29 train\nlaptop/image_0078.jpg 29 train\nlaptop/image_0068.jpg 29 train\nlaptop/image_0043.jpg 29 train\nlaptop/image_0044.jpg 29 train\nlaptop/image_0033.jpg 29 train\nlaptop/image_0073.jpg 29 train\nlaptop/image_0001.jpg 29 train\nlaptop/image_0080.jpg 29 train\nlaptop/image_0021.jpg 29 train\nlaptop/image_0057.jpg 29 train\nlaptop/image_0008.jpg 29 train\nlaptop/image_0035.jpg 29 train\nlaptop/image_0005.jpg 29 train\nlaptop/image_0059.jpg 29 train\nlaptop/image_0069.jpg 29 train\nlaptop/image_0034.jpg 29 train\nlaptop/image_0002.jpg 29 train\nlaptop/image_0048.jpg 29 train\nlaptop/image_0053.jpg 29 train\nlaptop/image_0020.jpg 29 train\nlaptop/image_0046.jpg 29 train\nlaptop/image_0041.jpg 29 train\nlaptop/image_0027.jpg 29 train\nlaptop/image_0014.jpg 29 train\nlaptop/image_0058.jpg 29 train\nlaptop/image_0081.jpg 29 train\nketch/image_0062.jpg 30 train\nketch/image_0037.jpg 30 train\nketch/image_0049.jpg 30 train\nketch/image_0083.jpg 30 train\nketch/image_0101.jpg 30 train\nketch/image_0051.jpg 30 train\nketch/image_0006.jpg 30 train\nketch/image_0016.jpg 30 train\nketch/image_0030.jpg 30 train\nketch/image_0091.jpg 30 train\nketch/image_0012.jpg 30 train\nketch/image_0066.jpg 30 train\nketch/image_0077.jpg 30 train\nketch/image_0025.jpg 30 train\nketch/image_0060.jpg 30 train\nketch/image_0052.jpg 30 train\nketch/image_0108.jpg 30 train\nketch/image_0007.jpg 30 train\nketch/image_0063.jpg 30 train\nketch/image_0019.jpg 30 train\nketch/image_0113.jpg 30 train\nketch/image_0031.jpg 30 train\nketch/image_0011.jpg 30 train\nketch/image_0061.jpg 30 train\nketch/image_0090.jpg 30 train\nketch/image_0050.jpg 30 train\nketch/image_0045.jpg 30 train\nketch/image_0017.jpg 30 train\nketch/image_0018.jpg 30 train\nketch/image_0013.jpg 30 train\nketch/image_0054.jpg 30 train\nketch/image_0094.jpg 30 train\nketch/image_0082.jpg 30 train\nketch/image_0097.jpg 30 train\nketch/image_0109.jpg 30 train\nketch/image_0079.jpg 30 train\nketch/image_0055.jpg 30 train\nketch/image_0071.jpg 30 train\nketch/image_0103.jpg 30 train\nketch/image_0024.jpg 30 train\nketch/image_0105.jpg 30 train\nketch/image_0074.jpg 30 train\nketch/image_0003.jpg 30 train\nketch/image_0009.jpg 30 train\nketch/image_0042.jpg 30 train\nketch/image_0093.jpg 30 train\nketch/image_0040.jpg 30 train\nketch/image_0067.jpg 30 train\nketch/image_0047.jpg 30 train\nketch/image_0064.jpg 30 train\nketch/image_0070.jpg 30 train\nketch/image_0112.jpg 30 train\nketch/image_0078.jpg 30 train\nketch/image_0068.jpg 30 train\nketch/image_0085.jpg 30 train\nketch/image_0110.jpg 30 train\nketch/image_0043.jpg 30 train\nketch/image_0044.jpg 30 train\nketch/image_0033.jpg 30 train\nketch/image_0073.jpg 30 train\nketch/image_0001.jpg 30 train\nketch/image_0080.jpg 30 train\nketch/image_0021.jpg 30 train\nketch/image_0095.jpg 30 train\nketch/image_0057.jpg 30 train\nketch/image_0114.jpg 30 train\nketch/image_0098.jpg 30 train\nketch/image_0008.jpg 30 train\nketch/image_0035.jpg 30 train\nketch/image_0102.jpg 30 train\nketch/image_0005.jpg 30 train\nketch/image_0087.jpg 30 train\nketch/image_0089.jpg 30 train\nketch/image_0059.jpg 30 train\nketch/image_0069.jpg 30 train\nketch/image_0099.jpg 30 train\nketch/image_0034.jpg 30 train\nketch/image_0002.jpg 30 train\nketch/image_0048.jpg 30 train\nketch/image_0053.jpg 30 train\nketch/image_0084.jpg 30 train\nketch/image_0020.jpg 30 train\nketch/image_0046.jpg 30 train\nketch/image_0041.jpg 30 train\nketch/image_0027.jpg 30 train\nketch/image_0014.jpg 30 train\nketch/image_0058.jpg 30 train\nketch/image_0104.jpg 30 train\nketch/image_0081.jpg 30 train\nketch/image_0088.jpg 30 train\nketch/image_0026.jpg 30 train\ngramophone/image_0037.jpg 31 train\ngramophone/image_0049.jpg 31 train\ngramophone/image_0051.jpg 31 train\ngramophone/image_0006.jpg 31 train\ngramophone/image_0016.jpg 31 train\ngramophone/image_0030.jpg 31 train\ngramophone/image_0012.jpg 31 train\ngramophone/image_0025.jpg 31 train\ngramophone/image_0007.jpg 31 train\ngramophone/image_0019.jpg 31 train\ngramophone/image_0031.jpg 31 train\ngramophone/image_0011.jpg 31 train\ngramophone/image_0050.jpg 31 train\ngramophone/image_0045.jpg 31 train\ngramophone/image_0017.jpg 31 train\ngramophone/image_0018.jpg 31 train\ngramophone/image_0013.jpg 31 train\ngramophone/image_0024.jpg 31 train\ngramophone/image_0003.jpg 31 train\ngramophone/image_0009.jpg 31 train\ngramophone/image_0042.jpg 31 train\ngramophone/image_0040.jpg 31 train\ngramophone/image_0047.jpg 31 train\ngramophone/image_0043.jpg 31 train\ngramophone/image_0044.jpg 31 train\ngramophone/image_0033.jpg 31 train\ngramophone/image_0001.jpg 31 train\ngramophone/image_0021.jpg 31 train\ngramophone/image_0008.jpg 31 train\ngramophone/image_0035.jpg 31 train\ngramophone/image_0005.jpg 31 train\ngramophone/image_0034.jpg 31 train\ngramophone/image_0002.jpg 31 train\ngramophone/image_0048.jpg 31 train\ngramophone/image_0020.jpg 31 train\ngramophone/image_0046.jpg 31 train\ngramophone/image_0041.jpg 31 train\ngramophone/image_0027.jpg 31 train\ngramophone/image_0014.jpg 31 train\ngramophone/image_0026.jpg 31 train\nmenorah/image_0062.jpg 32 train\nmenorah/image_0037.jpg 32 train\nmenorah/image_0049.jpg 32 train\nmenorah/image_0083.jpg 32 train\nmenorah/image_0051.jpg 32 train\nmenorah/image_0006.jpg 32 train\nmenorah/image_0016.jpg 32 train\nmenorah/image_0030.jpg 32 train\nmenorah/image_0012.jpg 32 train\nmenorah/image_0066.jpg 32 train\nmenorah/image_0077.jpg 32 train\nmenorah/image_0025.jpg 32 train\nmenorah/image_0060.jpg 32 train\nmenorah/image_0052.jpg 32 train\nmenorah/image_0007.jpg 32 train\nmenorah/image_0063.jpg 32 train\nmenorah/image_0019.jpg 32 train\nmenorah/image_0031.jpg 32 train\nmenorah/image_0011.jpg 32 train\nmenorah/image_0061.jpg 32 train\nmenorah/image_0050.jpg 32 train\nmenorah/image_0045.jpg 32 train\nmenorah/image_0017.jpg 32 train\nmenorah/image_0018.jpg 32 train\nmenorah/image_0013.jpg 32 train\nmenorah/image_0054.jpg 32 train\nmenorah/image_0082.jpg 32 train\nmenorah/image_0079.jpg 32 train\nmenorah/image_0055.jpg 32 train\nmenorah/image_0071.jpg 32 train\nmenorah/image_0024.jpg 32 train\nmenorah/image_0074.jpg 32 train\nmenorah/image_0003.jpg 32 train\nmenorah/image_0009.jpg 32 train\nmenorah/image_0042.jpg 32 train\nmenorah/image_0040.jpg 32 train\nmenorah/image_0067.jpg 32 train\nmenorah/image_0047.jpg 32 train\nmenorah/image_0064.jpg 32 train\nmenorah/image_0070.jpg 32 train\nmenorah/image_0078.jpg 32 train\nmenorah/image_0068.jpg 32 train\nmenorah/image_0085.jpg 32 train\nmenorah/image_0043.jpg 32 train\nmenorah/image_0044.jpg 32 train\nmenorah/image_0033.jpg 32 train\nmenorah/image_0073.jpg 32 train\nmenorah/image_0001.jpg 32 train\nmenorah/image_0080.jpg 32 train\nmenorah/image_0021.jpg 32 train\nmenorah/image_0057.jpg 32 train\nmenorah/image_0008.jpg 32 train\nmenorah/image_0035.jpg 32 train\nmenorah/image_0005.jpg 32 train\nmenorah/image_0087.jpg 32 train\nmenorah/image_0059.jpg 32 train\nmenorah/image_0069.jpg 32 train\nmenorah/image_0034.jpg 32 train\nmenorah/image_0002.jpg 32 train\nmenorah/image_0048.jpg 32 train\nmenorah/image_0053.jpg 32 train\nmenorah/image_0084.jpg 32 train\nmenorah/image_0020.jpg 32 train\nmenorah/image_0046.jpg 32 train\nmenorah/image_0041.jpg 32 train\nmenorah/image_0027.jpg 32 train\nmenorah/image_0014.jpg 32 train\nmenorah/image_0058.jpg 32 train\nmenorah/image_0081.jpg 32 train\neuphonium/image_0062.jpg 33 train\neuphonium/image_0037.jpg 33 train\neuphonium/image_0049.jpg 33 train\neuphonium/image_0051.jpg 33 train\neuphonium/image_0006.jpg 33 train\neuphonium/image_0016.jpg 33 train\neuphonium/image_0030.jpg 33 train\neuphonium/image_0012.jpg 33 train\neuphonium/image_0025.jpg 33 train\neuphonium/image_0060.jpg 33 train\neuphonium/image_0052.jpg 33 train\neuphonium/image_0007.jpg 33 train\neuphonium/image_0063.jpg 33 train\neuphonium/image_0019.jpg 33 train\neuphonium/image_0031.jpg 33 train\neuphonium/image_0011.jpg 33 train\neuphonium/image_0061.jpg 33 train\neuphonium/image_0050.jpg 33 train\neuphonium/image_0045.jpg 33 train\neuphonium/image_0017.jpg 33 train\neuphonium/image_0018.jpg 33 train\neuphonium/image_0013.jpg 33 train\neuphonium/image_0054.jpg 33 train\neuphonium/image_0055.jpg 33 train\neuphonium/image_0024.jpg 33 train\neuphonium/image_0003.jpg 33 train\neuphonium/image_0009.jpg 33 train\neuphonium/image_0042.jpg 33 train\neuphonium/image_0040.jpg 33 train\neuphonium/image_0047.jpg 33 train\neuphonium/image_0064.jpg 33 train\neuphonium/image_0043.jpg 33 train\neuphonium/image_0044.jpg 33 train\neuphonium/image_0033.jpg 33 train\neuphonium/image_0001.jpg 33 train\neuphonium/image_0021.jpg 33 train\neuphonium/image_0057.jpg 33 train\neuphonium/image_0008.jpg 33 train\neuphonium/image_0035.jpg 33 train\neuphonium/image_0005.jpg 33 train\neuphonium/image_0059.jpg 33 train\neuphonium/image_0034.jpg 33 train\neuphonium/image_0002.jpg 33 train\neuphonium/image_0048.jpg 33 train\neuphonium/image_0053.jpg 33 train\neuphonium/image_0020.jpg 33 train\neuphonium/image_0046.jpg 33 train\neuphonium/image_0041.jpg 33 train\neuphonium/image_0027.jpg 33 train\neuphonium/image_0014.jpg 33 train\neuphonium/image_0058.jpg 33 train\nrhino/image_0037.jpg 34 train\nrhino/image_0049.jpg 34 train\nrhino/image_0051.jpg 34 train\nrhino/image_0006.jpg 34 train\nrhino/image_0016.jpg 34 train\nrhino/image_0030.jpg 34 train\nrhino/image_0012.jpg 34 train\nrhino/image_0025.jpg 34 train\nrhino/image_0052.jpg 34 train\nrhino/image_0007.jpg 34 train\nrhino/image_0019.jpg 34 train\nrhino/image_0031.jpg 34 train\nrhino/image_0011.jpg 34 train\nrhino/image_0050.jpg 34 train\nrhino/image_0045.jpg 34 train\nrhino/image_0017.jpg 34 train\nrhino/image_0018.jpg 34 train\nrhino/image_0013.jpg 34 train\nrhino/image_0054.jpg 34 train\nrhino/image_0055.jpg 34 train\nrhino/image_0024.jpg 34 train\nrhino/image_0003.jpg 34 train\nrhino/image_0009.jpg 34 train\nrhino/image_0042.jpg 34 train\nrhino/image_0040.jpg 34 train\nrhino/image_0047.jpg 34 train\nrhino/image_0043.jpg 34 train\nrhino/image_0044.jpg 34 train\nrhino/image_0033.jpg 34 train\nrhino/image_0001.jpg 34 train\nrhino/image_0021.jpg 34 train\nrhino/image_0057.jpg 34 train\nrhino/image_0008.jpg 34 train\nrhino/image_0035.jpg 34 train\nrhino/image_0005.jpg 34 train\nrhino/image_0059.jpg 34 train\nrhino/image_0034.jpg 34 train\nrhino/image_0002.jpg 34 train\nrhino/image_0048.jpg 34 train\nrhino/image_0053.jpg 34 train\nrhino/image_0020.jpg 34 train\nrhino/image_0046.jpg 34 train\nrhino/image_0041.jpg 34 train\nrhino/image_0027.jpg 34 train\nrhino/image_0014.jpg 34 train\nrhino/image_0058.jpg 34 train\nrhino/image_0026.jpg 34 train\nbonsai/image_0062.jpg 35 train\nbonsai/image_0037.jpg 35 train\nbonsai/image_0049.jpg 35 train\nbonsai/image_0083.jpg 35 train\nbonsai/image_0101.jpg 35 train\nbonsai/image_0051.jpg 35 train\nbonsai/image_0006.jpg 35 train\nbonsai/image_0016.jpg 35 train\nbonsai/image_0030.jpg 35 train\nbonsai/image_0091.jpg 35 train\nbonsai/image_0012.jpg 35 train\nbonsai/image_0066.jpg 35 train\nbonsai/image_0077.jpg 35 train\nbonsai/image_0025.jpg 35 train\nbonsai/image_0115.jpg 35 train\nbonsai/image_0123.jpg 35 train\nbonsai/image_0119.jpg 35 train\nbonsai/image_0060.jpg 35 train\nbonsai/image_0052.jpg 35 train\nbonsai/image_0108.jpg 35 train\nbonsai/image_0007.jpg 35 train\nbonsai/image_0063.jpg 35 train\nbonsai/image_0019.jpg 35 train\nbonsai/image_0113.jpg 35 train\nbonsai/image_0127.jpg 35 train\nbonsai/image_0031.jpg 35 train\nbonsai/image_0011.jpg 35 train\nbonsai/image_0061.jpg 35 train\nbonsai/image_0090.jpg 35 train\nbonsai/image_0126.jpg 35 train\nbonsai/image_0050.jpg 35 train\nbonsai/image_0045.jpg 35 train\nbonsai/image_0017.jpg 35 train\nbonsai/image_0018.jpg 35 train\nbonsai/image_0013.jpg 35 train\nbonsai/image_0054.jpg 35 train\nbonsai/image_0094.jpg 35 train\nbonsai/image_0082.jpg 35 train\nbonsai/image_0097.jpg 35 train\nbonsai/image_0109.jpg 35 train\nbonsai/image_0079.jpg 35 train\nbonsai/image_0055.jpg 35 train\nbonsai/image_0071.jpg 35 train\nbonsai/image_0103.jpg 35 train\nbonsai/image_0024.jpg 35 train\nbonsai/image_0105.jpg 35 train\nbonsai/image_0074.jpg 35 train\nbonsai/image_0003.jpg 35 train\nbonsai/image_0009.jpg 35 train\nbonsai/image_0118.jpg 35 train\nbonsai/image_0042.jpg 35 train\nbonsai/image_0093.jpg 35 train\nbonsai/image_0040.jpg 35 train\nbonsai/image_0067.jpg 35 train\nbonsai/image_0122.jpg 35 train\nbonsai/image_0125.jpg 35 train\nbonsai/image_0047.jpg 35 train\nbonsai/image_0064.jpg 35 train\nbonsai/image_0070.jpg 35 train\nbonsai/image_0112.jpg 35 train\nbonsai/image_0078.jpg 35 train\nbonsai/image_0068.jpg 35 train\nbonsai/image_0085.jpg 35 train\nbonsai/image_0110.jpg 35 train\nbonsai/image_0043.jpg 35 train\nbonsai/image_0044.jpg 35 train\nbonsai/image_0033.jpg 35 train\nbonsai/image_0073.jpg 35 train\nbonsai/image_0001.jpg 35 train\nbonsai/image_0080.jpg 35 train\nbonsai/image_0021.jpg 35 train\nbonsai/image_0095.jpg 35 train\nbonsai/image_0057.jpg 35 train\nbonsai/image_0114.jpg 35 train\nbonsai/image_0098.jpg 35 train\nbonsai/image_0008.jpg 35 train\nbonsai/image_0035.jpg 35 train\nbonsai/image_0128.jpg 35 train\nbonsai/image_0102.jpg 35 train\nbonsai/image_0005.jpg 35 train\nbonsai/image_0087.jpg 35 train\nbonsai/image_0089.jpg 35 train\nbonsai/image_0059.jpg 35 train\nbonsai/image_0069.jpg 35 train\nbonsai/image_0099.jpg 35 train\nbonsai/image_0034.jpg 35 train\nbonsai/image_0121.jpg 35 train\nbonsai/image_0002.jpg 35 train\nbonsai/image_0048.jpg 35 train\nbonsai/image_0053.jpg 35 train\nbonsai/image_0084.jpg 35 train\nbonsai/image_0020.jpg 35 train\nbonsai/image_0046.jpg 35 train\nbonsai/image_0041.jpg 35 train\nbonsai/image_0027.jpg 35 train\nbonsai/image_0014.jpg 35 train\nbonsai/image_0058.jpg 35 train\nbonsai/image_0104.jpg 35 train\nbonsai/image_0081.jpg 35 train\nbonsai/image_0088.jpg 35 train\nbonsai/image_0026.jpg 35 train\nbonsai/image_0004.jpg 35 train\nchair/image_0062.jpg 36 train\nchair/image_0037.jpg 36 train\nchair/image_0049.jpg 36 train\nchair/image_0051.jpg 36 train\nchair/image_0006.jpg 36 train\nchair/image_0016.jpg 36 train\nchair/image_0030.jpg 36 train\nchair/image_0012.jpg 36 train\nchair/image_0025.jpg 36 train\nchair/image_0060.jpg 36 train\nchair/image_0052.jpg 36 train\nchair/image_0007.jpg 36 train\nchair/image_0019.jpg 36 train\nchair/image_0031.jpg 36 train\nchair/image_0011.jpg 36 train\nchair/image_0061.jpg 36 train\nchair/image_0050.jpg 36 train\nchair/image_0045.jpg 36 train\nchair/image_0017.jpg 36 train\nchair/image_0018.jpg 36 train\nchair/image_0013.jpg 36 train\nchair/image_0054.jpg 36 train\nchair/image_0055.jpg 36 train\nchair/image_0024.jpg 36 train\nchair/image_0003.jpg 36 train\nchair/image_0009.jpg 36 train\nchair/image_0042.jpg 36 train\nchair/image_0040.jpg 36 train\nchair/image_0047.jpg 36 train\nchair/image_0043.jpg 36 train\nchair/image_0044.jpg 36 train\nchair/image_0033.jpg 36 train\nchair/image_0001.jpg 36 train\nchair/image_0021.jpg 36 train\nchair/image_0057.jpg 36 train\nchair/image_0008.jpg 36 train\nchair/image_0035.jpg 36 train\nchair/image_0005.jpg 36 train\nchair/image_0059.jpg 36 train\nchair/image_0034.jpg 36 train\nchair/image_0002.jpg 36 train\nchair/image_0048.jpg 36 train\nchair/image_0053.jpg 36 train\nchair/image_0020.jpg 36 train\nchair/image_0046.jpg 36 train\nchair/image_0041.jpg 36 train\nchair/image_0027.jpg 36 train\nchair/image_0014.jpg 36 train\nchair/image_0058.jpg 36 train\nsnoopy/image_0006.jpg 37 train\nsnoopy/image_0016.jpg 37 train\nsnoopy/image_0030.jpg 37 train\nsnoopy/image_0012.jpg 37 train\nsnoopy/image_0025.jpg 37 train\nsnoopy/image_0007.jpg 37 train\nsnoopy/image_0019.jpg 37 train\nsnoopy/image_0031.jpg 37 train\nsnoopy/image_0011.jpg 37 train\nsnoopy/image_0017.jpg 37 train\nsnoopy/image_0018.jpg 37 train\nsnoopy/image_0013.jpg 37 train\nsnoopy/image_0024.jpg 37 train\nsnoopy/image_0003.jpg 37 train\nsnoopy/image_0009.jpg 37 train\nsnoopy/image_0033.jpg 37 train\nsnoopy/image_0001.jpg 37 train\nsnoopy/image_0021.jpg 37 train\nsnoopy/image_0008.jpg 37 train\nsnoopy/image_0035.jpg 37 train\nsnoopy/image_0005.jpg 37 train\nsnoopy/image_0034.jpg 37 train\nsnoopy/image_0002.jpg 37 train\nsnoopy/image_0020.jpg 37 train\nsnoopy/image_0027.jpg 37 train\nsnoopy/image_0014.jpg 37 train\nsnoopy/image_0026.jpg 37 train\nsnoopy/image_0004.jpg 37 train\nbuddha/image_0062.jpg 38 train\nbuddha/image_0037.jpg 38 train\nbuddha/image_0049.jpg 38 train\nbuddha/image_0083.jpg 38 train\nbuddha/image_0051.jpg 38 train\nbuddha/image_0006.jpg 38 train\nbuddha/image_0016.jpg 38 train\nbuddha/image_0030.jpg 38 train\nbuddha/image_0012.jpg 38 train\nbuddha/image_0066.jpg 38 train\nbuddha/image_0077.jpg 38 train\nbuddha/image_0025.jpg 38 train\nbuddha/image_0060.jpg 38 train\nbuddha/image_0052.jpg 38 train\nbuddha/image_0007.jpg 38 train\nbuddha/image_0063.jpg 38 train\nbuddha/image_0019.jpg 38 train\nbuddha/image_0031.jpg 38 train\nbuddha/image_0011.jpg 38 train\nbuddha/image_0061.jpg 38 train\nbuddha/image_0050.jpg 38 train\nbuddha/image_0045.jpg 38 train\nbuddha/image_0017.jpg 38 train\nbuddha/image_0018.jpg 38 train\nbuddha/image_0013.jpg 38 train\nbuddha/image_0054.jpg 38 train\nbuddha/image_0082.jpg 38 train\nbuddha/image_0079.jpg 38 train\nbuddha/image_0055.jpg 38 train\nbuddha/image_0071.jpg 38 train\nbuddha/image_0024.jpg 38 train\nbuddha/image_0074.jpg 38 train\nbuddha/image_0003.jpg 38 train\nbuddha/image_0009.jpg 38 train\nbuddha/image_0042.jpg 38 train\nbuddha/image_0040.jpg 38 train\nbuddha/image_0067.jpg 38 train\nbuddha/image_0047.jpg 38 train\nbuddha/image_0064.jpg 38 train\nbuddha/image_0070.jpg 38 train\nbuddha/image_0078.jpg 38 train\nbuddha/image_0068.jpg 38 train\nbuddha/image_0085.jpg 38 train\nbuddha/image_0043.jpg 38 train\nbuddha/image_0044.jpg 38 train\nbuddha/image_0033.jpg 38 train\nbuddha/image_0073.jpg 38 train\nbuddha/image_0001.jpg 38 train\nbuddha/image_0080.jpg 38 train\nbuddha/image_0021.jpg 38 train\nbuddha/image_0057.jpg 38 train\nbuddha/image_0008.jpg 38 train\nbuddha/image_0035.jpg 38 train\nbuddha/image_0005.jpg 38 train\nbuddha/image_0059.jpg 38 train\nbuddha/image_0069.jpg 38 train\nbuddha/image_0034.jpg 38 train\nbuddha/image_0002.jpg 38 train\nbuddha/image_0048.jpg 38 train\nbuddha/image_0053.jpg 38 train\nbuddha/image_0084.jpg 38 train\nbuddha/image_0020.jpg 38 train\nbuddha/image_0046.jpg 38 train\nbuddha/image_0041.jpg 38 train\nbuddha/image_0027.jpg 38 train\nbuddha/image_0014.jpg 38 train\nbuddha/image_0058.jpg 38 train\nbuddha/image_0081.jpg 38 train\nscissors/image_0037.jpg 39 train\nscissors/image_0006.jpg 39 train\nscissors/image_0016.jpg 39 train\nscissors/image_0030.jpg 39 train\nscissors/image_0012.jpg 39 train\nscissors/image_0025.jpg 39 train\nscissors/image_0007.jpg 39 train\nscissors/image_0019.jpg 39 train\nscissors/image_0031.jpg 39 train\nscissors/image_0011.jpg 39 train\nscissors/image_0017.jpg 39 train\nscissors/image_0018.jpg 39 train\nscissors/image_0013.jpg 39 train\nscissors/image_0024.jpg 39 train\nscissors/image_0003.jpg 39 train\nscissors/image_0009.jpg 39 train\nscissors/image_0033.jpg 39 train\nscissors/image_0001.jpg 39 train\nscissors/image_0021.jpg 39 train\nscissors/image_0008.jpg 39 train\nscissors/image_0035.jpg 39 train\nscissors/image_0005.jpg 39 train\nscissors/image_0034.jpg 39 train\nscissors/image_0002.jpg 39 train\nscissors/image_0020.jpg 39 train\nscissors/image_0027.jpg 39 train\nscissors/image_0014.jpg 39 train\nscissors/image_0026.jpg 39 train\nscissors/image_0004.jpg 39 train\nscissors/image_0015.jpg 39 train\nscissors/image_0029.jpg 39 train\ncougar_body/image_0037.jpg 40 train\ncougar_body/image_0006.jpg 40 train\ncougar_body/image_0016.jpg 40 train\ncougar_body/image_0030.jpg 40 train\ncougar_body/image_0012.jpg 40 train\ncougar_body/image_0025.jpg 40 train\ncougar_body/image_0007.jpg 40 train\ncougar_body/image_0019.jpg 40 train\ncougar_body/image_0031.jpg 40 train\ncougar_body/image_0011.jpg 40 train\ncougar_body/image_0045.jpg 40 train\ncougar_body/image_0017.jpg 40 train\ncougar_body/image_0018.jpg 40 train\ncougar_body/image_0013.jpg 40 train\ncougar_body/image_0024.jpg 40 train\ncougar_body/image_0003.jpg 40 train\ncougar_body/image_0009.jpg 40 train\ncougar_body/image_0042.jpg 40 train\ncougar_body/image_0040.jpg 40 train\ncougar_body/image_0047.jpg 40 train\ncougar_body/image_0043.jpg 40 train\ncougar_body/image_0044.jpg 40 train\ncougar_body/image_0033.jpg 40 train\ncougar_body/image_0001.jpg 40 train\ncougar_body/image_0021.jpg 40 train\ncougar_body/image_0008.jpg 40 train\ncougar_body/image_0035.jpg 40 train\ncougar_body/image_0005.jpg 40 train\ncougar_body/image_0034.jpg 40 train\ncougar_body/image_0002.jpg 40 train\ncougar_body/image_0020.jpg 40 train\ncougar_body/image_0046.jpg 40 train\ncougar_body/image_0041.jpg 40 train\ncougar_body/image_0027.jpg 40 train\ncougar_body/image_0014.jpg 40 train\ncougar_body/image_0026.jpg 40 train\ncougar_body/image_0004.jpg 40 train\nbinocular/image_0006.jpg 41 train\nbinocular/image_0016.jpg 41 train\nbinocular/image_0030.jpg 41 train\nbinocular/image_0012.jpg 41 train\nbinocular/image_0025.jpg 41 train\nbinocular/image_0007.jpg 41 train\nbinocular/image_0019.jpg 41 train\nbinocular/image_0031.jpg 41 train\nbinocular/image_0011.jpg 41 train\nbinocular/image_0017.jpg 41 train\nbinocular/image_0018.jpg 41 train\nbinocular/image_0013.jpg 41 train\nbinocular/image_0024.jpg 41 train\nbinocular/image_0003.jpg 41 train\nbinocular/image_0009.jpg 41 train\nbinocular/image_0033.jpg 41 train\nbinocular/image_0001.jpg 41 train\nbinocular/image_0021.jpg 41 train\nbinocular/image_0008.jpg 41 train\nbinocular/image_0005.jpg 41 train\nbinocular/image_0002.jpg 41 train\nbinocular/image_0020.jpg 41 train\nbinocular/image_0027.jpg 41 train\nbinocular/image_0014.jpg 41 train\nbinocular/image_0026.jpg 41 train\nbinocular/image_0004.jpg 41 train\ngrand_piano/image_0062.jpg 42 train\ngrand_piano/image_0037.jpg 42 train\ngrand_piano/image_0049.jpg 42 train\ngrand_piano/image_0083.jpg 42 train\ngrand_piano/image_0051.jpg 42 train\ngrand_piano/image_0006.jpg 42 train\ngrand_piano/image_0016.jpg 42 train\ngrand_piano/image_0030.jpg 42 train\ngrand_piano/image_0091.jpg 42 train\ngrand_piano/image_0012.jpg 42 train\ngrand_piano/image_0066.jpg 42 train\ngrand_piano/image_0077.jpg 42 train\ngrand_piano/image_0025.jpg 42 train\ngrand_piano/image_0060.jpg 42 train\ngrand_piano/image_0052.jpg 42 train\ngrand_piano/image_0007.jpg 42 train\ngrand_piano/image_0063.jpg 42 train\ngrand_piano/image_0019.jpg 42 train\ngrand_piano/image_0031.jpg 42 train\ngrand_piano/image_0011.jpg 42 train\ngrand_piano/image_0061.jpg 42 train\ngrand_piano/image_0090.jpg 42 train\ngrand_piano/image_0050.jpg 42 train\ngrand_piano/image_0045.jpg 42 train\ngrand_piano/image_0017.jpg 42 train\ngrand_piano/image_0018.jpg 42 train\ngrand_piano/image_0013.jpg 42 train\ngrand_piano/image_0054.jpg 42 train\ngrand_piano/image_0094.jpg 42 train\ngrand_piano/image_0082.jpg 42 train\ngrand_piano/image_0097.jpg 42 train\ngrand_piano/image_0079.jpg 42 train\ngrand_piano/image_0055.jpg 42 train\ngrand_piano/image_0071.jpg 42 train\ngrand_piano/image_0024.jpg 42 train\ngrand_piano/image_0074.jpg 42 train\ngrand_piano/image_0003.jpg 42 train\ngrand_piano/image_0009.jpg 42 train\ngrand_piano/image_0042.jpg 42 train\ngrand_piano/image_0093.jpg 42 train\ngrand_piano/image_0040.jpg 42 train\ngrand_piano/image_0067.jpg 42 train\ngrand_piano/image_0047.jpg 42 train\ngrand_piano/image_0064.jpg 42 train\ngrand_piano/image_0070.jpg 42 train\ngrand_piano/image_0078.jpg 42 train\ngrand_piano/image_0068.jpg 42 train\ngrand_piano/image_0085.jpg 42 train\ngrand_piano/image_0043.jpg 42 train\ngrand_piano/image_0044.jpg 42 train\ngrand_piano/image_0033.jpg 42 train\ngrand_piano/image_0073.jpg 42 train\ngrand_piano/image_0001.jpg 42 train\ngrand_piano/image_0080.jpg 42 train\ngrand_piano/image_0021.jpg 42 train\ngrand_piano/image_0095.jpg 42 train\ngrand_piano/image_0057.jpg 42 train\ngrand_piano/image_0098.jpg 42 train\ngrand_piano/image_0008.jpg 42 train\ngrand_piano/image_0035.jpg 42 train\ngrand_piano/image_0005.jpg 42 train\ngrand_piano/image_0087.jpg 42 train\ngrand_piano/image_0089.jpg 42 train\ngrand_piano/image_0059.jpg 42 train\ngrand_piano/image_0069.jpg 42 train\ngrand_piano/image_0099.jpg 42 train\ngrand_piano/image_0034.jpg 42 train\ngrand_piano/image_0002.jpg 42 train\ngrand_piano/image_0048.jpg 42 train\ngrand_piano/image_0053.jpg 42 train\ngrand_piano/image_0084.jpg 42 train\ngrand_piano/image_0020.jpg 42 train\ngrand_piano/image_0046.jpg 42 train\ngrand_piano/image_0041.jpg 42 train\ngrand_piano/image_0027.jpg 42 train\ngrand_piano/image_0014.jpg 42 train\ngrand_piano/image_0058.jpg 42 train\ngrand_piano/image_0081.jpg 42 train\ngrand_piano/image_0088.jpg 42 train\nibis/image_0062.jpg 43 train\nibis/image_0037.jpg 43 train\nibis/image_0049.jpg 43 train\nibis/image_0051.jpg 43 train\nibis/image_0006.jpg 43 train\nibis/image_0016.jpg 43 train\nibis/image_0030.jpg 43 train\nibis/image_0012.jpg 43 train\nibis/image_0066.jpg 43 train\nibis/image_0077.jpg 43 train\nibis/image_0025.jpg 43 train\nibis/image_0060.jpg 43 train\nibis/image_0052.jpg 43 train\nibis/image_0007.jpg 43 train\nibis/image_0063.jpg 43 train\nibis/image_0019.jpg 43 train\nibis/image_0031.jpg 43 train\nibis/image_0011.jpg 43 train\nibis/image_0061.jpg 43 train\nibis/image_0050.jpg 43 train\nibis/image_0045.jpg 43 train\nibis/image_0017.jpg 43 train\nibis/image_0018.jpg 43 train\nibis/image_0013.jpg 43 train\nibis/image_0054.jpg 43 train\nibis/image_0079.jpg 43 train\nibis/image_0055.jpg 43 train\nibis/image_0071.jpg 43 train\nibis/image_0024.jpg 43 train\nibis/image_0074.jpg 43 train\nibis/image_0003.jpg 43 train\nibis/image_0009.jpg 43 train\nibis/image_0042.jpg 43 train\nibis/image_0040.jpg 43 train\nibis/image_0067.jpg 43 train\nibis/image_0047.jpg 43 train\nibis/image_0064.jpg 43 train\nibis/image_0070.jpg 43 train\nibis/image_0078.jpg 43 train\nibis/image_0068.jpg 43 train\nibis/image_0043.jpg 43 train\nibis/image_0044.jpg 43 train\nibis/image_0033.jpg 43 train\nibis/image_0073.jpg 43 train\nibis/image_0001.jpg 43 train\nibis/image_0080.jpg 43 train\nibis/image_0021.jpg 43 train\nibis/image_0057.jpg 43 train\nibis/image_0008.jpg 43 train\nibis/image_0035.jpg 43 train\nibis/image_0005.jpg 43 train\nibis/image_0059.jpg 43 train\nibis/image_0069.jpg 43 train\nibis/image_0034.jpg 43 train\nibis/image_0002.jpg 43 train\nibis/image_0048.jpg 43 train\nibis/image_0053.jpg 43 train\nibis/image_0020.jpg 43 train\nibis/image_0046.jpg 43 train\nibis/image_0041.jpg 43 train\nibis/image_0027.jpg 43 train\nibis/image_0014.jpg 43 train\nibis/image_0058.jpg 43 train\nibis/image_0026.jpg 43 train\nbrontosaurus/image_0037.jpg 44 train\nbrontosaurus/image_0006.jpg 44 train\nbrontosaurus/image_0016.jpg 44 train\nbrontosaurus/image_0030.jpg 44 train\nbrontosaurus/image_0012.jpg 44 train\nbrontosaurus/image_0025.jpg 44 train\nbrontosaurus/image_0007.jpg 44 train\nbrontosaurus/image_0019.jpg 44 train\nbrontosaurus/image_0031.jpg 44 train\nbrontosaurus/image_0011.jpg 44 train\nbrontosaurus/image_0017.jpg 44 train\nbrontosaurus/image_0018.jpg 44 train\nbrontosaurus/image_0013.jpg 44 train\nbrontosaurus/image_0024.jpg 44 train\nbrontosaurus/image_0003.jpg 44 train\nbrontosaurus/image_0009.jpg 44 train\nbrontosaurus/image_0042.jpg 44 train\nbrontosaurus/image_0040.jpg 44 train\nbrontosaurus/image_0043.jpg 44 train\nbrontosaurus/image_0033.jpg 44 train\nbrontosaurus/image_0001.jpg 44 train\nbrontosaurus/image_0021.jpg 44 train\nbrontosaurus/image_0008.jpg 44 train\nbrontosaurus/image_0035.jpg 44 train\nbrontosaurus/image_0005.jpg 44 train\nbrontosaurus/image_0034.jpg 44 train\nbrontosaurus/image_0002.jpg 44 train\nbrontosaurus/image_0020.jpg 44 train\nbrontosaurus/image_0041.jpg 44 train\nbrontosaurus/image_0027.jpg 44 train\nbrontosaurus/image_0014.jpg 44 train\nbrontosaurus/image_0026.jpg 44 train\nbrontosaurus/image_0004.jpg 44 train\nbrontosaurus/image_0015.jpg 44 train\nsaxophone/image_0037.jpg 45 train\nsaxophone/image_0006.jpg 45 train\nsaxophone/image_0016.jpg 45 train\nsaxophone/image_0030.jpg 45 train\nsaxophone/image_0012.jpg 45 train\nsaxophone/image_0025.jpg 45 train\nsaxophone/image_0007.jpg 45 train\nsaxophone/image_0019.jpg 45 train\nsaxophone/image_0031.jpg 45 train\nsaxophone/image_0011.jpg 45 train\nsaxophone/image_0017.jpg 45 train\nsaxophone/image_0018.jpg 45 train\nsaxophone/image_0013.jpg 45 train\nsaxophone/image_0024.jpg 45 train\nsaxophone/image_0003.jpg 45 train\nsaxophone/image_0009.jpg 45 train\nsaxophone/image_0040.jpg 45 train\nsaxophone/image_0033.jpg 45 train\nsaxophone/image_0001.jpg 45 train\nsaxophone/image_0021.jpg 45 train\nsaxophone/image_0008.jpg 45 train\nsaxophone/image_0035.jpg 45 train\nsaxophone/image_0005.jpg 45 train\nsaxophone/image_0034.jpg 45 train\nsaxophone/image_0002.jpg 45 train\nsaxophone/image_0020.jpg 45 train\nsaxophone/image_0027.jpg 45 train\nsaxophone/image_0014.jpg 45 train\nsaxophone/image_0026.jpg 45 train\nsaxophone/image_0004.jpg 45 train\nsaxophone/image_0015.jpg 45 train\nsaxophone/image_0029.jpg 45 train\nstapler/image_0037.jpg 46 train\nstapler/image_0006.jpg 46 train\nstapler/image_0016.jpg 46 train\nstapler/image_0030.jpg 46 train\nstapler/image_0012.jpg 46 train\nstapler/image_0025.jpg 46 train\nstapler/image_0007.jpg 46 train\nstapler/image_0019.jpg 46 train\nstapler/image_0031.jpg 46 train\nstapler/image_0011.jpg 46 train\nstapler/image_0045.jpg 46 train\nstapler/image_0017.jpg 46 train\nstapler/image_0018.jpg 46 train\nstapler/image_0013.jpg 46 train\nstapler/image_0024.jpg 46 train\nstapler/image_0003.jpg 46 train\nstapler/image_0009.jpg 46 train\nstapler/image_0042.jpg 46 train\nstapler/image_0040.jpg 46 train\nstapler/image_0043.jpg 46 train\nstapler/image_0044.jpg 46 train\nstapler/image_0033.jpg 46 train\nstapler/image_0001.jpg 46 train\nstapler/image_0021.jpg 46 train\nstapler/image_0008.jpg 46 train\nstapler/image_0035.jpg 46 train\nstapler/image_0005.jpg 46 train\nstapler/image_0034.jpg 46 train\nstapler/image_0002.jpg 46 train\nstapler/image_0020.jpg 46 train\nstapler/image_0041.jpg 46 train\nstapler/image_0027.jpg 46 train\nstapler/image_0014.jpg 46 train\nstapler/image_0026.jpg 46 train\nstapler/image_0004.jpg 46 train\nstapler/image_0015.jpg 46 train\nelectric_guitar/image_0062.jpg 47 train\nelectric_guitar/image_0037.jpg 47 train\nelectric_guitar/image_0049.jpg 47 train\nelectric_guitar/image_0051.jpg 47 train\nelectric_guitar/image_0006.jpg 47 train\nelectric_guitar/image_0016.jpg 47 train\nelectric_guitar/image_0030.jpg 47 train\nelectric_guitar/image_0012.jpg 47 train\nelectric_guitar/image_0066.jpg 47 train\nelectric_guitar/image_0025.jpg 47 train\nelectric_guitar/image_0060.jpg 47 train\nelectric_guitar/image_0052.jpg 47 train\nelectric_guitar/image_0007.jpg 47 train\nelectric_guitar/image_0063.jpg 47 train\nelectric_guitar/image_0019.jpg 47 train\nelectric_guitar/image_0031.jpg 47 train\nelectric_guitar/image_0011.jpg 47 train\nelectric_guitar/image_0061.jpg 47 train\nelectric_guitar/image_0050.jpg 47 train\nelectric_guitar/image_0045.jpg 47 train\nelectric_guitar/image_0017.jpg 47 train\nelectric_guitar/image_0018.jpg 47 train\nelectric_guitar/image_0013.jpg 47 train\nelectric_guitar/image_0054.jpg 47 train\nelectric_guitar/image_0055.jpg 47 train\nelectric_guitar/image_0071.jpg 47 train\nelectric_guitar/image_0024.jpg 47 train\nelectric_guitar/image_0074.jpg 47 train\nelectric_guitar/image_0003.jpg 47 train\nelectric_guitar/image_0009.jpg 47 train\nelectric_guitar/image_0042.jpg 47 train\nelectric_guitar/image_0040.jpg 47 train\nelectric_guitar/image_0067.jpg 47 train\nelectric_guitar/image_0047.jpg 47 train\nelectric_guitar/image_0064.jpg 47 train\nelectric_guitar/image_0070.jpg 47 train\nelectric_guitar/image_0068.jpg 47 train\nelectric_guitar/image_0043.jpg 47 train\nelectric_guitar/image_0044.jpg 47 train\nelectric_guitar/image_0033.jpg 47 train\nelectric_guitar/image_0073.jpg 47 train\nelectric_guitar/image_0001.jpg 47 train\nelectric_guitar/image_0021.jpg 47 train\nelectric_guitar/image_0057.jpg 47 train\nelectric_guitar/image_0008.jpg 47 train\nelectric_guitar/image_0035.jpg 47 train\nelectric_guitar/image_0005.jpg 47 train\nelectric_guitar/image_0059.jpg 47 train\nelectric_guitar/image_0069.jpg 47 train\nelectric_guitar/image_0034.jpg 47 train\nelectric_guitar/image_0002.jpg 47 train\nelectric_guitar/image_0048.jpg 47 train\nelectric_guitar/image_0053.jpg 47 train\nelectric_guitar/image_0020.jpg 47 train\nelectric_guitar/image_0046.jpg 47 train\nelectric_guitar/image_0041.jpg 47 train\nelectric_guitar/image_0027.jpg 47 train\nelectric_guitar/image_0014.jpg 47 train\nelectric_guitar/image_0058.jpg 47 train\nelectric_guitar/image_0026.jpg 47 train\noctopus/image_0006.jpg 48 train\noctopus/image_0016.jpg 48 train\noctopus/image_0030.jpg 48 train\noctopus/image_0012.jpg 48 train\noctopus/image_0025.jpg 48 train\noctopus/image_0007.jpg 48 train\noctopus/image_0019.jpg 48 train\noctopus/image_0031.jpg 48 train\noctopus/image_0011.jpg 48 train\noctopus/image_0017.jpg 48 train\noctopus/image_0018.jpg 48 train\noctopus/image_0013.jpg 48 train\noctopus/image_0024.jpg 48 train\noctopus/image_0003.jpg 48 train\noctopus/image_0009.jpg 48 train\noctopus/image_0033.jpg 48 train\noctopus/image_0001.jpg 48 train\noctopus/image_0021.jpg 48 train\noctopus/image_0008.jpg 48 train\noctopus/image_0035.jpg 48 train\noctopus/image_0005.jpg 48 train\noctopus/image_0034.jpg 48 train\noctopus/image_0002.jpg 48 train\noctopus/image_0020.jpg 48 train\noctopus/image_0027.jpg 48 train\noctopus/image_0014.jpg 48 train\noctopus/image_0026.jpg 48 train\noctopus/image_0004.jpg 48 train\nkangaroo/image_0062.jpg 49 train\nkangaroo/image_0037.jpg 49 train\nkangaroo/image_0049.jpg 49 train\nkangaroo/image_0083.jpg 49 train\nkangaroo/image_0051.jpg 49 train\nkangaroo/image_0006.jpg 49 train\nkangaroo/image_0016.jpg 49 train\nkangaroo/image_0030.jpg 49 train\nkangaroo/image_0012.jpg 49 train\nkangaroo/image_0066.jpg 49 train\nkangaroo/image_0077.jpg 49 train\nkangaroo/image_0025.jpg 49 train\nkangaroo/image_0060.jpg 49 train\nkangaroo/image_0052.jpg 49 train\nkangaroo/image_0007.jpg 49 train\nkangaroo/image_0063.jpg 49 train\nkangaroo/image_0019.jpg 49 train\nkangaroo/image_0031.jpg 49 train\nkangaroo/image_0011.jpg 49 train\nkangaroo/image_0061.jpg 49 train\nkangaroo/image_0050.jpg 49 train\nkangaroo/image_0045.jpg 49 train\nkangaroo/image_0017.jpg 49 train\nkangaroo/image_0018.jpg 49 train\nkangaroo/image_0013.jpg 49 train\nkangaroo/image_0054.jpg 49 train\nkangaroo/image_0082.jpg 49 train\nkangaroo/image_0079.jpg 49 train\nkangaroo/image_0055.jpg 49 train\nkangaroo/image_0071.jpg 49 train\nkangaroo/image_0024.jpg 49 train\nkangaroo/image_0074.jpg 49 train\nkangaroo/image_0003.jpg 49 train\nkangaroo/image_0009.jpg 49 train\nkangaroo/image_0042.jpg 49 train\nkangaroo/image_0040.jpg 49 train\nkangaroo/image_0067.jpg 49 train\nkangaroo/image_0047.jpg 49 train\nkangaroo/image_0064.jpg 49 train\nkangaroo/image_0070.jpg 49 train\nkangaroo/image_0078.jpg 49 train\nkangaroo/image_0068.jpg 49 train\nkangaroo/image_0085.jpg 49 train\nkangaroo/image_0043.jpg 49 train\nkangaroo/image_0044.jpg 49 train\nkangaroo/image_0033.jpg 49 train\nkangaroo/image_0073.jpg 49 train\nkangaroo/image_0001.jpg 49 train\nkangaroo/image_0080.jpg 49 train\nkangaroo/image_0021.jpg 49 train\nkangaroo/image_0057.jpg 49 train\nkangaroo/image_0008.jpg 49 train\nkangaroo/image_0035.jpg 49 train\nkangaroo/image_0005.jpg 49 train\nkangaroo/image_0059.jpg 49 train\nkangaroo/image_0069.jpg 49 train\nkangaroo/image_0034.jpg 49 train\nkangaroo/image_0002.jpg 49 train\nkangaroo/image_0048.jpg 49 train\nkangaroo/image_0053.jpg 49 train\nkangaroo/image_0084.jpg 49 train\nkangaroo/image_0020.jpg 49 train\nkangaroo/image_0046.jpg 49 train\nkangaroo/image_0041.jpg 49 train\nkangaroo/image_0027.jpg 49 train\nkangaroo/image_0014.jpg 49 train\nkangaroo/image_0058.jpg 49 train\nkangaroo/image_0081.jpg 49 train\nokapi/image_0037.jpg 50 train\nokapi/image_0006.jpg 50 train\nokapi/image_0016.jpg 50 train\nokapi/image_0030.jpg 50 train\nokapi/image_0012.jpg 50 train\nokapi/image_0025.jpg 50 train\nokapi/image_0007.jpg 50 train\nokapi/image_0019.jpg 50 train\nokapi/image_0031.jpg 50 train\nokapi/image_0011.jpg 50 train\nokapi/image_0017.jpg 50 train\nokapi/image_0018.jpg 50 train\nokapi/image_0013.jpg 50 train\nokapi/image_0024.jpg 50 train\nokapi/image_0003.jpg 50 train\nokapi/image_0009.jpg 50 train\nokapi/image_0033.jpg 50 train\nokapi/image_0001.jpg 50 train\nokapi/image_0021.jpg 50 train\nokapi/image_0008.jpg 50 train\nokapi/image_0035.jpg 50 train\nokapi/image_0005.jpg 50 train\nokapi/image_0034.jpg 50 train\nokapi/image_0002.jpg 50 train\nokapi/image_0020.jpg 50 train\nokapi/image_0027.jpg 50 train\nokapi/image_0014.jpg 50 train\nokapi/image_0026.jpg 50 train\nokapi/image_0004.jpg 50 train\nokapi/image_0015.jpg 50 train\nokapi/image_0029.jpg 50 train\nsunflower/image_0062.jpg 51 train\nsunflower/image_0037.jpg 51 train\nsunflower/image_0049.jpg 51 train\nsunflower/image_0083.jpg 51 train\nsunflower/image_0051.jpg 51 train\nsunflower/image_0006.jpg 51 train\nsunflower/image_0016.jpg 51 train\nsunflower/image_0030.jpg 51 train\nsunflower/image_0012.jpg 51 train\nsunflower/image_0066.jpg 51 train\nsunflower/image_0077.jpg 51 train\nsunflower/image_0025.jpg 51 train\nsunflower/image_0060.jpg 51 train\nsunflower/image_0052.jpg 51 train\nsunflower/image_0007.jpg 51 train\nsunflower/image_0063.jpg 51 train\nsunflower/image_0019.jpg 51 train\nsunflower/image_0031.jpg 51 train\nsunflower/image_0011.jpg 51 train\nsunflower/image_0061.jpg 51 train\nsunflower/image_0050.jpg 51 train\nsunflower/image_0045.jpg 51 train\nsunflower/image_0017.jpg 51 train\nsunflower/image_0018.jpg 51 train\nsunflower/image_0013.jpg 51 train\nsunflower/image_0054.jpg 51 train\nsunflower/image_0082.jpg 51 train\nsunflower/image_0079.jpg 51 train\nsunflower/image_0055.jpg 51 train\nsunflower/image_0071.jpg 51 train\nsunflower/image_0024.jpg 51 train\nsunflower/image_0074.jpg 51 train\nsunflower/image_0003.jpg 51 train\nsunflower/image_0009.jpg 51 train\nsunflower/image_0042.jpg 51 train\nsunflower/image_0040.jpg 51 train\nsunflower/image_0067.jpg 51 train\nsunflower/image_0047.jpg 51 train\nsunflower/image_0064.jpg 51 train\nsunflower/image_0070.jpg 51 train\nsunflower/image_0078.jpg 51 train\nsunflower/image_0068.jpg 51 train\nsunflower/image_0085.jpg 51 train\nsunflower/image_0043.jpg 51 train\nsunflower/image_0044.jpg 51 train\nsunflower/image_0033.jpg 51 train\nsunflower/image_0073.jpg 51 train\nsunflower/image_0001.jpg 51 train\nsunflower/image_0080.jpg 51 train\nsunflower/image_0021.jpg 51 train\nsunflower/image_0057.jpg 51 train\nsunflower/image_0008.jpg 51 train\nsunflower/image_0035.jpg 51 train\nsunflower/image_0005.jpg 51 train\nsunflower/image_0059.jpg 51 train\nsunflower/image_0069.jpg 51 train\nsunflower/image_0034.jpg 51 train\nsunflower/image_0002.jpg 51 train\nsunflower/image_0048.jpg 51 train\nsunflower/image_0053.jpg 51 train\nsunflower/image_0084.jpg 51 train\nsunflower/image_0020.jpg 51 train\nsunflower/image_0046.jpg 51 train\nsunflower/image_0041.jpg 51 train\nsunflower/image_0027.jpg 51 train\nsunflower/image_0014.jpg 51 train\nsunflower/image_0058.jpg 51 train\nsunflower/image_0081.jpg 51 train\ngarfield/image_0006.jpg 52 train\ngarfield/image_0016.jpg 52 train\ngarfield/image_0030.jpg 52 train\ngarfield/image_0012.jpg 52 train\ngarfield/image_0025.jpg 52 train\ngarfield/image_0007.jpg 52 train\ngarfield/image_0019.jpg 52 train\ngarfield/image_0031.jpg 52 train\ngarfield/image_0011.jpg 52 train\ngarfield/image_0017.jpg 52 train\ngarfield/image_0018.jpg 52 train\ngarfield/image_0013.jpg 52 train\ngarfield/image_0024.jpg 52 train\ngarfield/image_0003.jpg 52 train\ngarfield/image_0009.jpg 52 train\ngarfield/image_0033.jpg 52 train\ngarfield/image_0001.jpg 52 train\ngarfield/image_0021.jpg 52 train\ngarfield/image_0008.jpg 52 train\ngarfield/image_0005.jpg 52 train\ngarfield/image_0034.jpg 52 train\ngarfield/image_0002.jpg 52 train\ngarfield/image_0020.jpg 52 train\ngarfield/image_0027.jpg 52 train\ngarfield/image_0014.jpg 52 train\ngarfield/image_0026.jpg 52 train\ngarfield/image_0004.jpg 52 train\nbass/image_0037.jpg 53 train\nbass/image_0049.jpg 53 train\nbass/image_0051.jpg 53 train\nbass/image_0006.jpg 53 train\nbass/image_0016.jpg 53 train\nbass/image_0030.jpg 53 train\nbass/image_0012.jpg 53 train\nbass/image_0025.jpg 53 train\nbass/image_0052.jpg 53 train\nbass/image_0007.jpg 53 train\nbass/image_0019.jpg 53 train\nbass/image_0031.jpg 53 train\nbass/image_0011.jpg 53 train\nbass/image_0050.jpg 53 train\nbass/image_0045.jpg 53 train\nbass/image_0017.jpg 53 train\nbass/image_0018.jpg 53 train\nbass/image_0013.jpg 53 train\nbass/image_0054.jpg 53 train\nbass/image_0024.jpg 53 train\nbass/image_0003.jpg 53 train\nbass/image_0009.jpg 53 train\nbass/image_0042.jpg 53 train\nbass/image_0040.jpg 53 train\nbass/image_0047.jpg 53 train\nbass/image_0043.jpg 53 train\nbass/image_0044.jpg 53 train\nbass/image_0033.jpg 53 train\nbass/image_0001.jpg 53 train\nbass/image_0021.jpg 53 train\nbass/image_0008.jpg 53 train\nbass/image_0035.jpg 53 train\nbass/image_0005.jpg 53 train\nbass/image_0034.jpg 53 train\nbass/image_0002.jpg 53 train\nbass/image_0048.jpg 53 train\nbass/image_0053.jpg 53 train\nbass/image_0020.jpg 53 train\nbass/image_0046.jpg 53 train\nbass/image_0041.jpg 53 train\nbass/image_0027.jpg 53 train\nbass/image_0014.jpg 53 train\nbass/image_0026.jpg 53 train\ncellphone/image_0037.jpg 54 train\ncellphone/image_0049.jpg 54 train\ncellphone/image_0051.jpg 54 train\ncellphone/image_0006.jpg 54 train\ncellphone/image_0016.jpg 54 train\ncellphone/image_0030.jpg 54 train\ncellphone/image_0012.jpg 54 train\ncellphone/image_0025.jpg 54 train\ncellphone/image_0052.jpg 54 train\ncellphone/image_0007.jpg 54 train\ncellphone/image_0019.jpg 54 train\ncellphone/image_0031.jpg 54 train\ncellphone/image_0011.jpg 54 train\ncellphone/image_0050.jpg 54 train\ncellphone/image_0045.jpg 54 train\ncellphone/image_0017.jpg 54 train\ncellphone/image_0018.jpg 54 train\ncellphone/image_0013.jpg 54 train\ncellphone/image_0054.jpg 54 train\ncellphone/image_0055.jpg 54 train\ncellphone/image_0024.jpg 54 train\ncellphone/image_0003.jpg 54 train\ncellphone/image_0009.jpg 54 train\ncellphone/image_0042.jpg 54 train\ncellphone/image_0040.jpg 54 train\ncellphone/image_0047.jpg 54 train\ncellphone/image_0043.jpg 54 train\ncellphone/image_0044.jpg 54 train\ncellphone/image_0033.jpg 54 train\ncellphone/image_0001.jpg 54 train\ncellphone/image_0021.jpg 54 train\ncellphone/image_0057.jpg 54 train\ncellphone/image_0008.jpg 54 train\ncellphone/image_0035.jpg 54 train\ncellphone/image_0005.jpg 54 train\ncellphone/image_0059.jpg 54 train\ncellphone/image_0034.jpg 54 train\ncellphone/image_0002.jpg 54 train\ncellphone/image_0048.jpg 54 train\ncellphone/image_0053.jpg 54 train\ncellphone/image_0020.jpg 54 train\ncellphone/image_0046.jpg 54 train\ncellphone/image_0041.jpg 54 train\ncellphone/image_0027.jpg 54 train\ncellphone/image_0014.jpg 54 train\ncellphone/image_0058.jpg 54 train\ncellphone/image_0026.jpg 54 train\nbrain/image_0062.jpg 55 train\nbrain/image_0037.jpg 55 train\nbrain/image_0049.jpg 55 train\nbrain/image_0083.jpg 55 train\nbrain/image_0051.jpg 55 train\nbrain/image_0006.jpg 55 train\nbrain/image_0016.jpg 55 train\nbrain/image_0030.jpg 55 train\nbrain/image_0091.jpg 55 train\nbrain/image_0012.jpg 55 train\nbrain/image_0066.jpg 55 train\nbrain/image_0077.jpg 55 train\nbrain/image_0025.jpg 55 train\nbrain/image_0060.jpg 55 train\nbrain/image_0052.jpg 55 train\nbrain/image_0007.jpg 55 train\nbrain/image_0063.jpg 55 train\nbrain/image_0019.jpg 55 train\nbrain/image_0031.jpg 55 train\nbrain/image_0011.jpg 55 train\nbrain/image_0061.jpg 55 train\nbrain/image_0090.jpg 55 train\nbrain/image_0050.jpg 55 train\nbrain/image_0045.jpg 55 train\nbrain/image_0017.jpg 55 train\nbrain/image_0018.jpg 55 train\nbrain/image_0013.jpg 55 train\nbrain/image_0054.jpg 55 train\nbrain/image_0094.jpg 55 train\nbrain/image_0082.jpg 55 train\nbrain/image_0097.jpg 55 train\nbrain/image_0079.jpg 55 train\nbrain/image_0055.jpg 55 train\nbrain/image_0071.jpg 55 train\nbrain/image_0024.jpg 55 train\nbrain/image_0074.jpg 55 train\nbrain/image_0003.jpg 55 train\nbrain/image_0009.jpg 55 train\nbrain/image_0042.jpg 55 train\nbrain/image_0093.jpg 55 train\nbrain/image_0040.jpg 55 train\nbrain/image_0067.jpg 55 train\nbrain/image_0047.jpg 55 train\nbrain/image_0064.jpg 55 train\nbrain/image_0070.jpg 55 train\nbrain/image_0078.jpg 55 train\nbrain/image_0068.jpg 55 train\nbrain/image_0085.jpg 55 train\nbrain/image_0043.jpg 55 train\nbrain/image_0044.jpg 55 train\nbrain/image_0033.jpg 55 train\nbrain/image_0073.jpg 55 train\nbrain/image_0001.jpg 55 train\nbrain/image_0080.jpg 55 train\nbrain/image_0021.jpg 55 train\nbrain/image_0095.jpg 55 train\nbrain/image_0057.jpg 55 train\nbrain/image_0098.jpg 55 train\nbrain/image_0008.jpg 55 train\nbrain/image_0035.jpg 55 train\nbrain/image_0005.jpg 55 train\nbrain/image_0087.jpg 55 train\nbrain/image_0089.jpg 55 train\nbrain/image_0059.jpg 55 train\nbrain/image_0069.jpg 55 train\nbrain/image_0034.jpg 55 train\nbrain/image_0002.jpg 55 train\nbrain/image_0048.jpg 55 train\nbrain/image_0053.jpg 55 train\nbrain/image_0084.jpg 55 train\nbrain/image_0020.jpg 55 train\nbrain/image_0046.jpg 55 train\nbrain/image_0041.jpg 55 train\nbrain/image_0027.jpg 55 train\nbrain/image_0014.jpg 55 train\nbrain/image_0058.jpg 55 train\nbrain/image_0081.jpg 55 train\nbrain/image_0088.jpg 55 train\nlobster/image_0037.jpg 56 train\nlobster/image_0006.jpg 56 train\nlobster/image_0016.jpg 56 train\nlobster/image_0030.jpg 56 train\nlobster/image_0012.jpg 56 train\nlobster/image_0025.jpg 56 train\nlobster/image_0007.jpg 56 train\nlobster/image_0019.jpg 56 train\nlobster/image_0031.jpg 56 train\nlobster/image_0011.jpg 56 train\nlobster/image_0017.jpg 56 train\nlobster/image_0018.jpg 56 train\nlobster/image_0013.jpg 56 train\nlobster/image_0024.jpg 56 train\nlobster/image_0003.jpg 56 train\nlobster/image_0009.jpg 56 train\nlobster/image_0040.jpg 56 train\nlobster/image_0033.jpg 56 train\nlobster/image_0001.jpg 56 train\nlobster/image_0021.jpg 56 train\nlobster/image_0008.jpg 56 train\nlobster/image_0035.jpg 56 train\nlobster/image_0005.jpg 56 train\nlobster/image_0034.jpg 56 train\nlobster/image_0002.jpg 56 train\nlobster/image_0020.jpg 56 train\nlobster/image_0041.jpg 56 train\nlobster/image_0027.jpg 56 train\nlobster/image_0014.jpg 56 train\nlobster/image_0026.jpg 56 train\nlobster/image_0004.jpg 56 train\nlobster/image_0015.jpg 56 train\nheadphone/image_0037.jpg 57 train\nheadphone/image_0006.jpg 57 train\nheadphone/image_0016.jpg 57 train\nheadphone/image_0030.jpg 57 train\nheadphone/image_0012.jpg 57 train\nheadphone/image_0025.jpg 57 train\nheadphone/image_0007.jpg 57 train\nheadphone/image_0019.jpg 57 train\nheadphone/image_0031.jpg 57 train\nheadphone/image_0011.jpg 57 train\nheadphone/image_0017.jpg 57 train\nheadphone/image_0018.jpg 57 train\nheadphone/image_0013.jpg 57 train\nheadphone/image_0024.jpg 57 train\nheadphone/image_0003.jpg 57 train\nheadphone/image_0009.jpg 57 train\nheadphone/image_0042.jpg 57 train\nheadphone/image_0040.jpg 57 train\nheadphone/image_0033.jpg 57 train\nheadphone/image_0001.jpg 57 train\nheadphone/image_0021.jpg 57 train\nheadphone/image_0008.jpg 57 train\nheadphone/image_0035.jpg 57 train\nheadphone/image_0005.jpg 57 train\nheadphone/image_0034.jpg 57 train\nheadphone/image_0002.jpg 57 train\nheadphone/image_0020.jpg 57 train\nheadphone/image_0041.jpg 57 train\nheadphone/image_0027.jpg 57 train\nheadphone/image_0014.jpg 57 train\nheadphone/image_0026.jpg 57 train\nheadphone/image_0004.jpg 57 train\nheadphone/image_0015.jpg 57 train\nbarrel/image_0037.jpg 58 train\nbarrel/image_0006.jpg 58 train\nbarrel/image_0016.jpg 58 train\nbarrel/image_0030.jpg 58 train\nbarrel/image_0012.jpg 58 train\nbarrel/image_0025.jpg 58 train\nbarrel/image_0007.jpg 58 train\nbarrel/image_0019.jpg 58 train\nbarrel/image_0031.jpg 58 train\nbarrel/image_0011.jpg 58 train\nbarrel/image_0045.jpg 58 train\nbarrel/image_0017.jpg 58 train\nbarrel/image_0018.jpg 58 train\nbarrel/image_0013.jpg 58 train\nbarrel/image_0024.jpg 58 train\nbarrel/image_0003.jpg 58 train\nbarrel/image_0009.jpg 58 train\nbarrel/image_0042.jpg 58 train\nbarrel/image_0040.jpg 58 train\nbarrel/image_0047.jpg 58 train\nbarrel/image_0043.jpg 58 train\nbarrel/image_0044.jpg 58 train\nbarrel/image_0033.jpg 58 train\nbarrel/image_0001.jpg 58 train\nbarrel/image_0021.jpg 58 train\nbarrel/image_0008.jpg 58 train\nbarrel/image_0035.jpg 58 train\nbarrel/image_0005.jpg 58 train\nbarrel/image_0034.jpg 58 train\nbarrel/image_0002.jpg 58 train\nbarrel/image_0020.jpg 58 train\nbarrel/image_0046.jpg 58 train\nbarrel/image_0041.jpg 58 train\nbarrel/image_0027.jpg 58 train\nbarrel/image_0014.jpg 58 train\nbarrel/image_0026.jpg 58 train\nbarrel/image_0004.jpg 58 train\npigeon/image_0037.jpg 59 train\npigeon/image_0006.jpg 59 train\npigeon/image_0016.jpg 59 train\npigeon/image_0030.jpg 59 train\npigeon/image_0012.jpg 59 train\npigeon/image_0025.jpg 59 train\npigeon/image_0007.jpg 59 train\npigeon/image_0019.jpg 59 train\npigeon/image_0031.jpg 59 train\npigeon/image_0011.jpg 59 train\npigeon/image_0045.jpg 59 train\npigeon/image_0017.jpg 59 train\npigeon/image_0018.jpg 59 train\npigeon/image_0013.jpg 59 train\npigeon/image_0024.jpg 59 train\npigeon/image_0003.jpg 59 train\npigeon/image_0009.jpg 59 train\npigeon/image_0042.jpg 59 train\npigeon/image_0040.jpg 59 train\npigeon/image_0043.jpg 59 train\npigeon/image_0044.jpg 59 train\npigeon/image_0033.jpg 59 train\npigeon/image_0001.jpg 59 train\npigeon/image_0021.jpg 59 train\npigeon/image_0008.jpg 59 train\npigeon/image_0035.jpg 59 train\npigeon/image_0005.jpg 59 train\npigeon/image_0034.jpg 59 train\npigeon/image_0002.jpg 59 train\npigeon/image_0020.jpg 59 train\npigeon/image_0041.jpg 59 train\npigeon/image_0027.jpg 59 train\npigeon/image_0014.jpg 59 train\npigeon/image_0026.jpg 59 train\npigeon/image_0004.jpg 59 train\npigeon/image_0015.jpg 59 train\ninline_skate/image_0006.jpg 60 train\ninline_skate/image_0016.jpg 60 train\ninline_skate/image_0030.jpg 60 train\ninline_skate/image_0012.jpg 60 train\ninline_skate/image_0025.jpg 60 train\ninline_skate/image_0007.jpg 60 train\ninline_skate/image_0019.jpg 60 train\ninline_skate/image_0031.jpg 60 train\ninline_skate/image_0011.jpg 60 train\ninline_skate/image_0017.jpg 60 train\ninline_skate/image_0018.jpg 60 train\ninline_skate/image_0013.jpg 60 train\ninline_skate/image_0024.jpg 60 train\ninline_skate/image_0003.jpg 60 train\ninline_skate/image_0009.jpg 60 train\ninline_skate/image_0001.jpg 60 train\ninline_skate/image_0021.jpg 60 train\ninline_skate/image_0008.jpg 60 train\ninline_skate/image_0005.jpg 60 train\ninline_skate/image_0002.jpg 60 train\ninline_skate/image_0020.jpg 60 train\ninline_skate/image_0027.jpg 60 train\ninline_skate/image_0014.jpg 60 train\ninline_skate/image_0026.jpg 60 train\ncannon/image_0037.jpg 61 train\ncannon/image_0006.jpg 61 train\ncannon/image_0016.jpg 61 train\ncannon/image_0030.jpg 61 train\ncannon/image_0012.jpg 61 train\ncannon/image_0025.jpg 61 train\ncannon/image_0007.jpg 61 train\ncannon/image_0019.jpg 61 train\ncannon/image_0031.jpg 61 train\ncannon/image_0011.jpg 61 train\ncannon/image_0017.jpg 61 train\ncannon/image_0018.jpg 61 train\ncannon/image_0013.jpg 61 train\ncannon/image_0024.jpg 61 train\ncannon/image_0003.jpg 61 train\ncannon/image_0009.jpg 61 train\ncannon/image_0042.jpg 61 train\ncannon/image_0040.jpg 61 train\ncannon/image_0043.jpg 61 train\ncannon/image_0033.jpg 61 train\ncannon/image_0001.jpg 61 train\ncannon/image_0021.jpg 61 train\ncannon/image_0008.jpg 61 train\ncannon/image_0035.jpg 61 train\ncannon/image_0005.jpg 61 train\ncannon/image_0034.jpg 61 train\ncannon/image_0002.jpg 61 train\ncannon/image_0020.jpg 61 train\ncannon/image_0041.jpg 61 train\ncannon/image_0027.jpg 61 train\ncannon/image_0014.jpg 61 train\ncannon/image_0026.jpg 61 train\ncannon/image_0004.jpg 61 train\ncannon/image_0015.jpg 61 train\nBACKGROUND_Google/image_0316.jpg 62 train\nBACKGROUND_Google/image_0258.jpg 62 train\nBACKGROUND_Google/image_0426.jpg 62 train\nBACKGROUND_Google/image_0302.jpg 62 train\nBACKGROUND_Google/image_0171.jpg 62 train\nBACKGROUND_Google/image_0062.jpg 62 train\nBACKGROUND_Google/image_0346.jpg 62 train\nBACKGROUND_Google/image_0435.jpg 62 train\nBACKGROUND_Google/image_0341.jpg 62 train\nBACKGROUND_Google/image_0340.jpg 62 train\nBACKGROUND_Google/image_0331.jpg 62 train\nBACKGROUND_Google/image_0037.jpg 62 train\nBACKGROUND_Google/image_0049.jpg 62 train\nBACKGROUND_Google/image_0337.jpg 62 train\nBACKGROUND_Google/image_0317.jpg 62 train\nBACKGROUND_Google/image_0357.jpg 62 train\nBACKGROUND_Google/image_0083.jpg 62 train\nBACKGROUND_Google/image_0101.jpg 62 train\nBACKGROUND_Google/image_0051.jpg 62 train\nBACKGROUND_Google/image_0006.jpg 62 train\nBACKGROUND_Google/image_0142.jpg 62 train\nBACKGROUND_Google/image_0250.jpg 62 train\nBACKGROUND_Google/image_0427.jpg 62 train\nBACKGROUND_Google/image_0396.jpg 62 train\nBACKGROUND_Google/image_0332.jpg 62 train\nBACKGROUND_Google/image_0448.jpg 62 train\nBACKGROUND_Google/image_0335.jpg 62 train\nBACKGROUND_Google/image_0201.jpg 62 train\nBACKGROUND_Google/image_0016.jpg 62 train\nBACKGROUND_Google/image_0373.jpg 62 train\nBACKGROUND_Google/image_0176.jpg 62 train\nBACKGROUND_Google/image_0411.jpg 62 train\nBACKGROUND_Google/image_0155.jpg 62 train\nBACKGROUND_Google/image_0030.jpg 62 train\nBACKGROUND_Google/image_0091.jpg 62 train\nBACKGROUND_Google/image_0397.jpg 62 train\nBACKGROUND_Google/image_0012.jpg 62 train\nBACKGROUND_Google/image_0066.jpg 62 train\nBACKGROUND_Google/image_0465.jpg 62 train\nBACKGROUND_Google/image_0464.jpg 62 train\nBACKGROUND_Google/image_0283.jpg 62 train\nBACKGROUND_Google/image_0318.jpg 62 train\nBACKGROUND_Google/image_0224.jpg 62 train\nBACKGROUND_Google/image_0148.jpg 62 train\nBACKGROUND_Google/image_0077.jpg 62 train\nBACKGROUND_Google/image_0129.jpg 62 train\nBACKGROUND_Google/image_0339.jpg 62 train\nBACKGROUND_Google/image_0025.jpg 62 train\nBACKGROUND_Google/image_0328.jpg 62 train\nBACKGROUND_Google/image_0139.jpg 62 train\nBACKGROUND_Google/image_0425.jpg 62 train\nBACKGROUND_Google/image_0292.jpg 62 train\nBACKGROUND_Google/image_0348.jpg 62 train\nBACKGROUND_Google/image_0115.jpg 62 train\nBACKGROUND_Google/image_0123.jpg 62 train\nBACKGROUND_Google/image_0119.jpg 62 train\nBACKGROUND_Google/image_0060.jpg 62 train\nBACKGROUND_Google/image_0269.jpg 62 train\nBACKGROUND_Google/image_0052.jpg 62 train\nBACKGROUND_Google/image_0108.jpg 62 train\nBACKGROUND_Google/image_0239.jpg 62 train\nBACKGROUND_Google/image_0442.jpg 62 train\nBACKGROUND_Google/image_0370.jpg 62 train\nBACKGROUND_Google/image_0326.jpg 62 train\nBACKGROUND_Google/image_0223.jpg 62 train\nBACKGROUND_Google/image_0172.jpg 62 train\nBACKGROUND_Google/image_0372.jpg 62 train\nBACKGROUND_Google/image_0344.jpg 62 train\nBACKGROUND_Google/image_0390.jpg 62 train\nBACKGROUND_Google/image_0007.jpg 62 train\nBACKGROUND_Google/image_0213.jpg 62 train\nBACKGROUND_Google/image_0312.jpg 62 train\nBACKGROUND_Google/image_0463.jpg 62 train\nBACKGROUND_Google/image_0063.jpg 62 train\nBACKGROUND_Google/image_0019.jpg 62 train\nBACKGROUND_Google/image_0407.jpg 62 train\nBACKGROUND_Google/image_0192.jpg 62 train\nBACKGROUND_Google/image_0384.jpg 62 train\nBACKGROUND_Google/image_0195.jpg 62 train\nBACKGROUND_Google/image_0113.jpg 62 train\nBACKGROUND_Google/image_0127.jpg 62 train\nBACKGROUND_Google/image_0215.jpg 62 train\nBACKGROUND_Google/image_0389.jpg 62 train\nBACKGROUND_Google/image_0031.jpg 62 train\nBACKGROUND_Google/image_0349.jpg 62 train\nBACKGROUND_Google/image_0228.jpg 62 train\nBACKGROUND_Google/image_0380.jpg 62 train\nBACKGROUND_Google/image_0219.jpg 62 train\nBACKGROUND_Google/image_0353.jpg 62 train\nBACKGROUND_Google/image_0011.jpg 62 train\nBACKGROUND_Google/image_0233.jpg 62 train\nBACKGROUND_Google/image_0447.jpg 62 train\nBACKGROUND_Google/image_0061.jpg 62 train\nBACKGROUND_Google/image_0444.jpg 62 train\nBACKGROUND_Google/image_0090.jpg 62 train\nBACKGROUND_Google/image_0351.jpg 62 train\nBACKGROUND_Google/image_0261.jpg 62 train\nBACKGROUND_Google/image_0229.jpg 62 train\nBACKGROUND_Google/image_0126.jpg 62 train\nBACKGROUND_Google/image_0240.jpg 62 train\nBACKGROUND_Google/image_0467.jpg 62 train\nBACKGROUND_Google/image_0455.jpg 62 train\nBACKGROUND_Google/image_0441.jpg 62 train\nBACKGROUND_Google/image_0305.jpg 62 train\nBACKGROUND_Google/image_0334.jpg 62 train\nBACKGROUND_Google/image_0354.jpg 62 train\nBACKGROUND_Google/image_0218.jpg 62 train\nBACKGROUND_Google/image_0366.jpg 62 train\nBACKGROUND_Google/image_0288.jpg 62 train\nBACKGROUND_Google/image_0216.jpg 62 train\nBACKGROUND_Google/image_0459.jpg 62 train\nBACKGROUND_Google/image_0429.jpg 62 train\nBACKGROUND_Google/image_0277.jpg 62 train\nBACKGROUND_Google/image_0358.jpg 62 train\nBACKGROUND_Google/image_0050.jpg 62 train\nBACKGROUND_Google/image_0246.jpg 62 train\nBACKGROUND_Google/image_0361.jpg 62 train\nBACKGROUND_Google/image_0177.jpg 62 train\nBACKGROUND_Google/image_0356.jpg 62 train\nBACKGROUND_Google/image_0196.jpg 62 train\nBACKGROUND_Google/image_0293.jpg 62 train\nBACKGROUND_Google/image_0406.jpg 62 train\nBACKGROUND_Google/image_0347.jpg 62 train\nBACKGROUND_Google/image_0443.jpg 62 train\nBACKGROUND_Google/image_0187.jpg 62 train\nBACKGROUND_Google/image_0234.jpg 62 train\nBACKGROUND_Google/image_0045.jpg 62 train\nBACKGROUND_Google/image_0303.jpg 62 train\nBACKGROUND_Google/image_0363.jpg 62 train\nBACKGROUND_Google/image_0017.jpg 62 train\nBACKGROUND_Google/image_0131.jpg 62 train\nBACKGROUND_Google/image_0296.jpg 62 train\nBACKGROUND_Google/image_0150.jpg 62 train\nBACKGROUND_Google/image_0018.jpg 62 train\nBACKGROUND_Google/image_0350.jpg 62 train\nBACKGROUND_Google/image_0209.jpg 62 train\nBACKGROUND_Google/image_0013.jpg 62 train\nBACKGROUND_Google/image_0054.jpg 62 train\nBACKGROUND_Google/image_0094.jpg 62 train\nBACKGROUND_Google/image_0082.jpg 62 train\nBACKGROUND_Google/image_0097.jpg 62 train\nBACKGROUND_Google/image_0130.jpg 62 train\nBACKGROUND_Google/image_0457.jpg 62 train\nBACKGROUND_Google/image_0382.jpg 62 train\nBACKGROUND_Google/image_0141.jpg 62 train\nBACKGROUND_Google/image_0144.jpg 62 train\nBACKGROUND_Google/image_0109.jpg 62 train\nBACKGROUND_Google/image_0079.jpg 62 train\nBACKGROUND_Google/image_0306.jpg 62 train\nBACKGROUND_Google/image_0398.jpg 62 train\nBACKGROUND_Google/image_0055.jpg 62 train\nBACKGROUND_Google/image_0367.jpg 62 train\nBACKGROUND_Google/image_0365.jpg 62 train\nBACKGROUND_Google/image_0217.jpg 62 train\nBACKGROUND_Google/image_0343.jpg 62 train\nBACKGROUND_Google/image_0432.jpg 62 train\nBACKGROUND_Google/image_0071.jpg 62 train\nBACKGROUND_Google/image_0282.jpg 62 train\nBACKGROUND_Google/image_0393.jpg 62 train\nBACKGROUND_Google/image_0287.jpg 62 train\nBACKGROUND_Google/image_0198.jpg 62 train\nBACKGROUND_Google/image_0262.jpg 62 train\nBACKGROUND_Google/image_0462.jpg 62 train\nBACKGROUND_Google/image_0168.jpg 62 train\nBACKGROUND_Google/image_0368.jpg 62 train\nBACKGROUND_Google/image_0420.jpg 62 train\nBACKGROUND_Google/image_0211.jpg 62 train\nBACKGROUND_Google/image_0270.jpg 62 train\nBACKGROUND_Google/image_0103.jpg 62 train\nBACKGROUND_Google/image_0024.jpg 62 train\nBACKGROUND_Google/image_0297.jpg 62 train\nBACKGROUND_Google/image_0188.jpg 62 train\nBACKGROUND_Google/image_0105.jpg 62 train\nBACKGROUND_Google/image_0074.jpg 62 train\nBACKGROUND_Google/image_0294.jpg 62 train\nBACKGROUND_Google/image_0440.jpg 62 train\nBACKGROUND_Google/image_0391.jpg 62 train\nBACKGROUND_Google/image_0189.jpg 62 train\nBACKGROUND_Google/image_0003.jpg 62 train\nBACKGROUND_Google/image_0009.jpg 62 train\nBACKGROUND_Google/image_0387.jpg 62 train\nBACKGROUND_Google/image_0275.jpg 62 train\nBACKGROUND_Google/image_0226.jpg 62 train\nBACKGROUND_Google/image_0185.jpg 62 train\nBACKGROUND_Google/image_0118.jpg 62 train\nBACKGROUND_Google/image_0304.jpg 62 train\nBACKGROUND_Google/image_0278.jpg 62 train\nBACKGROUND_Google/image_0230.jpg 62 train\nBACKGROUND_Google/image_0404.jpg 62 train\nBACKGROUND_Google/tmp 62 train\nBACKGROUND_Google/image_0460.jpg 62 train\nBACKGROUND_Google/image_0134.jpg 62 train\nBACKGROUND_Google/image_0249.jpg 62 train\nBACKGROUND_Google/image_0042.jpg 62 train\nBACKGROUND_Google/image_0352.jpg 62 train\nBACKGROUND_Google/image_0333.jpg 62 train\nBACKGROUND_Google/image_0221.jpg 62 train\nBACKGROUND_Google/image_0093.jpg 62 train\nBACKGROUND_Google/image_0266.jpg 62 train\nBACKGROUND_Google/image_0412.jpg 62 train\nBACKGROUND_Google/image_0301.jpg 62 train\nBACKGROUND_Google/image_0392.jpg 62 train\nBACKGROUND_Google/image_0254.jpg 62 train\nBACKGROUND_Google/image_0321.jpg 62 train\nBACKGROUND_Google/image_0386.jpg 62 train\nBACKGROUND_Google/image_0232.jpg 62 train\nBACKGROUND_Google/image_0324.jpg 62 train\nBACKGROUND_Google/image_0040.jpg 62 train\nBACKGROUND_Google/image_0067.jpg 62 train\nBACKGROUND_Google/image_0310.jpg 62 train\nBACKGROUND_Google/image_0330.jpg 62 train\nBACKGROUND_Google/image_0122.jpg 62 train\nBACKGROUND_Google/image_0205.jpg 62 train\nBACKGROUND_Google/image_0161.jpg 62 train\nBACKGROUND_Google/image_0146.jpg 62 train\nBACKGROUND_Google/image_0300.jpg 62 train\nBACKGROUND_Google/image_0369.jpg 62 train\nBACKGROUND_Google/image_0451.jpg 62 train\nBACKGROUND_Google/image_0461.jpg 62 train\nBACKGROUND_Google/image_0204.jpg 62 train\nBACKGROUND_Google/image_0125.jpg 62 train\nBACKGROUND_Google/image_0047.jpg 62 train\nBACKGROUND_Google/image_0285.jpg 62 train\nBACKGROUND_Google/image_0160.jpg 62 train\nBACKGROUND_Google/image_0422.jpg 62 train\nBACKGROUND_Google/image_0064.jpg 62 train\nBACKGROUND_Google/image_0070.jpg 62 train\nBACKGROUND_Google/image_0264.jpg 62 train\nBACKGROUND_Google/image_0112.jpg 62 train\nBACKGROUND_Google/image_0078.jpg 62 train\nBACKGROUND_Google/image_0068.jpg 62 train\nBACKGROUND_Google/image_0433.jpg 62 train\nBACKGROUND_Google/image_0085.jpg 62 train\nBACKGROUND_Google/image_0410.jpg 62 train\nBACKGROUND_Google/image_0110.jpg 62 train\nBACKGROUND_Google/image_0259.jpg 62 train\nBACKGROUND_Google/image_0143.jpg 62 train\nBACKGROUND_Google/image_0043.jpg 62 train\nBACKGROUND_Google/image_0162.jpg 62 train\nBACKGROUND_Google/image_0132.jpg 62 train\nBACKGROUND_Google/image_0157.jpg 62 train\nBACKGROUND_Google/image_0206.jpg 62 train\nBACKGROUND_Google/image_0376.jpg 62 train\nBACKGROUND_Google/image_0319.jpg 62 train\nBACKGROUND_Google/image_0371.jpg 62 train\nBACKGROUND_Google/image_0044.jpg 62 train\nBACKGROUND_Google/image_0199.jpg 62 train\nBACKGROUND_Google/image_0033.jpg 62 train\nBACKGROUND_Google/image_0136.jpg 62 train\nBACKGROUND_Google/image_0180.jpg 62 train\nBACKGROUND_Google/image_0260.jpg 62 train\nBACKGROUND_Google/image_0184.jpg 62 train\nBACKGROUND_Google/image_0073.jpg 62 train\nBACKGROUND_Google/image_0395.jpg 62 train\nBACKGROUND_Google/image_0257.jpg 62 train\nBACKGROUND_Google/image_0364.jpg 62 train\nBACKGROUND_Google/image_0001.jpg 62 train\nBACKGROUND_Google/image_0167.jpg 62 train\nBACKGROUND_Google/image_0284.jpg 62 train\nBACKGROUND_Google/image_0375.jpg 62 train\nBACKGROUND_Google/image_0080.jpg 62 train\nBACKGROUND_Google/image_0156.jpg 62 train\nBACKGROUND_Google/image_0021.jpg 62 train\nBACKGROUND_Google/image_0325.jpg 62 train\nBACKGROUND_Google/image_0267.jpg 62 train\nBACKGROUND_Google/image_0299.jpg 62 train\nBACKGROUND_Google/image_0421.jpg 62 train\nBACKGROUND_Google/image_0436.jpg 62 train\nBACKGROUND_Google/image_0360.jpg 62 train\nBACKGROUND_Google/image_0095.jpg 62 train\nBACKGROUND_Google/image_0057.jpg 62 train\nBACKGROUND_Google/image_0403.jpg 62 train\nBACKGROUND_Google/image_0114.jpg 62 train\nBACKGROUND_Google/image_0153.jpg 62 train\nBACKGROUND_Google/image_0416.jpg 62 train\nBACKGROUND_Google/image_0098.jpg 62 train\nBACKGROUND_Google/image_0256.jpg 62 train\nBACKGROUND_Google/image_0402.jpg 62 train\nBACKGROUND_Google/image_0203.jpg 62 train\nBACKGROUND_Google/image_0338.jpg 62 train\nBACKGROUND_Google/image_0265.jpg 62 train\nBACKGROUND_Google/image_0147.jpg 62 train\nBACKGROUND_Google/image_0008.jpg 62 train\nBACKGROUND_Google/image_0446.jpg 62 train\nBACKGROUND_Google/image_0207.jpg 62 train\nBACKGROUND_Google/image_0208.jpg 62 train\nBACKGROUND_Google/image_0222.jpg 62 train\nBACKGROUND_Google/image_0320.jpg 62 train\nBACKGROUND_Google/image_0227.jpg 62 train\nBACKGROUND_Google/image_0286.jpg 62 train\nBACKGROUND_Google/image_0035.jpg 62 train\nBACKGROUND_Google/image_0399.jpg 62 train\nBACKGROUND_Google/image_0166.jpg 62 train\nBACKGROUND_Google/image_0169.jpg 62 train\nBACKGROUND_Google/image_0191.jpg 62 train\nBACKGROUND_Google/image_0128.jpg 62 train\nBACKGROUND_Google/image_0414.jpg 62 train\nBACKGROUND_Google/image_0102.jpg 62 train\nBACKGROUND_Google/image_0434.jpg 62 train\nBACKGROUND_Google/image_0178.jpg 62 train\nBACKGROUND_Google/image_0431.jpg 62 train\nBACKGROUND_Google/image_0243.jpg 62 train\nBACKGROUND_Google/image_0005.jpg 62 train\nBACKGROUND_Google/image_0263.jpg 62 train\nBACKGROUND_Google/image_0173.jpg 62 train\nBACKGROUND_Google/image_0087.jpg 62 train\nBACKGROUND_Google/image_0417.jpg 62 train\nBACKGROUND_Google/image_0089.jpg 62 train\nBACKGROUND_Google/image_0059.jpg 62 train\nBACKGROUND_Google/image_0253.jpg 62 train\nBACKGROUND_Google/image_0159.jpg 62 train\nBACKGROUND_Google/image_0225.jpg 62 train\nBACKGROUND_Google/image_0445.jpg 62 train\nBACKGROUND_Google/image_0186.jpg 62 train\nBACKGROUND_Google/image_0388.jpg 62 train\nBACKGROUND_Google/image_0069.jpg 62 train\nBACKGROUND_Google/image_0315.jpg 62 train\nBACKGROUND_Google/image_0295.jpg 62 train\nBACKGROUND_Google/image_0200.jpg 62 train\nBACKGROUND_Google/image_0099.jpg 62 train\nBACKGROUND_Google/image_0336.jpg 62 train\nBACKGROUND_Google/image_0034.jpg 62 train\nBACKGROUND_Google/image_0409.jpg 62 train\nBACKGROUND_Google/image_0314.jpg 62 train\nBACKGROUND_Google/image_0276.jpg 62 train\nBACKGROUND_Google/image_0121.jpg 62 train\nBACKGROUND_Google/image_0438.jpg 62 train\nBACKGROUND_Google/image_0428.jpg 62 train\nBACKGROUND_Google/image_0181.jpg 62 train\nBACKGROUND_Google/image_0280.jpg 62 train\nBACKGROUND_Google/image_0183.jpg 62 train\nBACKGROUND_Google/image_0383.jpg 62 train\nBACKGROUND_Google/image_0175.jpg 62 train\nBACKGROUND_Google/image_0002.jpg 62 train\nBACKGROUND_Google/image_0291.jpg 62 train\nBACKGROUND_Google/image_0165.jpg 62 train\nBACKGROUND_Google/image_0154.jpg 62 train\nBACKGROUND_Google/image_0271.jpg 62 train\nBACKGROUND_Google/image_0193.jpg 62 train\nBACKGROUND_Google/image_0048.jpg 62 train\nBACKGROUND_Google/image_0437.jpg 62 train\nBACKGROUND_Google/image_0053.jpg 62 train\nBACKGROUND_Google/image_0255.jpg 62 train\nBACKGROUND_Google/image_0450.jpg 62 train\nBACKGROUND_Google/image_0084.jpg 62 train\nBACKGROUND_Google/image_0313.jpg 62 train\nBACKGROUND_Google/image_0020.jpg 62 train\nBACKGROUND_Google/image_0251.jpg 62 train\nBACKGROUND_Google/image_0046.jpg 62 train\nBACKGROUND_Google/image_0041.jpg 62 train\nBACKGROUND_Google/image_0027.jpg 62 train\nBACKGROUND_Google/image_0322.jpg 62 train\nBACKGROUND_Google/image_0458.jpg 62 train\nBACKGROUND_Google/image_0014.jpg 62 train\nBACKGROUND_Google/image_0058.jpg 62 train\nBACKGROUND_Google/image_0408.jpg 62 train\nBACKGROUND_Google/image_0329.jpg 62 train\nBACKGROUND_Google/image_0212.jpg 62 train\nBACKGROUND_Google/image_0163.jpg 62 train\nBACKGROUND_Google/image_0202.jpg 62 train\nBACKGROUND_Google/image_0104.jpg 62 train\nBACKGROUND_Google/image_0081.jpg 62 train\nBACKGROUND_Google/image_0272.jpg 62 train\nBACKGROUND_Google/image_0400.jpg 62 train\nBACKGROUND_Google/image_0237.jpg 62 train\nBACKGROUND_Google/image_0308.jpg 62 train\nBACKGROUND_Google/image_0149.jpg 62 train\nBACKGROUND_Google/image_0088.jpg 62 train\nBACKGROUND_Google/image_0309.jpg 62 train\nBACKGROUND_Google/image_0026.jpg 62 train\nBACKGROUND_Google/image_0231.jpg 62 train\nBACKGROUND_Google/image_0004.jpg 62 train\nBACKGROUND_Google/image_0015.jpg 62 train\nBACKGROUND_Google/image_0355.jpg 62 train\nmandolin/image_0037.jpg 63 train\nmandolin/image_0006.jpg 63 train\nmandolin/image_0016.jpg 63 train\nmandolin/image_0030.jpg 63 train\nmandolin/image_0012.jpg 63 train\nmandolin/image_0025.jpg 63 train\nmandolin/image_0007.jpg 63 train\nmandolin/image_0019.jpg 63 train\nmandolin/image_0031.jpg 63 train\nmandolin/image_0011.jpg 63 train\nmandolin/image_0017.jpg 63 train\nmandolin/image_0018.jpg 63 train\nmandolin/image_0013.jpg 63 train\nmandolin/image_0024.jpg 63 train\nmandolin/image_0003.jpg 63 train\nmandolin/image_0009.jpg 63 train\nmandolin/image_0042.jpg 63 train\nmandolin/image_0040.jpg 63 train\nmandolin/image_0043.jpg 63 train\nmandolin/image_0033.jpg 63 train\nmandolin/image_0001.jpg 63 train\nmandolin/image_0021.jpg 63 train\nmandolin/image_0008.jpg 63 train\nmandolin/image_0035.jpg 63 train\nmandolin/image_0005.jpg 63 train\nmandolin/image_0034.jpg 63 train\nmandolin/image_0002.jpg 63 train\nmandolin/image_0020.jpg 63 train\nmandolin/image_0041.jpg 63 train\nmandolin/image_0027.jpg 63 train\nmandolin/image_0014.jpg 63 train\nmandolin/image_0026.jpg 63 train\nmandolin/image_0004.jpg 63 train\nmandolin/image_0015.jpg 63 train\nMotorbikes/image_0316.jpg 64 train\nMotorbikes/image_0716.jpg 64 train\nMotorbikes/image_0258.jpg 64 train\nMotorbikes/image_0426.jpg 64 train\nMotorbikes/image_0511.jpg 64 train\nMotorbikes/image_0676.jpg 64 train\nMotorbikes/image_0528.jpg 64 train\nMotorbikes/image_0720.jpg 64 train\nMotorbikes/image_0556.jpg 64 train\nMotorbikes/image_0302.jpg 64 train\nMotorbikes/image_0171.jpg 64 train\nMotorbikes/image_0062.jpg 64 train\nMotorbikes/image_0504.jpg 64 train\nMotorbikes/image_0346.jpg 64 train\nMotorbikes/image_0546.jpg 64 train\nMotorbikes/image_0435.jpg 64 train\nMotorbikes/image_0341.jpg 64 train\nMotorbikes/image_0776.jpg 64 train\nMotorbikes/image_0340.jpg 64 train\nMotorbikes/image_0331.jpg 64 train\nMotorbikes/image_0037.jpg 64 train\nMotorbikes/image_0737.jpg 64 train\nMotorbikes/image_0786.jpg 64 train\nMotorbikes/image_0560.jpg 64 train\nMotorbikes/image_0472.jpg 64 train\nMotorbikes/image_0049.jpg 64 train\nMotorbikes/image_0337.jpg 64 train\nMotorbikes/image_0779.jpg 64 train\nMotorbikes/image_0317.jpg 64 train\nMotorbikes/image_0357.jpg 64 train\nMotorbikes/image_0083.jpg 64 train\nMotorbikes/image_0101.jpg 64 train\nMotorbikes/image_0051.jpg 64 train\nMotorbikes/image_0006.jpg 64 train\nMotorbikes/image_0142.jpg 64 train\nMotorbikes/image_0681.jpg 64 train\nMotorbikes/image_0250.jpg 64 train\nMotorbikes/image_0427.jpg 64 train\nMotorbikes/image_0486.jpg 64 train\nMotorbikes/image_0396.jpg 64 train\nMotorbikes/image_0733.jpg 64 train\nMotorbikes/image_0332.jpg 64 train\nMotorbikes/image_0448.jpg 64 train\nMotorbikes/image_0335.jpg 64 train\nMotorbikes/image_0752.jpg 64 train\nMotorbikes/image_0201.jpg 64 train\nMotorbikes/image_0505.jpg 64 train\nMotorbikes/image_0016.jpg 64 train\nMotorbikes/image_0373.jpg 64 train\nMotorbikes/image_0772.jpg 64 train\nMotorbikes/image_0565.jpg 64 train\nMotorbikes/image_0513.jpg 64 train\nMotorbikes/image_0709.jpg 64 train\nMotorbikes/image_0176.jpg 64 train\nMotorbikes/image_0411.jpg 64 train\nMotorbikes/image_0155.jpg 64 train\nMotorbikes/image_0030.jpg 64 train\nMotorbikes/image_0091.jpg 64 train\nMotorbikes/image_0664.jpg 64 train\nMotorbikes/image_0563.jpg 64 train\nMotorbikes/image_0397.jpg 64 train\nMotorbikes/image_0618.jpg 64 train\nMotorbikes/image_0012.jpg 64 train\nMotorbikes/image_0066.jpg 64 train\nMotorbikes/image_0657.jpg 64 train\nMotorbikes/image_0765.jpg 64 train\nMotorbikes/image_0585.jpg 64 train\nMotorbikes/image_0612.jpg 64 train\nMotorbikes/image_0544.jpg 64 train\nMotorbikes/image_0703.jpg 64 train\nMotorbikes/image_0465.jpg 64 train\nMotorbikes/image_0728.jpg 64 train\nMotorbikes/image_0464.jpg 64 train\nMotorbikes/image_0283.jpg 64 train\nMotorbikes/image_0551.jpg 64 train\nMotorbikes/image_0318.jpg 64 train\nMotorbikes/image_0224.jpg 64 train\nMotorbikes/image_0771.jpg 64 train\nMotorbikes/image_0148.jpg 64 train\nMotorbikes/image_0501.jpg 64 train\nMotorbikes/image_0496.jpg 64 train\nMotorbikes/image_0077.jpg 64 train\nMotorbikes/image_0129.jpg 64 train\nMotorbikes/image_0339.jpg 64 train\nMotorbikes/image_0554.jpg 64 train\nMotorbikes/image_0025.jpg 64 train\nMotorbikes/image_0328.jpg 64 train\nMotorbikes/image_0549.jpg 64 train\nMotorbikes/image_0659.jpg 64 train\nMotorbikes/image_0540.jpg 64 train\nMotorbikes/image_0518.jpg 64 train\nMotorbikes/image_0479.jpg 64 train\nMotorbikes/image_0539.jpg 64 train\nMotorbikes/image_0527.jpg 64 train\nMotorbikes/image_0783.jpg 64 train\nMotorbikes/image_0621.jpg 64 train\nMotorbikes/image_0476.jpg 64 train\nMotorbikes/image_0139.jpg 64 train\nMotorbikes/image_0519.jpg 64 train\nMotorbikes/image_0425.jpg 64 train\nMotorbikes/image_0292.jpg 64 train\nMotorbikes/image_0348.jpg 64 train\nMotorbikes/image_0499.jpg 64 train\nMotorbikes/image_0115.jpg 64 train\nMotorbikes/image_0123.jpg 64 train\nMotorbikes/image_0119.jpg 64 train\nMotorbikes/image_0670.jpg 64 train\nMotorbikes/image_0721.jpg 64 train\nMotorbikes/image_0639.jpg 64 train\nMotorbikes/image_0060.jpg 64 train\nMotorbikes/image_0269.jpg 64 train\nMotorbikes/image_0052.jpg 64 train\nMotorbikes/image_0108.jpg 64 train\nMotorbikes/image_0239.jpg 64 train\nMotorbikes/image_0442.jpg 64 train\nMotorbikes/image_0684.jpg 64 train\nMotorbikes/image_0370.jpg 64 train\nMotorbikes/image_0491.jpg 64 train\nMotorbikes/image_0469.jpg 64 train\nMotorbikes/image_0326.jpg 64 train\nMotorbikes/image_0223.jpg 64 train\nMotorbikes/image_0172.jpg 64 train\nMotorbikes/image_0372.jpg 64 train\nMotorbikes/image_0704.jpg 64 train\nMotorbikes/image_0344.jpg 64 train\nMotorbikes/image_0390.jpg 64 train\nMotorbikes/image_0007.jpg 64 train\nMotorbikes/image_0213.jpg 64 train\nMotorbikes/image_0739.jpg 64 train\nMotorbikes/image_0312.jpg 64 train\nMotorbikes/image_0463.jpg 64 train\nMotorbikes/image_0063.jpg 64 train\nMotorbikes/image_0019.jpg 64 train\nMotorbikes/image_0407.jpg 64 train\nMotorbikes/image_0192.jpg 64 train\nMotorbikes/image_0490.jpg 64 train\nMotorbikes/image_0533.jpg 64 train\nMotorbikes/image_0743.jpg 64 train\nMotorbikes/image_0384.jpg 64 train\nMotorbikes/image_0195.jpg 64 train\nMotorbikes/image_0113.jpg 64 train\nMotorbikes/image_0677.jpg 64 train\nMotorbikes/image_0127.jpg 64 train\nMotorbikes/image_0215.jpg 64 train\nMotorbikes/image_0389.jpg 64 train\nMotorbikes/image_0031.jpg 64 train\nMotorbikes/image_0349.jpg 64 train\nMotorbikes/image_0228.jpg 64 train\nMotorbikes/image_0380.jpg 64 train\nMotorbikes/image_0219.jpg 64 train\nMotorbikes/image_0796.jpg 64 train\nMotorbikes/image_0688.jpg 64 train\nMotorbikes/image_0597.jpg 64 train\nMotorbikes/image_0353.jpg 64 train\nMotorbikes/image_0011.jpg 64 train\nMotorbikes/image_0562.jpg 64 train\nMotorbikes/image_0233.jpg 64 train\nMotorbikes/image_0447.jpg 64 train\nMotorbikes/image_0747.jpg 64 train\nMotorbikes/image_0477.jpg 64 train\nMotorbikes/image_0680.jpg 64 train\nMotorbikes/image_0061.jpg 64 train\nMotorbikes/image_0444.jpg 64 train\nMotorbikes/image_0090.jpg 64 train\nMotorbikes/image_0745.jpg 64 train\nMotorbikes/image_0351.jpg 64 train\nMotorbikes/image_0261.jpg 64 train\nMotorbikes/image_0229.jpg 64 train\nMotorbikes/image_0126.jpg 64 train\nMotorbikes/image_0761.jpg 64 train\nMotorbikes/image_0775.jpg 64 train\nMotorbikes/image_0794.jpg 64 train\nMotorbikes/image_0240.jpg 64 train\nMotorbikes/image_0467.jpg 64 train\nMotorbikes/image_0768.jpg 64 train\nMotorbikes/image_0455.jpg 64 train\nMotorbikes/image_0441.jpg 64 train\nMotorbikes/image_0305.jpg 64 train\nMotorbikes/image_0572.jpg 64 train\nMotorbikes/image_0334.jpg 64 train\nMotorbikes/image_0602.jpg 64 train\nMotorbikes/image_0354.jpg 64 train\nMotorbikes/image_0542.jpg 64 train\nMotorbikes/image_0218.jpg 64 train\nMotorbikes/image_0573.jpg 64 train\nMotorbikes/image_0366.jpg 64 train\nMotorbikes/image_0288.jpg 64 train\nMotorbikes/image_0583.jpg 64 train\nMotorbikes/image_0216.jpg 64 train\nMotorbikes/image_0459.jpg 64 train\nMotorbikes/image_0429.jpg 64 train\nMotorbikes/image_0277.jpg 64 train\nMotorbikes/image_0623.jpg 64 train\nMotorbikes/image_0358.jpg 64 train\nMotorbikes/image_0588.jpg 64 train\nMotorbikes/image_0050.jpg 64 train\nMotorbikes/image_0246.jpg 64 train\nMotorbikes/image_0361.jpg 64 train\nMotorbikes/image_0177.jpg 64 train\nMotorbikes/image_0356.jpg 64 train\nMotorbikes/image_0648.jpg 64 train\nMotorbikes/image_0789.jpg 64 train\nMotorbikes/image_0196.jpg 64 train\nMotorbikes/image_0293.jpg 64 train\nMotorbikes/image_0406.jpg 64 train\nMotorbikes/image_0596.jpg 64 train\nMotorbikes/image_0640.jpg 64 train\nMotorbikes/image_0575.jpg 64 train\nMotorbikes/image_0625.jpg 64 train\nMotorbikes/image_0347.jpg 64 train\nMotorbikes/image_0521.jpg 64 train\nMotorbikes/image_0443.jpg 64 train\nMotorbikes/image_0187.jpg 64 train\nMotorbikes/image_0558.jpg 64 train\nMotorbikes/image_0234.jpg 64 train\nMotorbikes/image_0045.jpg 64 train\nMotorbikes/image_0303.jpg 64 train\nMotorbikes/image_0363.jpg 64 train\nMotorbikes/image_0679.jpg 64 train\nMotorbikes/image_0017.jpg 64 train\nMotorbikes/image_0131.jpg 64 train\nMotorbikes/image_0762.jpg 64 train\nMotorbikes/image_0492.jpg 64 train\nMotorbikes/image_0296.jpg 64 train\nMotorbikes/image_0555.jpg 64 train\nMotorbikes/image_0735.jpg 64 train\nMotorbikes/image_0150.jpg 64 train\nMotorbikes/image_0018.jpg 64 train\nMotorbikes/image_0350.jpg 64 train\nMotorbikes/image_0795.jpg 64 train\nMotorbikes/image_0209.jpg 64 train\nMotorbikes/image_0013.jpg 64 train\nMotorbikes/image_0570.jpg 64 train\nMotorbikes/image_0748.jpg 64 train\nMotorbikes/image_0569.jpg 64 train\nMotorbikes/image_0696.jpg 64 train\nMotorbikes/image_0614.jpg 64 train\nMotorbikes/image_0054.jpg 64 train\nMotorbikes/image_0629.jpg 64 train\nMotorbikes/image_0094.jpg 64 train\nMotorbikes/image_0613.jpg 64 train\nMotorbikes/image_0082.jpg 64 train\nMotorbikes/image_0097.jpg 64 train\nMotorbikes/image_0130.jpg 64 train\nMotorbikes/image_0457.jpg 64 train\nMotorbikes/image_0488.jpg 64 train\nMotorbikes/image_0382.jpg 64 train\nMotorbikes/image_0484.jpg 64 train\nMotorbikes/image_0141.jpg 64 train\nMotorbikes/image_0700.jpg 64 train\nMotorbikes/image_0144.jpg 64 train\nMotorbikes/image_0109.jpg 64 train\nMotorbikes/image_0645.jpg 64 train\nMotorbikes/image_0468.jpg 64 train\nMotorbikes/image_0515.jpg 64 train\nMotorbikes/image_0079.jpg 64 train\nMotorbikes/image_0306.jpg 64 train\nMotorbikes/image_0398.jpg 64 train\nMotorbikes/image_0055.jpg 64 train\nMotorbikes/image_0367.jpg 64 train\nMotorbikes/image_0365.jpg 64 train\nMotorbikes/image_0566.jpg 64 train\nMotorbikes/image_0217.jpg 64 train\nMotorbikes/image_0343.jpg 64 train\nMotorbikes/image_0432.jpg 64 train\nMotorbikes/image_0538.jpg 64 train\nMotorbikes/image_0600.jpg 64 train\nMotorbikes/image_0502.jpg 64 train\nMotorbikes/image_0661.jpg 64 train\nMotorbikes/image_0731.jpg 64 train\nMotorbikes/image_0071.jpg 64 train\nMotorbikes/image_0590.jpg 64 train\nMotorbikes/image_0282.jpg 64 train\nMotorbikes/image_0393.jpg 64 train\nMotorbikes/image_0287.jpg 64 train\nMotorbikes/image_0198.jpg 64 train\nMotorbikes/image_0262.jpg 64 train\nMotorbikes/image_0686.jpg 64 train\nMotorbikes/image_0462.jpg 64 train\nMotorbikes/image_0698.jpg 64 train\nMotorbikes/image_0168.jpg 64 train\nMotorbikes/image_0368.jpg 64 train\nMotorbikes/image_0420.jpg 64 train\nMotorbikes/image_0650.jpg 64 train\nMotorbikes/image_0763.jpg 64 train\nMotorbikes/image_0211.jpg 64 train\nMotorbikes/image_0537.jpg 64 train\nMotorbikes/image_0580.jpg 64 train\nMotorbikes/image_0627.jpg 64 train\nMotorbikes/image_0270.jpg 64 train\nMotorbikes/image_0594.jpg 64 train\nMotorbikes/image_0103.jpg 64 train\nMotorbikes/image_0643.jpg 64 train\nMotorbikes/image_0591.jpg 64 train\nMotorbikes/image_0024.jpg 64 train\nMotorbikes/image_0787.jpg 64 train\nMotorbikes/image_0297.jpg 64 train\nMotorbikes/image_0188.jpg 64 train\nMotorbikes/image_0792.jpg 64 train\nMotorbikes/image_0105.jpg 64 train\nMotorbikes/image_0074.jpg 64 train\nMotorbikes/image_0294.jpg 64 train\nMotorbikes/image_0440.jpg 64 train\nMotorbikes/image_0391.jpg 64 train\nMotorbikes/image_0189.jpg 64 train\nMotorbikes/image_0003.jpg 64 train\nMotorbikes/image_0009.jpg 64 train\nMotorbikes/image_0387.jpg 64 train\nMotorbikes/image_0275.jpg 64 train\nMotorbikes/image_0226.jpg 64 train\nMotorbikes/image_0628.jpg 64 train\nMotorbikes/image_0718.jpg 64 train\nMotorbikes/image_0714.jpg 64 train\nMotorbikes/image_0658.jpg 64 train\nMotorbikes/image_0601.jpg 64 train\nMotorbikes/image_0185.jpg 64 train\nMotorbikes/image_0790.jpg 64 train\nMotorbikes/image_0656.jpg 64 train\nMotorbikes/image_0567.jpg 64 train\nMotorbikes/image_0485.jpg 64 train\nMotorbikes/image_0541.jpg 64 train\nMotorbikes/image_0118.jpg 64 train\nMotorbikes/image_0652.jpg 64 train\nMotorbikes/image_0304.jpg 64 train\nMotorbikes/image_0278.jpg 64 train\nMotorbikes/image_0512.jpg 64 train\nMotorbikes/image_0230.jpg 64 train\nMotorbikes/image_0474.jpg 64 train\nMotorbikes/image_0553.jpg 64 train\nMotorbikes/image_0404.jpg 64 train\nMotorbikes/image_0487.jpg 64 train\nMotorbikes/image_0531.jpg 64 train\nMotorbikes/image_0638.jpg 64 train\nMotorbikes/image_0525.jpg 64 train\nMotorbikes/image_0482.jpg 64 train\nMotorbikes/image_0522.jpg 64 train\nMotorbikes/image_0460.jpg 64 train\nMotorbikes/image_0608.jpg 64 train\nMotorbikes/image_0134.jpg 64 train\nMotorbikes/image_0249.jpg 64 train\nMotorbikes/image_0042.jpg 64 train\nMotorbikes/image_0730.jpg 64 train\nMotorbikes/image_0352.jpg 64 train\nMotorbikes/image_0333.jpg 64 train\nMotorbikes/image_0221.jpg 64 train\nMotorbikes/image_0093.jpg 64 train\nMotorbikes/image_0266.jpg 64 train\nMotorbikes/image_0412.jpg 64 train\nMotorbikes/image_0759.jpg 64 train\nMotorbikes/image_0470.jpg 64 train\nMotorbikes/image_0497.jpg 64 train\nMotorbikes/image_0751.jpg 64 train\nMotorbikes/image_0301.jpg 64 train\nMotorbikes/image_0392.jpg 64 train\nMotorbikes/image_0473.jpg 64 train\nMotorbikes/image_0254.jpg 64 train\nMotorbikes/image_0561.jpg 64 train\nMotorbikes/image_0530.jpg 64 train\nMotorbikes/image_0321.jpg 64 train\nMotorbikes/image_0557.jpg 64 train\nMotorbikes/image_0386.jpg 64 train\nMotorbikes/image_0232.jpg 64 train\nMotorbikes/image_0324.jpg 64 train\nMotorbikes/image_0040.jpg 64 train\nMotorbikes/image_0067.jpg 64 train\nMotorbikes/image_0769.jpg 64 train\nMotorbikes/image_0310.jpg 64 train\nMotorbikes/image_0330.jpg 64 train\nMotorbikes/image_0548.jpg 64 train\nMotorbikes/image_0122.jpg 64 train\nMotorbikes/image_0671.jpg 64 train\nMotorbikes/image_0205.jpg 64 train\nMotorbikes/image_0161.jpg 64 train\nMotorbikes/image_0146.jpg 64 train\nMotorbikes/image_0300.jpg 64 train\nMotorbikes/image_0369.jpg 64 train\nMotorbikes/image_0451.jpg 64 train\nMotorbikes/image_0461.jpg 64 train\nMotorbikes/image_0204.jpg 64 train\nMotorbikes/image_0125.jpg 64 train\nMotorbikes/image_0047.jpg 64 train\nMotorbikes/image_0285.jpg 64 train\nMotorbikes/image_0160.jpg 64 train\nMotorbikes/image_0678.jpg 64 train\nMotorbikes/image_0422.jpg 64 train\nMotorbikes/image_0579.jpg 64 train\nMotorbikes/image_0064.jpg 64 train\nMotorbikes/image_0070.jpg 64 train\nMotorbikes/image_0552.jpg 64 train\nMotorbikes/image_0264.jpg 64 train\nMotorbikes/image_0112.jpg 64 train\nMotorbikes/image_0078.jpg 64 train\nMotorbikes/image_0637.jpg 64 train\nMotorbikes/image_0068.jpg 64 train\nMotorbikes/image_0506.jpg 64 train\nMotorbikes/image_0433.jpg 64 train\nMotorbikes/image_0547.jpg 64 train\nMotorbikes/image_0085.jpg 64 train\nMotorbikes/image_0410.jpg 64 train\nMotorbikes/image_0110.jpg 64 train\nMotorbikes/image_0722.jpg 64 train\nMotorbikes/image_0593.jpg 64 train\nMotorbikes/image_0259.jpg 64 train\nMotorbikes/image_0143.jpg 64 train\nMotorbikes/image_0043.jpg 64 train\nMotorbikes/image_0162.jpg 64 train\nMotorbikes/image_0132.jpg 64 train\nMotorbikes/image_0157.jpg 64 train\nMotorbikes/image_0475.jpg 64 train\nMotorbikes/image_0206.jpg 64 train\nMotorbikes/image_0376.jpg 64 train\nMotorbikes/image_0644.jpg 64 train\nMotorbikes/image_0319.jpg 64 train\nMotorbikes/image_0371.jpg 64 train\nMotorbikes/image_0044.jpg 64 train\nMotorbikes/image_0199.jpg 64 train\nMotorbikes/image_0729.jpg 64 train\nMotorbikes/image_0691.jpg 64 train\nMotorbikes/image_0033.jpg 64 train\nMotorbikes/image_0136.jpg 64 train\nMotorbikes/image_0180.jpg 64 train\nMotorbikes/image_0260.jpg 64 train\nMotorbikes/image_0604.jpg 64 train\nMotorbikes/image_0184.jpg 64 train\nMotorbikes/image_0719.jpg 64 train\nMotorbikes/image_0672.jpg 64 train\nMotorbikes/image_0784.jpg 64 train\nMotorbikes/image_0073.jpg 64 train\nMotorbikes/image_0395.jpg 64 train\nMotorbikes/image_0257.jpg 64 train\nMotorbikes/image_0364.jpg 64 train\nMotorbikes/image_0756.jpg 64 train\nMotorbikes/image_0001.jpg 64 train\nMotorbikes/image_0167.jpg 64 train\nMotorbikes/image_0284.jpg 64 train\nMotorbikes/image_0375.jpg 64 train\nMotorbikes/image_0080.jpg 64 train\nMotorbikes/image_0665.jpg 64 train\nMotorbikes/image_0156.jpg 64 train\nMotorbikes/image_0021.jpg 64 train\nMotorbikes/image_0571.jpg 64 train\nMotorbikes/image_0325.jpg 64 train\nMotorbikes/image_0517.jpg 64 train\nMotorbikes/image_0713.jpg 64 train\nMotorbikes/image_0516.jpg 64 train\nMotorbikes/image_0574.jpg 64 train\nMotorbikes/image_0471.jpg 64 train\nMotorbikes/image_0767.jpg 64 train\nMotorbikes/image_0581.jpg 64 train\nMotorbikes/image_0267.jpg 64 train\nMotorbikes/image_0299.jpg 64 train\nMotorbikes/image_0421.jpg 64 train\nMotorbikes/image_0495.jpg 64 train\nMotorbikes/image_0529.jpg 64 train\nMotorbikes/image_0436.jpg 64 train\nMotorbikes/image_0360.jpg 64 train\nMotorbikes/image_0649.jpg 64 train\nMotorbikes/image_0095.jpg 64 train\nMotorbikes/image_0057.jpg 64 train\nMotorbikes/image_0707.jpg 64 train\nMotorbikes/image_0797.jpg 64 train\nMotorbikes/image_0403.jpg 64 train\nMotorbikes/image_0114.jpg 64 train\nMotorbikes/image_0641.jpg 64 train\nMotorbikes/image_0595.jpg 64 train\nMotorbikes/image_0620.jpg 64 train\nMotorbikes/image_0153.jpg 64 train\nMotorbikes/image_0697.jpg 64 train\nMotorbikes/image_0654.jpg 64 train\nMotorbikes/image_0564.jpg 64 train\nMotorbikes/image_0416.jpg 64 train\nMotorbikes/image_0607.jpg 64 train\nMotorbikes/image_0098.jpg 64 train\nMotorbikes/image_0256.jpg 64 train\nMotorbikes/image_0402.jpg 64 train\nMotorbikes/image_0788.jpg 64 train\nMotorbikes/image_0203.jpg 64 train\nMotorbikes/image_0577.jpg 64 train\nMotorbikes/image_0764.jpg 64 train\nMotorbikes/image_0338.jpg 64 train\nMotorbikes/image_0582.jpg 64 train\nMotorbikes/image_0265.jpg 64 train\nMotorbikes/image_0147.jpg 64 train\nMotorbikes/image_0536.jpg 64 train\nMotorbikes/image_0008.jpg 64 train\nMotorbikes/image_0446.jpg 64 train\nMotorbikes/image_0514.jpg 64 train\nMotorbikes/image_0617.jpg 64 train\nMotorbikes/image_0619.jpg 64 train\nMotorbikes/image_0727.jpg 64 train\nMotorbikes/image_0589.jpg 64 train\nMotorbikes/image_0207.jpg 64 train\nMotorbikes/image_0208.jpg 64 train\nMotorbikes/image_0222.jpg 64 train\nMotorbikes/image_0320.jpg 64 train\nMotorbikes/image_0781.jpg 64 train\nMotorbikes/image_0532.jpg 64 train\nMotorbikes/image_0227.jpg 64 train\nMotorbikes/image_0651.jpg 64 train\nMotorbikes/image_0286.jpg 64 train\nMotorbikes/image_0035.jpg 64 train\nMotorbikes/image_0399.jpg 64 train\nMotorbikes/image_0166.jpg 64 train\nMotorbikes/image_0711.jpg 64 train\nMotorbikes/image_0669.jpg 64 train\nMotorbikes/image_0169.jpg 64 train\nMotorbikes/image_0191.jpg 64 train\nMotorbikes/image_0128.jpg 64 train\nMotorbikes/image_0493.jpg 64 train\nMotorbikes/image_0414.jpg 64 train\nMotorbikes/image_0606.jpg 64 train\nMotorbikes/image_0734.jpg 64 train\nMotorbikes/image_0683.jpg 64 train\nMotorbikes/image_0102.jpg 64 train\nMotorbikes/image_0523.jpg 64 train\nMotorbikes/image_0434.jpg 64 train\nMotorbikes/image_0699.jpg 64 train\nMotorbikes/image_0178.jpg 64 train\nMotorbikes/image_0431.jpg 64 train\nMotorbikes/image_0243.jpg 64 train\nMotorbikes/image_0005.jpg 64 train\nMotorbikes/image_0263.jpg 64 train\nMotorbikes/image_0780.jpg 64 train\nMotorbikes/image_0173.jpg 64 train\nMotorbikes/image_0622.jpg 64 train\nMotorbikes/image_0692.jpg 64 train\nMotorbikes/image_0087.jpg 64 train\nMotorbikes/image_0417.jpg 64 train\nMotorbikes/image_0089.jpg 64 train\nMotorbikes/image_0059.jpg 64 train\nMotorbikes/image_0673.jpg 64 train\nMotorbikes/image_0253.jpg 64 train\nMotorbikes/image_0159.jpg 64 train\nMotorbikes/image_0225.jpg 64 train\nMotorbikes/image_0445.jpg 64 train\nMotorbikes/image_0186.jpg 64 train\nMotorbikes/image_0706.jpg 64 train\nMotorbikes/image_0388.jpg 64 train\nMotorbikes/image_0694.jpg 64 train\nMotorbikes/image_0069.jpg 64 train\nMotorbikes/image_0758.jpg 64 train\nMotorbikes/image_0315.jpg 64 train\nMotorbikes/image_0295.jpg 64 train\nMotorbikes/image_0746.jpg 64 train\nMotorbikes/image_0200.jpg 64 train\nMotorbikes/image_0099.jpg 64 train\nMotorbikes/image_0738.jpg 64 train\nMotorbikes/image_0336.jpg 64 train\nMotorbikes/image_0034.jpg 64 train\nMotorbikes/image_0740.jpg 64 train\nMotorbikes/image_0712.jpg 64 train\nMotorbikes/image_0559.jpg 64 train\nMotorbikes/image_0409.jpg 64 train\nMotorbikes/image_0791.jpg 64 train\nMotorbikes/image_0635.jpg 64 train\nMotorbikes/image_0668.jpg 64 train\nMotorbikes/image_0314.jpg 64 train\nMotorbikes/image_0276.jpg 64 train\nMotorbikes/image_0121.jpg 64 train\nMotorbikes/image_0438.jpg 64 train\nMotorbikes/image_0766.jpg 64 train\nMotorbikes/image_0428.jpg 64 train\nMotorbikes/image_0568.jpg 64 train\nMotorbikes/image_0489.jpg 64 train\nMotorbikes/image_0181.jpg 64 train\nMotorbikes/image_0742.jpg 64 train\nMotorbikes/image_0280.jpg 64 train\nMotorbikes/image_0183.jpg 64 train\nMotorbikes/image_0605.jpg 64 train\nMotorbikes/image_0383.jpg 64 train\nMotorbikes/image_0175.jpg 64 train\nMotorbikes/image_0002.jpg 64 train\nMotorbikes/image_0291.jpg 64 train\nMotorbikes/image_0646.jpg 64 train\nMotorbikes/image_0165.jpg 64 train\nMotorbikes/image_0154.jpg 64 train\nMotorbikes/image_0271.jpg 64 train\nMotorbikes/image_0193.jpg 64 train\nMotorbikes/image_0048.jpg 64 train\nMotorbikes/image_0437.jpg 64 train\nMotorbikes/image_0480.jpg 64 train\nMotorbikes/image_0723.jpg 64 train\nMotorbikes/image_0053.jpg 64 train\nMotorbikes/image_0255.jpg 64 train\nMotorbikes/image_0450.jpg 64 train\nMotorbikes/image_0736.jpg 64 train\nMotorbikes/image_0616.jpg 64 train\nMotorbikes/image_0084.jpg 64 train\nMotorbikes/image_0313.jpg 64 train\nMotorbikes/image_0757.jpg 64 train\nMotorbikes/image_0020.jpg 64 train\nMotorbikes/image_0586.jpg 64 train\nMotorbikes/image_0494.jpg 64 train\nMotorbikes/image_0633.jpg 64 train\nMotorbikes/image_0251.jpg 64 train\nMotorbikes/image_0631.jpg 64 train\nMotorbikes/image_0046.jpg 64 train\nMotorbikes/image_0041.jpg 64 train\nMotorbikes/image_0027.jpg 64 train\nMotorbikes/image_0655.jpg 64 train\nMotorbikes/image_0322.jpg 64 train\nMotorbikes/image_0458.jpg 64 train\nMotorbikes/image_0014.jpg 64 train\nMotorbikes/image_0732.jpg 64 train\nMotorbikes/image_0058.jpg 64 train\nMotorbikes/image_0717.jpg 64 train\nMotorbikes/image_0408.jpg 64 train\nMotorbikes/image_0329.jpg 64 train\nMotorbikes/image_0212.jpg 64 train\nMotorbikes/image_0483.jpg 64 train\nMotorbikes/image_0611.jpg 64 train\nMotorbikes/image_0615.jpg 64 train\nMotorbikes/image_0741.jpg 64 train\nMotorbikes/image_0163.jpg 64 train\nMotorbikes/image_0202.jpg 64 train\nMotorbikes/image_0690.jpg 64 train\nMotorbikes/image_0104.jpg 64 train\nMotorbikes/image_0509.jpg 64 train\nMotorbikes/image_0081.jpg 64 train\nMotorbikes/image_0272.jpg 64 train\nMotorbikes/image_0689.jpg 64 train\nMotorbikes/image_0534.jpg 64 train\nMotorbikes/image_0400.jpg 64 train\nMotorbikes/image_0237.jpg 64 train\nMotorbikes/image_0543.jpg 64 train\nMotorbikes/image_0667.jpg 64 train\nMotorbikes/image_0308.jpg 64 train\nMotorbikes/image_0149.jpg 64 train\nMotorbikes/image_0500.jpg 64 train\nMotorbikes/image_0088.jpg 64 train\nMotorbikes/image_0309.jpg 64 train\nMotorbikes/image_0026.jpg 64 train\nMotorbikes/image_0231.jpg 64 train\nMotorbikes/image_0004.jpg 64 train\nMotorbikes/image_0015.jpg 64 train\nMotorbikes/image_0634.jpg 64 train\nMotorbikes/image_0675.jpg 64 train\nMotorbikes/image_0770.jpg 64 train\ndollar_bill/image_0037.jpg 65 train\ndollar_bill/image_0049.jpg 65 train\ndollar_bill/image_0051.jpg 65 train\ndollar_bill/image_0006.jpg 65 train\ndollar_bill/image_0016.jpg 65 train\ndollar_bill/image_0030.jpg 65 train\ndollar_bill/image_0012.jpg 65 train\ndollar_bill/image_0025.jpg 65 train\ndollar_bill/image_0052.jpg 65 train\ndollar_bill/image_0007.jpg 65 train\ndollar_bill/image_0019.jpg 65 train\ndollar_bill/image_0031.jpg 65 train\ndollar_bill/image_0011.jpg 65 train\ndollar_bill/image_0050.jpg 65 train\ndollar_bill/image_0045.jpg 65 train\ndollar_bill/image_0017.jpg 65 train\ndollar_bill/image_0018.jpg 65 train\ndollar_bill/image_0013.jpg 65 train\ndollar_bill/image_0024.jpg 65 train\ndollar_bill/image_0003.jpg 65 train\ndollar_bill/image_0009.jpg 65 train\ndollar_bill/image_0042.jpg 65 train\ndollar_bill/image_0040.jpg 65 train\ndollar_bill/image_0047.jpg 65 train\ndollar_bill/image_0043.jpg 65 train\ndollar_bill/image_0044.jpg 65 train\ndollar_bill/image_0033.jpg 65 train\ndollar_bill/image_0001.jpg 65 train\ndollar_bill/image_0021.jpg 65 train\ndollar_bill/image_0008.jpg 65 train\ndollar_bill/image_0035.jpg 65 train\ndollar_bill/image_0005.jpg 65 train\ndollar_bill/image_0034.jpg 65 train\ndollar_bill/image_0002.jpg 65 train\ndollar_bill/image_0048.jpg 65 train\ndollar_bill/image_0020.jpg 65 train\ndollar_bill/image_0046.jpg 65 train\ndollar_bill/image_0041.jpg 65 train\ndollar_bill/image_0027.jpg 65 train\ndollar_bill/image_0014.jpg 65 train\ndollar_bill/image_0026.jpg 65 train\nnautilus/image_0037.jpg 66 train\nnautilus/image_0049.jpg 66 train\nnautilus/image_0051.jpg 66 train\nnautilus/image_0006.jpg 66 train\nnautilus/image_0016.jpg 66 train\nnautilus/image_0030.jpg 66 train\nnautilus/image_0012.jpg 66 train\nnautilus/image_0025.jpg 66 train\nnautilus/image_0052.jpg 66 train\nnautilus/image_0007.jpg 66 train\nnautilus/image_0019.jpg 66 train\nnautilus/image_0031.jpg 66 train\nnautilus/image_0011.jpg 66 train\nnautilus/image_0050.jpg 66 train\nnautilus/image_0045.jpg 66 train\nnautilus/image_0017.jpg 66 train\nnautilus/image_0018.jpg 66 train\nnautilus/image_0013.jpg 66 train\nnautilus/image_0054.jpg 66 train\nnautilus/image_0055.jpg 66 train\nnautilus/image_0024.jpg 66 train\nnautilus/image_0003.jpg 66 train\nnautilus/image_0009.jpg 66 train\nnautilus/image_0042.jpg 66 train\nnautilus/image_0040.jpg 66 train\nnautilus/image_0047.jpg 66 train\nnautilus/image_0043.jpg 66 train\nnautilus/image_0044.jpg 66 train\nnautilus/image_0033.jpg 66 train\nnautilus/image_0001.jpg 66 train\nnautilus/image_0021.jpg 66 train\nnautilus/image_0008.jpg 66 train\nnautilus/image_0035.jpg 66 train\nnautilus/image_0005.jpg 66 train\nnautilus/image_0034.jpg 66 train\nnautilus/image_0002.jpg 66 train\nnautilus/image_0048.jpg 66 train\nnautilus/image_0053.jpg 66 train\nnautilus/image_0020.jpg 66 train\nnautilus/image_0046.jpg 66 train\nnautilus/image_0041.jpg 66 train\nnautilus/image_0027.jpg 66 train\nnautilus/image_0014.jpg 66 train\nnautilus/image_0026.jpg 66 train\ncrab/image_0062.jpg 67 train\ncrab/image_0037.jpg 67 train\ncrab/image_0049.jpg 67 train\ncrab/image_0051.jpg 67 train\ncrab/image_0006.jpg 67 train\ncrab/image_0016.jpg 67 train\ncrab/image_0030.jpg 67 train\ncrab/image_0012.jpg 67 train\ncrab/image_0066.jpg 67 train\ncrab/image_0025.jpg 67 train\ncrab/image_0060.jpg 67 train\ncrab/image_0052.jpg 67 train\ncrab/image_0007.jpg 67 train\ncrab/image_0063.jpg 67 train\ncrab/image_0019.jpg 67 train\ncrab/image_0031.jpg 67 train\ncrab/image_0011.jpg 67 train\ncrab/image_0061.jpg 67 train\ncrab/image_0050.jpg 67 train\ncrab/image_0045.jpg 67 train\ncrab/image_0017.jpg 67 train\ncrab/image_0018.jpg 67 train\ncrab/image_0013.jpg 67 train\ncrab/image_0054.jpg 67 train\ncrab/image_0055.jpg 67 train\ncrab/image_0071.jpg 67 train\ncrab/image_0024.jpg 67 train\ncrab/image_0003.jpg 67 train\ncrab/image_0009.jpg 67 train\ncrab/image_0042.jpg 67 train\ncrab/image_0040.jpg 67 train\ncrab/image_0067.jpg 67 train\ncrab/image_0047.jpg 67 train\ncrab/image_0064.jpg 67 train\ncrab/image_0070.jpg 67 train\ncrab/image_0068.jpg 67 train\ncrab/image_0043.jpg 67 train\ncrab/image_0044.jpg 67 train\ncrab/image_0033.jpg 67 train\ncrab/image_0073.jpg 67 train\ncrab/image_0001.jpg 67 train\ncrab/image_0021.jpg 67 train\ncrab/image_0057.jpg 67 train\ncrab/image_0008.jpg 67 train\ncrab/image_0035.jpg 67 train\ncrab/image_0005.jpg 67 train\ncrab/image_0059.jpg 67 train\ncrab/image_0069.jpg 67 train\ncrab/image_0034.jpg 67 train\ncrab/image_0002.jpg 67 train\ncrab/image_0048.jpg 67 train\ncrab/image_0053.jpg 67 train\ncrab/image_0020.jpg 67 train\ncrab/image_0046.jpg 67 train\ncrab/image_0041.jpg 67 train\ncrab/image_0027.jpg 67 train\ncrab/image_0014.jpg 67 train\ncrab/image_0058.jpg 67 train\naccordion/image_0037.jpg 68 train\naccordion/image_0049.jpg 68 train\naccordion/image_0051.jpg 68 train\naccordion/image_0006.jpg 68 train\naccordion/image_0016.jpg 68 train\naccordion/image_0030.jpg 68 train\naccordion/image_0012.jpg 68 train\naccordion/image_0025.jpg 68 train\naccordion/image_0052.jpg 68 train\naccordion/image_0007.jpg 68 train\naccordion/image_0019.jpg 68 train\naccordion/image_0031.jpg 68 train\naccordion/image_0011.jpg 68 train\naccordion/image_0050.jpg 68 train\naccordion/image_0045.jpg 68 train\naccordion/image_0017.jpg 68 train\naccordion/image_0018.jpg 68 train\naccordion/image_0013.jpg 68 train\naccordion/image_0054.jpg 68 train\naccordion/image_0055.jpg 68 train\naccordion/image_0024.jpg 68 train\naccordion/image_0003.jpg 68 train\naccordion/image_0009.jpg 68 train\naccordion/image_0042.jpg 68 train\naccordion/image_0040.jpg 68 train\naccordion/image_0047.jpg 68 train\naccordion/image_0043.jpg 68 train\naccordion/image_0044.jpg 68 train\naccordion/image_0033.jpg 68 train\naccordion/image_0001.jpg 68 train\naccordion/image_0021.jpg 68 train\naccordion/image_0008.jpg 68 train\naccordion/image_0035.jpg 68 train\naccordion/image_0005.jpg 68 train\naccordion/image_0034.jpg 68 train\naccordion/image_0002.jpg 68 train\naccordion/image_0048.jpg 68 train\naccordion/image_0053.jpg 68 train\naccordion/image_0020.jpg 68 train\naccordion/image_0046.jpg 68 train\naccordion/image_0041.jpg 68 train\naccordion/image_0027.jpg 68 train\naccordion/image_0014.jpg 68 train\naccordion/image_0026.jpg 68 train\ncrayfish/image_0062.jpg 69 train\ncrayfish/image_0037.jpg 69 train\ncrayfish/image_0049.jpg 69 train\ncrayfish/image_0051.jpg 69 train\ncrayfish/image_0006.jpg 69 train\ncrayfish/image_0016.jpg 69 train\ncrayfish/image_0030.jpg 69 train\ncrayfish/image_0012.jpg 69 train\ncrayfish/image_0066.jpg 69 train\ncrayfish/image_0025.jpg 69 train\ncrayfish/image_0060.jpg 69 train\ncrayfish/image_0052.jpg 69 train\ncrayfish/image_0007.jpg 69 train\ncrayfish/image_0063.jpg 69 train\ncrayfish/image_0019.jpg 69 train\ncrayfish/image_0031.jpg 69 train\ncrayfish/image_0011.jpg 69 train\ncrayfish/image_0061.jpg 69 train\ncrayfish/image_0050.jpg 69 train\ncrayfish/image_0045.jpg 69 train\ncrayfish/image_0017.jpg 69 train\ncrayfish/image_0018.jpg 69 train\ncrayfish/image_0013.jpg 69 train\ncrayfish/image_0054.jpg 69 train\ncrayfish/image_0055.jpg 69 train\ncrayfish/image_0024.jpg 69 train\ncrayfish/image_0003.jpg 69 train\ncrayfish/image_0009.jpg 69 train\ncrayfish/image_0042.jpg 69 train\ncrayfish/image_0040.jpg 69 train\ncrayfish/image_0067.jpg 69 train\ncrayfish/image_0047.jpg 69 train\ncrayfish/image_0064.jpg 69 train\ncrayfish/image_0070.jpg 69 train\ncrayfish/image_0068.jpg 69 train\ncrayfish/image_0043.jpg 69 train\ncrayfish/image_0044.jpg 69 train\ncrayfish/image_0033.jpg 69 train\ncrayfish/image_0001.jpg 69 train\ncrayfish/image_0021.jpg 69 train\ncrayfish/image_0057.jpg 69 train\ncrayfish/image_0008.jpg 69 train\ncrayfish/image_0035.jpg 69 train\ncrayfish/image_0005.jpg 69 train\ncrayfish/image_0059.jpg 69 train\ncrayfish/image_0069.jpg 69 train\ncrayfish/image_0034.jpg 69 train\ncrayfish/image_0002.jpg 69 train\ncrayfish/image_0048.jpg 69 train\ncrayfish/image_0053.jpg 69 train\ncrayfish/image_0020.jpg 69 train\ncrayfish/image_0046.jpg 69 train\ncrayfish/image_0041.jpg 69 train\ncrayfish/image_0027.jpg 69 train\ncrayfish/image_0014.jpg 69 train\ncrayfish/image_0058.jpg 69 train\nflamingo_head/image_0037.jpg 70 train\nflamingo_head/image_0006.jpg 70 train\nflamingo_head/image_0016.jpg 70 train\nflamingo_head/image_0030.jpg 70 train\nflamingo_head/image_0012.jpg 70 train\nflamingo_head/image_0025.jpg 70 train\nflamingo_head/image_0007.jpg 70 train\nflamingo_head/image_0019.jpg 70 train\nflamingo_head/image_0031.jpg 70 train\nflamingo_head/image_0011.jpg 70 train\nflamingo_head/image_0045.jpg 70 train\nflamingo_head/image_0017.jpg 70 train\nflamingo_head/image_0018.jpg 70 train\nflamingo_head/image_0013.jpg 70 train\nflamingo_head/image_0024.jpg 70 train\nflamingo_head/image_0003.jpg 70 train\nflamingo_head/image_0009.jpg 70 train\nflamingo_head/image_0042.jpg 70 train\nflamingo_head/image_0040.jpg 70 train\nflamingo_head/image_0043.jpg 70 train\nflamingo_head/image_0044.jpg 70 train\nflamingo_head/image_0033.jpg 70 train\nflamingo_head/image_0001.jpg 70 train\nflamingo_head/image_0021.jpg 70 train\nflamingo_head/image_0008.jpg 70 train\nflamingo_head/image_0035.jpg 70 train\nflamingo_head/image_0005.jpg 70 train\nflamingo_head/image_0034.jpg 70 train\nflamingo_head/image_0002.jpg 70 train\nflamingo_head/image_0020.jpg 70 train\nflamingo_head/image_0041.jpg 70 train\nflamingo_head/image_0027.jpg 70 train\nflamingo_head/image_0014.jpg 70 train\nflamingo_head/image_0026.jpg 70 train\nflamingo_head/image_0004.jpg 70 train\nflamingo_head/image_0015.jpg 70 train\nemu/image_0037.jpg 71 train\nemu/image_0049.jpg 71 train\nemu/image_0051.jpg 71 train\nemu/image_0006.jpg 71 train\nemu/image_0016.jpg 71 train\nemu/image_0030.jpg 71 train\nemu/image_0012.jpg 71 train\nemu/image_0025.jpg 71 train\nemu/image_0052.jpg 71 train\nemu/image_0007.jpg 71 train\nemu/image_0019.jpg 71 train\nemu/image_0031.jpg 71 train\nemu/image_0011.jpg 71 train\nemu/image_0050.jpg 71 train\nemu/image_0045.jpg 71 train\nemu/image_0017.jpg 71 train\nemu/image_0018.jpg 71 train\nemu/image_0013.jpg 71 train\nemu/image_0024.jpg 71 train\nemu/image_0003.jpg 71 train\nemu/image_0009.jpg 71 train\nemu/image_0042.jpg 71 train\nemu/image_0040.jpg 71 train\nemu/image_0047.jpg 71 train\nemu/image_0043.jpg 71 train\nemu/image_0044.jpg 71 train\nemu/image_0033.jpg 71 train\nemu/image_0001.jpg 71 train\nemu/image_0021.jpg 71 train\nemu/image_0008.jpg 71 train\nemu/image_0035.jpg 71 train\nemu/image_0005.jpg 71 train\nemu/image_0034.jpg 71 train\nemu/image_0002.jpg 71 train\nemu/image_0048.jpg 71 train\nemu/image_0053.jpg 71 train\nemu/image_0020.jpg 71 train\nemu/image_0046.jpg 71 train\nemu/image_0041.jpg 71 train\nemu/image_0027.jpg 71 train\nemu/image_0014.jpg 71 train\nemu/image_0026.jpg 71 train\ntrilobite/image_0062.jpg 72 train\ntrilobite/image_0037.jpg 72 train\ntrilobite/image_0049.jpg 72 train\ntrilobite/image_0083.jpg 72 train\ntrilobite/image_0051.jpg 72 train\ntrilobite/image_0006.jpg 72 train\ntrilobite/image_0016.jpg 72 train\ntrilobite/image_0030.jpg 72 train\ntrilobite/image_0012.jpg 72 train\ntrilobite/image_0066.jpg 72 train\ntrilobite/image_0077.jpg 72 train\ntrilobite/image_0025.jpg 72 train\ntrilobite/image_0060.jpg 72 train\ntrilobite/image_0052.jpg 72 train\ntrilobite/image_0007.jpg 72 train\ntrilobite/image_0063.jpg 72 train\ntrilobite/image_0019.jpg 72 train\ntrilobite/image_0031.jpg 72 train\ntrilobite/image_0011.jpg 72 train\ntrilobite/image_0061.jpg 72 train\ntrilobite/image_0050.jpg 72 train\ntrilobite/image_0045.jpg 72 train\ntrilobite/image_0017.jpg 72 train\ntrilobite/image_0018.jpg 72 train\ntrilobite/image_0013.jpg 72 train\ntrilobite/image_0054.jpg 72 train\ntrilobite/image_0082.jpg 72 train\ntrilobite/image_0079.jpg 72 train\ntrilobite/image_0055.jpg 72 train\ntrilobite/image_0071.jpg 72 train\ntrilobite/image_0024.jpg 72 train\ntrilobite/image_0074.jpg 72 train\ntrilobite/image_0003.jpg 72 train\ntrilobite/image_0009.jpg 72 train\ntrilobite/image_0042.jpg 72 train\ntrilobite/image_0040.jpg 72 train\ntrilobite/image_0067.jpg 72 train\ntrilobite/image_0047.jpg 72 train\ntrilobite/image_0064.jpg 72 train\ntrilobite/image_0070.jpg 72 train\ntrilobite/image_0078.jpg 72 train\ntrilobite/image_0068.jpg 72 train\ntrilobite/image_0085.jpg 72 train\ntrilobite/image_0043.jpg 72 train\ntrilobite/image_0044.jpg 72 train\ntrilobite/image_0033.jpg 72 train\ntrilobite/image_0073.jpg 72 train\ntrilobite/image_0001.jpg 72 train\ntrilobite/image_0080.jpg 72 train\ntrilobite/image_0021.jpg 72 train\ntrilobite/image_0057.jpg 72 train\ntrilobite/image_0008.jpg 72 train\ntrilobite/image_0035.jpg 72 train\ntrilobite/image_0005.jpg 72 train\ntrilobite/image_0059.jpg 72 train\ntrilobite/image_0069.jpg 72 train\ntrilobite/image_0034.jpg 72 train\ntrilobite/image_0002.jpg 72 train\ntrilobite/image_0048.jpg 72 train\ntrilobite/image_0053.jpg 72 train\ntrilobite/image_0084.jpg 72 train\ntrilobite/image_0020.jpg 72 train\ntrilobite/image_0046.jpg 72 train\ntrilobite/image_0041.jpg 72 train\ntrilobite/image_0027.jpg 72 train\ntrilobite/image_0014.jpg 72 train\ntrilobite/image_0058.jpg 72 train\ntrilobite/image_0081.jpg 72 train\ncamera/image_0037.jpg 73 train\ncamera/image_0049.jpg 73 train\ncamera/image_0006.jpg 73 train\ncamera/image_0016.jpg 73 train\ncamera/image_0030.jpg 73 train\ncamera/image_0012.jpg 73 train\ncamera/image_0025.jpg 73 train\ncamera/image_0007.jpg 73 train\ncamera/image_0019.jpg 73 train\ncamera/image_0031.jpg 73 train\ncamera/image_0011.jpg 73 train\ncamera/image_0050.jpg 73 train\ncamera/image_0045.jpg 73 train\ncamera/image_0017.jpg 73 train\ncamera/image_0018.jpg 73 train\ncamera/image_0013.jpg 73 train\ncamera/image_0024.jpg 73 train\ncamera/image_0003.jpg 73 train\ncamera/image_0009.jpg 73 train\ncamera/image_0042.jpg 73 train\ncamera/image_0040.jpg 73 train\ncamera/image_0047.jpg 73 train\ncamera/image_0043.jpg 73 train\ncamera/image_0044.jpg 73 train\ncamera/image_0033.jpg 73 train\ncamera/image_0001.jpg 73 train\ncamera/image_0021.jpg 73 train\ncamera/image_0008.jpg 73 train\ncamera/image_0035.jpg 73 train\ncamera/image_0005.jpg 73 train\ncamera/image_0034.jpg 73 train\ncamera/image_0002.jpg 73 train\ncamera/image_0048.jpg 73 train\ncamera/image_0020.jpg 73 train\ncamera/image_0046.jpg 73 train\ncamera/image_0041.jpg 73 train\ncamera/image_0027.jpg 73 train\ncamera/image_0014.jpg 73 train\ncamera/image_0026.jpg 73 train\ncamera/image_0004.jpg 73 train\nplatypus/image_0006.jpg 74 train\nplatypus/image_0016.jpg 74 train\nplatypus/image_0030.jpg 74 train\nplatypus/image_0012.jpg 74 train\nplatypus/image_0025.jpg 74 train\nplatypus/image_0007.jpg 74 train\nplatypus/image_0019.jpg 74 train\nplatypus/image_0031.jpg 74 train\nplatypus/image_0011.jpg 74 train\nplatypus/image_0017.jpg 74 train\nplatypus/image_0018.jpg 74 train\nplatypus/image_0013.jpg 74 train\nplatypus/image_0024.jpg 74 train\nplatypus/image_0003.jpg 74 train\nplatypus/image_0009.jpg 74 train\nplatypus/image_0033.jpg 74 train\nplatypus/image_0001.jpg 74 train\nplatypus/image_0021.jpg 74 train\nplatypus/image_0008.jpg 74 train\nplatypus/image_0005.jpg 74 train\nplatypus/image_0034.jpg 74 train\nplatypus/image_0002.jpg 74 train\nplatypus/image_0020.jpg 74 train\nplatypus/image_0027.jpg 74 train\nplatypus/image_0014.jpg 74 train\nplatypus/image_0026.jpg 74 train\nplatypus/image_0004.jpg 74 train\nchandelier/image_0062.jpg 75 train\nchandelier/image_0037.jpg 75 train\nchandelier/image_0049.jpg 75 train\nchandelier/image_0083.jpg 75 train\nchandelier/image_0101.jpg 75 train\nchandelier/image_0051.jpg 75 train\nchandelier/image_0006.jpg 75 train\nchandelier/image_0016.jpg 75 train\nchandelier/image_0030.jpg 75 train\nchandelier/image_0091.jpg 75 train\nchandelier/image_0012.jpg 75 train\nchandelier/image_0066.jpg 75 train\nchandelier/image_0077.jpg 75 train\nchandelier/image_0025.jpg 75 train\nchandelier/image_0060.jpg 75 train\nchandelier/image_0052.jpg 75 train\nchandelier/image_0007.jpg 75 train\nchandelier/image_0063.jpg 75 train\nchandelier/image_0019.jpg 75 train\nchandelier/image_0031.jpg 75 train\nchandelier/image_0011.jpg 75 train\nchandelier/image_0061.jpg 75 train\nchandelier/image_0090.jpg 75 train\nchandelier/image_0050.jpg 75 train\nchandelier/image_0045.jpg 75 train\nchandelier/image_0017.jpg 75 train\nchandelier/image_0018.jpg 75 train\nchandelier/image_0013.jpg 75 train\nchandelier/image_0054.jpg 75 train\nchandelier/image_0094.jpg 75 train\nchandelier/image_0082.jpg 75 train\nchandelier/image_0097.jpg 75 train\nchandelier/image_0079.jpg 75 train\nchandelier/image_0055.jpg 75 train\nchandelier/image_0071.jpg 75 train\nchandelier/image_0103.jpg 75 train\nchandelier/image_0024.jpg 75 train\nchandelier/image_0105.jpg 75 train\nchandelier/image_0074.jpg 75 train\nchandelier/image_0003.jpg 75 train\nchandelier/image_0009.jpg 75 train\nchandelier/image_0042.jpg 75 train\nchandelier/image_0093.jpg 75 train\nchandelier/image_0040.jpg 75 train\nchandelier/image_0067.jpg 75 train\nchandelier/image_0047.jpg 75 train\nchandelier/image_0064.jpg 75 train\nchandelier/image_0070.jpg 75 train\nchandelier/image_0078.jpg 75 train\nchandelier/image_0068.jpg 75 train\nchandelier/image_0085.jpg 75 train\nchandelier/image_0043.jpg 75 train\nchandelier/image_0044.jpg 75 train\nchandelier/image_0033.jpg 75 train\nchandelier/image_0073.jpg 75 train\nchandelier/image_0001.jpg 75 train\nchandelier/image_0080.jpg 75 train\nchandelier/image_0021.jpg 75 train\nchandelier/image_0095.jpg 75 train\nchandelier/image_0057.jpg 75 train\nchandelier/image_0098.jpg 75 train\nchandelier/image_0008.jpg 75 train\nchandelier/image_0035.jpg 75 train\nchandelier/image_0102.jpg 75 train\nchandelier/image_0005.jpg 75 train\nchandelier/image_0087.jpg 75 train\nchandelier/image_0089.jpg 75 train\nchandelier/image_0059.jpg 75 train\nchandelier/image_0069.jpg 75 train\nchandelier/image_0099.jpg 75 train\nchandelier/image_0034.jpg 75 train\nchandelier/image_0002.jpg 75 train\nchandelier/image_0048.jpg 75 train\nchandelier/image_0053.jpg 75 train\nchandelier/image_0084.jpg 75 train\nchandelier/image_0020.jpg 75 train\nchandelier/image_0046.jpg 75 train\nchandelier/image_0041.jpg 75 train\nchandelier/image_0027.jpg 75 train\nchandelier/image_0014.jpg 75 train\nchandelier/image_0058.jpg 75 train\nchandelier/image_0104.jpg 75 train\nchandelier/image_0081.jpg 75 train\nchandelier/image_0088.jpg 75 train\nchandelier/image_0026.jpg 75 train\ncrocodile/image_0037.jpg 76 train\ncrocodile/image_0049.jpg 76 train\ncrocodile/image_0006.jpg 76 train\ncrocodile/image_0016.jpg 76 train\ncrocodile/image_0030.jpg 76 train\ncrocodile/image_0012.jpg 76 train\ncrocodile/image_0025.jpg 76 train\ncrocodile/image_0007.jpg 76 train\ncrocodile/image_0019.jpg 76 train\ncrocodile/image_0031.jpg 76 train\ncrocodile/image_0011.jpg 76 train\ncrocodile/image_0050.jpg 76 train\ncrocodile/image_0045.jpg 76 train\ncrocodile/image_0017.jpg 76 train\ncrocodile/image_0018.jpg 76 train\ncrocodile/image_0013.jpg 76 train\ncrocodile/image_0024.jpg 76 train\ncrocodile/image_0003.jpg 76 train\ncrocodile/image_0009.jpg 76 train\ncrocodile/image_0042.jpg 76 train\ncrocodile/image_0040.jpg 76 train\ncrocodile/image_0047.jpg 76 train\ncrocodile/image_0043.jpg 76 train\ncrocodile/image_0044.jpg 76 train\ncrocodile/image_0033.jpg 76 train\ncrocodile/image_0001.jpg 76 train\ncrocodile/image_0021.jpg 76 train\ncrocodile/image_0008.jpg 76 train\ncrocodile/image_0035.jpg 76 train\ncrocodile/image_0005.jpg 76 train\ncrocodile/image_0034.jpg 76 train\ncrocodile/image_0002.jpg 76 train\ncrocodile/image_0048.jpg 76 train\ncrocodile/image_0020.jpg 76 train\ncrocodile/image_0046.jpg 76 train\ncrocodile/image_0041.jpg 76 train\ncrocodile/image_0027.jpg 76 train\ncrocodile/image_0014.jpg 76 train\ncrocodile/image_0026.jpg 76 train\ncrocodile/image_0004.jpg 76 train\ncar_side/image_0062.jpg 77 train\ncar_side/image_0037.jpg 77 train\ncar_side/image_0049.jpg 77 train\ncar_side/image_0083.jpg 77 train\ncar_side/image_0101.jpg 77 train\ncar_side/image_0051.jpg 77 train\ncar_side/image_0006.jpg 77 train\ncar_side/image_0016.jpg 77 train\ncar_side/image_0030.jpg 77 train\ncar_side/image_0091.jpg 77 train\ncar_side/image_0012.jpg 77 train\ncar_side/image_0066.jpg 77 train\ncar_side/image_0077.jpg 77 train\ncar_side/image_0025.jpg 77 train\ncar_side/image_0115.jpg 77 train\ncar_side/image_0123.jpg 77 train\ncar_side/image_0119.jpg 77 train\ncar_side/image_0060.jpg 77 train\ncar_side/image_0052.jpg 77 train\ncar_side/image_0108.jpg 77 train\ncar_side/image_0007.jpg 77 train\ncar_side/image_0063.jpg 77 train\ncar_side/image_0019.jpg 77 train\ncar_side/image_0113.jpg 77 train\ncar_side/image_0031.jpg 77 train\ncar_side/image_0011.jpg 77 train\ncar_side/image_0061.jpg 77 train\ncar_side/image_0090.jpg 77 train\ncar_side/image_0050.jpg 77 train\ncar_side/image_0045.jpg 77 train\ncar_side/image_0017.jpg 77 train\ncar_side/image_0018.jpg 77 train\ncar_side/image_0013.jpg 77 train\ncar_side/image_0054.jpg 77 train\ncar_side/image_0094.jpg 77 train\ncar_side/image_0082.jpg 77 train\ncar_side/image_0097.jpg 77 train\ncar_side/image_0109.jpg 77 train\ncar_side/image_0079.jpg 77 train\ncar_side/image_0055.jpg 77 train\ncar_side/image_0071.jpg 77 train\ncar_side/image_0103.jpg 77 train\ncar_side/image_0024.jpg 77 train\ncar_side/image_0105.jpg 77 train\ncar_side/image_0074.jpg 77 train\ncar_side/image_0003.jpg 77 train\ncar_side/image_0009.jpg 77 train\ncar_side/image_0118.jpg 77 train\ncar_side/image_0042.jpg 77 train\ncar_side/image_0093.jpg 77 train\ncar_side/image_0040.jpg 77 train\ncar_side/image_0067.jpg 77 train\ncar_side/image_0122.jpg 77 train\ncar_side/image_0047.jpg 77 train\ncar_side/image_0064.jpg 77 train\ncar_side/image_0070.jpg 77 train\ncar_side/image_0112.jpg 77 train\ncar_side/image_0078.jpg 77 train\ncar_side/image_0068.jpg 77 train\ncar_side/image_0085.jpg 77 train\ncar_side/image_0110.jpg 77 train\ncar_side/image_0043.jpg 77 train\ncar_side/image_0044.jpg 77 train\ncar_side/image_0033.jpg 77 train\ncar_side/image_0073.jpg 77 train\ncar_side/image_0001.jpg 77 train\ncar_side/image_0080.jpg 77 train\ncar_side/image_0021.jpg 77 train\ncar_side/image_0095.jpg 77 train\ncar_side/image_0057.jpg 77 train\ncar_side/image_0114.jpg 77 train\ncar_side/image_0098.jpg 77 train\ncar_side/image_0008.jpg 77 train\ncar_side/image_0035.jpg 77 train\ncar_side/image_0102.jpg 77 train\ncar_side/image_0005.jpg 77 train\ncar_side/image_0087.jpg 77 train\ncar_side/image_0089.jpg 77 train\ncar_side/image_0059.jpg 77 train\ncar_side/image_0069.jpg 77 train\ncar_side/image_0099.jpg 77 train\ncar_side/image_0034.jpg 77 train\ncar_side/image_0121.jpg 77 train\ncar_side/image_0002.jpg 77 train\ncar_side/image_0048.jpg 77 train\ncar_side/image_0053.jpg 77 train\ncar_side/image_0084.jpg 77 train\ncar_side/image_0020.jpg 77 train\ncar_side/image_0046.jpg 77 train\ncar_side/image_0041.jpg 77 train\ncar_side/image_0027.jpg 77 train\ncar_side/image_0014.jpg 77 train\ncar_side/image_0058.jpg 77 train\ncar_side/image_0104.jpg 77 train\ncar_side/image_0081.jpg 77 train\ncar_side/image_0088.jpg 77 train\ncar_side/image_0026.jpg 77 train\ncar_side/image_0004.jpg 77 train\njoshua_tree/image_0062.jpg 78 train\njoshua_tree/image_0037.jpg 78 train\njoshua_tree/image_0049.jpg 78 train\njoshua_tree/image_0051.jpg 78 train\njoshua_tree/image_0006.jpg 78 train\njoshua_tree/image_0016.jpg 78 train\njoshua_tree/image_0030.jpg 78 train\njoshua_tree/image_0012.jpg 78 train\njoshua_tree/image_0025.jpg 78 train\njoshua_tree/image_0060.jpg 78 train\njoshua_tree/image_0052.jpg 78 train\njoshua_tree/image_0007.jpg 78 train\njoshua_tree/image_0063.jpg 78 train\njoshua_tree/image_0019.jpg 78 train\njoshua_tree/image_0031.jpg 78 train\njoshua_tree/image_0011.jpg 78 train\njoshua_tree/image_0061.jpg 78 train\njoshua_tree/image_0050.jpg 78 train\njoshua_tree/image_0045.jpg 78 train\njoshua_tree/image_0017.jpg 78 train\njoshua_tree/image_0018.jpg 78 train\njoshua_tree/image_0013.jpg 78 train\njoshua_tree/image_0054.jpg 78 train\njoshua_tree/image_0055.jpg 78 train\njoshua_tree/image_0024.jpg 78 train\njoshua_tree/image_0003.jpg 78 train\njoshua_tree/image_0009.jpg 78 train\njoshua_tree/image_0042.jpg 78 train\njoshua_tree/image_0040.jpg 78 train\njoshua_tree/image_0047.jpg 78 train\njoshua_tree/image_0064.jpg 78 train\njoshua_tree/image_0043.jpg 78 train\njoshua_tree/image_0044.jpg 78 train\njoshua_tree/image_0033.jpg 78 train\njoshua_tree/image_0001.jpg 78 train\njoshua_tree/image_0021.jpg 78 train\njoshua_tree/image_0057.jpg 78 train\njoshua_tree/image_0008.jpg 78 train\njoshua_tree/image_0035.jpg 78 train\njoshua_tree/image_0005.jpg 78 train\njoshua_tree/image_0059.jpg 78 train\njoshua_tree/image_0034.jpg 78 train\njoshua_tree/image_0002.jpg 78 train\njoshua_tree/image_0048.jpg 78 train\njoshua_tree/image_0053.jpg 78 train\njoshua_tree/image_0020.jpg 78 train\njoshua_tree/image_0046.jpg 78 train\njoshua_tree/image_0041.jpg 78 train\njoshua_tree/image_0027.jpg 78 train\njoshua_tree/image_0014.jpg 78 train\njoshua_tree/image_0058.jpg 78 train\ntick/image_0037.jpg 79 train\ntick/image_0049.jpg 79 train\ntick/image_0006.jpg 79 train\ntick/image_0016.jpg 79 train\ntick/image_0030.jpg 79 train\ntick/image_0012.jpg 79 train\ntick/image_0025.jpg 79 train\ntick/image_0007.jpg 79 train\ntick/image_0019.jpg 79 train\ntick/image_0031.jpg 79 train\ntick/image_0011.jpg 79 train\ntick/image_0045.jpg 79 train\ntick/image_0017.jpg 79 train\ntick/image_0018.jpg 79 train\ntick/image_0013.jpg 79 train\ntick/image_0024.jpg 79 train\ntick/image_0003.jpg 79 train\ntick/image_0009.jpg 79 train\ntick/image_0042.jpg 79 train\ntick/image_0040.jpg 79 train\ntick/image_0047.jpg 79 train\ntick/image_0043.jpg 79 train\ntick/image_0044.jpg 79 train\ntick/image_0033.jpg 79 train\ntick/image_0001.jpg 79 train\ntick/image_0021.jpg 79 train\ntick/image_0008.jpg 79 train\ntick/image_0035.jpg 79 train\ntick/image_0005.jpg 79 train\ntick/image_0034.jpg 79 train\ntick/image_0002.jpg 79 train\ntick/image_0048.jpg 79 train\ntick/image_0020.jpg 79 train\ntick/image_0046.jpg 79 train\ntick/image_0041.jpg 79 train\ntick/image_0027.jpg 79 train\ntick/image_0014.jpg 79 train\ntick/image_0026.jpg 79 train\ntick/image_0004.jpg 79 train\nhawksbill/image_0062.jpg 80 train\nhawksbill/image_0037.jpg 80 train\nhawksbill/image_0049.jpg 80 train\nhawksbill/image_0083.jpg 80 train\nhawksbill/image_0051.jpg 80 train\nhawksbill/image_0006.jpg 80 train\nhawksbill/image_0016.jpg 80 train\nhawksbill/image_0030.jpg 80 train\nhawksbill/image_0091.jpg 80 train\nhawksbill/image_0012.jpg 80 train\nhawksbill/image_0066.jpg 80 train\nhawksbill/image_0077.jpg 80 train\nhawksbill/image_0025.jpg 80 train\nhawksbill/image_0060.jpg 80 train\nhawksbill/image_0052.jpg 80 train\nhawksbill/image_0007.jpg 80 train\nhawksbill/image_0063.jpg 80 train\nhawksbill/image_0019.jpg 80 train\nhawksbill/image_0031.jpg 80 train\nhawksbill/image_0011.jpg 80 train\nhawksbill/image_0061.jpg 80 train\nhawksbill/image_0090.jpg 80 train\nhawksbill/image_0050.jpg 80 train\nhawksbill/image_0045.jpg 80 train\nhawksbill/image_0017.jpg 80 train\nhawksbill/image_0018.jpg 80 train\nhawksbill/image_0013.jpg 80 train\nhawksbill/image_0054.jpg 80 train\nhawksbill/image_0094.jpg 80 train\nhawksbill/image_0082.jpg 80 train\nhawksbill/image_0097.jpg 80 train\nhawksbill/image_0079.jpg 80 train\nhawksbill/image_0055.jpg 80 train\nhawksbill/image_0071.jpg 80 train\nhawksbill/image_0024.jpg 80 train\nhawksbill/image_0074.jpg 80 train\nhawksbill/image_0003.jpg 80 train\nhawksbill/image_0009.jpg 80 train\nhawksbill/image_0042.jpg 80 train\nhawksbill/image_0093.jpg 80 train\nhawksbill/image_0040.jpg 80 train\nhawksbill/image_0067.jpg 80 train\nhawksbill/image_0047.jpg 80 train\nhawksbill/image_0064.jpg 80 train\nhawksbill/image_0070.jpg 80 train\nhawksbill/image_0078.jpg 80 train\nhawksbill/image_0068.jpg 80 train\nhawksbill/image_0085.jpg 80 train\nhawksbill/image_0043.jpg 80 train\nhawksbill/image_0044.jpg 80 train\nhawksbill/image_0033.jpg 80 train\nhawksbill/image_0073.jpg 80 train\nhawksbill/image_0001.jpg 80 train\nhawksbill/image_0080.jpg 80 train\nhawksbill/image_0021.jpg 80 train\nhawksbill/image_0095.jpg 80 train\nhawksbill/image_0057.jpg 80 train\nhawksbill/image_0098.jpg 80 train\nhawksbill/image_0008.jpg 80 train\nhawksbill/image_0035.jpg 80 train\nhawksbill/image_0005.jpg 80 train\nhawksbill/image_0087.jpg 80 train\nhawksbill/image_0089.jpg 80 train\nhawksbill/image_0059.jpg 80 train\nhawksbill/image_0069.jpg 80 train\nhawksbill/image_0099.jpg 80 train\nhawksbill/image_0034.jpg 80 train\nhawksbill/image_0002.jpg 80 train\nhawksbill/image_0048.jpg 80 train\nhawksbill/image_0053.jpg 80 train\nhawksbill/image_0084.jpg 80 train\nhawksbill/image_0020.jpg 80 train\nhawksbill/image_0046.jpg 80 train\nhawksbill/image_0041.jpg 80 train\nhawksbill/image_0027.jpg 80 train\nhawksbill/image_0014.jpg 80 train\nhawksbill/image_0058.jpg 80 train\nhawksbill/image_0081.jpg 80 train\nhawksbill/image_0088.jpg 80 train\nhawksbill/image_0026.jpg 80 train\nhelicopter/image_0062.jpg 81 train\nhelicopter/image_0037.jpg 81 train\nhelicopter/image_0049.jpg 81 train\nhelicopter/image_0083.jpg 81 train\nhelicopter/image_0051.jpg 81 train\nhelicopter/image_0006.jpg 81 train\nhelicopter/image_0016.jpg 81 train\nhelicopter/image_0030.jpg 81 train\nhelicopter/image_0012.jpg 81 train\nhelicopter/image_0066.jpg 81 train\nhelicopter/image_0077.jpg 81 train\nhelicopter/image_0025.jpg 81 train\nhelicopter/image_0060.jpg 81 train\nhelicopter/image_0052.jpg 81 train\nhelicopter/image_0007.jpg 81 train\nhelicopter/image_0063.jpg 81 train\nhelicopter/image_0019.jpg 81 train\nhelicopter/image_0031.jpg 81 train\nhelicopter/image_0011.jpg 81 train\nhelicopter/image_0061.jpg 81 train\nhelicopter/image_0050.jpg 81 train\nhelicopter/image_0045.jpg 81 train\nhelicopter/image_0017.jpg 81 train\nhelicopter/image_0018.jpg 81 train\nhelicopter/image_0013.jpg 81 train\nhelicopter/image_0054.jpg 81 train\nhelicopter/image_0082.jpg 81 train\nhelicopter/image_0079.jpg 81 train\nhelicopter/image_0055.jpg 81 train\nhelicopter/image_0071.jpg 81 train\nhelicopter/image_0024.jpg 81 train\nhelicopter/image_0074.jpg 81 train\nhelicopter/image_0003.jpg 81 train\nhelicopter/image_0009.jpg 81 train\nhelicopter/image_0042.jpg 81 train\nhelicopter/image_0040.jpg 81 train\nhelicopter/image_0067.jpg 81 train\nhelicopter/image_0047.jpg 81 train\nhelicopter/image_0064.jpg 81 train\nhelicopter/image_0070.jpg 81 train\nhelicopter/image_0078.jpg 81 train\nhelicopter/image_0068.jpg 81 train\nhelicopter/image_0085.jpg 81 train\nhelicopter/image_0043.jpg 81 train\nhelicopter/image_0044.jpg 81 train\nhelicopter/image_0033.jpg 81 train\nhelicopter/image_0073.jpg 81 train\nhelicopter/image_0001.jpg 81 train\nhelicopter/image_0080.jpg 81 train\nhelicopter/image_0021.jpg 81 train\nhelicopter/image_0057.jpg 81 train\nhelicopter/image_0008.jpg 81 train\nhelicopter/image_0035.jpg 81 train\nhelicopter/image_0005.jpg 81 train\nhelicopter/image_0087.jpg 81 train\nhelicopter/image_0059.jpg 81 train\nhelicopter/image_0069.jpg 81 train\nhelicopter/image_0034.jpg 81 train\nhelicopter/image_0002.jpg 81 train\nhelicopter/image_0048.jpg 81 train\nhelicopter/image_0053.jpg 81 train\nhelicopter/image_0084.jpg 81 train\nhelicopter/image_0020.jpg 81 train\nhelicopter/image_0046.jpg 81 train\nhelicopter/image_0041.jpg 81 train\nhelicopter/image_0027.jpg 81 train\nhelicopter/image_0014.jpg 81 train\nhelicopter/image_0058.jpg 81 train\nhelicopter/image_0081.jpg 81 train\nhelicopter/image_0088.jpg 81 train\npagoda/image_0037.jpg 82 train\npagoda/image_0006.jpg 82 train\npagoda/image_0016.jpg 82 train\npagoda/image_0030.jpg 82 train\npagoda/image_0012.jpg 82 train\npagoda/image_0025.jpg 82 train\npagoda/image_0007.jpg 82 train\npagoda/image_0019.jpg 82 train\npagoda/image_0031.jpg 82 train\npagoda/image_0011.jpg 82 train\npagoda/image_0045.jpg 82 train\npagoda/image_0017.jpg 82 train\npagoda/image_0018.jpg 82 train\npagoda/image_0013.jpg 82 train\npagoda/image_0024.jpg 82 train\npagoda/image_0003.jpg 82 train\npagoda/image_0009.jpg 82 train\npagoda/image_0042.jpg 82 train\npagoda/image_0040.jpg 82 train\npagoda/image_0047.jpg 82 train\npagoda/image_0043.jpg 82 train\npagoda/image_0044.jpg 82 train\npagoda/image_0033.jpg 82 train\npagoda/image_0001.jpg 82 train\npagoda/image_0021.jpg 82 train\npagoda/image_0008.jpg 82 train\npagoda/image_0035.jpg 82 train\npagoda/image_0005.jpg 82 train\npagoda/image_0034.jpg 82 train\npagoda/image_0002.jpg 82 train\npagoda/image_0020.jpg 82 train\npagoda/image_0046.jpg 82 train\npagoda/image_0041.jpg 82 train\npagoda/image_0027.jpg 82 train\npagoda/image_0014.jpg 82 train\npagoda/image_0026.jpg 82 train\npagoda/image_0004.jpg 82 train\newer/image_0062.jpg 83 train\newer/image_0037.jpg 83 train\newer/image_0049.jpg 83 train\newer/image_0083.jpg 83 train\newer/image_0051.jpg 83 train\newer/image_0006.jpg 83 train\newer/image_0016.jpg 83 train\newer/image_0030.jpg 83 train\newer/image_0012.jpg 83 train\newer/image_0066.jpg 83 train\newer/image_0077.jpg 83 train\newer/image_0025.jpg 83 train\newer/image_0060.jpg 83 train\newer/image_0052.jpg 83 train\newer/image_0007.jpg 83 train\newer/image_0063.jpg 83 train\newer/image_0019.jpg 83 train\newer/image_0031.jpg 83 train\newer/image_0011.jpg 83 train\newer/image_0061.jpg 83 train\newer/image_0050.jpg 83 train\newer/image_0045.jpg 83 train\newer/image_0017.jpg 83 train\newer/image_0018.jpg 83 train\newer/image_0013.jpg 83 train\newer/image_0054.jpg 83 train\newer/image_0082.jpg 83 train\newer/image_0079.jpg 83 train\newer/image_0055.jpg 83 train\newer/image_0071.jpg 83 train\newer/image_0024.jpg 83 train\newer/image_0074.jpg 83 train\newer/image_0003.jpg 83 train\newer/image_0009.jpg 83 train\newer/image_0042.jpg 83 train\newer/image_0040.jpg 83 train\newer/image_0067.jpg 83 train\newer/image_0047.jpg 83 train\newer/image_0064.jpg 83 train\newer/image_0070.jpg 83 train\newer/image_0078.jpg 83 train\newer/image_0068.jpg 83 train\newer/image_0085.jpg 83 train\newer/image_0043.jpg 83 train\newer/image_0044.jpg 83 train\newer/image_0033.jpg 83 train\newer/image_0073.jpg 83 train\newer/image_0001.jpg 83 train\newer/image_0080.jpg 83 train\newer/image_0021.jpg 83 train\newer/image_0057.jpg 83 train\newer/image_0008.jpg 83 train\newer/image_0035.jpg 83 train\newer/image_0005.jpg 83 train\newer/image_0059.jpg 83 train\newer/image_0069.jpg 83 train\newer/image_0034.jpg 83 train\newer/image_0002.jpg 83 train\newer/image_0048.jpg 83 train\newer/image_0053.jpg 83 train\newer/image_0084.jpg 83 train\newer/image_0020.jpg 83 train\newer/image_0046.jpg 83 train\newer/image_0041.jpg 83 train\newer/image_0027.jpg 83 train\newer/image_0014.jpg 83 train\newer/image_0058.jpg 83 train\newer/image_0081.jpg 83 train\npanda/image_0037.jpg 84 train\npanda/image_0006.jpg 84 train\npanda/image_0016.jpg 84 train\npanda/image_0030.jpg 84 train\npanda/image_0012.jpg 84 train\npanda/image_0025.jpg 84 train\npanda/image_0007.jpg 84 train\npanda/image_0019.jpg 84 train\npanda/image_0031.jpg 84 train\npanda/image_0011.jpg 84 train\npanda/image_0017.jpg 84 train\npanda/image_0018.jpg 84 train\npanda/image_0013.jpg 84 train\npanda/image_0024.jpg 84 train\npanda/image_0003.jpg 84 train\npanda/image_0009.jpg 84 train\npanda/image_0033.jpg 84 train\npanda/image_0001.jpg 84 train\npanda/image_0021.jpg 84 train\npanda/image_0008.jpg 84 train\npanda/image_0035.jpg 84 train\npanda/image_0005.jpg 84 train\npanda/image_0034.jpg 84 train\npanda/image_0002.jpg 84 train\npanda/image_0020.jpg 84 train\npanda/image_0027.jpg 84 train\npanda/image_0014.jpg 84 train\npanda/image_0026.jpg 84 train\npanda/image_0004.jpg 84 train\npanda/image_0015.jpg 84 train\npizza/image_0037.jpg 85 train\npizza/image_0049.jpg 85 train\npizza/image_0051.jpg 85 train\npizza/image_0006.jpg 85 train\npizza/image_0016.jpg 85 train\npizza/image_0030.jpg 85 train\npizza/image_0012.jpg 85 train\npizza/image_0025.jpg 85 train\npizza/image_0052.jpg 85 train\npizza/image_0007.jpg 85 train\npizza/image_0019.jpg 85 train\npizza/image_0031.jpg 85 train\npizza/image_0011.jpg 85 train\npizza/image_0050.jpg 85 train\npizza/image_0045.jpg 85 train\npizza/image_0017.jpg 85 train\npizza/image_0018.jpg 85 train\npizza/image_0013.jpg 85 train\npizza/image_0024.jpg 85 train\npizza/image_0003.jpg 85 train\npizza/image_0009.jpg 85 train\npizza/image_0042.jpg 85 train\npizza/image_0040.jpg 85 train\npizza/image_0047.jpg 85 train\npizza/image_0043.jpg 85 train\npizza/image_0044.jpg 85 train\npizza/image_0033.jpg 85 train\npizza/image_0001.jpg 85 train\npizza/image_0021.jpg 85 train\npizza/image_0008.jpg 85 train\npizza/image_0035.jpg 85 train\npizza/image_0005.jpg 85 train\npizza/image_0034.jpg 85 train\npizza/image_0002.jpg 85 train\npizza/image_0048.jpg 85 train\npizza/image_0053.jpg 85 train\npizza/image_0020.jpg 85 train\npizza/image_0046.jpg 85 train\npizza/image_0041.jpg 85 train\npizza/image_0027.jpg 85 train\npizza/image_0014.jpg 85 train\npizza/image_0026.jpg 85 train\ncup/image_0037.jpg 86 train\ncup/image_0049.jpg 86 train\ncup/image_0051.jpg 86 train\ncup/image_0006.jpg 86 train\ncup/image_0016.jpg 86 train\ncup/image_0030.jpg 86 train\ncup/image_0012.jpg 86 train\ncup/image_0025.jpg 86 train\ncup/image_0052.jpg 86 train\ncup/image_0007.jpg 86 train\ncup/image_0019.jpg 86 train\ncup/image_0031.jpg 86 train\ncup/image_0011.jpg 86 train\ncup/image_0050.jpg 86 train\ncup/image_0045.jpg 86 train\ncup/image_0017.jpg 86 train\ncup/image_0018.jpg 86 train\ncup/image_0013.jpg 86 train\ncup/image_0054.jpg 86 train\ncup/image_0055.jpg 86 train\ncup/image_0024.jpg 86 train\ncup/image_0003.jpg 86 train\ncup/image_0009.jpg 86 train\ncup/image_0042.jpg 86 train\ncup/image_0040.jpg 86 train\ncup/image_0047.jpg 86 train\ncup/image_0043.jpg 86 train\ncup/image_0044.jpg 86 train\ncup/image_0033.jpg 86 train\ncup/image_0001.jpg 86 train\ncup/image_0021.jpg 86 train\ncup/image_0057.jpg 86 train\ncup/image_0008.jpg 86 train\ncup/image_0035.jpg 86 train\ncup/image_0005.jpg 86 train\ncup/image_0034.jpg 86 train\ncup/image_0002.jpg 86 train\ncup/image_0048.jpg 86 train\ncup/image_0053.jpg 86 train\ncup/image_0020.jpg 86 train\ncup/image_0046.jpg 86 train\ncup/image_0041.jpg 86 train\ncup/image_0027.jpg 86 train\ncup/image_0014.jpg 86 train\ncup/image_0026.jpg 86 train\nanchor/image_0037.jpg 87 train\nanchor/image_0006.jpg 87 train\nanchor/image_0016.jpg 87 train\nanchor/image_0030.jpg 87 train\nanchor/image_0012.jpg 87 train\nanchor/image_0025.jpg 87 train\nanchor/image_0007.jpg 87 train\nanchor/image_0019.jpg 87 train\nanchor/image_0031.jpg 87 train\nanchor/image_0011.jpg 87 train\nanchor/image_0017.jpg 87 train\nanchor/image_0018.jpg 87 train\nanchor/image_0013.jpg 87 train\nanchor/image_0024.jpg 87 train\nanchor/image_0003.jpg 87 train\nanchor/image_0009.jpg 87 train\nanchor/image_0042.jpg 87 train\nanchor/image_0040.jpg 87 train\nanchor/image_0033.jpg 87 train\nanchor/image_0001.jpg 87 train\nanchor/image_0021.jpg 87 train\nanchor/image_0008.jpg 87 train\nanchor/image_0035.jpg 87 train\nanchor/image_0005.jpg 87 train\nanchor/image_0034.jpg 87 train\nanchor/image_0002.jpg 87 train\nanchor/image_0020.jpg 87 train\nanchor/image_0041.jpg 87 train\nanchor/image_0027.jpg 87 train\nanchor/image_0014.jpg 87 train\nanchor/image_0026.jpg 87 train\nanchor/image_0004.jpg 87 train\nanchor/image_0015.jpg 87 train\nhedgehog/image_0037.jpg 88 train\nhedgehog/image_0049.jpg 88 train\nhedgehog/image_0051.jpg 88 train\nhedgehog/image_0006.jpg 88 train\nhedgehog/image_0016.jpg 88 train\nhedgehog/image_0030.jpg 88 train\nhedgehog/image_0012.jpg 88 train\nhedgehog/image_0025.jpg 88 train\nhedgehog/image_0052.jpg 88 train\nhedgehog/image_0007.jpg 88 train\nhedgehog/image_0019.jpg 88 train\nhedgehog/image_0031.jpg 88 train\nhedgehog/image_0011.jpg 88 train\nhedgehog/image_0050.jpg 88 train\nhedgehog/image_0045.jpg 88 train\nhedgehog/image_0017.jpg 88 train\nhedgehog/image_0018.jpg 88 train\nhedgehog/image_0013.jpg 88 train\nhedgehog/image_0054.jpg 88 train\nhedgehog/image_0024.jpg 88 train\nhedgehog/image_0003.jpg 88 train\nhedgehog/image_0009.jpg 88 train\nhedgehog/image_0042.jpg 88 train\nhedgehog/image_0040.jpg 88 train\nhedgehog/image_0047.jpg 88 train\nhedgehog/image_0043.jpg 88 train\nhedgehog/image_0044.jpg 88 train\nhedgehog/image_0033.jpg 88 train\nhedgehog/image_0001.jpg 88 train\nhedgehog/image_0021.jpg 88 train\nhedgehog/image_0008.jpg 88 train\nhedgehog/image_0035.jpg 88 train\nhedgehog/image_0005.jpg 88 train\nhedgehog/image_0034.jpg 88 train\nhedgehog/image_0002.jpg 88 train\nhedgehog/image_0048.jpg 88 train\nhedgehog/image_0053.jpg 88 train\nhedgehog/image_0020.jpg 88 train\nhedgehog/image_0046.jpg 88 train\nhedgehog/image_0041.jpg 88 train\nhedgehog/image_0027.jpg 88 train\nhedgehog/image_0014.jpg 88 train\nhedgehog/image_0026.jpg 88 train\nflamingo/image_0062.jpg 89 train\nflamingo/image_0037.jpg 89 train\nflamingo/image_0049.jpg 89 train\nflamingo/image_0051.jpg 89 train\nflamingo/image_0006.jpg 89 train\nflamingo/image_0016.jpg 89 train\nflamingo/image_0030.jpg 89 train\nflamingo/image_0012.jpg 89 train\nflamingo/image_0066.jpg 89 train\nflamingo/image_0025.jpg 89 train\nflamingo/image_0060.jpg 89 train\nflamingo/image_0052.jpg 89 train\nflamingo/image_0007.jpg 89 train\nflamingo/image_0063.jpg 89 train\nflamingo/image_0019.jpg 89 train\nflamingo/image_0031.jpg 89 train\nflamingo/image_0011.jpg 89 train\nflamingo/image_0061.jpg 89 train\nflamingo/image_0050.jpg 89 train\nflamingo/image_0045.jpg 89 train\nflamingo/image_0017.jpg 89 train\nflamingo/image_0018.jpg 89 train\nflamingo/image_0013.jpg 89 train\nflamingo/image_0054.jpg 89 train\nflamingo/image_0055.jpg 89 train\nflamingo/image_0024.jpg 89 train\nflamingo/image_0003.jpg 89 train\nflamingo/image_0009.jpg 89 train\nflamingo/image_0042.jpg 89 train\nflamingo/image_0040.jpg 89 train\nflamingo/image_0067.jpg 89 train\nflamingo/image_0047.jpg 89 train\nflamingo/image_0064.jpg 89 train\nflamingo/image_0043.jpg 89 train\nflamingo/image_0044.jpg 89 train\nflamingo/image_0033.jpg 89 train\nflamingo/image_0001.jpg 89 train\nflamingo/image_0021.jpg 89 train\nflamingo/image_0057.jpg 89 train\nflamingo/image_0008.jpg 89 train\nflamingo/image_0035.jpg 89 train\nflamingo/image_0005.jpg 89 train\nflamingo/image_0059.jpg 89 train\nflamingo/image_0034.jpg 89 train\nflamingo/image_0002.jpg 89 train\nflamingo/image_0048.jpg 89 train\nflamingo/image_0053.jpg 89 train\nflamingo/image_0020.jpg 89 train\nflamingo/image_0046.jpg 89 train\nflamingo/image_0041.jpg 89 train\nflamingo/image_0027.jpg 89 train\nflamingo/image_0014.jpg 89 train\nflamingo/image_0058.jpg 89 train\nstegosaurus/image_0037.jpg 90 train\nstegosaurus/image_0049.jpg 90 train\nstegosaurus/image_0051.jpg 90 train\nstegosaurus/image_0006.jpg 90 train\nstegosaurus/image_0016.jpg 90 train\nstegosaurus/image_0030.jpg 90 train\nstegosaurus/image_0012.jpg 90 train\nstegosaurus/image_0025.jpg 90 train\nstegosaurus/image_0052.jpg 90 train\nstegosaurus/image_0007.jpg 90 train\nstegosaurus/image_0019.jpg 90 train\nstegosaurus/image_0031.jpg 90 train\nstegosaurus/image_0011.jpg 90 train\nstegosaurus/image_0050.jpg 90 train\nstegosaurus/image_0045.jpg 90 train\nstegosaurus/image_0017.jpg 90 train\nstegosaurus/image_0018.jpg 90 train\nstegosaurus/image_0013.jpg 90 train\nstegosaurus/image_0054.jpg 90 train\nstegosaurus/image_0055.jpg 90 train\nstegosaurus/image_0024.jpg 90 train\nstegosaurus/image_0003.jpg 90 train\nstegosaurus/image_0009.jpg 90 train\nstegosaurus/image_0042.jpg 90 train\nstegosaurus/image_0040.jpg 90 train\nstegosaurus/image_0047.jpg 90 train\nstegosaurus/image_0043.jpg 90 train\nstegosaurus/image_0044.jpg 90 train\nstegosaurus/image_0033.jpg 90 train\nstegosaurus/image_0001.jpg 90 train\nstegosaurus/image_0021.jpg 90 train\nstegosaurus/image_0057.jpg 90 train\nstegosaurus/image_0008.jpg 90 train\nstegosaurus/image_0035.jpg 90 train\nstegosaurus/image_0005.jpg 90 train\nstegosaurus/image_0059.jpg 90 train\nstegosaurus/image_0034.jpg 90 train\nstegosaurus/image_0002.jpg 90 train\nstegosaurus/image_0048.jpg 90 train\nstegosaurus/image_0053.jpg 90 train\nstegosaurus/image_0020.jpg 90 train\nstegosaurus/image_0046.jpg 90 train\nstegosaurus/image_0041.jpg 90 train\nstegosaurus/image_0027.jpg 90 train\nstegosaurus/image_0014.jpg 90 train\nstegosaurus/image_0058.jpg 90 train\nstegosaurus/image_0026.jpg 90 train\nferry/image_0062.jpg 91 train\nferry/image_0037.jpg 91 train\nferry/image_0049.jpg 91 train\nferry/image_0051.jpg 91 train\nferry/image_0006.jpg 91 train\nferry/image_0016.jpg 91 train\nferry/image_0030.jpg 91 train\nferry/image_0012.jpg 91 train\nferry/image_0066.jpg 91 train\nferry/image_0025.jpg 91 train\nferry/image_0060.jpg 91 train\nferry/image_0052.jpg 91 train\nferry/image_0007.jpg 91 train\nferry/image_0063.jpg 91 train\nferry/image_0019.jpg 91 train\nferry/image_0031.jpg 91 train\nferry/image_0011.jpg 91 train\nferry/image_0061.jpg 91 train\nferry/image_0050.jpg 91 train\nferry/image_0045.jpg 91 train\nferry/image_0017.jpg 91 train\nferry/image_0018.jpg 91 train\nferry/image_0013.jpg 91 train\nferry/image_0054.jpg 91 train\nferry/image_0055.jpg 91 train\nferry/image_0024.jpg 91 train\nferry/image_0003.jpg 91 train\nferry/image_0009.jpg 91 train\nferry/image_0042.jpg 91 train\nferry/image_0040.jpg 91 train\nferry/image_0067.jpg 91 train\nferry/image_0047.jpg 91 train\nferry/image_0064.jpg 91 train\nferry/image_0043.jpg 91 train\nferry/image_0044.jpg 91 train\nferry/image_0033.jpg 91 train\nferry/image_0001.jpg 91 train\nferry/image_0021.jpg 91 train\nferry/image_0057.jpg 91 train\nferry/image_0008.jpg 91 train\nferry/image_0035.jpg 91 train\nferry/image_0005.jpg 91 train\nferry/image_0059.jpg 91 train\nferry/image_0034.jpg 91 train\nferry/image_0002.jpg 91 train\nferry/image_0048.jpg 91 train\nferry/image_0053.jpg 91 train\nferry/image_0020.jpg 91 train\nferry/image_0046.jpg 91 train\nferry/image_0041.jpg 91 train\nferry/image_0027.jpg 91 train\nferry/image_0014.jpg 91 train\nferry/image_0058.jpg 91 train\ndalmatian/image_0062.jpg 92 train\ndalmatian/image_0037.jpg 92 train\ndalmatian/image_0049.jpg 92 train\ndalmatian/image_0051.jpg 92 train\ndalmatian/image_0006.jpg 92 train\ndalmatian/image_0016.jpg 92 train\ndalmatian/image_0030.jpg 92 train\ndalmatian/image_0012.jpg 92 train\ndalmatian/image_0066.jpg 92 train\ndalmatian/image_0025.jpg 92 train\ndalmatian/image_0060.jpg 92 train\ndalmatian/image_0052.jpg 92 train\ndalmatian/image_0007.jpg 92 train\ndalmatian/image_0063.jpg 92 train\ndalmatian/image_0019.jpg 92 train\ndalmatian/image_0031.jpg 92 train\ndalmatian/image_0011.jpg 92 train\ndalmatian/image_0061.jpg 92 train\ndalmatian/image_0050.jpg 92 train\ndalmatian/image_0045.jpg 92 train\ndalmatian/image_0017.jpg 92 train\ndalmatian/image_0018.jpg 92 train\ndalmatian/image_0013.jpg 92 train\ndalmatian/image_0054.jpg 92 train\ndalmatian/image_0055.jpg 92 train\ndalmatian/image_0024.jpg 92 train\ndalmatian/image_0003.jpg 92 train\ndalmatian/image_0009.jpg 92 train\ndalmatian/image_0042.jpg 92 train\ndalmatian/image_0040.jpg 92 train\ndalmatian/image_0067.jpg 92 train\ndalmatian/image_0047.jpg 92 train\ndalmatian/image_0064.jpg 92 train\ndalmatian/image_0043.jpg 92 train\ndalmatian/image_0044.jpg 92 train\ndalmatian/image_0033.jpg 92 train\ndalmatian/image_0001.jpg 92 train\ndalmatian/image_0021.jpg 92 train\ndalmatian/image_0057.jpg 92 train\ndalmatian/image_0008.jpg 92 train\ndalmatian/image_0035.jpg 92 train\ndalmatian/image_0005.jpg 92 train\ndalmatian/image_0059.jpg 92 train\ndalmatian/image_0034.jpg 92 train\ndalmatian/image_0002.jpg 92 train\ndalmatian/image_0048.jpg 92 train\ndalmatian/image_0053.jpg 92 train\ndalmatian/image_0020.jpg 92 train\ndalmatian/image_0046.jpg 92 train\ndalmatian/image_0041.jpg 92 train\ndalmatian/image_0027.jpg 92 train\ndalmatian/image_0014.jpg 92 train\ndalmatian/image_0058.jpg 92 train\nwheelchair/image_0037.jpg 93 train\nwheelchair/image_0049.jpg 93 train\nwheelchair/image_0051.jpg 93 train\nwheelchair/image_0006.jpg 93 train\nwheelchair/image_0016.jpg 93 train\nwheelchair/image_0030.jpg 93 train\nwheelchair/image_0012.jpg 93 train\nwheelchair/image_0025.jpg 93 train\nwheelchair/image_0052.jpg 93 train\nwheelchair/image_0007.jpg 93 train\nwheelchair/image_0019.jpg 93 train\nwheelchair/image_0031.jpg 93 train\nwheelchair/image_0011.jpg 93 train\nwheelchair/image_0050.jpg 93 train\nwheelchair/image_0045.jpg 93 train\nwheelchair/image_0017.jpg 93 train\nwheelchair/image_0018.jpg 93 train\nwheelchair/image_0013.jpg 93 train\nwheelchair/image_0054.jpg 93 train\nwheelchair/image_0055.jpg 93 train\nwheelchair/image_0024.jpg 93 train\nwheelchair/image_0003.jpg 93 train\nwheelchair/image_0009.jpg 93 train\nwheelchair/image_0042.jpg 93 train\nwheelchair/image_0040.jpg 93 train\nwheelchair/image_0047.jpg 93 train\nwheelchair/image_0043.jpg 93 train\nwheelchair/image_0044.jpg 93 train\nwheelchair/image_0033.jpg 93 train\nwheelchair/image_0001.jpg 93 train\nwheelchair/image_0021.jpg 93 train\nwheelchair/image_0057.jpg 93 train\nwheelchair/image_0008.jpg 93 train\nwheelchair/image_0035.jpg 93 train\nwheelchair/image_0005.jpg 93 train\nwheelchair/image_0059.jpg 93 train\nwheelchair/image_0034.jpg 93 train\nwheelchair/image_0002.jpg 93 train\nwheelchair/image_0048.jpg 93 train\nwheelchair/image_0053.jpg 93 train\nwheelchair/image_0020.jpg 93 train\nwheelchair/image_0046.jpg 93 train\nwheelchair/image_0041.jpg 93 train\nwheelchair/image_0027.jpg 93 train\nwheelchair/image_0014.jpg 93 train\nwheelchair/image_0058.jpg 93 train\nwheelchair/image_0026.jpg 93 train\nwatch/image_0171.jpg 94 train\nwatch/image_0062.jpg 94 train\nwatch/image_0037.jpg 94 train\nwatch/image_0049.jpg 94 train\nwatch/image_0083.jpg 94 train\nwatch/image_0101.jpg 94 train\nwatch/image_0051.jpg 94 train\nwatch/image_0006.jpg 94 train\nwatch/image_0142.jpg 94 train\nwatch/image_0201.jpg 94 train\nwatch/image_0016.jpg 94 train\nwatch/image_0176.jpg 94 train\nwatch/image_0155.jpg 94 train\nwatch/image_0030.jpg 94 train\nwatch/image_0091.jpg 94 train\nwatch/image_0012.jpg 94 train\nwatch/image_0066.jpg 94 train\nwatch/image_0224.jpg 94 train\nwatch/image_0148.jpg 94 train\nwatch/image_0077.jpg 94 train\nwatch/image_0129.jpg 94 train\nwatch/image_0025.jpg 94 train\nwatch/image_0139.jpg 94 train\nwatch/image_0115.jpg 94 train\nwatch/image_0123.jpg 94 train\nwatch/image_0119.jpg 94 train\nwatch/image_0060.jpg 94 train\nwatch/image_0052.jpg 94 train\nwatch/image_0108.jpg 94 train\nwatch/image_0239.jpg 94 train\nwatch/image_0223.jpg 94 train\nwatch/image_0172.jpg 94 train\nwatch/image_0007.jpg 94 train\nwatch/image_0213.jpg 94 train\nwatch/image_0063.jpg 94 train\nwatch/image_0019.jpg 94 train\nwatch/image_0192.jpg 94 train\nwatch/image_0195.jpg 94 train\nwatch/image_0113.jpg 94 train\nwatch/image_0127.jpg 94 train\nwatch/image_0215.jpg 94 train\nwatch/image_0031.jpg 94 train\nwatch/image_0228.jpg 94 train\nwatch/image_0219.jpg 94 train\nwatch/image_0011.jpg 94 train\nwatch/image_0233.jpg 94 train\nwatch/image_0061.jpg 94 train\nwatch/image_0090.jpg 94 train\nwatch/image_0229.jpg 94 train\nwatch/image_0126.jpg 94 train\nwatch/image_0218.jpg 94 train\nwatch/image_0216.jpg 94 train\nwatch/image_0050.jpg 94 train\nwatch/image_0177.jpg 94 train\nwatch/image_0196.jpg 94 train\nwatch/image_0187.jpg 94 train\nwatch/image_0234.jpg 94 train\nwatch/image_0045.jpg 94 train\nwatch/image_0017.jpg 94 train\nwatch/image_0131.jpg 94 train\nwatch/image_0150.jpg 94 train\nwatch/image_0018.jpg 94 train\nwatch/image_0209.jpg 94 train\nwatch/image_0013.jpg 94 train\nwatch/image_0054.jpg 94 train\nwatch/image_0094.jpg 94 train\nwatch/image_0082.jpg 94 train\nwatch/image_0097.jpg 94 train\nwatch/image_0130.jpg 94 train\nwatch/image_0141.jpg 94 train\nwatch/image_0144.jpg 94 train\nwatch/image_0109.jpg 94 train\nwatch/image_0079.jpg 94 train\nwatch/image_0055.jpg 94 train\nwatch/image_0217.jpg 94 train\nwatch/image_0071.jpg 94 train\nwatch/image_0198.jpg 94 train\nwatch/image_0168.jpg 94 train\nwatch/image_0211.jpg 94 train\nwatch/image_0103.jpg 94 train\nwatch/image_0024.jpg 94 train\nwatch/image_0188.jpg 94 train\nwatch/image_0105.jpg 94 train\nwatch/image_0074.jpg 94 train\nwatch/image_0189.jpg 94 train\nwatch/image_0003.jpg 94 train\nwatch/image_0009.jpg 94 train\nwatch/image_0226.jpg 94 train\nwatch/image_0185.jpg 94 train\nwatch/image_0118.jpg 94 train\nwatch/image_0230.jpg 94 train\nwatch/image_0134.jpg 94 train\nwatch/image_0042.jpg 94 train\nwatch/image_0221.jpg 94 train\nwatch/image_0093.jpg 94 train\nwatch/image_0232.jpg 94 train\nwatch/image_0040.jpg 94 train\nwatch/image_0067.jpg 94 train\nwatch/image_0122.jpg 94 train\nwatch/image_0205.jpg 94 train\nwatch/image_0161.jpg 94 train\nwatch/image_0146.jpg 94 train\nwatch/image_0204.jpg 94 train\nwatch/image_0125.jpg 94 train\nwatch/image_0047.jpg 94 train\nwatch/image_0160.jpg 94 train\nwatch/image_0064.jpg 94 train\nwatch/image_0070.jpg 94 train\nwatch/image_0112.jpg 94 train\nwatch/image_0078.jpg 94 train\nwatch/image_0068.jpg 94 train\nwatch/image_0085.jpg 94 train\nwatch/image_0110.jpg 94 train\nwatch/image_0143.jpg 94 train\nwatch/image_0043.jpg 94 train\nwatch/image_0162.jpg 94 train\nwatch/image_0132.jpg 94 train\nwatch/image_0157.jpg 94 train\nwatch/image_0206.jpg 94 train\nwatch/image_0044.jpg 94 train\nwatch/image_0199.jpg 94 train\nwatch/image_0033.jpg 94 train\nwatch/image_0136.jpg 94 train\nwatch/image_0180.jpg 94 train\nwatch/image_0184.jpg 94 train\nwatch/image_0073.jpg 94 train\nwatch/image_0001.jpg 94 train\nwatch/image_0167.jpg 94 train\nwatch/image_0080.jpg 94 train\nwatch/image_0156.jpg 94 train\nwatch/image_0021.jpg 94 train\nwatch/image_0095.jpg 94 train\nwatch/image_0057.jpg 94 train\nwatch/image_0114.jpg 94 train\nwatch/image_0153.jpg 94 train\nwatch/image_0098.jpg 94 train\nwatch/image_0203.jpg 94 train\nwatch/image_0147.jpg 94 train\nwatch/image_0008.jpg 94 train\nwatch/image_0207.jpg 94 train\nwatch/image_0208.jpg 94 train\nwatch/image_0222.jpg 94 train\nwatch/image_0227.jpg 94 train\nwatch/image_0035.jpg 94 train\nwatch/image_0166.jpg 94 train\nwatch/image_0169.jpg 94 train\nwatch/image_0191.jpg 94 train\nwatch/image_0128.jpg 94 train\nwatch/image_0102.jpg 94 train\nwatch/image_0178.jpg 94 train\nwatch/image_0005.jpg 94 train\nwatch/image_0173.jpg 94 train\nwatch/image_0087.jpg 94 train\nwatch/image_0089.jpg 94 train\nwatch/image_0059.jpg 94 train\nwatch/image_0159.jpg 94 train\nwatch/image_0225.jpg 94 train\nwatch/image_0186.jpg 94 train\nwatch/image_0069.jpg 94 train\nwatch/image_0200.jpg 94 train\nwatch/image_0099.jpg 94 train\nwatch/image_0034.jpg 94 train\nwatch/image_0121.jpg 94 train\nwatch/image_0181.jpg 94 train\nwatch/image_0183.jpg 94 train\nwatch/image_0175.jpg 94 train\nwatch/image_0002.jpg 94 train\nwatch/image_0165.jpg 94 train\nwatch/image_0154.jpg 94 train\nwatch/image_0193.jpg 94 train\nwatch/image_0048.jpg 94 train\nwatch/image_0053.jpg 94 train\nwatch/image_0084.jpg 94 train\nwatch/image_0020.jpg 94 train\nwatch/image_0046.jpg 94 train\nwatch/image_0041.jpg 94 train\nwatch/image_0027.jpg 94 train\nwatch/image_0014.jpg 94 train\nwatch/image_0058.jpg 94 train\nwatch/image_0212.jpg 94 train\nwatch/image_0163.jpg 94 train\nwatch/image_0202.jpg 94 train\nwatch/image_0104.jpg 94 train\nwatch/image_0081.jpg 94 train\nwatch/image_0237.jpg 94 train\nwatch/image_0149.jpg 94 train\nwatch/image_0088.jpg 94 train\nwatch/image_0026.jpg 94 train\nwatch/image_0231.jpg 94 train\nwatch/image_0004.jpg 94 train\nwatch/image_0015.jpg 94 train\nsea_horse/image_0037.jpg 95 train\nsea_horse/image_0049.jpg 95 train\nsea_horse/image_0051.jpg 95 train\nsea_horse/image_0006.jpg 95 train\nsea_horse/image_0016.jpg 95 train\nsea_horse/image_0030.jpg 95 train\nsea_horse/image_0012.jpg 95 train\nsea_horse/image_0025.jpg 95 train\nsea_horse/image_0052.jpg 95 train\nsea_horse/image_0007.jpg 95 train\nsea_horse/image_0019.jpg 95 train\nsea_horse/image_0031.jpg 95 train\nsea_horse/image_0011.jpg 95 train\nsea_horse/image_0050.jpg 95 train\nsea_horse/image_0045.jpg 95 train\nsea_horse/image_0017.jpg 95 train\nsea_horse/image_0018.jpg 95 train\nsea_horse/image_0013.jpg 95 train\nsea_horse/image_0054.jpg 95 train\nsea_horse/image_0055.jpg 95 train\nsea_horse/image_0024.jpg 95 train\nsea_horse/image_0003.jpg 95 train\nsea_horse/image_0009.jpg 95 train\nsea_horse/image_0042.jpg 95 train\nsea_horse/image_0040.jpg 95 train\nsea_horse/image_0047.jpg 95 train\nsea_horse/image_0043.jpg 95 train\nsea_horse/image_0044.jpg 95 train\nsea_horse/image_0033.jpg 95 train\nsea_horse/image_0001.jpg 95 train\nsea_horse/image_0021.jpg 95 train\nsea_horse/image_0057.jpg 95 train\nsea_horse/image_0008.jpg 95 train\nsea_horse/image_0035.jpg 95 train\nsea_horse/image_0005.jpg 95 train\nsea_horse/image_0034.jpg 95 train\nsea_horse/image_0002.jpg 95 train\nsea_horse/image_0048.jpg 95 train\nsea_horse/image_0053.jpg 95 train\nsea_horse/image_0020.jpg 95 train\nsea_horse/image_0046.jpg 95 train\nsea_horse/image_0041.jpg 95 train\nsea_horse/image_0027.jpg 95 train\nsea_horse/image_0014.jpg 95 train\nsea_horse/image_0026.jpg 95 train\npyramid/image_0037.jpg 96 train\npyramid/image_0049.jpg 96 train\npyramid/image_0051.jpg 96 train\npyramid/image_0006.jpg 96 train\npyramid/image_0016.jpg 96 train\npyramid/image_0030.jpg 96 train\npyramid/image_0012.jpg 96 train\npyramid/image_0025.jpg 96 train\npyramid/image_0052.jpg 96 train\npyramid/image_0007.jpg 96 train\npyramid/image_0019.jpg 96 train\npyramid/image_0031.jpg 96 train\npyramid/image_0011.jpg 96 train\npyramid/image_0050.jpg 96 train\npyramid/image_0045.jpg 96 train\npyramid/image_0017.jpg 96 train\npyramid/image_0018.jpg 96 train\npyramid/image_0013.jpg 96 train\npyramid/image_0054.jpg 96 train\npyramid/image_0055.jpg 96 train\npyramid/image_0024.jpg 96 train\npyramid/image_0003.jpg 96 train\npyramid/image_0009.jpg 96 train\npyramid/image_0042.jpg 96 train\npyramid/image_0040.jpg 96 train\npyramid/image_0047.jpg 96 train\npyramid/image_0043.jpg 96 train\npyramid/image_0044.jpg 96 train\npyramid/image_0033.jpg 96 train\npyramid/image_0001.jpg 96 train\npyramid/image_0021.jpg 96 train\npyramid/image_0057.jpg 96 train\npyramid/image_0008.jpg 96 train\npyramid/image_0035.jpg 96 train\npyramid/image_0005.jpg 96 train\npyramid/image_0034.jpg 96 train\npyramid/image_0002.jpg 96 train\npyramid/image_0048.jpg 96 train\npyramid/image_0053.jpg 96 train\npyramid/image_0020.jpg 96 train\npyramid/image_0046.jpg 96 train\npyramid/image_0041.jpg 96 train\npyramid/image_0027.jpg 96 train\npyramid/image_0014.jpg 96 train\npyramid/image_0026.jpg 96 train\nstrawberry/image_0006.jpg 97 train\nstrawberry/image_0016.jpg 97 train\nstrawberry/image_0030.jpg 97 train\nstrawberry/image_0012.jpg 97 train\nstrawberry/image_0025.jpg 97 train\nstrawberry/image_0007.jpg 97 train\nstrawberry/image_0019.jpg 97 train\nstrawberry/image_0031.jpg 97 train\nstrawberry/image_0011.jpg 97 train\nstrawberry/image_0017.jpg 97 train\nstrawberry/image_0018.jpg 97 train\nstrawberry/image_0013.jpg 97 train\nstrawberry/image_0024.jpg 97 train\nstrawberry/image_0003.jpg 97 train\nstrawberry/image_0009.jpg 97 train\nstrawberry/image_0033.jpg 97 train\nstrawberry/image_0001.jpg 97 train\nstrawberry/image_0021.jpg 97 train\nstrawberry/image_0008.jpg 97 train\nstrawberry/image_0035.jpg 97 train\nstrawberry/image_0005.jpg 97 train\nstrawberry/image_0034.jpg 97 train\nstrawberry/image_0002.jpg 97 train\nstrawberry/image_0020.jpg 97 train\nstrawberry/image_0027.jpg 97 train\nstrawberry/image_0014.jpg 97 train\nstrawberry/image_0026.jpg 97 train\nstrawberry/image_0004.jpg 97 train\nFaces_easy/image_0316.jpg 98 train\nFaces_easy/image_0258.jpg 98 train\nFaces_easy/image_0426.jpg 98 train\nFaces_easy/image_0302.jpg 98 train\nFaces_easy/image_0171.jpg 98 train\nFaces_easy/image_0062.jpg 98 train\nFaces_easy/image_0346.jpg 98 train\nFaces_easy/image_0435.jpg 98 train\nFaces_easy/image_0341.jpg 98 train\nFaces_easy/image_0340.jpg 98 train\nFaces_easy/image_0331.jpg 98 train\nFaces_easy/image_0037.jpg 98 train\nFaces_easy/image_0049.jpg 98 train\nFaces_easy/image_0337.jpg 98 train\nFaces_easy/image_0317.jpg 98 train\nFaces_easy/image_0357.jpg 98 train\nFaces_easy/image_0083.jpg 98 train\nFaces_easy/image_0101.jpg 98 train\nFaces_easy/image_0051.jpg 98 train\nFaces_easy/image_0006.jpg 98 train\nFaces_easy/image_0142.jpg 98 train\nFaces_easy/image_0250.jpg 98 train\nFaces_easy/image_0427.jpg 98 train\nFaces_easy/image_0396.jpg 98 train\nFaces_easy/image_0332.jpg 98 train\nFaces_easy/image_0335.jpg 98 train\nFaces_easy/image_0201.jpg 98 train\nFaces_easy/image_0016.jpg 98 train\nFaces_easy/image_0373.jpg 98 train\nFaces_easy/image_0176.jpg 98 train\nFaces_easy/image_0411.jpg 98 train\nFaces_easy/image_0155.jpg 98 train\nFaces_easy/image_0030.jpg 98 train\nFaces_easy/image_0091.jpg 98 train\nFaces_easy/image_0397.jpg 98 train\nFaces_easy/image_0012.jpg 98 train\nFaces_easy/image_0066.jpg 98 train\nFaces_easy/image_0283.jpg 98 train\nFaces_easy/image_0318.jpg 98 train\nFaces_easy/image_0224.jpg 98 train\nFaces_easy/image_0148.jpg 98 train\nFaces_easy/image_0077.jpg 98 train\nFaces_easy/image_0129.jpg 98 train\nFaces_easy/image_0339.jpg 98 train\nFaces_easy/image_0025.jpg 98 train\nFaces_easy/image_0328.jpg 98 train\nFaces_easy/image_0139.jpg 98 train\nFaces_easy/image_0425.jpg 98 train\nFaces_easy/image_0292.jpg 98 train\nFaces_easy/image_0348.jpg 98 train\nFaces_easy/image_0115.jpg 98 train\nFaces_easy/image_0123.jpg 98 train\nFaces_easy/image_0119.jpg 98 train\nFaces_easy/image_0060.jpg 98 train\nFaces_easy/image_0269.jpg 98 train\nFaces_easy/image_0052.jpg 98 train\nFaces_easy/image_0108.jpg 98 train\nFaces_easy/image_0239.jpg 98 train\nFaces_easy/image_0370.jpg 98 train\nFaces_easy/image_0326.jpg 98 train\nFaces_easy/image_0223.jpg 98 train\nFaces_easy/image_0172.jpg 98 train\nFaces_easy/image_0372.jpg 98 train\nFaces_easy/image_0344.jpg 98 train\nFaces_easy/image_0390.jpg 98 train\nFaces_easy/image_0007.jpg 98 train\nFaces_easy/image_0213.jpg 98 train\nFaces_easy/image_0312.jpg 98 train\nFaces_easy/image_0063.jpg 98 train\nFaces_easy/image_0019.jpg 98 train\nFaces_easy/image_0407.jpg 98 train\nFaces_easy/image_0192.jpg 98 train\nFaces_easy/image_0384.jpg 98 train\nFaces_easy/image_0195.jpg 98 train\nFaces_easy/image_0113.jpg 98 train\nFaces_easy/image_0127.jpg 98 train\nFaces_easy/image_0215.jpg 98 train\nFaces_easy/image_0389.jpg 98 train\nFaces_easy/image_0031.jpg 98 train\nFaces_easy/image_0349.jpg 98 train\nFaces_easy/image_0228.jpg 98 train\nFaces_easy/image_0380.jpg 98 train\nFaces_easy/image_0219.jpg 98 train\nFaces_easy/image_0353.jpg 98 train\nFaces_easy/image_0011.jpg 98 train\nFaces_easy/image_0233.jpg 98 train\nFaces_easy/image_0061.jpg 98 train\nFaces_easy/image_0090.jpg 98 train\nFaces_easy/image_0351.jpg 98 train\nFaces_easy/image_0261.jpg 98 train\nFaces_easy/image_0229.jpg 98 train\nFaces_easy/image_0126.jpg 98 train\nFaces_easy/image_0240.jpg 98 train\nFaces_easy/image_0305.jpg 98 train\nFaces_easy/image_0334.jpg 98 train\nFaces_easy/image_0354.jpg 98 train\nFaces_easy/image_0218.jpg 98 train\nFaces_easy/image_0366.jpg 98 train\nFaces_easy/image_0288.jpg 98 train\nFaces_easy/image_0216.jpg 98 train\nFaces_easy/image_0429.jpg 98 train\nFaces_easy/image_0277.jpg 98 train\nFaces_easy/image_0358.jpg 98 train\nFaces_easy/image_0050.jpg 98 train\nFaces_easy/image_0246.jpg 98 train\nFaces_easy/image_0361.jpg 98 train\nFaces_easy/image_0177.jpg 98 train\nFaces_easy/image_0356.jpg 98 train\nFaces_easy/image_0196.jpg 98 train\nFaces_easy/image_0293.jpg 98 train\nFaces_easy/image_0406.jpg 98 train\nFaces_easy/image_0347.jpg 98 train\nFaces_easy/image_0187.jpg 98 train\nFaces_easy/image_0234.jpg 98 train\nFaces_easy/image_0045.jpg 98 train\nFaces_easy/image_0303.jpg 98 train\nFaces_easy/image_0363.jpg 98 train\nFaces_easy/image_0017.jpg 98 train\nFaces_easy/image_0131.jpg 98 train\nFaces_easy/image_0296.jpg 98 train\nFaces_easy/image_0150.jpg 98 train\nFaces_easy/image_0018.jpg 98 train\nFaces_easy/image_0350.jpg 98 train\nFaces_easy/image_0209.jpg 98 train\nFaces_easy/image_0013.jpg 98 train\nFaces_easy/image_0054.jpg 98 train\nFaces_easy/image_0094.jpg 98 train\nFaces_easy/image_0082.jpg 98 train\nFaces_easy/image_0097.jpg 98 train\nFaces_easy/image_0130.jpg 98 train\nFaces_easy/image_0382.jpg 98 train\nFaces_easy/image_0141.jpg 98 train\nFaces_easy/image_0144.jpg 98 train\nFaces_easy/image_0109.jpg 98 train\nFaces_easy/image_0079.jpg 98 train\nFaces_easy/image_0306.jpg 98 train\nFaces_easy/image_0398.jpg 98 train\nFaces_easy/image_0055.jpg 98 train\nFaces_easy/image_0367.jpg 98 train\nFaces_easy/image_0365.jpg 98 train\nFaces_easy/image_0217.jpg 98 train\nFaces_easy/image_0343.jpg 98 train\nFaces_easy/image_0432.jpg 98 train\nFaces_easy/image_0071.jpg 98 train\nFaces_easy/image_0282.jpg 98 train\nFaces_easy/image_0393.jpg 98 train\nFaces_easy/image_0287.jpg 98 train\nFaces_easy/image_0198.jpg 98 train\nFaces_easy/image_0262.jpg 98 train\nFaces_easy/image_0168.jpg 98 train\nFaces_easy/image_0368.jpg 98 train\nFaces_easy/image_0420.jpg 98 train\nFaces_easy/image_0211.jpg 98 train\nFaces_easy/image_0270.jpg 98 train\nFaces_easy/image_0103.jpg 98 train\nFaces_easy/image_0024.jpg 98 train\nFaces_easy/image_0297.jpg 98 train\nFaces_easy/image_0188.jpg 98 train\nFaces_easy/image_0105.jpg 98 train\nFaces_easy/image_0074.jpg 98 train\nFaces_easy/image_0294.jpg 98 train\nFaces_easy/image_0391.jpg 98 train\nFaces_easy/image_0189.jpg 98 train\nFaces_easy/image_0003.jpg 98 train\nFaces_easy/image_0009.jpg 98 train\nFaces_easy/image_0387.jpg 98 train\nFaces_easy/image_0275.jpg 98 train\nFaces_easy/image_0226.jpg 98 train\nFaces_easy/image_0185.jpg 98 train\nFaces_easy/image_0118.jpg 98 train\nFaces_easy/image_0304.jpg 98 train\nFaces_easy/image_0278.jpg 98 train\nFaces_easy/image_0230.jpg 98 train\nFaces_easy/image_0404.jpg 98 train\nFaces_easy/image_0134.jpg 98 train\nFaces_easy/image_0249.jpg 98 train\nFaces_easy/image_0042.jpg 98 train\nFaces_easy/image_0352.jpg 98 train\nFaces_easy/image_0333.jpg 98 train\nFaces_easy/image_0221.jpg 98 train\nFaces_easy/image_0093.jpg 98 train\nFaces_easy/image_0266.jpg 98 train\nFaces_easy/image_0412.jpg 98 train\nFaces_easy/image_0301.jpg 98 train\nFaces_easy/image_0392.jpg 98 train\nFaces_easy/image_0254.jpg 98 train\nFaces_easy/image_0321.jpg 98 train\nFaces_easy/image_0386.jpg 98 train\nFaces_easy/image_0232.jpg 98 train\nFaces_easy/image_0324.jpg 98 train\nFaces_easy/image_0040.jpg 98 train\nFaces_easy/image_0067.jpg 98 train\nFaces_easy/image_0310.jpg 98 train\nFaces_easy/image_0330.jpg 98 train\nFaces_easy/image_0122.jpg 98 train\nFaces_easy/image_0205.jpg 98 train\nFaces_easy/image_0161.jpg 98 train\nFaces_easy/image_0146.jpg 98 train\nFaces_easy/image_0300.jpg 98 train\nFaces_easy/image_0369.jpg 98 train\nFaces_easy/image_0204.jpg 98 train\nFaces_easy/image_0125.jpg 98 train\nFaces_easy/image_0047.jpg 98 train\nFaces_easy/image_0285.jpg 98 train\nFaces_easy/image_0160.jpg 98 train\nFaces_easy/image_0422.jpg 98 train\nFaces_easy/image_0064.jpg 98 train\nFaces_easy/image_0070.jpg 98 train\nFaces_easy/image_0264.jpg 98 train\nFaces_easy/image_0112.jpg 98 train\nFaces_easy/image_0078.jpg 98 train\nFaces_easy/image_0068.jpg 98 train\nFaces_easy/image_0433.jpg 98 train\nFaces_easy/image_0085.jpg 98 train\nFaces_easy/image_0410.jpg 98 train\nFaces_easy/image_0110.jpg 98 train\nFaces_easy/image_0259.jpg 98 train\nFaces_easy/image_0143.jpg 98 train\nFaces_easy/image_0043.jpg 98 train\nFaces_easy/image_0162.jpg 98 train\nFaces_easy/image_0132.jpg 98 train\nFaces_easy/image_0157.jpg 98 train\nFaces_easy/image_0206.jpg 98 train\nFaces_easy/image_0376.jpg 98 train\nFaces_easy/image_0319.jpg 98 train\nFaces_easy/image_0371.jpg 98 train\nFaces_easy/image_0044.jpg 98 train\nFaces_easy/image_0199.jpg 98 train\nFaces_easy/image_0033.jpg 98 train\nFaces_easy/image_0136.jpg 98 train\nFaces_easy/image_0180.jpg 98 train\nFaces_easy/image_0260.jpg 98 train\nFaces_easy/image_0184.jpg 98 train\nFaces_easy/image_0073.jpg 98 train\nFaces_easy/image_0395.jpg 98 train\nFaces_easy/image_0257.jpg 98 train\nFaces_easy/image_0364.jpg 98 train\nFaces_easy/image_0001.jpg 98 train\nFaces_easy/image_0167.jpg 98 train\nFaces_easy/image_0284.jpg 98 train\nFaces_easy/image_0375.jpg 98 train\nFaces_easy/image_0080.jpg 98 train\nFaces_easy/image_0156.jpg 98 train\nFaces_easy/image_0021.jpg 98 train\nFaces_easy/image_0325.jpg 98 train\nFaces_easy/image_0267.jpg 98 train\nFaces_easy/image_0299.jpg 98 train\nFaces_easy/image_0421.jpg 98 train\nFaces_easy/image_0360.jpg 98 train\nFaces_easy/image_0095.jpg 98 train\nFaces_easy/image_0057.jpg 98 train\nFaces_easy/image_0403.jpg 98 train\nFaces_easy/image_0114.jpg 98 train\nFaces_easy/image_0153.jpg 98 train\nFaces_easy/image_0416.jpg 98 train\nFaces_easy/image_0098.jpg 98 train\nFaces_easy/image_0256.jpg 98 train\nFaces_easy/image_0402.jpg 98 train\nFaces_easy/image_0203.jpg 98 train\nFaces_easy/image_0338.jpg 98 train\nFaces_easy/image_0265.jpg 98 train\nFaces_easy/image_0147.jpg 98 train\nFaces_easy/image_0008.jpg 98 train\nFaces_easy/image_0207.jpg 98 train\nFaces_easy/image_0208.jpg 98 train\nFaces_easy/image_0222.jpg 98 train\nFaces_easy/image_0320.jpg 98 train\nFaces_easy/image_0227.jpg 98 train\nFaces_easy/image_0286.jpg 98 train\nFaces_easy/image_0035.jpg 98 train\nFaces_easy/image_0399.jpg 98 train\nFaces_easy/image_0166.jpg 98 train\nFaces_easy/image_0169.jpg 98 train\nFaces_easy/image_0191.jpg 98 train\nFaces_easy/image_0128.jpg 98 train\nFaces_easy/image_0414.jpg 98 train\nFaces_easy/image_0102.jpg 98 train\nFaces_easy/image_0434.jpg 98 train\nFaces_easy/image_0178.jpg 98 train\nFaces_easy/image_0431.jpg 98 train\nFaces_easy/image_0243.jpg 98 train\nFaces_easy/image_0005.jpg 98 train\nFaces_easy/image_0263.jpg 98 train\nFaces_easy/image_0173.jpg 98 train\nFaces_easy/image_0087.jpg 98 train\nFaces_easy/image_0417.jpg 98 train\nFaces_easy/image_0089.jpg 98 train\nFaces_easy/image_0059.jpg 98 train\nFaces_easy/image_0253.jpg 98 train\nFaces_easy/image_0159.jpg 98 train\nFaces_easy/image_0225.jpg 98 train\nFaces_easy/image_0186.jpg 98 train\nFaces_easy/image_0388.jpg 98 train\nFaces_easy/image_0069.jpg 98 train\nFaces_easy/image_0315.jpg 98 train\nFaces_easy/image_0295.jpg 98 train\nFaces_easy/image_0200.jpg 98 train\nFaces_easy/image_0099.jpg 98 train\nFaces_easy/image_0336.jpg 98 train\nFaces_easy/image_0034.jpg 98 train\nFaces_easy/image_0409.jpg 98 train\nFaces_easy/image_0314.jpg 98 train\nFaces_easy/image_0276.jpg 98 train\nFaces_easy/image_0121.jpg 98 train\nFaces_easy/image_0428.jpg 98 train\nFaces_easy/image_0181.jpg 98 train\nFaces_easy/image_0280.jpg 98 train\nFaces_easy/image_0183.jpg 98 train\nFaces_easy/image_0383.jpg 98 train\nFaces_easy/image_0175.jpg 98 train\nFaces_easy/image_0002.jpg 98 train\nFaces_easy/image_0291.jpg 98 train\nFaces_easy/image_0165.jpg 98 train\nFaces_easy/image_0154.jpg 98 train\nFaces_easy/image_0271.jpg 98 train\nFaces_easy/image_0193.jpg 98 train\nFaces_easy/image_0048.jpg 98 train\nFaces_easy/image_0053.jpg 98 train\nFaces_easy/image_0255.jpg 98 train\nFaces_easy/image_0084.jpg 98 train\nFaces_easy/image_0313.jpg 98 train\nFaces_easy/image_0020.jpg 98 train\nFaces_easy/image_0251.jpg 98 train\nFaces_easy/image_0046.jpg 98 train\nFaces_easy/image_0041.jpg 98 train\nFaces_easy/image_0027.jpg 98 train\nFaces_easy/image_0322.jpg 98 train\nFaces_easy/image_0014.jpg 98 train\nFaces_easy/image_0058.jpg 98 train\nFaces_easy/image_0408.jpg 98 train\nFaces_easy/image_0329.jpg 98 train\nFaces_easy/image_0212.jpg 98 train\nFaces_easy/image_0163.jpg 98 train\nFaces_easy/image_0202.jpg 98 train\nFaces_easy/image_0104.jpg 98 train\nFaces_easy/image_0081.jpg 98 train\nFaces_easy/image_0272.jpg 98 train\nFaces_easy/image_0400.jpg 98 train\nFaces_easy/image_0237.jpg 98 train\nFaces_easy/image_0308.jpg 98 train\nFaces_easy/image_0149.jpg 98 train\nFaces_easy/image_0088.jpg 98 train\nFaces_easy/image_0309.jpg 98 train\nFaces_easy/image_0026.jpg 98 train\nFaces_easy/image_0231.jpg 98 train\nFaces_easy/image_0004.jpg 98 train\nFaces_easy/image_0015.jpg 98 train\nFaces_easy/image_0355.jpg 98 train\ncougar_face/image_0062.jpg 99 train\ncougar_face/image_0037.jpg 99 train\ncougar_face/image_0049.jpg 99 train\ncougar_face/image_0051.jpg 99 train\ncougar_face/image_0006.jpg 99 train\ncougar_face/image_0016.jpg 99 train\ncougar_face/image_0030.jpg 99 train\ncougar_face/image_0012.jpg 99 train\ncougar_face/image_0066.jpg 99 train\ncougar_face/image_0025.jpg 99 train\ncougar_face/image_0060.jpg 99 train\ncougar_face/image_0052.jpg 99 train\ncougar_face/image_0007.jpg 99 train\ncougar_face/image_0063.jpg 99 train\ncougar_face/image_0019.jpg 99 train\ncougar_face/image_0031.jpg 99 train\ncougar_face/image_0011.jpg 99 train\ncougar_face/image_0061.jpg 99 train\ncougar_face/image_0050.jpg 99 train\ncougar_face/image_0045.jpg 99 train\ncougar_face/image_0017.jpg 99 train\ncougar_face/image_0018.jpg 99 train\ncougar_face/image_0013.jpg 99 train\ncougar_face/image_0054.jpg 99 train\ncougar_face/image_0055.jpg 99 train\ncougar_face/image_0024.jpg 99 train\ncougar_face/image_0003.jpg 99 train\ncougar_face/image_0009.jpg 99 train\ncougar_face/image_0042.jpg 99 train\ncougar_face/image_0040.jpg 99 train\ncougar_face/image_0067.jpg 99 train\ncougar_face/image_0047.jpg 99 train\ncougar_face/image_0064.jpg 99 train\ncougar_face/image_0068.jpg 99 train\ncougar_face/image_0043.jpg 99 train\ncougar_face/image_0044.jpg 99 train\ncougar_face/image_0033.jpg 99 train\ncougar_face/image_0001.jpg 99 train\ncougar_face/image_0021.jpg 99 train\ncougar_face/image_0057.jpg 99 train\ncougar_face/image_0008.jpg 99 train\ncougar_face/image_0035.jpg 99 train\ncougar_face/image_0005.jpg 99 train\ncougar_face/image_0059.jpg 99 train\ncougar_face/image_0069.jpg 99 train\ncougar_face/image_0034.jpg 99 train\ncougar_face/image_0002.jpg 99 train\ncougar_face/image_0048.jpg 99 train\ncougar_face/image_0053.jpg 99 train\ncougar_face/image_0020.jpg 99 train\ncougar_face/image_0046.jpg 99 train\ncougar_face/image_0041.jpg 99 train\ncougar_face/image_0027.jpg 99 train\ncougar_face/image_0014.jpg 99 train\ncougar_face/image_0058.jpg 99 train\nmayfly/image_0037.jpg 100 train\nmayfly/image_0006.jpg 100 train\nmayfly/image_0016.jpg 100 train\nmayfly/image_0030.jpg 100 train\nmayfly/image_0012.jpg 100 train\nmayfly/image_0025.jpg 100 train\nmayfly/image_0007.jpg 100 train\nmayfly/image_0019.jpg 100 train\nmayfly/image_0031.jpg 100 train\nmayfly/image_0011.jpg 100 train\nmayfly/image_0017.jpg 100 train\nmayfly/image_0018.jpg 100 train\nmayfly/image_0013.jpg 100 train\nmayfly/image_0024.jpg 100 train\nmayfly/image_0003.jpg 100 train\nmayfly/image_0009.jpg 100 train\nmayfly/image_0040.jpg 100 train\nmayfly/image_0033.jpg 100 train\nmayfly/image_0001.jpg 100 train\nmayfly/image_0021.jpg 100 train\nmayfly/image_0008.jpg 100 train\nmayfly/image_0035.jpg 100 train\nmayfly/image_0005.jpg 100 train\nmayfly/image_0034.jpg 100 train\nmayfly/image_0002.jpg 100 train\nmayfly/image_0020.jpg 100 train\nmayfly/image_0027.jpg 100 train\nmayfly/image_0014.jpg 100 train\nmayfly/image_0026.jpg 100 train\nmayfly/image_0004.jpg 100 train\nmayfly/image_0015.jpg 100 train\nmayfly/image_0029.jpg 100 train\nwrench/image_0037.jpg 101 train\nwrench/image_0006.jpg 101 train\nwrench/image_0016.jpg 101 train\nwrench/image_0030.jpg 101 train\nwrench/image_0012.jpg 101 train\nwrench/image_0025.jpg 101 train\nwrench/image_0007.jpg 101 train\nwrench/image_0019.jpg 101 train\nwrench/image_0031.jpg 101 train\nwrench/image_0011.jpg 101 train\nwrench/image_0017.jpg 101 train\nwrench/image_0018.jpg 101 train\nwrench/image_0013.jpg 101 train\nwrench/image_0024.jpg 101 train\nwrench/image_0003.jpg 101 train\nwrench/image_0009.jpg 101 train\nwrench/image_0033.jpg 101 train\nwrench/image_0001.jpg 101 train\nwrench/image_0021.jpg 101 train\nwrench/image_0008.jpg 101 train\nwrench/image_0035.jpg 101 train\nwrench/image_0005.jpg 101 train\nwrench/image_0034.jpg 101 train\nwrench/image_0002.jpg 101 train\nwrench/image_0020.jpg 101 train\nwrench/image_0027.jpg 101 train\nwrench/image_0014.jpg 101 train\nwrench/image_0026.jpg 101 train\nwrench/image_0004.jpg 101 train\nwrench/image_0015.jpg 101 train\nwrench/image_0029.jpg 101 train\nairplanes/image_0029.jpg 0 test\nairplanes/image_0662.jpg 0 test\nairplanes/image_0647.jpg 0 test\nairplanes/image_0798.jpg 0 test\nairplanes/image_0327.jpg 0 test\nairplanes/image_0120.jpg 0 test\nairplanes/image_0281.jpg 0 test\nairplanes/image_0603.jpg 0 test\nairplanes/image_0653.jpg 0 test\nairplanes/image_0381.jpg 0 test\nairplanes/image_0220.jpg 0 test\nairplanes/image_0345.jpg 0 test\nairplanes/image_0520.jpg 0 test\nairplanes/image_0685.jpg 0 test\nairplanes/image_0503.jpg 0 test\nairplanes/image_0023.jpg 0 test\nairplanes/image_0135.jpg 0 test\nairplanes/image_0452.jpg 0 test\nairplanes/image_0584.jpg 0 test\nairplanes/image_0423.jpg 0 test\nairplanes/image_0782.jpg 0 test\nairplanes/image_0454.jpg 0 test\nairplanes/image_0466.jpg 0 test\nairplanes/image_0111.jpg 0 test\nairplanes/image_0124.jpg 0 test\nairplanes/image_0279.jpg 0 test\nairplanes/image_0478.jpg 0 test\nairplanes/image_0642.jpg 0 test\nairplanes/image_0666.jpg 0 test\nairplanes/image_0708.jpg 0 test\nairplanes/image_0424.jpg 0 test\nairplanes/image_0418.jpg 0 test\nairplanes/image_0379.jpg 0 test\nairplanes/image_0626.jpg 0 test\nairplanes/image_0610.jpg 0 test\nairplanes/image_0273.jpg 0 test\nairplanes/image_0587.jpg 0 test\nairplanes/image_0072.jpg 0 test\nairplanes/image_0598.jpg 0 test\nairplanes/image_0508.jpg 0 test\nairplanes/image_0145.jpg 0 test\nairplanes/image_0750.jpg 0 test\nairplanes/image_0214.jpg 0 test\nairplanes/image_0174.jpg 0 test\nairplanes/image_0754.jpg 0 test\nairplanes/image_0726.jpg 0 test\nairplanes/image_0133.jpg 0 test\nairplanes/image_0152.jpg 0 test\nairplanes/image_0289.jpg 0 test\nairplanes/image_0674.jpg 0 test\nairplanes/image_0609.jpg 0 test\nairplanes/image_0342.jpg 0 test\nairplanes/image_0298.jpg 0 test\nairplanes/image_0710.jpg 0 test\nairplanes/image_0010.jpg 0 test\nairplanes/image_0248.jpg 0 test\nairplanes/image_0245.jpg 0 test\nairplanes/image_0056.jpg 0 test\nairplanes/image_0022.jpg 0 test\nairplanes/image_0687.jpg 0 test\nairplanes/image_0307.jpg 0 test\nairplanes/image_0755.jpg 0 test\nairplanes/image_0164.jpg 0 test\nairplanes/image_0377.jpg 0 test\nairplanes/image_0715.jpg 0 test\nairplanes/image_0065.jpg 0 test\nairplanes/image_0244.jpg 0 test\nairplanes/image_0777.jpg 0 test\nairplanes/image_0100.jpg 0 test\nairplanes/image_0137.jpg 0 test\nairplanes/image_0274.jpg 0 test\nairplanes/image_0323.jpg 0 test\nairplanes/image_0576.jpg 0 test\nairplanes/image_0526.jpg 0 test\nairplanes/image_0036.jpg 0 test\nairplanes/image_0785.jpg 0 test\nairplanes/image_0599.jpg 0 test\nairplanes/image_0086.jpg 0 test\nairplanes/image_0032.jpg 0 test\nairplanes/image_0405.jpg 0 test\nairplanes/image_0394.jpg 0 test\nairplanes/image_0076.jpg 0 test\nairplanes/image_0682.jpg 0 test\nairplanes/image_0439.jpg 0 test\nairplanes/image_0705.jpg 0 test\nairplanes/image_0252.jpg 0 test\nairplanes/image_0235.jpg 0 test\nairplanes/image_0510.jpg 0 test\nairplanes/image_0190.jpg 0 test\nairplanes/image_0290.jpg 0 test\nairplanes/image_0453.jpg 0 test\nairplanes/image_0106.jpg 0 test\nairplanes/image_0140.jpg 0 test\nairplanes/image_0415.jpg 0 test\nairplanes/image_0117.jpg 0 test\nairplanes/image_0038.jpg 0 test\nairplanes/image_0773.jpg 0 test\nairplanes/image_0663.jpg 0 test\nairplanes/image_0624.jpg 0 test\nairplanes/image_0210.jpg 0 test\nairplanes/image_0456.jpg 0 test\nairplanes/image_0107.jpg 0 test\nairplanes/image_0247.jpg 0 test\nairplanes/image_0800.jpg 0 test\nairplanes/image_0362.jpg 0 test\nairplanes/image_0760.jpg 0 test\nairplanes/image_0535.jpg 0 test\nairplanes/image_0158.jpg 0 test\nairplanes/image_0498.jpg 0 test\nairplanes/image_0430.jpg 0 test\nairplanes/image_0268.jpg 0 test\nairplanes/image_0660.jpg 0 test\nairplanes/image_0630.jpg 0 test\nairplanes/image_0592.jpg 0 test\nairplanes/image_0242.jpg 0 test\nairplanes/image_0401.jpg 0 test\nairplanes/image_0179.jpg 0 test\nairplanes/image_0170.jpg 0 test\nairplanes/image_0524.jpg 0 test\nairplanes/image_0378.jpg 0 test\nairplanes/image_0753.jpg 0 test\nairplanes/image_0507.jpg 0 test\nairplanes/image_0749.jpg 0 test\nairplanes/image_0039.jpg 0 test\nairplanes/image_0075.jpg 0 test\nairplanes/image_0744.jpg 0 test\nairplanes/image_0194.jpg 0 test\nairplanes/image_0695.jpg 0 test\nairplanes/image_0693.jpg 0 test\nairplanes/image_0096.jpg 0 test\nairplanes/image_0481.jpg 0 test\nairplanes/image_0238.jpg 0 test\nairplanes/image_0701.jpg 0 test\nairplanes/image_0236.jpg 0 test\nairplanes/image_0311.jpg 0 test\nairplanes/image_0724.jpg 0 test\nairplanes/image_0116.jpg 0 test\nairplanes/image_0419.jpg 0 test\nairplanes/image_0702.jpg 0 test\nairplanes/image_0725.jpg 0 test\nairplanes/image_0138.jpg 0 test\nairplanes/image_0774.jpg 0 test\nairplanes/image_0793.jpg 0 test\nairplanes/image_0449.jpg 0 test\nairplanes/image_0778.jpg 0 test\nairplanes/image_0197.jpg 0 test\nairplanes/image_0578.jpg 0 test\nairplanes/image_0241.jpg 0 test\nairplanes/image_0028.jpg 0 test\nairplanes/image_0151.jpg 0 test\nairplanes/image_0385.jpg 0 test\nairplanes/image_0550.jpg 0 test\nairplanes/image_0636.jpg 0 test\nairplanes/image_0545.jpg 0 test\nairplanes/image_0092.jpg 0 test\nairplanes/image_0632.jpg 0 test\nairplanes/image_0182.jpg 0 test\nairplanes/image_0413.jpg 0 test\nairplanes/image_0359.jpg 0 test\nairplanes/image_0374.jpg 0 test\nant/image_0029.jpg 1 test\nant/image_0023.jpg 1 test\nant/image_0010.jpg 1 test\nant/image_0022.jpg 1 test\nant/image_0036.jpg 1 test\nant/image_0032.jpg 1 test\nant/image_0038.jpg 1 test\nant/image_0039.jpg 1 test\nant/image_0028.jpg 1 test\nFaces/image_0029.jpg 2 test\nFaces/image_0327.jpg 2 test\nFaces/image_0120.jpg 2 test\nFaces/image_0281.jpg 2 test\nFaces/image_0381.jpg 2 test\nFaces/image_0220.jpg 2 test\nFaces/image_0345.jpg 2 test\nFaces/image_0023.jpg 2 test\nFaces/image_0135.jpg 2 test\nFaces/image_0423.jpg 2 test\nFaces/image_0111.jpg 2 test\nFaces/image_0124.jpg 2 test\nFaces/image_0279.jpg 2 test\nFaces/image_0424.jpg 2 test\nFaces/image_0418.jpg 2 test\nFaces/image_0379.jpg 2 test\nFaces/image_0273.jpg 2 test\nFaces/image_0072.jpg 2 test\nFaces/image_0145.jpg 2 test\nFaces/image_0214.jpg 2 test\nFaces/image_0174.jpg 2 test\nFaces/image_0133.jpg 2 test\nFaces/image_0152.jpg 2 test\nFaces/image_0289.jpg 2 test\nFaces/image_0342.jpg 2 test\nFaces/image_0298.jpg 2 test\nFaces/image_0010.jpg 2 test\nFaces/image_0248.jpg 2 test\nFaces/image_0245.jpg 2 test\nFaces/image_0056.jpg 2 test\nFaces/image_0022.jpg 2 test\nFaces/image_0307.jpg 2 test\nFaces/image_0164.jpg 2 test\nFaces/image_0377.jpg 2 test\nFaces/image_0065.jpg 2 test\nFaces/image_0244.jpg 2 test\nFaces/image_0100.jpg 2 test\nFaces/image_0137.jpg 2 test\nFaces/image_0274.jpg 2 test\nFaces/image_0323.jpg 2 test\nFaces/image_0036.jpg 2 test\nFaces/image_0086.jpg 2 test\nFaces/image_0032.jpg 2 test\nFaces/image_0405.jpg 2 test\nFaces/image_0394.jpg 2 test\nFaces/image_0076.jpg 2 test\nFaces/image_0252.jpg 2 test\nFaces/image_0235.jpg 2 test\nFaces/image_0190.jpg 2 test\nFaces/image_0290.jpg 2 test\nFaces/image_0106.jpg 2 test\nFaces/image_0140.jpg 2 test\nFaces/image_0415.jpg 2 test\nFaces/image_0117.jpg 2 test\nFaces/image_0038.jpg 2 test\nFaces/image_0210.jpg 2 test\nFaces/image_0107.jpg 2 test\nFaces/image_0247.jpg 2 test\nFaces/image_0362.jpg 2 test\nFaces/image_0158.jpg 2 test\nFaces/image_0430.jpg 2 test\nFaces/image_0268.jpg 2 test\nFaces/image_0242.jpg 2 test\nFaces/image_0401.jpg 2 test\nFaces/image_0179.jpg 2 test\nFaces/image_0170.jpg 2 test\nFaces/image_0378.jpg 2 test\nFaces/image_0039.jpg 2 test\nFaces/image_0075.jpg 2 test\nFaces/image_0194.jpg 2 test\nFaces/image_0096.jpg 2 test\nFaces/image_0238.jpg 2 test\nFaces/image_0236.jpg 2 test\nFaces/image_0311.jpg 2 test\nFaces/image_0116.jpg 2 test\nFaces/image_0419.jpg 2 test\nFaces/image_0138.jpg 2 test\nFaces/image_0197.jpg 2 test\nFaces/image_0241.jpg 2 test\nFaces/image_0028.jpg 2 test\nFaces/image_0151.jpg 2 test\nFaces/image_0385.jpg 2 test\nFaces/image_0092.jpg 2 test\nFaces/image_0182.jpg 2 test\nFaces/image_0413.jpg 2 test\nFaces/image_0359.jpg 2 test\nFaces/image_0374.jpg 2 test\nrooster/image_0015.jpg 3 test\nrooster/image_0029.jpg 3 test\nrooster/image_0023.jpg 3 test\nrooster/image_0010.jpg 3 test\nrooster/image_0022.jpg 3 test\nrooster/image_0036.jpg 3 test\nrooster/image_0032.jpg 3 test\nrooster/image_0038.jpg 3 test\nrooster/image_0039.jpg 3 test\nrooster/image_0028.jpg 3 test\nwater_lilly/image_0015.jpg 4 test\nwater_lilly/image_0029.jpg 4 test\nwater_lilly/image_0023.jpg 4 test\nwater_lilly/image_0010.jpg 4 test\nwater_lilly/image_0022.jpg 4 test\nwater_lilly/image_0036.jpg 4 test\nwater_lilly/image_0032.jpg 4 test\nwater_lilly/image_0028.jpg 4 test\nelephant/image_0026.jpg 5 test\nelephant/image_0004.jpg 5 test\nelephant/image_0015.jpg 5 test\nelephant/image_0029.jpg 5 test\nelephant/image_0023.jpg 5 test\nelephant/image_0010.jpg 5 test\nelephant/image_0056.jpg 5 test\nelephant/image_0022.jpg 5 test\nelephant/image_0036.jpg 5 test\nelephant/image_0032.jpg 5 test\nelephant/image_0038.jpg 5 test\nelephant/image_0039.jpg 5 test\nelephant/image_0028.jpg 5 test\numbrella/image_0004.jpg 6 test\numbrella/image_0015.jpg 6 test\numbrella/image_0029.jpg 6 test\numbrella/image_0023.jpg 6 test\numbrella/image_0072.jpg 6 test\numbrella/image_0010.jpg 6 test\numbrella/image_0056.jpg 6 test\numbrella/image_0022.jpg 6 test\numbrella/image_0065.jpg 6 test\numbrella/image_0036.jpg 6 test\numbrella/image_0032.jpg 6 test\numbrella/image_0038.jpg 6 test\numbrella/image_0039.jpg 6 test\numbrella/image_0075.jpg 6 test\numbrella/image_0028.jpg 6 test\ndolphin/image_0004.jpg 7 test\ndolphin/image_0015.jpg 7 test\ndolphin/image_0029.jpg 7 test\ndolphin/image_0023.jpg 7 test\ndolphin/image_0010.jpg 7 test\ndolphin/image_0056.jpg 7 test\ndolphin/image_0022.jpg 7 test\ndolphin/image_0065.jpg 7 test\ndolphin/image_0036.jpg 7 test\ndolphin/image_0032.jpg 7 test\ndolphin/image_0038.jpg 7 test\ndolphin/image_0039.jpg 7 test\ndolphin/image_0028.jpg 7 test\ngerenuk/image_0015.jpg 8 test\ngerenuk/image_0029.jpg 8 test\ngerenuk/image_0023.jpg 8 test\ngerenuk/image_0010.jpg 8 test\ngerenuk/image_0022.jpg 8 test\ngerenuk/image_0032.jpg 8 test\ngerenuk/image_0028.jpg 8 test\ndragonfly/image_0026.jpg 9 test\ndragonfly/image_0004.jpg 9 test\ndragonfly/image_0015.jpg 9 test\ndragonfly/image_0029.jpg 9 test\ndragonfly/image_0023.jpg 9 test\ndragonfly/image_0010.jpg 9 test\ndragonfly/image_0056.jpg 9 test\ndragonfly/image_0022.jpg 9 test\ndragonfly/image_0065.jpg 9 test\ndragonfly/image_0036.jpg 9 test\ndragonfly/image_0032.jpg 9 test\ndragonfly/image_0038.jpg 9 test\ndragonfly/image_0039.jpg 9 test\ndragonfly/image_0028.jpg 9 test\nyin_yang/image_0004.jpg 10 test\nyin_yang/image_0015.jpg 10 test\nyin_yang/image_0029.jpg 10 test\nyin_yang/image_0023.jpg 10 test\nyin_yang/image_0010.jpg 10 test\nyin_yang/image_0056.jpg 10 test\nyin_yang/image_0022.jpg 10 test\nyin_yang/image_0036.jpg 10 test\nyin_yang/image_0032.jpg 10 test\nyin_yang/image_0038.jpg 10 test\nyin_yang/image_0039.jpg 10 test\nyin_yang/image_0028.jpg 10 test\nstarfish/image_0026.jpg 11 test\nstarfish/image_0004.jpg 11 test\nstarfish/image_0015.jpg 11 test\nstarfish/image_0029.jpg 11 test\nstarfish/image_0023.jpg 11 test\nstarfish/image_0072.jpg 11 test\nstarfish/image_0010.jpg 11 test\nstarfish/image_0056.jpg 11 test\nstarfish/image_0022.jpg 11 test\nstarfish/image_0065.jpg 11 test\nstarfish/image_0036.jpg 11 test\nstarfish/image_0086.jpg 11 test\nstarfish/image_0032.jpg 11 test\nstarfish/image_0076.jpg 11 test\nstarfish/image_0038.jpg 11 test\nstarfish/image_0039.jpg 11 test\nstarfish/image_0075.jpg 11 test\nstarfish/image_0028.jpg 11 test\nceiling_fan/image_0015.jpg 12 test\nceiling_fan/image_0029.jpg 12 test\nceiling_fan/image_0023.jpg 12 test\nceiling_fan/image_0010.jpg 12 test\nceiling_fan/image_0022.jpg 12 test\nceiling_fan/image_0036.jpg 12 test\nceiling_fan/image_0032.jpg 12 test\nceiling_fan/image_0038.jpg 12 test\nceiling_fan/image_0039.jpg 12 test\nceiling_fan/image_0028.jpg 12 test\nsoccer_ball/image_0026.jpg 13 test\nsoccer_ball/image_0004.jpg 13 test\nsoccer_ball/image_0015.jpg 13 test\nsoccer_ball/image_0029.jpg 13 test\nsoccer_ball/image_0023.jpg 13 test\nsoccer_ball/image_0010.jpg 13 test\nsoccer_ball/image_0056.jpg 13 test\nsoccer_ball/image_0022.jpg 13 test\nsoccer_ball/image_0036.jpg 13 test\nsoccer_ball/image_0032.jpg 13 test\nsoccer_ball/image_0038.jpg 13 test\nsoccer_ball/image_0039.jpg 13 test\nsoccer_ball/image_0028.jpg 13 test\nLeopards/image_0023.jpg 14 test\nLeopards/image_0135.jpg 14 test\nLeopards/image_0111.jpg 14 test\nLeopards/image_0124.jpg 14 test\nLeopards/image_0072.jpg 14 test\nLeopards/image_0145.jpg 14 test\nLeopards/image_0174.jpg 14 test\nLeopards/image_0133.jpg 14 test\nLeopards/image_0152.jpg 14 test\nLeopards/image_0010.jpg 14 test\nLeopards/image_0056.jpg 14 test\nLeopards/image_0022.jpg 14 test\nLeopards/image_0164.jpg 14 test\nLeopards/image_0065.jpg 14 test\nLeopards/image_0100.jpg 14 test\nLeopards/image_0137.jpg 14 test\nLeopards/image_0036.jpg 14 test\nLeopards/image_0086.jpg 14 test\nLeopards/image_0032.jpg 14 test\nLeopards/image_0076.jpg 14 test\nLeopards/image_0190.jpg 14 test\nLeopards/image_0106.jpg 14 test\nLeopards/image_0140.jpg 14 test\nLeopards/image_0117.jpg 14 test\nLeopards/image_0038.jpg 14 test\nLeopards/image_0107.jpg 14 test\nLeopards/image_0158.jpg 14 test\nLeopards/image_0179.jpg 14 test\nLeopards/image_0170.jpg 14 test\nLeopards/image_0039.jpg 14 test\nLeopards/image_0075.jpg 14 test\nLeopards/image_0194.jpg 14 test\nLeopards/image_0096.jpg 14 test\nLeopards/image_0116.jpg 14 test\nLeopards/image_0138.jpg 14 test\nLeopards/image_0197.jpg 14 test\nLeopards/image_0028.jpg 14 test\nLeopards/image_0151.jpg 14 test\nLeopards/image_0092.jpg 14 test\nLeopards/image_0182.jpg 14 test\nscorpion/image_0026.jpg 15 test\nscorpion/image_0004.jpg 15 test\nscorpion/image_0015.jpg 15 test\nscorpion/image_0029.jpg 15 test\nscorpion/image_0023.jpg 15 test\nscorpion/image_0072.jpg 15 test\nscorpion/image_0010.jpg 15 test\nscorpion/image_0056.jpg 15 test\nscorpion/image_0022.jpg 15 test\nscorpion/image_0065.jpg 15 test\nscorpion/image_0036.jpg 15 test\nscorpion/image_0032.jpg 15 test\nscorpion/image_0076.jpg 15 test\nscorpion/image_0038.jpg 15 test\nscorpion/image_0039.jpg 15 test\nscorpion/image_0075.jpg 15 test\nscorpion/image_0028.jpg 15 test\nllama/image_0004.jpg 16 test\nllama/image_0015.jpg 16 test\nllama/image_0029.jpg 16 test\nllama/image_0023.jpg 16 test\nllama/image_0072.jpg 16 test\nllama/image_0010.jpg 16 test\nllama/image_0056.jpg 16 test\nllama/image_0022.jpg 16 test\nllama/image_0065.jpg 16 test\nllama/image_0036.jpg 16 test\nllama/image_0032.jpg 16 test\nllama/image_0076.jpg 16 test\nllama/image_0038.jpg 16 test\nllama/image_0039.jpg 16 test\nllama/image_0075.jpg 16 test\nllama/image_0028.jpg 16 test\nwild_cat/image_0015.jpg 17 test\nwild_cat/image_0029.jpg 17 test\nwild_cat/image_0023.jpg 17 test\nwild_cat/image_0010.jpg 17 test\nwild_cat/image_0022.jpg 17 test\nwild_cat/image_0032.jpg 17 test\nwild_cat/image_0028.jpg 17 test\nlamp/image_0026.jpg 18 test\nlamp/image_0004.jpg 18 test\nlamp/image_0015.jpg 18 test\nlamp/image_0029.jpg 18 test\nlamp/image_0023.jpg 18 test\nlamp/image_0010.jpg 18 test\nlamp/image_0056.jpg 18 test\nlamp/image_0022.jpg 18 test\nlamp/image_0036.jpg 18 test\nlamp/image_0032.jpg 18 test\nlamp/image_0038.jpg 18 test\nlamp/image_0039.jpg 18 test\nlamp/image_0028.jpg 18 test\nlotus/image_0026.jpg 19 test\nlotus/image_0004.jpg 19 test\nlotus/image_0015.jpg 19 test\nlotus/image_0029.jpg 19 test\nlotus/image_0023.jpg 19 test\nlotus/image_0010.jpg 19 test\nlotus/image_0056.jpg 19 test\nlotus/image_0022.jpg 19 test\nlotus/image_0065.jpg 19 test\nlotus/image_0036.jpg 19 test\nlotus/image_0032.jpg 19 test\nlotus/image_0038.jpg 19 test\nlotus/image_0039.jpg 19 test\nlotus/image_0028.jpg 19 test\ncrocodile_head/image_0004.jpg 20 test\ncrocodile_head/image_0015.jpg 20 test\ncrocodile_head/image_0029.jpg 20 test\ncrocodile_head/image_0023.jpg 20 test\ncrocodile_head/image_0010.jpg 20 test\ncrocodile_head/image_0022.jpg 20 test\ncrocodile_head/image_0036.jpg 20 test\ncrocodile_head/image_0032.jpg 20 test\ncrocodile_head/image_0038.jpg 20 test\ncrocodile_head/image_0039.jpg 20 test\ncrocodile_head/image_0028.jpg 20 test\nbutterfly/image_0088.jpg 21 test\nbutterfly/image_0026.jpg 21 test\nbutterfly/image_0004.jpg 21 test\nbutterfly/image_0015.jpg 21 test\nbutterfly/image_0029.jpg 21 test\nbutterfly/image_0023.jpg 21 test\nbutterfly/image_0072.jpg 21 test\nbutterfly/image_0010.jpg 21 test\nbutterfly/image_0056.jpg 21 test\nbutterfly/image_0022.jpg 21 test\nbutterfly/image_0065.jpg 21 test\nbutterfly/image_0036.jpg 21 test\nbutterfly/image_0086.jpg 21 test\nbutterfly/image_0032.jpg 21 test\nbutterfly/image_0076.jpg 21 test\nbutterfly/image_0038.jpg 21 test\nbutterfly/image_0039.jpg 21 test\nbutterfly/image_0075.jpg 21 test\nbutterfly/image_0028.jpg 21 test\nwindsor_chair/image_0004.jpg 22 test\nwindsor_chair/image_0015.jpg 22 test\nwindsor_chair/image_0029.jpg 22 test\nwindsor_chair/image_0023.jpg 22 test\nwindsor_chair/image_0010.jpg 22 test\nwindsor_chair/image_0056.jpg 22 test\nwindsor_chair/image_0022.jpg 22 test\nwindsor_chair/image_0036.jpg 22 test\nwindsor_chair/image_0032.jpg 22 test\nwindsor_chair/image_0038.jpg 22 test\nwindsor_chair/image_0039.jpg 22 test\nwindsor_chair/image_0028.jpg 22 test\nmetronome/image_0015.jpg 23 test\nmetronome/image_0029.jpg 23 test\nmetronome/image_0023.jpg 23 test\nmetronome/image_0010.jpg 23 test\nmetronome/image_0022.jpg 23 test\nmetronome/image_0032.jpg 23 test\nmetronome/image_0028.jpg 23 test\nrevolver/image_0026.jpg 24 test\nrevolver/image_0004.jpg 24 test\nrevolver/image_0015.jpg 24 test\nrevolver/image_0029.jpg 24 test\nrevolver/image_0023.jpg 24 test\nrevolver/image_0072.jpg 24 test\nrevolver/image_0010.jpg 24 test\nrevolver/image_0056.jpg 24 test\nrevolver/image_0022.jpg 24 test\nrevolver/image_0065.jpg 24 test\nrevolver/image_0036.jpg 24 test\nrevolver/image_0032.jpg 24 test\nrevolver/image_0076.jpg 24 test\nrevolver/image_0038.jpg 24 test\nrevolver/image_0039.jpg 24 test\nrevolver/image_0075.jpg 24 test\nrevolver/image_0028.jpg 24 test\nschooner/image_0026.jpg 25 test\nschooner/image_0004.jpg 25 test\nschooner/image_0015.jpg 25 test\nschooner/image_0029.jpg 25 test\nschooner/image_0023.jpg 25 test\nschooner/image_0010.jpg 25 test\nschooner/image_0056.jpg 25 test\nschooner/image_0022.jpg 25 test\nschooner/image_0036.jpg 25 test\nschooner/image_0032.jpg 25 test\nschooner/image_0038.jpg 25 test\nschooner/image_0039.jpg 25 test\nschooner/image_0028.jpg 25 test\nminaret/image_0004.jpg 26 test\nminaret/image_0015.jpg 26 test\nminaret/image_0029.jpg 26 test\nminaret/image_0023.jpg 26 test\nminaret/image_0072.jpg 26 test\nminaret/image_0010.jpg 26 test\nminaret/image_0056.jpg 26 test\nminaret/image_0022.jpg 26 test\nminaret/image_0065.jpg 26 test\nminaret/image_0036.jpg 26 test\nminaret/image_0032.jpg 26 test\nminaret/image_0076.jpg 26 test\nminaret/image_0038.jpg 26 test\nminaret/image_0039.jpg 26 test\nminaret/image_0075.jpg 26 test\nminaret/image_0028.jpg 26 test\nstop_sign/image_0026.jpg 27 test\nstop_sign/image_0004.jpg 27 test\nstop_sign/image_0015.jpg 27 test\nstop_sign/image_0029.jpg 27 test\nstop_sign/image_0023.jpg 27 test\nstop_sign/image_0010.jpg 27 test\nstop_sign/image_0056.jpg 27 test\nstop_sign/image_0022.jpg 27 test\nstop_sign/image_0036.jpg 27 test\nstop_sign/image_0032.jpg 27 test\nstop_sign/image_0038.jpg 27 test\nstop_sign/image_0039.jpg 27 test\nstop_sign/image_0028.jpg 27 test\nbeaver/image_0015.jpg 28 test\nbeaver/image_0029.jpg 28 test\nbeaver/image_0023.jpg 28 test\nbeaver/image_0010.jpg 28 test\nbeaver/image_0022.jpg 28 test\nbeaver/image_0036.jpg 28 test\nbeaver/image_0032.jpg 28 test\nbeaver/image_0038.jpg 28 test\nbeaver/image_0039.jpg 28 test\nbeaver/image_0028.jpg 28 test\nlaptop/image_0026.jpg 29 test\nlaptop/image_0004.jpg 29 test\nlaptop/image_0015.jpg 29 test\nlaptop/image_0029.jpg 29 test\nlaptop/image_0023.jpg 29 test\nlaptop/image_0072.jpg 29 test\nlaptop/image_0010.jpg 29 test\nlaptop/image_0056.jpg 29 test\nlaptop/image_0022.jpg 29 test\nlaptop/image_0065.jpg 29 test\nlaptop/image_0036.jpg 29 test\nlaptop/image_0032.jpg 29 test\nlaptop/image_0076.jpg 29 test\nlaptop/image_0038.jpg 29 test\nlaptop/image_0039.jpg 29 test\nlaptop/image_0075.jpg 29 test\nlaptop/image_0028.jpg 29 test\nketch/image_0004.jpg 30 test\nketch/image_0015.jpg 30 test\nketch/image_0029.jpg 30 test\nketch/image_0023.jpg 30 test\nketch/image_0111.jpg 30 test\nketch/image_0072.jpg 30 test\nketch/image_0010.jpg 30 test\nketch/image_0056.jpg 30 test\nketch/image_0022.jpg 30 test\nketch/image_0065.jpg 30 test\nketch/image_0100.jpg 30 test\nketch/image_0036.jpg 30 test\nketch/image_0086.jpg 30 test\nketch/image_0032.jpg 30 test\nketch/image_0076.jpg 30 test\nketch/image_0106.jpg 30 test\nketch/image_0038.jpg 30 test\nketch/image_0107.jpg 30 test\nketch/image_0039.jpg 30 test\nketch/image_0075.jpg 30 test\nketch/image_0096.jpg 30 test\nketch/image_0028.jpg 30 test\nketch/image_0092.jpg 30 test\ngramophone/image_0004.jpg 31 test\ngramophone/image_0015.jpg 31 test\ngramophone/image_0029.jpg 31 test\ngramophone/image_0023.jpg 31 test\ngramophone/image_0010.jpg 31 test\ngramophone/image_0022.jpg 31 test\ngramophone/image_0036.jpg 31 test\ngramophone/image_0032.jpg 31 test\ngramophone/image_0038.jpg 31 test\ngramophone/image_0039.jpg 31 test\ngramophone/image_0028.jpg 31 test\nmenorah/image_0026.jpg 32 test\nmenorah/image_0004.jpg 32 test\nmenorah/image_0015.jpg 32 test\nmenorah/image_0029.jpg 32 test\nmenorah/image_0023.jpg 32 test\nmenorah/image_0072.jpg 32 test\nmenorah/image_0010.jpg 32 test\nmenorah/image_0056.jpg 32 test\nmenorah/image_0022.jpg 32 test\nmenorah/image_0065.jpg 32 test\nmenorah/image_0036.jpg 32 test\nmenorah/image_0086.jpg 32 test\nmenorah/image_0032.jpg 32 test\nmenorah/image_0076.jpg 32 test\nmenorah/image_0038.jpg 32 test\nmenorah/image_0039.jpg 32 test\nmenorah/image_0075.jpg 32 test\nmenorah/image_0028.jpg 32 test\neuphonium/image_0026.jpg 33 test\neuphonium/image_0004.jpg 33 test\neuphonium/image_0015.jpg 33 test\neuphonium/image_0029.jpg 33 test\neuphonium/image_0023.jpg 33 test\neuphonium/image_0010.jpg 33 test\neuphonium/image_0056.jpg 33 test\neuphonium/image_0022.jpg 33 test\neuphonium/image_0036.jpg 33 test\neuphonium/image_0032.jpg 33 test\neuphonium/image_0038.jpg 33 test\neuphonium/image_0039.jpg 33 test\neuphonium/image_0028.jpg 33 test\nrhino/image_0004.jpg 34 test\nrhino/image_0015.jpg 34 test\nrhino/image_0029.jpg 34 test\nrhino/image_0023.jpg 34 test\nrhino/image_0010.jpg 34 test\nrhino/image_0056.jpg 34 test\nrhino/image_0022.jpg 34 test\nrhino/image_0036.jpg 34 test\nrhino/image_0032.jpg 34 test\nrhino/image_0038.jpg 34 test\nrhino/image_0039.jpg 34 test\nrhino/image_0028.jpg 34 test\nbonsai/image_0015.jpg 35 test\nbonsai/image_0029.jpg 35 test\nbonsai/image_0120.jpg 35 test\nbonsai/image_0023.jpg 35 test\nbonsai/image_0111.jpg 35 test\nbonsai/image_0124.jpg 35 test\nbonsai/image_0072.jpg 35 test\nbonsai/image_0010.jpg 35 test\nbonsai/image_0056.jpg 35 test\nbonsai/image_0022.jpg 35 test\nbonsai/image_0065.jpg 35 test\nbonsai/image_0100.jpg 35 test\nbonsai/image_0036.jpg 35 test\nbonsai/image_0086.jpg 35 test\nbonsai/image_0032.jpg 35 test\nbonsai/image_0076.jpg 35 test\nbonsai/image_0106.jpg 35 test\nbonsai/image_0117.jpg 35 test\nbonsai/image_0038.jpg 35 test\nbonsai/image_0107.jpg 35 test\nbonsai/image_0039.jpg 35 test\nbonsai/image_0075.jpg 35 test\nbonsai/image_0096.jpg 35 test\nbonsai/image_0116.jpg 35 test\nbonsai/image_0028.jpg 35 test\nbonsai/image_0092.jpg 35 test\nchair/image_0026.jpg 36 test\nchair/image_0004.jpg 36 test\nchair/image_0015.jpg 36 test\nchair/image_0029.jpg 36 test\nchair/image_0023.jpg 36 test\nchair/image_0010.jpg 36 test\nchair/image_0056.jpg 36 test\nchair/image_0022.jpg 36 test\nchair/image_0036.jpg 36 test\nchair/image_0032.jpg 36 test\nchair/image_0038.jpg 36 test\nchair/image_0039.jpg 36 test\nchair/image_0028.jpg 36 test\nsnoopy/image_0015.jpg 37 test\nsnoopy/image_0029.jpg 37 test\nsnoopy/image_0023.jpg 37 test\nsnoopy/image_0010.jpg 37 test\nsnoopy/image_0022.jpg 37 test\nsnoopy/image_0032.jpg 37 test\nsnoopy/image_0028.jpg 37 test\nbuddha/image_0026.jpg 38 test\nbuddha/image_0004.jpg 38 test\nbuddha/image_0015.jpg 38 test\nbuddha/image_0029.jpg 38 test\nbuddha/image_0023.jpg 38 test\nbuddha/image_0072.jpg 38 test\nbuddha/image_0010.jpg 38 test\nbuddha/image_0056.jpg 38 test\nbuddha/image_0022.jpg 38 test\nbuddha/image_0065.jpg 38 test\nbuddha/image_0036.jpg 38 test\nbuddha/image_0032.jpg 38 test\nbuddha/image_0076.jpg 38 test\nbuddha/image_0038.jpg 38 test\nbuddha/image_0039.jpg 38 test\nbuddha/image_0075.jpg 38 test\nbuddha/image_0028.jpg 38 test\nscissors/image_0023.jpg 39 test\nscissors/image_0010.jpg 39 test\nscissors/image_0022.jpg 39 test\nscissors/image_0036.jpg 39 test\nscissors/image_0032.jpg 39 test\nscissors/image_0038.jpg 39 test\nscissors/image_0039.jpg 39 test\nscissors/image_0028.jpg 39 test\ncougar_body/image_0015.jpg 40 test\ncougar_body/image_0029.jpg 40 test\ncougar_body/image_0023.jpg 40 test\ncougar_body/image_0010.jpg 40 test\ncougar_body/image_0022.jpg 40 test\ncougar_body/image_0036.jpg 40 test\ncougar_body/image_0032.jpg 40 test\ncougar_body/image_0038.jpg 40 test\ncougar_body/image_0039.jpg 40 test\ncougar_body/image_0028.jpg 40 test\nbinocular/image_0015.jpg 41 test\nbinocular/image_0029.jpg 41 test\nbinocular/image_0023.jpg 41 test\nbinocular/image_0010.jpg 41 test\nbinocular/image_0022.jpg 41 test\nbinocular/image_0032.jpg 41 test\nbinocular/image_0028.jpg 41 test\ngrand_piano/image_0026.jpg 42 test\ngrand_piano/image_0004.jpg 42 test\ngrand_piano/image_0015.jpg 42 test\ngrand_piano/image_0029.jpg 42 test\ngrand_piano/image_0023.jpg 42 test\ngrand_piano/image_0072.jpg 42 test\ngrand_piano/image_0010.jpg 42 test\ngrand_piano/image_0056.jpg 42 test\ngrand_piano/image_0022.jpg 42 test\ngrand_piano/image_0065.jpg 42 test\ngrand_piano/image_0036.jpg 42 test\ngrand_piano/image_0086.jpg 42 test\ngrand_piano/image_0032.jpg 42 test\ngrand_piano/image_0076.jpg 42 test\ngrand_piano/image_0038.jpg 42 test\ngrand_piano/image_0039.jpg 42 test\ngrand_piano/image_0075.jpg 42 test\ngrand_piano/image_0096.jpg 42 test\ngrand_piano/image_0028.jpg 42 test\ngrand_piano/image_0092.jpg 42 test\nibis/image_0004.jpg 43 test\nibis/image_0015.jpg 43 test\nibis/image_0029.jpg 43 test\nibis/image_0023.jpg 43 test\nibis/image_0072.jpg 43 test\nibis/image_0010.jpg 43 test\nibis/image_0056.jpg 43 test\nibis/image_0022.jpg 43 test\nibis/image_0065.jpg 43 test\nibis/image_0036.jpg 43 test\nibis/image_0032.jpg 43 test\nibis/image_0076.jpg 43 test\nibis/image_0038.jpg 43 test\nibis/image_0039.jpg 43 test\nibis/image_0075.jpg 43 test\nibis/image_0028.jpg 43 test\nbrontosaurus/image_0029.jpg 44 test\nbrontosaurus/image_0023.jpg 44 test\nbrontosaurus/image_0010.jpg 44 test\nbrontosaurus/image_0022.jpg 44 test\nbrontosaurus/image_0036.jpg 44 test\nbrontosaurus/image_0032.jpg 44 test\nbrontosaurus/image_0038.jpg 44 test\nbrontosaurus/image_0039.jpg 44 test\nbrontosaurus/image_0028.jpg 44 test\nsaxophone/image_0023.jpg 45 test\nsaxophone/image_0010.jpg 45 test\nsaxophone/image_0022.jpg 45 test\nsaxophone/image_0036.jpg 45 test\nsaxophone/image_0032.jpg 45 test\nsaxophone/image_0038.jpg 45 test\nsaxophone/image_0039.jpg 45 test\nsaxophone/image_0028.jpg 45 test\nstapler/image_0029.jpg 46 test\nstapler/image_0023.jpg 46 test\nstapler/image_0010.jpg 46 test\nstapler/image_0022.jpg 46 test\nstapler/image_0036.jpg 46 test\nstapler/image_0032.jpg 46 test\nstapler/image_0038.jpg 46 test\nstapler/image_0039.jpg 46 test\nstapler/image_0028.jpg 46 test\nelectric_guitar/image_0004.jpg 47 test\nelectric_guitar/image_0015.jpg 47 test\nelectric_guitar/image_0029.jpg 47 test\nelectric_guitar/image_0023.jpg 47 test\nelectric_guitar/image_0072.jpg 47 test\nelectric_guitar/image_0010.jpg 47 test\nelectric_guitar/image_0056.jpg 47 test\nelectric_guitar/image_0022.jpg 47 test\nelectric_guitar/image_0065.jpg 47 test\nelectric_guitar/image_0036.jpg 47 test\nelectric_guitar/image_0032.jpg 47 test\nelectric_guitar/image_0038.jpg 47 test\nelectric_guitar/image_0039.jpg 47 test\nelectric_guitar/image_0075.jpg 47 test\nelectric_guitar/image_0028.jpg 47 test\noctopus/image_0015.jpg 48 test\noctopus/image_0029.jpg 48 test\noctopus/image_0023.jpg 48 test\noctopus/image_0010.jpg 48 test\noctopus/image_0022.jpg 48 test\noctopus/image_0032.jpg 48 test\noctopus/image_0028.jpg 48 test\nkangaroo/image_0026.jpg 49 test\nkangaroo/image_0004.jpg 49 test\nkangaroo/image_0015.jpg 49 test\nkangaroo/image_0029.jpg 49 test\nkangaroo/image_0023.jpg 49 test\nkangaroo/image_0072.jpg 49 test\nkangaroo/image_0010.jpg 49 test\nkangaroo/image_0056.jpg 49 test\nkangaroo/image_0022.jpg 49 test\nkangaroo/image_0065.jpg 49 test\nkangaroo/image_0036.jpg 49 test\nkangaroo/image_0086.jpg 49 test\nkangaroo/image_0032.jpg 49 test\nkangaroo/image_0076.jpg 49 test\nkangaroo/image_0038.jpg 49 test\nkangaroo/image_0039.jpg 49 test\nkangaroo/image_0075.jpg 49 test\nkangaroo/image_0028.jpg 49 test\nokapi/image_0023.jpg 50 test\nokapi/image_0010.jpg 50 test\nokapi/image_0022.jpg 50 test\nokapi/image_0036.jpg 50 test\nokapi/image_0032.jpg 50 test\nokapi/image_0038.jpg 50 test\nokapi/image_0039.jpg 50 test\nokapi/image_0028.jpg 50 test\nsunflower/image_0026.jpg 51 test\nsunflower/image_0004.jpg 51 test\nsunflower/image_0015.jpg 51 test\nsunflower/image_0029.jpg 51 test\nsunflower/image_0023.jpg 51 test\nsunflower/image_0072.jpg 51 test\nsunflower/image_0010.jpg 51 test\nsunflower/image_0056.jpg 51 test\nsunflower/image_0022.jpg 51 test\nsunflower/image_0065.jpg 51 test\nsunflower/image_0036.jpg 51 test\nsunflower/image_0032.jpg 51 test\nsunflower/image_0076.jpg 51 test\nsunflower/image_0038.jpg 51 test\nsunflower/image_0039.jpg 51 test\nsunflower/image_0075.jpg 51 test\nsunflower/image_0028.jpg 51 test\ngarfield/image_0015.jpg 52 test\ngarfield/image_0029.jpg 52 test\ngarfield/image_0023.jpg 52 test\ngarfield/image_0010.jpg 52 test\ngarfield/image_0022.jpg 52 test\ngarfield/image_0032.jpg 52 test\ngarfield/image_0028.jpg 52 test\nbass/image_0004.jpg 53 test\nbass/image_0015.jpg 53 test\nbass/image_0029.jpg 53 test\nbass/image_0023.jpg 53 test\nbass/image_0010.jpg 53 test\nbass/image_0022.jpg 53 test\nbass/image_0036.jpg 53 test\nbass/image_0032.jpg 53 test\nbass/image_0038.jpg 53 test\nbass/image_0039.jpg 53 test\nbass/image_0028.jpg 53 test\ncellphone/image_0004.jpg 54 test\ncellphone/image_0015.jpg 54 test\ncellphone/image_0029.jpg 54 test\ncellphone/image_0023.jpg 54 test\ncellphone/image_0010.jpg 54 test\ncellphone/image_0056.jpg 54 test\ncellphone/image_0022.jpg 54 test\ncellphone/image_0036.jpg 54 test\ncellphone/image_0032.jpg 54 test\ncellphone/image_0038.jpg 54 test\ncellphone/image_0039.jpg 54 test\ncellphone/image_0028.jpg 54 test\nbrain/image_0026.jpg 55 test\nbrain/image_0004.jpg 55 test\nbrain/image_0015.jpg 55 test\nbrain/image_0029.jpg 55 test\nbrain/image_0023.jpg 55 test\nbrain/image_0072.jpg 55 test\nbrain/image_0010.jpg 55 test\nbrain/image_0056.jpg 55 test\nbrain/image_0022.jpg 55 test\nbrain/image_0065.jpg 55 test\nbrain/image_0036.jpg 55 test\nbrain/image_0086.jpg 55 test\nbrain/image_0032.jpg 55 test\nbrain/image_0076.jpg 55 test\nbrain/image_0038.jpg 55 test\nbrain/image_0039.jpg 55 test\nbrain/image_0075.jpg 55 test\nbrain/image_0096.jpg 55 test\nbrain/image_0028.jpg 55 test\nbrain/image_0092.jpg 55 test\nlobster/image_0029.jpg 56 test\nlobster/image_0023.jpg 56 test\nlobster/image_0010.jpg 56 test\nlobster/image_0022.jpg 56 test\nlobster/image_0036.jpg 56 test\nlobster/image_0032.jpg 56 test\nlobster/image_0038.jpg 56 test\nlobster/image_0039.jpg 56 test\nlobster/image_0028.jpg 56 test\nheadphone/image_0029.jpg 57 test\nheadphone/image_0023.jpg 57 test\nheadphone/image_0010.jpg 57 test\nheadphone/image_0022.jpg 57 test\nheadphone/image_0036.jpg 57 test\nheadphone/image_0032.jpg 57 test\nheadphone/image_0038.jpg 57 test\nheadphone/image_0039.jpg 57 test\nheadphone/image_0028.jpg 57 test\nbarrel/image_0015.jpg 58 test\nbarrel/image_0029.jpg 58 test\nbarrel/image_0023.jpg 58 test\nbarrel/image_0010.jpg 58 test\nbarrel/image_0022.jpg 58 test\nbarrel/image_0036.jpg 58 test\nbarrel/image_0032.jpg 58 test\nbarrel/image_0038.jpg 58 test\nbarrel/image_0039.jpg 58 test\nbarrel/image_0028.jpg 58 test\npigeon/image_0029.jpg 59 test\npigeon/image_0023.jpg 59 test\npigeon/image_0010.jpg 59 test\npigeon/image_0022.jpg 59 test\npigeon/image_0036.jpg 59 test\npigeon/image_0032.jpg 59 test\npigeon/image_0038.jpg 59 test\npigeon/image_0039.jpg 59 test\npigeon/image_0028.jpg 59 test\ninline_skate/image_0004.jpg 60 test\ninline_skate/image_0015.jpg 60 test\ninline_skate/image_0029.jpg 60 test\ninline_skate/image_0023.jpg 60 test\ninline_skate/image_0010.jpg 60 test\ninline_skate/image_0022.jpg 60 test\ninline_skate/image_0028.jpg 60 test\ncannon/image_0029.jpg 61 test\ncannon/image_0023.jpg 61 test\ncannon/image_0010.jpg 61 test\ncannon/image_0022.jpg 61 test\ncannon/image_0036.jpg 61 test\ncannon/image_0032.jpg 61 test\ncannon/image_0038.jpg 61 test\ncannon/image_0039.jpg 61 test\ncannon/image_0028.jpg 61 test\nBACKGROUND_Google/image_0029.jpg 62 test\nBACKGROUND_Google/image_0327.jpg 62 test\nBACKGROUND_Google/image_0120.jpg 62 test\nBACKGROUND_Google/image_0281.jpg 62 test\nBACKGROUND_Google/image_0381.jpg 62 test\nBACKGROUND_Google/image_0220.jpg 62 test\nBACKGROUND_Google/image_0345.jpg 62 test\nBACKGROUND_Google/image_0023.jpg 62 test\nBACKGROUND_Google/image_0135.jpg 62 test\nBACKGROUND_Google/image_0452.jpg 62 test\nBACKGROUND_Google/image_0423.jpg 62 test\nBACKGROUND_Google/image_0454.jpg 62 test\nBACKGROUND_Google/image_0466.jpg 62 test\nBACKGROUND_Google/image_0111.jpg 62 test\nBACKGROUND_Google/image_0124.jpg 62 test\nBACKGROUND_Google/image_0279.jpg 62 test\nBACKGROUND_Google/image_0424.jpg 62 test\nBACKGROUND_Google/image_0418.jpg 62 test\nBACKGROUND_Google/image_0379.jpg 62 test\nBACKGROUND_Google/image_0273.jpg 62 test\nBACKGROUND_Google/image_0072.jpg 62 test\nBACKGROUND_Google/image_0145.jpg 62 test\nBACKGROUND_Google/image_0214.jpg 62 test\nBACKGROUND_Google/image_0174.jpg 62 test\nBACKGROUND_Google/image_0133.jpg 62 test\nBACKGROUND_Google/image_0152.jpg 62 test\nBACKGROUND_Google/image_0289.jpg 62 test\nBACKGROUND_Google/image_0342.jpg 62 test\nBACKGROUND_Google/image_0298.jpg 62 test\nBACKGROUND_Google/image_0010.jpg 62 test\nBACKGROUND_Google/image_0248.jpg 62 test\nBACKGROUND_Google/image_0245.jpg 62 test\nBACKGROUND_Google/image_0056.jpg 62 test\nBACKGROUND_Google/image_0022.jpg 62 test\nBACKGROUND_Google/image_0307.jpg 62 test\nBACKGROUND_Google/image_0164.jpg 62 test\nBACKGROUND_Google/image_0377.jpg 62 test\nBACKGROUND_Google/image_0065.jpg 62 test\nBACKGROUND_Google/image_0244.jpg 62 test\nBACKGROUND_Google/image_0100.jpg 62 test\nBACKGROUND_Google/image_0137.jpg 62 test\nBACKGROUND_Google/image_0274.jpg 62 test\nBACKGROUND_Google/image_0323.jpg 62 test\nBACKGROUND_Google/image_0036.jpg 62 test\nBACKGROUND_Google/image_0086.jpg 62 test\nBACKGROUND_Google/image_0032.jpg 62 test\nBACKGROUND_Google/image_0405.jpg 62 test\nBACKGROUND_Google/image_0394.jpg 62 test\nBACKGROUND_Google/image_0076.jpg 62 test\nBACKGROUND_Google/image_0439.jpg 62 test\nBACKGROUND_Google/image_0252.jpg 62 test\nBACKGROUND_Google/image_0235.jpg 62 test\nBACKGROUND_Google/image_0190.jpg 62 test\nBACKGROUND_Google/image_0290.jpg 62 test\nBACKGROUND_Google/image_0453.jpg 62 test\nBACKGROUND_Google/image_0106.jpg 62 test\nBACKGROUND_Google/image_0140.jpg 62 test\nBACKGROUND_Google/image_0415.jpg 62 test\nBACKGROUND_Google/image_0117.jpg 62 test\nBACKGROUND_Google/image_0038.jpg 62 test\nBACKGROUND_Google/image_0210.jpg 62 test\nBACKGROUND_Google/image_0456.jpg 62 test\nBACKGROUND_Google/image_0107.jpg 62 test\nBACKGROUND_Google/image_0247.jpg 62 test\nBACKGROUND_Google/image_0362.jpg 62 test\nBACKGROUND_Google/image_0158.jpg 62 test\nBACKGROUND_Google/image_0430.jpg 62 test\nBACKGROUND_Google/image_0268.jpg 62 test\nBACKGROUND_Google/image_0242.jpg 62 test\nBACKGROUND_Google/image_0401.jpg 62 test\nBACKGROUND_Google/image_0179.jpg 62 test\nBACKGROUND_Google/image_0170.jpg 62 test\nBACKGROUND_Google/image_0378.jpg 62 test\nBACKGROUND_Google/image_0039.jpg 62 test\nBACKGROUND_Google/image_0075.jpg 62 test\nBACKGROUND_Google/image_0194.jpg 62 test\nBACKGROUND_Google/image_0096.jpg 62 test\nBACKGROUND_Google/image_0238.jpg 62 test\nBACKGROUND_Google/image_0236.jpg 62 test\nBACKGROUND_Google/image_0311.jpg 62 test\nBACKGROUND_Google/image_0116.jpg 62 test\nBACKGROUND_Google/image_0419.jpg 62 test\nBACKGROUND_Google/image_0138.jpg 62 test\nBACKGROUND_Google/image_0449.jpg 62 test\nBACKGROUND_Google/image_0197.jpg 62 test\nBACKGROUND_Google/image_0241.jpg 62 test\nBACKGROUND_Google/image_0028.jpg 62 test\nBACKGROUND_Google/image_0151.jpg 62 test\nBACKGROUND_Google/image_0385.jpg 62 test\nBACKGROUND_Google/image_0092.jpg 62 test\nBACKGROUND_Google/image_0182.jpg 62 test\nBACKGROUND_Google/image_0413.jpg 62 test\nBACKGROUND_Google/image_0359.jpg 62 test\nBACKGROUND_Google/image_0374.jpg 62 test\nmandolin/image_0029.jpg 63 test\nmandolin/image_0023.jpg 63 test\nmandolin/image_0010.jpg 63 test\nmandolin/image_0022.jpg 63 test\nmandolin/image_0036.jpg 63 test\nmandolin/image_0032.jpg 63 test\nmandolin/image_0038.jpg 63 test\nmandolin/image_0039.jpg 63 test\nmandolin/image_0028.jpg 63 test\nMotorbikes/image_0355.jpg 64 test\nMotorbikes/image_0029.jpg 64 test\nMotorbikes/image_0662.jpg 64 test\nMotorbikes/image_0647.jpg 64 test\nMotorbikes/image_0798.jpg 64 test\nMotorbikes/image_0327.jpg 64 test\nMotorbikes/image_0120.jpg 64 test\nMotorbikes/image_0281.jpg 64 test\nMotorbikes/image_0603.jpg 64 test\nMotorbikes/image_0653.jpg 64 test\nMotorbikes/image_0381.jpg 64 test\nMotorbikes/image_0220.jpg 64 test\nMotorbikes/image_0345.jpg 64 test\nMotorbikes/image_0520.jpg 64 test\nMotorbikes/image_0685.jpg 64 test\nMotorbikes/image_0503.jpg 64 test\nMotorbikes/image_0023.jpg 64 test\nMotorbikes/image_0135.jpg 64 test\nMotorbikes/image_0452.jpg 64 test\nMotorbikes/image_0584.jpg 64 test\nMotorbikes/image_0423.jpg 64 test\nMotorbikes/image_0782.jpg 64 test\nMotorbikes/image_0454.jpg 64 test\nMotorbikes/image_0466.jpg 64 test\nMotorbikes/image_0111.jpg 64 test\nMotorbikes/image_0124.jpg 64 test\nMotorbikes/image_0279.jpg 64 test\nMotorbikes/image_0478.jpg 64 test\nMotorbikes/image_0642.jpg 64 test\nMotorbikes/image_0666.jpg 64 test\nMotorbikes/image_0708.jpg 64 test\nMotorbikes/image_0424.jpg 64 test\nMotorbikes/image_0418.jpg 64 test\nMotorbikes/image_0379.jpg 64 test\nMotorbikes/image_0626.jpg 64 test\nMotorbikes/image_0610.jpg 64 test\nMotorbikes/image_0273.jpg 64 test\nMotorbikes/image_0587.jpg 64 test\nMotorbikes/image_0072.jpg 64 test\nMotorbikes/image_0598.jpg 64 test\nMotorbikes/image_0508.jpg 64 test\nMotorbikes/image_0145.jpg 64 test\nMotorbikes/image_0750.jpg 64 test\nMotorbikes/image_0214.jpg 64 test\nMotorbikes/image_0174.jpg 64 test\nMotorbikes/image_0754.jpg 64 test\nMotorbikes/image_0726.jpg 64 test\nMotorbikes/image_0133.jpg 64 test\nMotorbikes/image_0152.jpg 64 test\nMotorbikes/image_0289.jpg 64 test\nMotorbikes/image_0674.jpg 64 test\nMotorbikes/image_0609.jpg 64 test\nMotorbikes/image_0342.jpg 64 test\nMotorbikes/image_0298.jpg 64 test\nMotorbikes/image_0710.jpg 64 test\nMotorbikes/image_0010.jpg 64 test\nMotorbikes/image_0248.jpg 64 test\nMotorbikes/image_0245.jpg 64 test\nMotorbikes/image_0056.jpg 64 test\nMotorbikes/image_0022.jpg 64 test\nMotorbikes/image_0687.jpg 64 test\nMotorbikes/image_0307.jpg 64 test\nMotorbikes/image_0755.jpg 64 test\nMotorbikes/image_0164.jpg 64 test\nMotorbikes/image_0377.jpg 64 test\nMotorbikes/image_0715.jpg 64 test\nMotorbikes/image_0065.jpg 64 test\nMotorbikes/image_0244.jpg 64 test\nMotorbikes/image_0777.jpg 64 test\nMotorbikes/image_0100.jpg 64 test\nMotorbikes/image_0137.jpg 64 test\nMotorbikes/image_0274.jpg 64 test\nMotorbikes/image_0323.jpg 64 test\nMotorbikes/image_0576.jpg 64 test\nMotorbikes/image_0526.jpg 64 test\nMotorbikes/image_0036.jpg 64 test\nMotorbikes/image_0785.jpg 64 test\nMotorbikes/image_0599.jpg 64 test\nMotorbikes/image_0086.jpg 64 test\nMotorbikes/image_0032.jpg 64 test\nMotorbikes/image_0405.jpg 64 test\nMotorbikes/image_0394.jpg 64 test\nMotorbikes/image_0076.jpg 64 test\nMotorbikes/image_0682.jpg 64 test\nMotorbikes/image_0439.jpg 64 test\nMotorbikes/image_0705.jpg 64 test\nMotorbikes/image_0252.jpg 64 test\nMotorbikes/image_0235.jpg 64 test\nMotorbikes/image_0510.jpg 64 test\nMotorbikes/image_0190.jpg 64 test\nMotorbikes/image_0290.jpg 64 test\nMotorbikes/image_0453.jpg 64 test\nMotorbikes/image_0106.jpg 64 test\nMotorbikes/image_0140.jpg 64 test\nMotorbikes/image_0415.jpg 64 test\nMotorbikes/image_0117.jpg 64 test\nMotorbikes/image_0038.jpg 64 test\nMotorbikes/image_0773.jpg 64 test\nMotorbikes/image_0663.jpg 64 test\nMotorbikes/image_0624.jpg 64 test\nMotorbikes/image_0210.jpg 64 test\nMotorbikes/image_0456.jpg 64 test\nMotorbikes/image_0107.jpg 64 test\nMotorbikes/image_0247.jpg 64 test\nMotorbikes/image_0362.jpg 64 test\nMotorbikes/image_0760.jpg 64 test\nMotorbikes/image_0535.jpg 64 test\nMotorbikes/image_0158.jpg 64 test\nMotorbikes/image_0498.jpg 64 test\nMotorbikes/image_0430.jpg 64 test\nMotorbikes/image_0268.jpg 64 test\nMotorbikes/image_0660.jpg 64 test\nMotorbikes/image_0630.jpg 64 test\nMotorbikes/image_0592.jpg 64 test\nMotorbikes/image_0242.jpg 64 test\nMotorbikes/image_0401.jpg 64 test\nMotorbikes/image_0179.jpg 64 test\nMotorbikes/image_0170.jpg 64 test\nMotorbikes/image_0524.jpg 64 test\nMotorbikes/image_0378.jpg 64 test\nMotorbikes/image_0753.jpg 64 test\nMotorbikes/image_0507.jpg 64 test\nMotorbikes/image_0749.jpg 64 test\nMotorbikes/image_0039.jpg 64 test\nMotorbikes/image_0075.jpg 64 test\nMotorbikes/image_0744.jpg 64 test\nMotorbikes/image_0194.jpg 64 test\nMotorbikes/image_0695.jpg 64 test\nMotorbikes/image_0693.jpg 64 test\nMotorbikes/image_0096.jpg 64 test\nMotorbikes/image_0481.jpg 64 test\nMotorbikes/image_0238.jpg 64 test\nMotorbikes/image_0701.jpg 64 test\nMotorbikes/image_0236.jpg 64 test\nMotorbikes/image_0311.jpg 64 test\nMotorbikes/image_0724.jpg 64 test\nMotorbikes/image_0116.jpg 64 test\nMotorbikes/image_0419.jpg 64 test\nMotorbikes/image_0702.jpg 64 test\nMotorbikes/image_0725.jpg 64 test\nMotorbikes/image_0138.jpg 64 test\nMotorbikes/image_0774.jpg 64 test\nMotorbikes/image_0793.jpg 64 test\nMotorbikes/image_0449.jpg 64 test\nMotorbikes/image_0778.jpg 64 test\nMotorbikes/image_0197.jpg 64 test\nMotorbikes/image_0578.jpg 64 test\nMotorbikes/image_0241.jpg 64 test\nMotorbikes/image_0028.jpg 64 test\nMotorbikes/image_0151.jpg 64 test\nMotorbikes/image_0385.jpg 64 test\nMotorbikes/image_0550.jpg 64 test\nMotorbikes/image_0636.jpg 64 test\nMotorbikes/image_0545.jpg 64 test\nMotorbikes/image_0092.jpg 64 test\nMotorbikes/image_0632.jpg 64 test\nMotorbikes/image_0182.jpg 64 test\nMotorbikes/image_0413.jpg 64 test\nMotorbikes/image_0359.jpg 64 test\nMotorbikes/image_0374.jpg 64 test\ndollar_bill/image_0004.jpg 65 test\ndollar_bill/image_0015.jpg 65 test\ndollar_bill/image_0029.jpg 65 test\ndollar_bill/image_0023.jpg 65 test\ndollar_bill/image_0010.jpg 65 test\ndollar_bill/image_0022.jpg 65 test\ndollar_bill/image_0036.jpg 65 test\ndollar_bill/image_0032.jpg 65 test\ndollar_bill/image_0038.jpg 65 test\ndollar_bill/image_0039.jpg 65 test\ndollar_bill/image_0028.jpg 65 test\nnautilus/image_0004.jpg 66 test\nnautilus/image_0015.jpg 66 test\nnautilus/image_0029.jpg 66 test\nnautilus/image_0023.jpg 66 test\nnautilus/image_0010.jpg 66 test\nnautilus/image_0022.jpg 66 test\nnautilus/image_0036.jpg 66 test\nnautilus/image_0032.jpg 66 test\nnautilus/image_0038.jpg 66 test\nnautilus/image_0039.jpg 66 test\nnautilus/image_0028.jpg 66 test\ncrab/image_0026.jpg 67 test\ncrab/image_0004.jpg 67 test\ncrab/image_0015.jpg 67 test\ncrab/image_0029.jpg 67 test\ncrab/image_0023.jpg 67 test\ncrab/image_0072.jpg 67 test\ncrab/image_0010.jpg 67 test\ncrab/image_0056.jpg 67 test\ncrab/image_0022.jpg 67 test\ncrab/image_0065.jpg 67 test\ncrab/image_0036.jpg 67 test\ncrab/image_0032.jpg 67 test\ncrab/image_0038.jpg 67 test\ncrab/image_0039.jpg 67 test\ncrab/image_0028.jpg 67 test\naccordion/image_0004.jpg 68 test\naccordion/image_0015.jpg 68 test\naccordion/image_0029.jpg 68 test\naccordion/image_0023.jpg 68 test\naccordion/image_0010.jpg 68 test\naccordion/image_0022.jpg 68 test\naccordion/image_0036.jpg 68 test\naccordion/image_0032.jpg 68 test\naccordion/image_0038.jpg 68 test\naccordion/image_0039.jpg 68 test\naccordion/image_0028.jpg 68 test\ncrayfish/image_0026.jpg 69 test\ncrayfish/image_0004.jpg 69 test\ncrayfish/image_0015.jpg 69 test\ncrayfish/image_0029.jpg 69 test\ncrayfish/image_0023.jpg 69 test\ncrayfish/image_0010.jpg 69 test\ncrayfish/image_0056.jpg 69 test\ncrayfish/image_0022.jpg 69 test\ncrayfish/image_0065.jpg 69 test\ncrayfish/image_0036.jpg 69 test\ncrayfish/image_0032.jpg 69 test\ncrayfish/image_0038.jpg 69 test\ncrayfish/image_0039.jpg 69 test\ncrayfish/image_0028.jpg 69 test\nflamingo_head/image_0029.jpg 70 test\nflamingo_head/image_0023.jpg 70 test\nflamingo_head/image_0010.jpg 70 test\nflamingo_head/image_0022.jpg 70 test\nflamingo_head/image_0036.jpg 70 test\nflamingo_head/image_0032.jpg 70 test\nflamingo_head/image_0038.jpg 70 test\nflamingo_head/image_0039.jpg 70 test\nflamingo_head/image_0028.jpg 70 test\nemu/image_0004.jpg 71 test\nemu/image_0015.jpg 71 test\nemu/image_0029.jpg 71 test\nemu/image_0023.jpg 71 test\nemu/image_0010.jpg 71 test\nemu/image_0022.jpg 71 test\nemu/image_0036.jpg 71 test\nemu/image_0032.jpg 71 test\nemu/image_0038.jpg 71 test\nemu/image_0039.jpg 71 test\nemu/image_0028.jpg 71 test\ntrilobite/image_0026.jpg 72 test\ntrilobite/image_0004.jpg 72 test\ntrilobite/image_0015.jpg 72 test\ntrilobite/image_0029.jpg 72 test\ntrilobite/image_0023.jpg 72 test\ntrilobite/image_0072.jpg 72 test\ntrilobite/image_0010.jpg 72 test\ntrilobite/image_0056.jpg 72 test\ntrilobite/image_0022.jpg 72 test\ntrilobite/image_0065.jpg 72 test\ntrilobite/image_0036.jpg 72 test\ntrilobite/image_0086.jpg 72 test\ntrilobite/image_0032.jpg 72 test\ntrilobite/image_0076.jpg 72 test\ntrilobite/image_0038.jpg 72 test\ntrilobite/image_0039.jpg 72 test\ntrilobite/image_0075.jpg 72 test\ntrilobite/image_0028.jpg 72 test\ncamera/image_0015.jpg 73 test\ncamera/image_0029.jpg 73 test\ncamera/image_0023.jpg 73 test\ncamera/image_0010.jpg 73 test\ncamera/image_0022.jpg 73 test\ncamera/image_0036.jpg 73 test\ncamera/image_0032.jpg 73 test\ncamera/image_0038.jpg 73 test\ncamera/image_0039.jpg 73 test\ncamera/image_0028.jpg 73 test\nplatypus/image_0015.jpg 74 test\nplatypus/image_0029.jpg 74 test\nplatypus/image_0023.jpg 74 test\nplatypus/image_0010.jpg 74 test\nplatypus/image_0022.jpg 74 test\nplatypus/image_0032.jpg 74 test\nplatypus/image_0028.jpg 74 test\nchandelier/image_0004.jpg 75 test\nchandelier/image_0015.jpg 75 test\nchandelier/image_0029.jpg 75 test\nchandelier/image_0023.jpg 75 test\nchandelier/image_0072.jpg 75 test\nchandelier/image_0010.jpg 75 test\nchandelier/image_0056.jpg 75 test\nchandelier/image_0022.jpg 75 test\nchandelier/image_0065.jpg 75 test\nchandelier/image_0100.jpg 75 test\nchandelier/image_0036.jpg 75 test\nchandelier/image_0086.jpg 75 test\nchandelier/image_0032.jpg 75 test\nchandelier/image_0076.jpg 75 test\nchandelier/image_0106.jpg 75 test\nchandelier/image_0038.jpg 75 test\nchandelier/image_0107.jpg 75 test\nchandelier/image_0039.jpg 75 test\nchandelier/image_0075.jpg 75 test\nchandelier/image_0096.jpg 75 test\nchandelier/image_0028.jpg 75 test\nchandelier/image_0092.jpg 75 test\ncrocodile/image_0015.jpg 76 test\ncrocodile/image_0029.jpg 76 test\ncrocodile/image_0023.jpg 76 test\ncrocodile/image_0010.jpg 76 test\ncrocodile/image_0022.jpg 76 test\ncrocodile/image_0036.jpg 76 test\ncrocodile/image_0032.jpg 76 test\ncrocodile/image_0038.jpg 76 test\ncrocodile/image_0039.jpg 76 test\ncrocodile/image_0028.jpg 76 test\ncar_side/image_0015.jpg 77 test\ncar_side/image_0029.jpg 77 test\ncar_side/image_0120.jpg 77 test\ncar_side/image_0023.jpg 77 test\ncar_side/image_0111.jpg 77 test\ncar_side/image_0072.jpg 77 test\ncar_side/image_0010.jpg 77 test\ncar_side/image_0056.jpg 77 test\ncar_side/image_0022.jpg 77 test\ncar_side/image_0065.jpg 77 test\ncar_side/image_0100.jpg 77 test\ncar_side/image_0036.jpg 77 test\ncar_side/image_0086.jpg 77 test\ncar_side/image_0032.jpg 77 test\ncar_side/image_0076.jpg 77 test\ncar_side/image_0106.jpg 77 test\ncar_side/image_0117.jpg 77 test\ncar_side/image_0038.jpg 77 test\ncar_side/image_0107.jpg 77 test\ncar_side/image_0039.jpg 77 test\ncar_side/image_0075.jpg 77 test\ncar_side/image_0096.jpg 77 test\ncar_side/image_0116.jpg 77 test\ncar_side/image_0028.jpg 77 test\ncar_side/image_0092.jpg 77 test\njoshua_tree/image_0026.jpg 78 test\njoshua_tree/image_0004.jpg 78 test\njoshua_tree/image_0015.jpg 78 test\njoshua_tree/image_0029.jpg 78 test\njoshua_tree/image_0023.jpg 78 test\njoshua_tree/image_0010.jpg 78 test\njoshua_tree/image_0056.jpg 78 test\njoshua_tree/image_0022.jpg 78 test\njoshua_tree/image_0036.jpg 78 test\njoshua_tree/image_0032.jpg 78 test\njoshua_tree/image_0038.jpg 78 test\njoshua_tree/image_0039.jpg 78 test\njoshua_tree/image_0028.jpg 78 test\ntick/image_0015.jpg 79 test\ntick/image_0029.jpg 79 test\ntick/image_0023.jpg 79 test\ntick/image_0010.jpg 79 test\ntick/image_0022.jpg 79 test\ntick/image_0036.jpg 79 test\ntick/image_0032.jpg 79 test\ntick/image_0038.jpg 79 test\ntick/image_0039.jpg 79 test\ntick/image_0028.jpg 79 test\nhawksbill/image_0004.jpg 80 test\nhawksbill/image_0015.jpg 80 test\nhawksbill/image_0029.jpg 80 test\nhawksbill/image_0023.jpg 80 test\nhawksbill/image_0072.jpg 80 test\nhawksbill/image_0010.jpg 80 test\nhawksbill/image_0056.jpg 80 test\nhawksbill/image_0022.jpg 80 test\nhawksbill/image_0065.jpg 80 test\nhawksbill/image_0100.jpg 80 test\nhawksbill/image_0036.jpg 80 test\nhawksbill/image_0086.jpg 80 test\nhawksbill/image_0032.jpg 80 test\nhawksbill/image_0076.jpg 80 test\nhawksbill/image_0038.jpg 80 test\nhawksbill/image_0039.jpg 80 test\nhawksbill/image_0075.jpg 80 test\nhawksbill/image_0096.jpg 80 test\nhawksbill/image_0028.jpg 80 test\nhawksbill/image_0092.jpg 80 test\nhelicopter/image_0026.jpg 81 test\nhelicopter/image_0004.jpg 81 test\nhelicopter/image_0015.jpg 81 test\nhelicopter/image_0029.jpg 81 test\nhelicopter/image_0023.jpg 81 test\nhelicopter/image_0072.jpg 81 test\nhelicopter/image_0010.jpg 81 test\nhelicopter/image_0056.jpg 81 test\nhelicopter/image_0022.jpg 81 test\nhelicopter/image_0065.jpg 81 test\nhelicopter/image_0036.jpg 81 test\nhelicopter/image_0086.jpg 81 test\nhelicopter/image_0032.jpg 81 test\nhelicopter/image_0076.jpg 81 test\nhelicopter/image_0038.jpg 81 test\nhelicopter/image_0039.jpg 81 test\nhelicopter/image_0075.jpg 81 test\nhelicopter/image_0028.jpg 81 test\npagoda/image_0015.jpg 82 test\npagoda/image_0029.jpg 82 test\npagoda/image_0023.jpg 82 test\npagoda/image_0010.jpg 82 test\npagoda/image_0022.jpg 82 test\npagoda/image_0036.jpg 82 test\npagoda/image_0032.jpg 82 test\npagoda/image_0038.jpg 82 test\npagoda/image_0039.jpg 82 test\npagoda/image_0028.jpg 82 test\newer/image_0026.jpg 83 test\newer/image_0004.jpg 83 test\newer/image_0015.jpg 83 test\newer/image_0029.jpg 83 test\newer/image_0023.jpg 83 test\newer/image_0072.jpg 83 test\newer/image_0010.jpg 83 test\newer/image_0056.jpg 83 test\newer/image_0022.jpg 83 test\newer/image_0065.jpg 83 test\newer/image_0036.jpg 83 test\newer/image_0032.jpg 83 test\newer/image_0076.jpg 83 test\newer/image_0038.jpg 83 test\newer/image_0039.jpg 83 test\newer/image_0075.jpg 83 test\newer/image_0028.jpg 83 test\npanda/image_0029.jpg 84 test\npanda/image_0023.jpg 84 test\npanda/image_0010.jpg 84 test\npanda/image_0022.jpg 84 test\npanda/image_0036.jpg 84 test\npanda/image_0032.jpg 84 test\npanda/image_0038.jpg 84 test\npanda/image_0028.jpg 84 test\npizza/image_0004.jpg 85 test\npizza/image_0015.jpg 85 test\npizza/image_0029.jpg 85 test\npizza/image_0023.jpg 85 test\npizza/image_0010.jpg 85 test\npizza/image_0022.jpg 85 test\npizza/image_0036.jpg 85 test\npizza/image_0032.jpg 85 test\npizza/image_0038.jpg 85 test\npizza/image_0039.jpg 85 test\npizza/image_0028.jpg 85 test\ncup/image_0004.jpg 86 test\ncup/image_0015.jpg 86 test\ncup/image_0029.jpg 86 test\ncup/image_0023.jpg 86 test\ncup/image_0010.jpg 86 test\ncup/image_0056.jpg 86 test\ncup/image_0022.jpg 86 test\ncup/image_0036.jpg 86 test\ncup/image_0032.jpg 86 test\ncup/image_0038.jpg 86 test\ncup/image_0039.jpg 86 test\ncup/image_0028.jpg 86 test\nanchor/image_0029.jpg 87 test\nanchor/image_0023.jpg 87 test\nanchor/image_0010.jpg 87 test\nanchor/image_0022.jpg 87 test\nanchor/image_0036.jpg 87 test\nanchor/image_0032.jpg 87 test\nanchor/image_0038.jpg 87 test\nanchor/image_0039.jpg 87 test\nanchor/image_0028.jpg 87 test\nhedgehog/image_0004.jpg 88 test\nhedgehog/image_0015.jpg 88 test\nhedgehog/image_0029.jpg 88 test\nhedgehog/image_0023.jpg 88 test\nhedgehog/image_0010.jpg 88 test\nhedgehog/image_0022.jpg 88 test\nhedgehog/image_0036.jpg 88 test\nhedgehog/image_0032.jpg 88 test\nhedgehog/image_0038.jpg 88 test\nhedgehog/image_0039.jpg 88 test\nhedgehog/image_0028.jpg 88 test\nflamingo/image_0026.jpg 89 test\nflamingo/image_0004.jpg 89 test\nflamingo/image_0015.jpg 89 test\nflamingo/image_0029.jpg 89 test\nflamingo/image_0023.jpg 89 test\nflamingo/image_0010.jpg 89 test\nflamingo/image_0056.jpg 89 test\nflamingo/image_0022.jpg 89 test\nflamingo/image_0065.jpg 89 test\nflamingo/image_0036.jpg 89 test\nflamingo/image_0032.jpg 89 test\nflamingo/image_0038.jpg 89 test\nflamingo/image_0039.jpg 89 test\nflamingo/image_0028.jpg 89 test\nstegosaurus/image_0004.jpg 90 test\nstegosaurus/image_0015.jpg 90 test\nstegosaurus/image_0029.jpg 90 test\nstegosaurus/image_0023.jpg 90 test\nstegosaurus/image_0010.jpg 90 test\nstegosaurus/image_0056.jpg 90 test\nstegosaurus/image_0022.jpg 90 test\nstegosaurus/image_0036.jpg 90 test\nstegosaurus/image_0032.jpg 90 test\nstegosaurus/image_0038.jpg 90 test\nstegosaurus/image_0039.jpg 90 test\nstegosaurus/image_0028.jpg 90 test\nferry/image_0026.jpg 91 test\nferry/image_0004.jpg 91 test\nferry/image_0015.jpg 91 test\nferry/image_0029.jpg 91 test\nferry/image_0023.jpg 91 test\nferry/image_0010.jpg 91 test\nferry/image_0056.jpg 91 test\nferry/image_0022.jpg 91 test\nferry/image_0065.jpg 91 test\nferry/image_0036.jpg 91 test\nferry/image_0032.jpg 91 test\nferry/image_0038.jpg 91 test\nferry/image_0039.jpg 91 test\nferry/image_0028.jpg 91 test\ndalmatian/image_0026.jpg 92 test\ndalmatian/image_0004.jpg 92 test\ndalmatian/image_0015.jpg 92 test\ndalmatian/image_0029.jpg 92 test\ndalmatian/image_0023.jpg 92 test\ndalmatian/image_0010.jpg 92 test\ndalmatian/image_0056.jpg 92 test\ndalmatian/image_0022.jpg 92 test\ndalmatian/image_0065.jpg 92 test\ndalmatian/image_0036.jpg 92 test\ndalmatian/image_0032.jpg 92 test\ndalmatian/image_0038.jpg 92 test\ndalmatian/image_0039.jpg 92 test\ndalmatian/image_0028.jpg 92 test\nwheelchair/image_0004.jpg 93 test\nwheelchair/image_0015.jpg 93 test\nwheelchair/image_0029.jpg 93 test\nwheelchair/image_0023.jpg 93 test\nwheelchair/image_0010.jpg 93 test\nwheelchair/image_0056.jpg 93 test\nwheelchair/image_0022.jpg 93 test\nwheelchair/image_0036.jpg 93 test\nwheelchair/image_0032.jpg 93 test\nwheelchair/image_0038.jpg 93 test\nwheelchair/image_0039.jpg 93 test\nwheelchair/image_0028.jpg 93 test\nwatch/image_0029.jpg 94 test\nwatch/image_0120.jpg 94 test\nwatch/image_0220.jpg 94 test\nwatch/image_0023.jpg 94 test\nwatch/image_0135.jpg 94 test\nwatch/image_0111.jpg 94 test\nwatch/image_0124.jpg 94 test\nwatch/image_0072.jpg 94 test\nwatch/image_0145.jpg 94 test\nwatch/image_0214.jpg 94 test\nwatch/image_0174.jpg 94 test\nwatch/image_0133.jpg 94 test\nwatch/image_0152.jpg 94 test\nwatch/image_0010.jpg 94 test\nwatch/image_0056.jpg 94 test\nwatch/image_0022.jpg 94 test\nwatch/image_0164.jpg 94 test\nwatch/image_0065.jpg 94 test\nwatch/image_0100.jpg 94 test\nwatch/image_0137.jpg 94 test\nwatch/image_0036.jpg 94 test\nwatch/image_0086.jpg 94 test\nwatch/image_0032.jpg 94 test\nwatch/image_0076.jpg 94 test\nwatch/image_0235.jpg 94 test\nwatch/image_0190.jpg 94 test\nwatch/image_0106.jpg 94 test\nwatch/image_0140.jpg 94 test\nwatch/image_0117.jpg 94 test\nwatch/image_0038.jpg 94 test\nwatch/image_0210.jpg 94 test\nwatch/image_0107.jpg 94 test\nwatch/image_0158.jpg 94 test\nwatch/image_0179.jpg 94 test\nwatch/image_0170.jpg 94 test\nwatch/image_0039.jpg 94 test\nwatch/image_0075.jpg 94 test\nwatch/image_0194.jpg 94 test\nwatch/image_0096.jpg 94 test\nwatch/image_0238.jpg 94 test\nwatch/image_0236.jpg 94 test\nwatch/image_0116.jpg 94 test\nwatch/image_0138.jpg 94 test\nwatch/image_0197.jpg 94 test\nwatch/image_0028.jpg 94 test\nwatch/image_0151.jpg 94 test\nwatch/image_0092.jpg 94 test\nwatch/image_0182.jpg 94 test\nsea_horse/image_0004.jpg 95 test\nsea_horse/image_0015.jpg 95 test\nsea_horse/image_0029.jpg 95 test\nsea_horse/image_0023.jpg 95 test\nsea_horse/image_0010.jpg 95 test\nsea_horse/image_0056.jpg 95 test\nsea_horse/image_0022.jpg 95 test\nsea_horse/image_0036.jpg 95 test\nsea_horse/image_0032.jpg 95 test\nsea_horse/image_0038.jpg 95 test\nsea_horse/image_0039.jpg 95 test\nsea_horse/image_0028.jpg 95 test\npyramid/image_0004.jpg 96 test\npyramid/image_0015.jpg 96 test\npyramid/image_0029.jpg 96 test\npyramid/image_0023.jpg 96 test\npyramid/image_0010.jpg 96 test\npyramid/image_0056.jpg 96 test\npyramid/image_0022.jpg 96 test\npyramid/image_0036.jpg 96 test\npyramid/image_0032.jpg 96 test\npyramid/image_0038.jpg 96 test\npyramid/image_0039.jpg 96 test\npyramid/image_0028.jpg 96 test\nstrawberry/image_0015.jpg 97 test\nstrawberry/image_0029.jpg 97 test\nstrawberry/image_0023.jpg 97 test\nstrawberry/image_0010.jpg 97 test\nstrawberry/image_0022.jpg 97 test\nstrawberry/image_0032.jpg 97 test\nstrawberry/image_0028.jpg 97 test\nFaces_easy/image_0029.jpg 98 test\nFaces_easy/image_0327.jpg 98 test\nFaces_easy/image_0120.jpg 98 test\nFaces_easy/image_0281.jpg 98 test\nFaces_easy/image_0381.jpg 98 test\nFaces_easy/image_0220.jpg 98 test\nFaces_easy/image_0345.jpg 98 test\nFaces_easy/image_0023.jpg 98 test\nFaces_easy/image_0135.jpg 98 test\nFaces_easy/image_0423.jpg 98 test\nFaces_easy/image_0111.jpg 98 test\nFaces_easy/image_0124.jpg 98 test\nFaces_easy/image_0279.jpg 98 test\nFaces_easy/image_0424.jpg 98 test\nFaces_easy/image_0418.jpg 98 test\nFaces_easy/image_0379.jpg 98 test\nFaces_easy/image_0273.jpg 98 test\nFaces_easy/image_0072.jpg 98 test\nFaces_easy/image_0145.jpg 98 test\nFaces_easy/image_0214.jpg 98 test\nFaces_easy/image_0174.jpg 98 test\nFaces_easy/image_0133.jpg 98 test\nFaces_easy/image_0152.jpg 98 test\nFaces_easy/image_0289.jpg 98 test\nFaces_easy/image_0342.jpg 98 test\nFaces_easy/image_0298.jpg 98 test\nFaces_easy/image_0010.jpg 98 test\nFaces_easy/image_0248.jpg 98 test\nFaces_easy/image_0245.jpg 98 test\nFaces_easy/image_0056.jpg 98 test\nFaces_easy/image_0022.jpg 98 test\nFaces_easy/image_0307.jpg 98 test\nFaces_easy/image_0164.jpg 98 test\nFaces_easy/image_0377.jpg 98 test\nFaces_easy/image_0065.jpg 98 test\nFaces_easy/image_0244.jpg 98 test\nFaces_easy/image_0100.jpg 98 test\nFaces_easy/image_0137.jpg 98 test\nFaces_easy/image_0274.jpg 98 test\nFaces_easy/image_0323.jpg 98 test\nFaces_easy/image_0036.jpg 98 test\nFaces_easy/image_0086.jpg 98 test\nFaces_easy/image_0032.jpg 98 test\nFaces_easy/image_0405.jpg 98 test\nFaces_easy/image_0394.jpg 98 test\nFaces_easy/image_0076.jpg 98 test\nFaces_easy/image_0252.jpg 98 test\nFaces_easy/image_0235.jpg 98 test\nFaces_easy/image_0190.jpg 98 test\nFaces_easy/image_0290.jpg 98 test\nFaces_easy/image_0106.jpg 98 test\nFaces_easy/image_0140.jpg 98 test\nFaces_easy/image_0415.jpg 98 test\nFaces_easy/image_0117.jpg 98 test\nFaces_easy/image_0038.jpg 98 test\nFaces_easy/image_0210.jpg 98 test\nFaces_easy/image_0107.jpg 98 test\nFaces_easy/image_0247.jpg 98 test\nFaces_easy/image_0362.jpg 98 test\nFaces_easy/image_0158.jpg 98 test\nFaces_easy/image_0430.jpg 98 test\nFaces_easy/image_0268.jpg 98 test\nFaces_easy/image_0242.jpg 98 test\nFaces_easy/image_0401.jpg 98 test\nFaces_easy/image_0179.jpg 98 test\nFaces_easy/image_0170.jpg 98 test\nFaces_easy/image_0378.jpg 98 test\nFaces_easy/image_0039.jpg 98 test\nFaces_easy/image_0075.jpg 98 test\nFaces_easy/image_0194.jpg 98 test\nFaces_easy/image_0096.jpg 98 test\nFaces_easy/image_0238.jpg 98 test\nFaces_easy/image_0236.jpg 98 test\nFaces_easy/image_0311.jpg 98 test\nFaces_easy/image_0116.jpg 98 test\nFaces_easy/image_0419.jpg 98 test\nFaces_easy/image_0138.jpg 98 test\nFaces_easy/image_0197.jpg 98 test\nFaces_easy/image_0241.jpg 98 test\nFaces_easy/image_0028.jpg 98 test\nFaces_easy/image_0151.jpg 98 test\nFaces_easy/image_0385.jpg 98 test\nFaces_easy/image_0092.jpg 98 test\nFaces_easy/image_0182.jpg 98 test\nFaces_easy/image_0413.jpg 98 test\nFaces_easy/image_0359.jpg 98 test\nFaces_easy/image_0374.jpg 98 test\ncougar_face/image_0026.jpg 99 test\ncougar_face/image_0004.jpg 99 test\ncougar_face/image_0015.jpg 99 test\ncougar_face/image_0029.jpg 99 test\ncougar_face/image_0023.jpg 99 test\ncougar_face/image_0010.jpg 99 test\ncougar_face/image_0056.jpg 99 test\ncougar_face/image_0022.jpg 99 test\ncougar_face/image_0065.jpg 99 test\ncougar_face/image_0036.jpg 99 test\ncougar_face/image_0032.jpg 99 test\ncougar_face/image_0038.jpg 99 test\ncougar_face/image_0039.jpg 99 test\ncougar_face/image_0028.jpg 99 test\nmayfly/image_0023.jpg 100 test\nmayfly/image_0010.jpg 100 test\nmayfly/image_0022.jpg 100 test\nmayfly/image_0036.jpg 100 test\nmayfly/image_0032.jpg 100 test\nmayfly/image_0038.jpg 100 test\nmayfly/image_0039.jpg 100 test\nmayfly/image_0028.jpg 100 test\nwrench/image_0023.jpg 101 test\nwrench/image_0010.jpg 101 test\nwrench/image_0022.jpg 101 test\nwrench/image_0036.jpg 101 test\nwrench/image_0032.jpg 101 test\nwrench/image_0038.jpg 101 test\nwrench/image_0039.jpg 101 test\nwrench/image_0028.jpg 101 test\n"
  },
  {
    "path": "cpp/dcgan/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\nproject(dcgan)\n\nfind_package(Torch REQUIRED)\n\noption(DOWNLOAD_MNIST \"Download the MNIST dataset from the internet\" ON)\nif (DOWNLOAD_MNIST)\n  message(STATUS \"Downloading MNIST dataset\")\n  execute_process(\n    COMMAND python ${CMAKE_CURRENT_LIST_DIR}/../tools/download_mnist.py\n      -d ${CMAKE_BINARY_DIR}/data\n    ERROR_VARIABLE DOWNLOAD_ERROR)\n  if (DOWNLOAD_ERROR)\n    message(FATAL_ERROR \"Error downloading MNIST dataset: ${DOWNLOAD_ERROR}\")\n  endif()\nendif()\n\nadd_executable(dcgan dcgan.cpp)\ntarget_link_libraries(dcgan \"${TORCH_LIBRARIES}\")\nset_property(TARGET dcgan PROPERTY CXX_STANDARD 17)\n\nif (MSVC)\n  file(GLOB TORCH_DLLS \"${TORCH_INSTALL_PREFIX}/lib/*.dll\")\n  add_custom_command(TARGET dcgan\n                     POST_BUILD\n                     COMMAND ${CMAKE_COMMAND} -E copy_if_different\n                     ${TORCH_DLLS}\n                     $<TARGET_FILE_DIR:dcgan>)\nendif (MSVC)\n"
  },
  {
    "path": "cpp/dcgan/README.md",
    "content": "# DCGAN Example with the PyTorch C++ Frontend\n\nThis folder contains an example of training a DCGAN to generate MNIST digits\nwith the PyTorch C++ frontend.\n\nThe entire training code is contained in `dcgan.cpp`.\n\nYou can find the commands to install argparse [here](https://github.com/pytorch/examples/blob/main/.github/workflows/main_cpp.yml#L34).\n\nTo build the code, run the following commands from your terminal:\n\n```shell\n$ cd dcgan\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..\n$ make\n```\n\nwhere `/path/to/libtorch` should be the path to the unzipped _LibTorch_\ndistribution, which you can get from the [PyTorch\nhomepage](https://pytorch.org/get-started/locally/).\n\nExecute the compiled binary to train the model:\n\n```shell\n$ ./dcgan\n[ 1/30][200/938] D_loss: 0.4953 | G_loss: 4.0195\n-> checkpoint 1\n[ 1/30][400/938] D_loss: 0.3610 | G_loss: 4.8148\n-> checkpoint 2\n[ 1/30][600/938] D_loss: 0.4072 | G_loss: 4.36760\n-> checkpoint 3\n[ 1/30][800/938] D_loss: 0.4444 | G_loss: 4.0250\n-> checkpoint 4\n[ 2/30][200/938] D_loss: 0.3761 | G_loss: 3.8790\n-> checkpoint 5\n[ 2/30][400/938] D_loss: 0.3977 | G_loss: 3.3315\n-> checkpoint 6\n[ 2/30][600/938] D_loss: 0.3815 | G_loss: 3.5696\n-> checkpoint 7\n[ 2/30][800/938] D_loss: 0.4039 | G_loss: 3.2759\n-> checkpoint 8\n[ 3/30][200/938] D_loss: 0.4236 | G_loss: 4.5132\n-> checkpoint 9\n[ 3/30][400/938] D_loss: 0.3645 | G_loss: 3.9759\n-> checkpoint 10\n...\n```\n\nWe can also specify the `--epochs` to change the number of epochs to train as follows:\n\n```shell\n$ ./dcgan --epochs 10\n```\nWithout specifying the `--epochs` flag, the default number of epochs to train is 30.\n\n\nThe training script periodically generates image samples. Use the\n`display_samples.py` script situated in this folder to generate a plot image.\nFor example:\n\n```shell\n$ python display_samples.py -i dcgan-sample-10.pt\nSaved out.png\n```\n"
  },
  {
    "path": "cpp/dcgan/dcgan.cpp",
    "content": "#include <torch/torch.h>\n#include <argparse/argparse.hpp>\n#include <cmath>\n#include <cstdio>\n#include <iostream>\n\n// The size of the noise vector fed to the generator.\nconst int64_t kNoiseSize = 100;\n\n// The batch size for training.\nconst int64_t kBatchSize = 64;\n\n// Where to find the MNIST dataset.\nconst char* kDataFolder = \"./data\";\n\n// After how many batches to create a new checkpoint periodically.\nconst int64_t kCheckpointEvery = 200;\n\n// How many images to sample at every checkpoint.\nconst int64_t kNumberOfSamplesPerCheckpoint = 10;\n\n// Set to `true` to restore models and optimizers from previously saved\n// checkpoints.\nconst bool kRestoreFromCheckpoint = false;\n\n// After how many batches to log a new update with the loss value.\nconst int64_t kLogInterval = 10;\n\nusing namespace torch;\n\nstruct DCGANGeneratorImpl : nn::Module {\n  DCGANGeneratorImpl(int kNoiseSize)\n      : conv1(nn::ConvTranspose2dOptions(kNoiseSize, 256, 4)\n                  .bias(false)),\n        batch_norm1(256),\n        conv2(nn::ConvTranspose2dOptions(256, 128, 3)\n                  .stride(2)\n                  .padding(1)\n                  .bias(false)),\n        batch_norm2(128),\n        conv3(nn::ConvTranspose2dOptions(128, 64, 4)\n                  .stride(2)\n                  .padding(1)\n                  .bias(false)),\n        batch_norm3(64),\n        conv4(nn::ConvTranspose2dOptions(64, 1, 4)\n                  .stride(2)\n                  .padding(1)\n                  .bias(false))\n {\n   // register_module() is needed if we want to use the parameters() method later on\n   register_module(\"conv1\", conv1);\n   register_module(\"conv2\", conv2);\n   register_module(\"conv3\", conv3);\n   register_module(\"conv4\", conv4);\n   register_module(\"batch_norm1\", batch_norm1);\n   register_module(\"batch_norm2\", batch_norm2);\n   register_module(\"batch_norm3\", batch_norm3);\n }\n\n torch::Tensor forward(torch::Tensor x) {\n   x = torch::relu(batch_norm1(conv1(x)));\n   x = torch::relu(batch_norm2(conv2(x)));\n   x = torch::relu(batch_norm3(conv3(x)));\n   x = torch::tanh(conv4(x));\n   return x;\n }\n\n nn::ConvTranspose2d conv1, conv2, conv3, conv4;\n nn::BatchNorm2d batch_norm1, batch_norm2, batch_norm3;\n};\n\nTORCH_MODULE(DCGANGenerator);\n\nnn::Sequential create_discriminator() {\n    return nn::Sequential(\n     // Layer 1\n     nn::Conv2d(nn::Conv2dOptions(1, 64, 4).stride(2).padding(1).bias(false)),\n     nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),\n     // Layer 2\n     nn::Conv2d(nn::Conv2dOptions(64, 128, 4).stride(2).padding(1).bias(false)),\n     nn::BatchNorm2d(128),\n     nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),\n     // Layer 3\n     nn::Conv2d(\n         nn::Conv2dOptions(128, 256, 4).stride(2).padding(1).bias(false)),\n     nn::BatchNorm2d(256),\n     nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),\n     // Layer 4\n     nn::Conv2d(nn::Conv2dOptions(256, 1, 3).stride(1).padding(0).bias(false)),\n     nn::Sigmoid());\n}\n\nint main(int argc, const char* argv[]) {\n  argparse::ArgumentParser parser(\"cpp/dcgan example\");\n  parser.add_argument(\"--epochs\")\n      .help(\"Number of epochs to train\")\n      .default_value(std::int64_t{30})\n      .scan<'i', int64_t>();\n  try {\n    parser.parse_args(argc, argv);\n  } catch (const std::exception& err) {\n    std::cout << err.what() << std::endl;\n    std::cout << parser;\n    std::exit(1);\n  }\n  // The number of epochs to train, default value is 30.\n  const int64_t kNumberOfEpochs = parser.get<int64_t>(\"--epochs\");\n  std::cout << \"Traning with number of epochs: \" << kNumberOfEpochs\n            << std::endl;\n\n  torch::manual_seed(1);\n\n  // Create the device we pass around based on whether CUDA is available.\n  torch::Device device(torch::kCPU);\n  if (torch::cuda::is_available()) {\n    std::cout << \"CUDA is available! Training on GPU.\" << std::endl;\n    device = torch::Device(torch::kCUDA);\n  }\n\n  DCGANGenerator generator(kNoiseSize);\n  generator->to(device);\n\n  nn::Sequential discriminator = create_discriminator();\n  discriminator->to(device);\n\n  // Assume the MNIST dataset is available under `kDataFolder`;\n  auto dataset = torch::data::datasets::MNIST(kDataFolder)\n                     .map(torch::data::transforms::Normalize<>(0.5, 0.5))\n                     .map(torch::data::transforms::Stack<>());\n  const int64_t batches_per_epoch = static_cast<int64_t>(\n      std::ceil(dataset.size().value() / static_cast<double>(kBatchSize)));\n\n  auto data_loader = torch::data::make_data_loader(\n      std::move(dataset),\n      torch::data::DataLoaderOptions().batch_size(kBatchSize).workers(2));\n\n  torch::optim::Adam generator_optimizer(\n      generator->parameters(), torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));\n  torch::optim::Adam discriminator_optimizer(\n      discriminator->parameters(), torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));\n\n  if (kRestoreFromCheckpoint) {\n    torch::load(generator, \"generator-checkpoint.pt\");\n    torch::load(generator_optimizer, \"generator-optimizer-checkpoint.pt\");\n    torch::load(discriminator, \"discriminator-checkpoint.pt\");\n    torch::load(\n        discriminator_optimizer, \"discriminator-optimizer-checkpoint.pt\");\n  }\n\n  int64_t checkpoint_counter = 1;\n  for (int64_t epoch = 1; epoch <= kNumberOfEpochs; ++epoch) {\n    int64_t batch_index = 0;\n    for (const torch::data::Example<>& batch : *data_loader) {\n      // Train discriminator with real images.\n      discriminator->zero_grad();\n      torch::Tensor real_images = batch.data.to(device);\n      torch::Tensor real_labels =\n          torch::empty(batch.data.size(0), device).uniform_(0.8, 1.0);\n      torch::Tensor real_output = discriminator->forward(real_images).reshape(real_labels.sizes());\n      torch::Tensor d_loss_real =\n          torch::binary_cross_entropy(real_output, real_labels);\n      d_loss_real.backward();\n\n      // Train discriminator with fake images.\n      torch::Tensor noise =\n          torch::randn({batch.data.size(0), kNoiseSize, 1, 1}, device);\n      torch::Tensor fake_images = generator->forward(noise);\n      torch::Tensor fake_labels = torch::zeros(batch.data.size(0), device);\n      torch::Tensor fake_output = discriminator->forward(fake_images.detach()).reshape(fake_labels.sizes());\n      torch::Tensor d_loss_fake =\n          torch::binary_cross_entropy(fake_output, fake_labels);\n      d_loss_fake.backward();\n\n      torch::Tensor d_loss = d_loss_real + d_loss_fake;\n      discriminator_optimizer.step();\n\n      // Train generator.\n      generator->zero_grad();\n      fake_labels.fill_(1);\n      fake_output = discriminator->forward(fake_images).reshape(fake_labels.sizes());\n      torch::Tensor g_loss =\n          torch::binary_cross_entropy(fake_output, fake_labels);\n      g_loss.backward();\n      generator_optimizer.step();\n      batch_index++;\n      if (batch_index % kLogInterval == 0) {\n        std::printf(\n            \"\\r[%2ld/%2ld][%3ld/%3ld] D_loss: %.4f | G_loss: %.4f\\n\",\n            epoch,\n            kNumberOfEpochs,\n            batch_index,\n            batches_per_epoch,\n            d_loss.item<float>(),\n            g_loss.item<float>());\n      }\n\n      if (batch_index % kCheckpointEvery == 0) {\n        // Checkpoint the model and optimizer state.\n        torch::save(generator, \"generator-checkpoint.pt\");\n        torch::save(generator_optimizer, \"generator-optimizer-checkpoint.pt\");\n        torch::save(discriminator, \"discriminator-checkpoint.pt\");\n        torch::save(\n            discriminator_optimizer, \"discriminator-optimizer-checkpoint.pt\");\n        // Sample the generator and save the images.\n        torch::Tensor samples = generator->forward(torch::randn(\n            {kNumberOfSamplesPerCheckpoint, kNoiseSize, 1, 1}, device));\n        torch::save(\n            (samples + 1.0) / 2.0,\n            torch::str(\"dcgan-sample-\", checkpoint_counter, \".pt\"));\n        std::cout << \"\\n-> checkpoint \" << ++checkpoint_counter << '\\n';\n      }\n    }\n  }\n\n  std::cout << \"Training complete!\" << std::endl;\n}\n"
  },
  {
    "path": "cpp/dcgan/display_samples.py",
    "content": "from __future__ import print_function\nfrom __future__ import unicode_literals\n\nimport argparse\n\nimport matplotlib.pyplot as plt\nimport torch\n\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"-i\", \"--sample-file\", required=True)\nparser.add_argument(\"-o\", \"--out-file\", default=\"out.png\")\nparser.add_argument(\"-d\", \"--dimension\", type=int, default=3)\noptions = parser.parse_args()\n\nmodule = torch.jit.load(options.sample_file)\nimages = list(module.parameters())[0]\n\nfor index in range(options.dimension * options.dimension):\n    image = images[index].detach().cpu().reshape(28, 28).mul(255).to(torch.uint8)\n    array = image.numpy()\n    axis = plt.subplot(options.dimension, options.dimension, 1 + index)\n    plt.imshow(array, cmap=\"gray\")\n    axis.get_xaxis().set_visible(False)\n    axis.get_yaxis().set_visible(False)\n\nplt.savefig(options.out_file)\nprint(\"Saved \", options.out_file)\n"
  },
  {
    "path": "cpp/distributed/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\nproject(dist-mnist)\n\nfind_package(Torch REQUIRED)\n\nfind_package(MPI REQUIRED)\n\ninclude_directories(SYSTEM ${MPI_C_INCLUDE_PATH} ${MPI_CXX_INCLUDE_PATH})\n\nadd_executable(dist-mnist dist-mnist.cpp)\ntarget_link_libraries(dist-mnist ${TORCH_LIBRARIES})\ntarget_link_libraries(dist-mnist ${MPI_LIBRARIES})\ntarget_link_libraries(dist-mnist ${CMAKE_PREFIX_PATH}/lib/libc10d.a)\n\nif(MPI_COMPILE_FLAGS)\n  set_target_properties(dist-mnist PROPERTIES\n    COMPILE_FLAGS \"${MPI_COMPILE_FLAGS}\")\nendif()\n\nif(MPI_LINK_FLAGS)\n  set_target_properties(dist-mnist PROPERTIES\n    LINK_FLAGS \"${MPI_LINK_FLAGS}\")\nendif()\n"
  },
  {
    "path": "cpp/distributed/README.md",
    "content": "# Distributed Training on MNIST using PyTorch C++ Frontend (Libtorch)\n\nThis folder contains an example of data-parallel training of a convolutional neural network on the MNIST dataset. For parallelization, Message Passing Interface (MPI) is used.\n\nThe entire code is contained in dist-mnist.cpp\n\nYou can find instructions on how to install MPI [here] (https://www.open-mpi.org/faq/?category=building). This code was tested on Open MPI but it should run on other MPI distributions as well such as MPICH, MVAPICH, etc.\n\nTo build the code, run the following commands from the terminal:\n\n```shell\n$ cd distributed\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..\n$ make\n```\n\nwhere /path/to/libtorch should be the path to the unzipped LibTorch distribution. Note that the LibTorch from the [PyTorch homepage] ((https://pytorch.org/get-started/locally/) does not include MPI headers and cannot be used for this example. You have to compile LibTorch manually - a set of guidelines is provided [here] (https://gist.github.com/lasagnaphil/3e0099816837318e8e8bcab7edcfd5d9), however this may vary for different systems.\n\nTo run the code,\n\n```shell\nmpirun -np {NUM-PROCS} ./dist-mnist\n```\n"
  },
  {
    "path": "cpp/distributed/dist-mnist.cpp",
    "content": "#include <c10d/ProcessGroupMPI.hpp>\n#include <torch/torch.h>\n#include <iostream>\n\n// Define a Convolutional Module\nstruct Model : torch::nn::Module {\n  Model()\n      : conv1(torch::nn::Conv2dOptions(1, 10, 5)),\n        conv2(torch::nn::Conv2dOptions(10, 20, 5)),\n        fc1(320, 50),\n        fc2(50, 10) {\n    register_module(\"conv1\", conv1);\n    register_module(\"conv2\", conv2);\n    register_module(\"conv2_drop\", conv2_drop);\n    register_module(\"fc1\", fc1);\n    register_module(\"fc2\", fc2);\n  }\n\n  torch::Tensor forward(torch::Tensor x) {\n    x = torch::relu(torch::max_pool2d(conv1->forward(x), 2));\n    x = torch::relu(\n        torch::max_pool2d(conv2_drop->forward(conv2->forward(x)), 2));\n    x = x.view({-1, 320});\n    x = torch::relu(fc1->forward(x));\n    x = torch::dropout(x, 0.5, is_training());\n    x = fc2->forward(x);\n    return torch::log_softmax(x, 1);\n  }\n\n  torch::nn::Conv2d conv1;\n  torch::nn::Conv2d conv2;\n  torch::nn::Dropout2d conv2_drop;\n  torch::nn::Linear fc1;\n  torch::nn::Linear fc2;\n};\n\nvoid waitWork(\n    std::shared_ptr<c10d::ProcessGroupMPI> pg,\n    std::vector<std::shared_ptr<c10d::ProcessGroup::Work>> works) {\n  for (auto& work : works) {\n    try {\n      work->wait();\n    } catch (const std::exception& ex) {\n      std::cerr << \"Exception received: \" << ex.what() << std::endl;\n      pg->abort();\n    }\n  }\n}\n\nint main(int argc, char* argv[]) {\n  // Creating MPI Process Group\n  auto pg = c10d::ProcessGroupMPI::createProcessGroupMPI();\n\n  // Retrieving MPI environment variables\n  auto numranks = pg->getSize();\n  auto rank = pg->getRank();\n\n  // TRAINING\n  // Read train dataset\n  const char* kDataRoot = \"../data\";\n  auto train_dataset =\n      torch::data::datasets::MNIST(kDataRoot)\n          .map(torch::data::transforms::Normalize<>(0.1307, 0.3081))\n          .map(torch::data::transforms::Stack<>());\n\n  // Distributed Random Sampler\n  auto data_sampler = torch::data::samplers::DistributedRandomSampler(\n      train_dataset.size().value(), numranks, rank, false);\n\n  auto num_train_samples_per_proc = train_dataset.size().value() / numranks;\n\n  // Generate dataloader\n  auto total_batch_size = 64;\n  auto batch_size_per_proc =\n      total_batch_size / numranks; // effective batch size in each processor\n  auto data_loader = torch::data::make_data_loader(\n      std::move(train_dataset), data_sampler, batch_size_per_proc);\n\n  // setting manual seed\n  torch::manual_seed(0);\n\n  auto model = std::make_shared<Model>();\n\n  auto learning_rate = 1e-2;\n\n  torch::optim::SGD optimizer(model->parameters(), learning_rate);\n\n  // Number of epochs\n  size_t num_epochs = 10;\n\n  for (size_t epoch = 1; epoch <= num_epochs; ++epoch) {\n    size_t num_correct = 0;\n\n    for (auto& batch : *data_loader) {\n      auto ip = batch.data;\n      auto op = batch.target.squeeze();\n\n      // convert to required formats\n      ip = ip.to(torch::kF32);\n      op = op.to(torch::kLong);\n\n      // Reset gradients\n      model->zero_grad();\n\n      // Execute forward pass\n      auto prediction = model->forward(ip);\n\n      auto loss = torch::nll_loss(torch::log_softmax(prediction, 1), op);\n\n      // Backpropagation\n      loss.backward();\n\n      // Averaging the gradients of the parameters in all the processors\n      // Note: This may lag behind DistributedDataParallel (DDP) in performance\n      // since this synchronizes parameters after backward pass while DDP\n      // overlaps synchronizing parameters and computing gradients in backward\n      // pass\n      std::vector<std::shared_ptr<::c10d::ProcessGroup::Work>> works;\n      for (auto& param : model->named_parameters()) {\n        std::vector<torch::Tensor> tmp = {param.value().grad()};\n        auto work = pg->allreduce(tmp);\n        works.push_back(std::move(work));\n      }\n\n      waitWork(pg, works);\n\n      for (auto& param : model->named_parameters()) {\n        param.value().grad().data() = param.value().grad().data() / numranks;\n      }\n\n      // Update parameters\n      optimizer.step();\n\n      auto guess = prediction.argmax(1);\n      num_correct += torch::sum(guess.eq_(op)).item<int64_t>();\n    } // end batch loader\n\n    auto accuracy = 100.0 * num_correct / num_train_samples_per_proc;\n\n    std::cout << \"Accuracy in rank \" << rank << \" in epoch \" << epoch << \" - \"\n              << accuracy << std::endl;\n\n  } // end epoch\n\n  // TESTING ONLY IN RANK 0\n  if (rank == 0) {\n    auto test_dataset =\n        torch::data::datasets::MNIST(\n            kDataRoot, torch::data::datasets::MNIST::Mode::kTest)\n            .map(torch::data::transforms::Normalize<>(0.1307, 0.3081))\n            .map(torch::data::transforms::Stack<>());\n\n    auto num_test_samples = test_dataset.size().value();\n    auto test_loader = torch::data::make_data_loader(\n        std::move(test_dataset), num_test_samples);\n\n    model->eval(); // enable eval mode to prevent backprop\n\n    size_t num_correct = 0;\n\n    for (auto& batch : *test_loader) {\n      auto ip = batch.data;\n      auto op = batch.target.squeeze();\n\n      // convert to required format\n      ip = ip.to(torch::kF32);\n      op = op.to(torch::kLong);\n\n      auto prediction = model->forward(ip);\n\n      auto loss = torch::nll_loss(torch::log_softmax(prediction, 1), op);\n\n      std::cout << \"Test loss - \" << loss.item<float>() << std::endl;\n\n      auto guess = prediction.argmax(1);\n\n      num_correct += torch::sum(guess.eq_(op)).item<int64_t>();\n\n    } // end test loader\n\n    std::cout << \"Num correct - \" << num_correct << std::endl;\n    std::cout << \"Test Accuracy - \" << 100.0 * num_correct / num_test_samples\n              << std::endl;\n  } // end rank 0\n}\n"
  },
  {
    "path": "cpp/mnist/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\nproject(mnist)\nset(CMAKE_CXX_STANDARD 17)\n\nfind_package(Torch REQUIRED)\n\noption(DOWNLOAD_MNIST \"Download the MNIST dataset from the internet\" ON)\nif (DOWNLOAD_MNIST)\n  message(STATUS \"Downloading MNIST dataset\")\n  execute_process(\n    COMMAND python ${CMAKE_CURRENT_LIST_DIR}/../tools/download_mnist.py\n      -d ${CMAKE_BINARY_DIR}/data\n    ERROR_VARIABLE DOWNLOAD_ERROR)\n  if (DOWNLOAD_ERROR)\n    message(FATAL_ERROR \"Error downloading MNIST dataset: ${DOWNLOAD_ERROR}\")\n  endif()\nendif()\n\nadd_executable(mnist mnist.cpp)\ntarget_compile_features(mnist PUBLIC cxx_range_for)\ntarget_link_libraries(mnist ${TORCH_LIBRARIES})\n\nif (MSVC)\n  file(GLOB TORCH_DLLS \"${TORCH_INSTALL_PREFIX}/lib/*.dll\")\n  add_custom_command(TARGET mnist\n                     POST_BUILD\n                     COMMAND ${CMAKE_COMMAND} -E copy_if_different\n                     ${TORCH_DLLS}\n                     $<TARGET_FILE_DIR:mnist>)\nendif (MSVC)\n"
  },
  {
    "path": "cpp/mnist/README.md",
    "content": "# MNIST Example with the PyTorch C++ Frontend\n\nThis folder contains an example of training a computer vision model to recognize\ndigits in images from the MNIST dataset, using the PyTorch C++ frontend.\n\nThe entire training code is contained in `mnist.cpp`.\n\nTo build the code, run the following commands from your terminal:\n\n```shell\n$ cd mnist\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..\n$ make\n```\n\nwhere `/path/to/libtorch` should be the path to the unzipped _LibTorch_\ndistribution, which you can get from the [PyTorch\nhomepage](https://pytorch.org/get-started/locally/).\n\nExecute the compiled binary to train the model:\n\n```shell\n$ ./mnist\nTrain Epoch: 1 [59584/60000] Loss: 0.4232\nTest set: Average loss: 0.1989 | Accuracy: 0.940\nTrain Epoch: 2 [59584/60000] Loss: 0.1926\nTest set: Average loss: 0.1338 | Accuracy: 0.959\nTrain Epoch: 3 [59584/60000] Loss: 0.1390\nTest set: Average loss: 0.0997 | Accuracy: 0.969\nTrain Epoch: 4 [59584/60000] Loss: 0.1239\nTest set: Average loss: 0.0875 | Accuracy: 0.972\n...\n```\n"
  },
  {
    "path": "cpp/mnist/mnist.cpp",
    "content": "#include <torch/torch.h>\n\n#include <cstddef>\n#include <cstdio>\n#include <iostream>\n#include <string>\n#include <vector>\n\n// Where to find the MNIST dataset.\nconst char* kDataRoot = \"./data\";\n\n// The batch size for training.\nconst int64_t kTrainBatchSize = 64;\n\n// The batch size for testing.\nconst int64_t kTestBatchSize = 1000;\n\n// The number of epochs to train.\nconst int64_t kNumberOfEpochs = 10;\n\n// After how many batches to log a new update with the loss value.\nconst int64_t kLogInterval = 10;\n\nstruct Net : torch::nn::Module {\n  Net()\n      : conv1(torch::nn::Conv2dOptions(1, 10, /*kernel_size=*/5)),\n        conv2(torch::nn::Conv2dOptions(10, 20, /*kernel_size=*/5)),\n        fc1(320, 50),\n        fc2(50, 10) {\n    register_module(\"conv1\", conv1);\n    register_module(\"conv2\", conv2);\n    register_module(\"conv2_drop\", conv2_drop);\n    register_module(\"fc1\", fc1);\n    register_module(\"fc2\", fc2);\n  }\n\n  torch::Tensor forward(torch::Tensor x) {\n    x = torch::relu(torch::max_pool2d(conv1->forward(x), 2));\n    x = torch::relu(\n        torch::max_pool2d(conv2_drop->forward(conv2->forward(x)), 2));\n    x = x.view({-1, 320});\n    x = torch::relu(fc1->forward(x));\n    x = torch::dropout(x, /*p=*/0.5, /*training=*/is_training());\n    x = fc2->forward(x);\n    return torch::log_softmax(x, /*dim=*/1);\n  }\n\n  torch::nn::Conv2d conv1;\n  torch::nn::Conv2d conv2;\n  torch::nn::Dropout2d conv2_drop;\n  torch::nn::Linear fc1;\n  torch::nn::Linear fc2;\n};\n\ntemplate <typename DataLoader>\nvoid train(\n    size_t epoch,\n    Net& model,\n    torch::Device device,\n    DataLoader& data_loader,\n    torch::optim::Optimizer& optimizer,\n    size_t dataset_size) {\n  model.train();\n  size_t batch_idx = 0;\n  for (auto& batch : data_loader) {\n    auto data = batch.data.to(device), targets = batch.target.to(device);\n    optimizer.zero_grad();\n    auto output = model.forward(data);\n    auto loss = torch::nll_loss(output, targets);\n    AT_ASSERT(!std::isnan(loss.template item<float>()));\n    loss.backward();\n    optimizer.step();\n\n    if (batch_idx++ % kLogInterval == 0) {\n      std::printf(\n          \"\\rTrain Epoch: %ld [%5ld/%5ld] Loss: %.4f\",\n          epoch,\n          batch_idx * batch.data.size(0),\n          dataset_size,\n          loss.template item<float>());\n    }\n  }\n}\n\ntemplate <typename DataLoader>\nvoid test(\n    Net& model,\n    torch::Device device,\n    DataLoader& data_loader,\n    size_t dataset_size) {\n  torch::NoGradGuard no_grad;\n  model.eval();\n  double test_loss = 0;\n  int32_t correct = 0;\n  for (const auto& batch : data_loader) {\n    auto data = batch.data.to(device), targets = batch.target.to(device);\n    auto output = model.forward(data);\n    test_loss += torch::nll_loss(\n                     output,\n                     targets,\n                     /*weight=*/{},\n                     torch::Reduction::Sum)\n                     .template item<float>();\n    auto pred = output.argmax(1);\n    correct += pred.eq(targets).sum().template item<int64_t>();\n  }\n\n  test_loss /= dataset_size;\n  std::printf(\n      \"\\nTest set: Average loss: %.4f | Accuracy: %.3f\\n\",\n      test_loss,\n      static_cast<double>(correct) / dataset_size);\n}\n\nauto main() -> int {\n  torch::manual_seed(1);\n\n  torch::DeviceType device_type;\n  if (torch::cuda::is_available()) {\n    std::cout << \"CUDA available! Training on GPU.\" << std::endl;\n    device_type = torch::kCUDA;\n  } else {\n    std::cout << \"Training on CPU.\" << std::endl;\n    device_type = torch::kCPU;\n  }\n  torch::Device device(device_type);\n\n  Net model;\n  model.to(device);\n\n  auto train_dataset = torch::data::datasets::MNIST(kDataRoot)\n                           .map(torch::data::transforms::Normalize<>(0.1307, 0.3081))\n                           .map(torch::data::transforms::Stack<>());\n  const size_t train_dataset_size = train_dataset.size().value();\n  auto train_loader =\n      torch::data::make_data_loader<torch::data::samplers::SequentialSampler>(\n          std::move(train_dataset), kTrainBatchSize);\n\n  auto test_dataset = torch::data::datasets::MNIST(\n                          kDataRoot, torch::data::datasets::MNIST::Mode::kTest)\n                          .map(torch::data::transforms::Normalize<>(0.1307, 0.3081))\n                          .map(torch::data::transforms::Stack<>());\n  const size_t test_dataset_size = test_dataset.size().value();\n  auto test_loader =\n      torch::data::make_data_loader(std::move(test_dataset), kTestBatchSize);\n\n  torch::optim::SGD optimizer(\n      model.parameters(), torch::optim::SGDOptions(0.01).momentum(0.5));\n\n  for (size_t epoch = 1; epoch <= kNumberOfEpochs; ++epoch) {\n    train(epoch, model, device, *train_loader, optimizer, train_dataset_size);\n    test(model, device, *test_loader, test_dataset_size);\n  }\n}\n"
  },
  {
    "path": "cpp/regression/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\n\nproject(regression)\nset(CMAKE_CXX_STANDARD 17)\n\nfind_package(Torch REQUIRED)\n\nadd_executable(${PROJECT_NAME} \"regression.cpp\")\ntarget_link_libraries(${PROJECT_NAME} \"${TORCH_LIBRARIES}\")\n\n# The following code block is suggested to be used on Windows.\n# According to https://github.com/pytorch/pytorch/issues/25457,\n# the DLLs need to be copied to avoid memory errors.\nif (MSVC)\n  file(GLOB TORCH_DLLS \"${TORCH_INSTALL_PREFIX}/lib/*.dll\")\n  add_custom_command(TARGET ${PROJECT_NAME}\n                     POST_BUILD\n                     COMMAND ${CMAKE_COMMAND} -E copy_if_different\n                     ${TORCH_DLLS}\n                     $<TARGET_FILE_DIR:${PROJECT_NAME}>)\nendif (MSVC)\n"
  },
  {
    "path": "cpp/regression/README.md",
    "content": "# Linear regression example\n\nTrains a single fully-connected layer to fit a 4th degree polynomial.\n\nTo build the code, run the following commands from your terminal:\n\n```shell\n$ cd regression\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..\n$ make\n```\n\nwhere `/path/to/libtorch` should be the path to the unzipped _LibTorch_\ndistribution, which you can get from the [PyTorch\nhomepage](https://pytorch.org/get-started/locally/).\n\nExecute the compiled binary to run:\n\n```shell\n$ ./regression\nLoss: 0.000301158 after 584 batches\n==> Learned function:\ty = 11.6441 x^4 -3.10164 x^3 2.19786 x^2 -3.83606 x^1 + 4.37066\n==> Actual function:\ty = 11.669 x^4 -3.16023 x^3 2.19182 x^2 -3.81505 x^1 + 4.38219\n...\n```\n"
  },
  {
    "path": "cpp/regression/regression.cpp",
    "content": "#include <torch/torch.h>\n#include <iostream>\n#include <sstream>\n#include <string>\n#include <vector>\n\n#define POLY_DEGREE 4\n\n// Builds features i.e. a matrix with columns [x, x^2, x^3, x^4].\ntorch::Tensor make_features(torch::Tensor x) {\n  x = x.unsqueeze(1);\n  std::vector<torch::Tensor> xs;\n  for (int64_t i = 0; i < POLY_DEGREE; ++i)\n    xs.push_back(x.pow(i + 1));\n  return torch::cat(xs, 1);\n}\n\n// Approximated function.\ntorch::Tensor f(\n    torch::Tensor x,\n    torch::Tensor W_target,\n    torch::Tensor b_target) {\n  return x.mm(W_target) + b_target.item();\n}\n\n// Creates a string description of a polynomial.\nstd::string poly_desc(torch::Tensor W, torch::Tensor b) {\n  auto size = W.size(0);\n  std::ostringstream stream;\n\n  stream << \"y = \";\n  for (int64_t i = 0; i < size; ++i)\n    stream << W[i].item<float>() << \" x^\" << size - i << \" \";\n  stream << \"+ \" << b[0].item<float>();\n  return stream.str();\n}\n\n// Builds a batch i.e. (x, f(x)) pair.\nstd::pair<torch::Tensor, torch::Tensor> get_batch(\n    torch::Tensor W_target,\n    torch::Tensor b_target,\n    int64_t batch_size = 32) {\n  auto random = torch::randn({batch_size});\n  auto x = make_features(random);\n  auto y = f(x, W_target, b_target);\n  return std::make_pair(x, y);\n}\n\nint main() {\n  auto W_target = torch::randn({POLY_DEGREE, 1}) * 5;\n  auto b_target = torch::randn({1}) * 5;\n\n  // Define the model and optimizer\n  auto fc = torch::nn::Linear(W_target.size(0), 1);\n  torch::optim::SGD optim(fc->parameters(), .1);\n\n  float loss = 0;\n  int64_t batch_idx = 0;\n\n  while (++batch_idx) {\n    // Get data\n    torch::Tensor batch_x, batch_y;\n    std::tie(batch_x, batch_y) = get_batch(W_target, b_target);\n\n    // Reset gradients\n    optim.zero_grad();\n\n    // Forward pass\n    auto output = torch::smooth_l1_loss(fc(batch_x), batch_y);\n    loss = output.item<float>();\n\n    // Backward pass\n    output.backward();\n\n    // Apply gradients\n    optim.step();\n\n    // Stop criterion\n    if (loss < 1e-3f)\n      break;\n  }\n\n  std::cout << \"Loss: \" << loss << \" after \" << batch_idx << \" batches\"\n            << std::endl;\n  std::cout << \"==> Learned function:\\t\"\n            << poly_desc(fc->weight.view({-1}), fc->bias) << std::endl;\n  std::cout << \"==> Actual function:\\t\"\n            << poly_desc(W_target.view({-1}), b_target) << std::endl;\n\n  return 0;\n}\n"
  },
  {
    "path": "cpp/tools/InstallingOpenCV.md",
    "content": "# Installing OpenCV \n\n## Linux with Package Manager\n\n### Arch Linux\n\n```shell\npacman -Syu base-devel opencv\n```\n\n### Fedora\n\n```shell\nsudo dnf install opencv opencv-dev\n```\n\n## Linux From Source\n\nRequired Packages:\n\n```shell\nsudo apt-get install build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev\n```\n\nOptional Packages:\n\n```shell\nsudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev\n```\n\nBuilding from Source:\n\n```shell\ngit clone https://github.com/opencv/opencv.git\ngit clone https://github.com/opencv/opencv_contrib.git\n\ncd opencv && mkdir build && cd build\ncmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..\nmake -j8 # runs 8 jobs in parallel\nsudo make install\n```\n\n## Windows\n\nYou can download the pre-built libraries from [OpenCV releases](https://github.com/opencv/opencv/releases) and install them easily.\n"
  },
  {
    "path": "cpp/tools/download_mnist.py",
    "content": "from __future__ import division\nfrom __future__ import print_function\n\nimport argparse\nimport gzip\nimport os\nimport sys\nimport urllib\n\ntry:\n    from urllib.error import URLError\n    from urllib.request import urlretrieve\nexcept ImportError:\n    from urllib2 import URLError\n    from urllib import urlretrieve\n\nRESOURCES = [\n    'train-images-idx3-ubyte.gz',\n    'train-labels-idx1-ubyte.gz',\n    't10k-images-idx3-ubyte.gz',\n    't10k-labels-idx1-ubyte.gz',\n]\n\n\ndef report_download_progress(chunk_number, chunk_size, file_size):\n    if file_size != -1:\n        percent = min(1, (chunk_number * chunk_size) / file_size)\n        bar = '#' * int(64 * percent)\n        sys.stdout.write('\\r0% |{:<64}| {}%'.format(bar, int(percent * 100)))\n\n\ndef download(destination_path, url, quiet):\n    if os.path.exists(destination_path):\n        if not quiet:\n            print('{} already exists, skipping ...'.format(destination_path))\n    else:\n        print('Downloading {} ...'.format(url))\n        try:\n            hook = None if quiet else report_download_progress\n            urlretrieve(url, destination_path, reporthook=hook)\n        except URLError:\n            raise RuntimeError('Error downloading resource!')\n        finally:\n            if not quiet:\n                # Just a newline.\n                print()\n\n\ndef unzip(zipped_path, quiet):\n    unzipped_path = os.path.splitext(zipped_path)[0]\n    if os.path.exists(unzipped_path):\n        if not quiet:\n            print('{} already exists, skipping ... '.format(unzipped_path))\n        return\n    with gzip.open(zipped_path, 'rb') as zipped_file:\n        with open(unzipped_path, 'wb') as unzipped_file:\n            unzipped_file.write(zipped_file.read())\n            if not quiet:\n                print('Unzipped {} ...'.format(zipped_path))\n\n\ndef main():\n    parser = argparse.ArgumentParser(\n        description='Download the MNIST dataset from the internet')\n    parser.add_argument(\n        '-d', '--destination', default='.', help='Destination directory')\n    parser.add_argument(\n        '-q',\n        '--quiet',\n        action='store_true',\n        help=\"Don't report about progress\")\n    options = parser.parse_args()\n\n    if not os.path.exists(options.destination):\n        os.makedirs(options.destination)\n\n    try:\n        for resource in RESOURCES:\n            path = os.path.join(options.destination, resource)\n            # url = 'http://yann.lecun.com/exdb/mnist/{}'.format(resource)\n            url = 'https://ossci-datasets.s3.amazonaws.com/mnist/{}'.format(resource)\n            download(path, url, options.quiet)\n            unzip(path, options.quiet)\n    except KeyboardInterrupt:\n        print('Interrupted')\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "cpp/transfer-learning/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.5)\nproject(example)\n\nfind_package(Torch REQUIRED)\nfind_package(OpenCV 4.1.0 REQUIRED)\n\ninclude_directories(${OpenCV_INCLUDE_DIRS})\n\nadd_executable(example main.cpp main.h)\nadd_executable(classify classify.cpp)\n\ntarget_link_libraries(example ${OpenCV_LIBS})\ntarget_link_libraries(example \"${TORCH_LIBRARIES}\")\ntarget_link_libraries(classify ${OpenCV_LIBS})\ntarget_link_libraries(classify \"${TORCH_LIBRARIES}\")\n\nset_property(TARGET classify PROPERTY CXX_STANDARD 17)\nset_property(TARGET example PROPERTY CXX_STANDARD 17)\n"
  },
  {
    "path": "cpp/transfer-learning/README.md",
    "content": "# Transfer Learning on Dogs vs Cats Dataset using Libtorch and OpenCV\n\nTransfer Learning on Dogs vs Cats dataset using PyTorch C++ API.\n\n## Usage\n\nFor **training**:\n\n1. Remove final layer of `ResNet18` pre-trained model and convert to `torch.jit` module: `python3 convert.py`.\n2. Create build directory: `mkdir build && cd build`\n3. `cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..`\n4. `make`\n5. Run training code: `./example <path_to_scripting_model>`\n\nFor **prediction**:\n\n1. `cd build`\n2. `./classify <path_image> <path_to_resnet18_model_without_fc_layer> <model_linear_trained>` : `./classify <path_image> ../resnet18_without_last_layer.pt model_linear.pt`\n\nDetailed blog on applying Transfer Learning using Libtorch: https://krshrimali.github.io/Applying-Transfer-Learning-Dogs-Cats/.\n"
  },
  {
    "path": "cpp/transfer-learning/classify.cpp",
    "content": "//\n//  classify.cpp\n//  transfer-learning\n//\n//  Created by Kushashwa Ravi Shrimali on 15/08/19.\n//\n\n#include <iostream>\n#include <torch/torch.h>\n#include <opencv2/opencv.hpp>\n#include <torch/script.h>\n#include <dirent.h>\n\n// Utility function to load image from given folder\n// File type accepted: .jpg\nstd::vector<std::string> load_images(std::string folder_name) {\n    std::vector<std::string> list_images;\n    std::string base_name = folder_name;\n    DIR* dir;\n    struct dirent *ent;\n    if((dir = opendir(base_name.c_str())) != NULL) {\n        while((ent = readdir(dir)) != NULL) {\n            std::string filename = ent->d_name;\n            if(filename.length() > 4 && filename.substr(filename.length() - 3) == \"jpg\") {\n                std::string newf = base_name + filename;\n                list_images.push_back(newf);\n            }\n        }\n    }\n    return list_images;\n}\n\nvoid print_probabilities(std::string loc, std::string model_path, std::string model_path_linear) {\n    // Load image with OpenCV.\n    cv::Mat img = cv::imread(loc);\n    cv::resize(img, img, cv::Size(224, 224), cv::INTER_CUBIC);\n    // Convert the image and label to a tensor.\n    torch::Tensor img_tensor = torch::from_blob(img.data, {1, img.rows, img.cols, 3}, torch::kByte);\n    img_tensor = img_tensor.permute({0, 3, 1, 2}); // convert to CxHxW\n    img_tensor = img_tensor.to(torch::kF32);\n    \n    // Load the model.\n    torch::jit::script::Module model;\n    model = torch::jit::load(model_path);\n    \n    torch::nn::Linear model_linear(512, 2);\n    torch::load(model_linear, model_path_linear);\n    \n    // Predict the probabilities for the classes.\n    std::vector<torch::jit::IValue> input;\n    input.push_back(img_tensor);\n    torch::Tensor prob = model.forward(input).toTensor();\n    prob = prob.view({prob.size(0), -1});\n    prob = model_linear(prob);\n    \n    std::cout << \"Printing for location: \" << loc << std::endl;\n    std::cout << \"Cat prob: \" << *(prob.data<float>())*100. << std::endl;\n    std::cout << \"Dog prob: \" << *(prob.data<float>()+1)*100. << std::endl;\n}\n\nint main(int arc, char** argv)\n{\n    // argv[1] should is the test image\n    std::string location = argv[1];\n    \n    // argv[2] contains pre-trained model without last layer\n    // argv[3] contains trained last FC layer\n    std::string model_path = argv[2];\n    std::string model_path_linear = argv[3];\n    \n    // Load the model.\n    // You can also use: auto model = torch::jit::load(model_path);\n    torch::jit::script::Module model = torch::jit::load(model_path);\n    \n    // Print probabilities for dog and cat classes\n    print_probabilities(location, model_path, model_path_linear);\n    return 0;\n}\n"
  },
  {
    "path": "cpp/transfer-learning/convert.py",
    "content": "\"\"\"\nThis python script converts the network into Script Module\n\"\"\"\nimport torch\nfrom torchvision import models\n\n# Download and load the pre-trained model\nmodel = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1)\n\n# Set upgrading the gradients to False\nfor param in model.parameters():\n\tparam.requires_grad = False\n\n# Save the model except the final FC Layer\nresnet18 = torch.nn.Sequential(*list(model.children())[:-1])\n\nexample_input = torch.rand(1, 3, 224, 224)\nscript_module = torch.jit.trace(resnet18, example_input)\nscript_module.save('resnet18_without_last_layer.pt')\n"
  },
  {
    "path": "cpp/transfer-learning/main.cpp",
    "content": "//\n//  main.cpp\n//  transfer-learning\n//\n//  Created by Kushashwa Ravi Shrimali on 12/08/19.\n//\n\n#include \"main.h\"\n\ntorch::Tensor read_data(std::string location) {\n    /*\n     Function to return image read at location given as type torch::Tensor\n     Resizes image to (224, 224, 3)\n     Parameters\n     ===========\n     1. location (std::string type) - required to load image from the location\n     \n     Returns\n     ===========\n     torch::Tensor type - image read as tensor\n    */\n    cv::Mat img = cv::imread(location, 1);\n    cv::resize(img, img, cv::Size(224, 224), cv::INTER_CUBIC);\n    torch::Tensor img_tensor = torch::from_blob(img.data, {img.rows, img.cols, 3}, torch::kByte);\n    img_tensor = img_tensor.permute({2, 0, 1});\n    return img_tensor.clone();\n}\n\ntorch::Tensor read_label(int label) {\n    /*\n     Function to return label from int (0, 1 for binary and 0, 1, ..., n-1 for n-class classification) as type torch::Tensor\n     Parameters\n     ===========\n     1. label (int type) - required to convert int to tensor\n     \n     Returns\n     ===========\n     torch::Tensor type - label read as tensor\n    */\n    torch::Tensor label_tensor = torch::full({1}, label);\n    return label_tensor.clone();\n}\n\nstd::vector<torch::Tensor> process_images(std::vector<std::string> list_images) {\n    /*\n     Function returns vector of tensors (images) read from the list of images in a folder\n     Parameters\n     ===========\n     1. list_images (std::vector<std::string> type) - list of image paths in a folder to be read\n     \n     Returns\n     ===========\n     std::vector<torch::Tensor> type - Images read as tensors\n     */\n    std::vector<torch::Tensor> states;\n    for(std::vector<std::string>::iterator it = list_images.begin(); it != list_images.end(); ++it) {\n        torch::Tensor img = read_data(*it);\n        states.push_back(img);\n    }\n    return states;\n}\n\nstd::vector<torch::Tensor> process_labels(std::vector<int> list_labels) {\n    /*\n     Function returns vector of tensors (labels) read from the list of labels\n     Parameters\n     ===========\n     1. list_labels (std::vector<int> list_labels) -\n     \n     Returns\n     ===========\n     std::vector<torch::Tensor> type - returns vector of tensors (labels)\n     */\n    std::vector<torch::Tensor> labels;\n    for(std::vector<int>::iterator it = list_labels.begin(); it != list_labels.end(); ++it) {\n        torch::Tensor label = read_label(*it);\n        labels.push_back(label);\n    }\n    return labels;\n}\n\nstd::pair<std::vector<std::string>,std::vector<int>> load_data_from_folder(std::vector<std::string> folders_name) {\n    /*\n     Function to load data from given folder(s) name(s) (folders_name)\n     Returns pair of vectors of string (image locations) and int (respective labels)\n     Parameters\n     ===========\n     1. folders_name (std::vector<std::string> type) - name of folders as a vector to load data from\n     \n     Returns\n     ===========\n     std::pair<std::vector<std::string>, std::vector<int>> type - returns pair of vector of strings (image paths) and respective labels' vector (int label)\n     */\n    std::vector<std::string> list_images;\n    std::vector<int> list_labels;\n    int label = 0;\n    for(auto const& value: folders_name) {\n        std::string base_name = value + \"/\";\n        // cout << \"Reading from: \" << base_name << endl;\n        DIR* dir;\n        struct dirent *ent;\n        if((dir = opendir(base_name.c_str())) != NULL) {\n            while((ent = readdir(dir)) != NULL) {\n                std::string filename = ent->d_name;\n                if(filename.length() > 4 && filename.substr(filename.length() - 3) == \"jpg\") {\n                    // cout << base_name + ent->d_name << endl;\n                    // cv::Mat temp = cv::imread(base_name + \"/\" + ent->d_name, 1);\n                    list_images.push_back(base_name + ent->d_name);\n                    list_labels.push_back(label);\n                }\n            }\n            closedir(dir);\n        } else {\n            std::cout << \"Could not open directory\" << std::endl;\n            // return EXIT_FAILURE;\n        }\n        label += 1;\n    }\n    return std::make_pair(list_images, list_labels);\n}\n\ntemplate<typename Dataloader>\nvoid train(torch::jit::script::Module net, torch::nn::Linear lin, Dataloader& data_loader, torch::optim::Optimizer& optimizer, size_t dataset_size) {\n    /*\n     This function trains the network on our data loader using optimizer.\n     \n     Also saves the model as model.pt after every epoch.\n     Parameters\n     ===========\n     1. net (torch::jit::script::Module type) - Pre-trained model without last FC layer\n     2. lin (torch::nn::Linear type) - last FC layer with revised out_features depending on the number of classes\n     3. data_loader (DataLoader& type) - Training data loader\n     4. optimizer (torch::optim::Optimizer& type) - Optimizer like Adam, SGD etc.\n     5. size_t (dataset_size type) - Size of training dataset\n     \n     Returns\n     ===========\n     Nothing (void)\n     */\n    float best_accuracy = 0.0; \n    int batch_index = 0;\n    \n    for(int i=0; i<25; i++) {\n        float mse = 0;\n        float Acc = 0.0;\n        \n        for(auto& batch: *data_loader) {\n            auto data = batch.data;\n            auto target = batch.target.squeeze();\n            \n            // Should be of length: batch_size\n            data = data.to(torch::kF32);\n            target = target.to(torch::kInt64);\n            \n            std::vector<torch::jit::IValue> input;\n            input.push_back(data);\n            optimizer.zero_grad();\n            \n            auto output = net.forward(input).toTensor();\n            // For transfer learning\n            output = output.view({output.size(0), -1});\n            output = lin(output);\n            \n            auto loss = torch::nll_loss(torch::log_softmax(output, 1), target);\n            \n            loss.backward();\n            optimizer.step();\n            \n            auto acc = output.argmax(1).eq(target).sum();\n            \n            Acc += acc.template item<float>();\n            mse += loss.template item<float>();\n            \n            batch_index += 1;\n        }\n\n        mse = mse/float(batch_index); // Take mean of loss\n        std::cout << \"Epoch: \" << i  << \", \" << \"Accuracy: \" << Acc/dataset_size << \", \" << \"MSE: \" << mse << std::endl;\n\n        test(net, lin, data_loader, dataset_size);\n\n        if(Acc/dataset_size > best_accuracy) {\n            best_accuracy = Acc/dataset_size;\n            std::cout << \"Saving model\" << std::endl;\n            net.save(\"model.pt\");\n            torch::save(lin, \"model_linear.pt\");\n        }\n    }\n}\n\ntemplate<typename Dataloader>\nvoid test(torch::jit::script::Module network, torch::nn::Linear lin, Dataloader& loader, size_t data_size) {\n    /*\n     Function to test the network on test data\n     \n     Parameters\n     ===========\n     1. network (torch::jit::script::Module type) - Pre-trained model without last FC layer\n     2. lin (torch::nn::Linear type) - last FC layer with revised out_features depending on the number of classes\n     3. loader (Dataloader& type) - test data loader\n     4. data_size (size_t type) - test data size\n     \n     Returns\n     ===========\n     Nothing (void)\n     */\n    network.eval();\n    \n    float Loss = 0, Acc = 0;\n    \n    for (const auto& batch : *loader) {\n        auto data = batch.data;\n        auto targets = batch.target.squeeze();\n        \n        data = data.to(torch::kF32);\n        targets = targets.to(torch::kInt64);\n\n        std::vector<torch::jit::IValue> input;\n        input.push_back(data);\n\n        auto output = network.forward(input).toTensor();\n        output = output.view({output.size(0), -1});\n        output = lin(output);\n        \n        auto loss = torch::nll_loss(torch::log_softmax(output, 1), targets);\n        auto acc = output.argmax(1).eq(targets).sum();\n        Loss += loss.template item<float>();\n        Acc += acc.template item<float>();\n    }\n    \n    std::cout << \"Test Loss: \" << Loss/data_size << \", Acc:\" << Acc/data_size << std::endl;\n}\n\nint main(int argc, const char * argv[]) {\n    // Set folder names for cat and dog images\n    std::string cats_name = \"/Users/krshrimali/Documents/krshrimali-blogs/dataset/train/cat_test\";\n    std::string dogs_name = \"/Users/krshrimali/Documents/krshrimali-blogs/dataset/train/dog_test\";\n    \n    std::vector<std::string> folders_name;\n    folders_name.push_back(cats_name);\n    folders_name.push_back(dogs_name);\n    \n    // Get paths of images and labels as int from the folder paths\n    std::pair<std::vector<std::string>, std::vector<int>> pair_images_labels = load_data_from_folder(folders_name);\n    \n    std::vector<std::string> list_images = pair_images_labels.first;\n    std::vector<int> list_labels = pair_images_labels.second;\n    \n    // Initialize CustomDataset class and read data\n    auto custom_dataset = CustomDataset(list_images, list_labels).map(torch::data::transforms::Stack<>());\n\n    // Load pre-trained model\n    // You can also use: auto module = torch::jit::load(argv[1]);\n    torch::jit::script::Module module = torch::jit::load(argv[1]);\n    \n    // Resource: https://discuss.pytorch.org/t/how-to-load-the-prebuilt-resnet-models-or-any-other-prebuilt-models/40269/8\n    // For VGG: 512 * 14 * 14, 2\n\n    torch::nn::Linear lin(512, 2); // the last layer of resnet, which we want to replace, has dimensions 512x1000\n    torch::optim::Adam opt(lin->parameters(), torch::optim::AdamOptions(1e-3 /*learning rate*/));\n\n    auto data_loader = torch::data::make_data_loader<torch::data::samplers::RandomSampler>(std::move(custom_dataset), 4);\n\n    train(module, lin, data_loader, opt, custom_dataset.size().value());\n    return 0;\n}\n"
  },
  {
    "path": "cpp/transfer-learning/main.h",
    "content": "//\n//  main.h\n//  transfer-learning\n//\n//  Created by Kushashwa Ravi Shrimali on 15/08/19.\n//\n\n#ifndef main_h\n#define main_h\n\n#include <iostream>\n#include <opencv2/opencv.hpp>\n#include <torch/torch.h>\n#include <dirent.h>\n#include <torch/script.h>\n\n// Function to return image read at location given as type torch::Tensor\n// Resizes image to (224, 224, 3)\ntorch::Tensor read_data(std::string location);\n\n// Function to return label from int (0, 1 for binary and 0, 1, ..., n-1 for n-class classification) as type torch::Tensor\ntorch::Tensor read_label(int label);\n\n// Function returns vector of tensors (images) read from the list of images in a folder\nstd::vector<torch::Tensor> process_images(std::vector<std::string> list_images);\n\n// Function returns vector of tensors (labels) read from the list of labels\nstd::vector<torch::Tensor> process_labels(std::vector<int> list_labels);\n\n// Function to load data from given folder(s) name(s) (folders_name)\n// Returns pair of vectors of string (image locations) and int (respective labels)\nstd::pair<std::vector<std::string>, std::vector<int>> load_data_from_folder(std::vector<std::string> folders_name);\n\n// Function to train the network on train data\ntemplate<typename Dataloader>\nvoid train(torch::jit::script::Module net, torch::nn::Linear lin, Dataloader& data_loader, torch::optim::Optimizer& optimizer, size_t dataset_size);\n\n// Function to test the network on test data\ntemplate<typename Dataloader>\nvoid test(torch::jit::script::Module network, torch::nn::Linear lin, Dataloader& loader, size_t data_size);\n\n// Custom Dataset class\nclass CustomDataset : public torch::data::Dataset<CustomDataset> {\nprivate:\n    /* data */\n    // Should be 2 tensors\n    std::vector<torch::Tensor> states, labels;\n    size_t ds_size;\npublic:\n    CustomDataset(std::vector<std::string> list_images, std::vector<int> list_labels) {\n        states = process_images(list_images);\n        labels = process_labels(list_labels);\n        ds_size = states.size();\n    };\n    \n    torch::data::Example<> get(size_t index) override {\n        /* This should return {torch::Tensor, torch::Tensor} */\n        torch::Tensor sample_img = states.at(index);\n        torch::Tensor sample_label = labels.at(index);\n        return {sample_img.clone(), sample_label.clone()};\n    };\n    \n    torch::optional<size_t> size() const override {\n        return ds_size;\n    };\n};\n\n#endif /* main_h */\n"
  },
  {
    "path": "dcgan/README.md",
    "content": "# Deep Convolution Generative Adversarial Networks\n\nThis example implements the paper [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](http://arxiv.org/abs/1511.06434)\n\nThe implementation is very close to the Torch implementation [dcgan.torch](https://github.com/soumith/dcgan.torch)\n\nAfter every 100 training iterations, the files `real_samples.png` and `fake_samples.png` are written to disk\nwith the samples from the generative model.\n\nAfter every epoch, models are saved to: `netG_epoch_%d.pth` and `netD_epoch_%d.pth`\n\n## Downloading the dataset\n\nYou can download the LSUN dataset by cloning [this repo](https://github.com/fyu/lsun) and running\n\n```\npython download.py -c bedroom\n```\n\n## Usage\n\n```\nusage: main.py [-h] --dataset DATASET [--dataroot DATAROOT]\n               [--workers WORKERS] [--batchSize BATCHSIZE]\n               [--imageSize IMAGESIZE] [--nz NZ] [--ngf NGF] [--ndf NDF]\n               [--niter NITER] [--lr LR] [--beta1 BETA1] [--dry-run]\n               [--ngpu NGPU] [--netG NETG] [--netD NETD] [--outf OUTF]\n               [--manualSeed MANUALSEED] [--classes CLASSES] [--accel]\n\noptions:\n  -h, --help            show this help message and exit\n  --dataset DATASET     cifar10 | lsun | mnist |imagenet | folder | lfw | fake\n  --dataroot DATAROOT   path to dataset\n  --workers WORKERS     number of data loading workers\n  --batchSize BATCHSIZE\n                        input batch size\n  --imageSize IMAGESIZE\n                        the height / width of the input image to network\n  --nz NZ               size of the latent z vector\n  --ngf NGF             number of generator filters\n  --ndf NDF             number of discriminator filters\n  --niter NITER         number of epochs to train for\n  --lr LR               learning rate, default=0.0002\n  --beta1 BETA1         beta1 for adam. default=0.5\n  --dry-run             check a single training cycle works\n  --ngpu NGPU           number of GPUs to use\n  --netG NETG           path to netG (to continue training)\n  --netD NETD           path to netD (to continue training)\n  --outf OUTF           folder to output images and model checkpoints\n  --manualSeed MANUALSEED\n                        manual seed\n  --classes CLASSES     comma separated list of classes for the lsun data set\n  --accel               enables accelerator\n```\n"
  },
  {
    "path": "dcgan/main.py",
    "content": "from __future__ import print_function\nimport argparse\nimport os\nimport random\nimport torch\nimport torch.nn as nn\nimport torch.nn.parallel\nimport torch.backends.cudnn as cudnn\nimport torch.optim as optim\nimport torch.utils.data\nimport torchvision.datasets as dset\nimport torchvision.transforms as transforms\nimport torchvision.utils as vutils\n\n\nparser = argparse.ArgumentParser()\nparser.add_argument('--dataset', required=True, help='cifar10 | lsun | mnist |imagenet | folder | lfw | fake')\nparser.add_argument('--dataroot', required=False, help='path to dataset')\nparser.add_argument('--workers', type=int, help='number of data loading workers', default=2)\nparser.add_argument('--batchSize', type=int, default=64, help='input batch size')\nparser.add_argument('--imageSize', type=int, default=64, help='the height / width of the input image to network')\nparser.add_argument('--nz', type=int, default=100, help='size of the latent z vector')\nparser.add_argument('--ngf', type=int, default=64, help='number of generator filters')\nparser.add_argument('--ndf', type=int, default=64, help='number of discriminator filters')\nparser.add_argument('--niter', type=int, default=25, help='number of epochs to train for')\nparser.add_argument('--lr', type=float, default=0.0002, help='learning rate, default=0.0002')\nparser.add_argument('--beta1', type=float, default=0.5, help='beta1 for adam. default=0.5')\nparser.add_argument('--dry-run', action='store_true', help='check a single training cycle works')\nparser.add_argument('--ngpu', type=int, default=1, help='number of GPUs to use')\nparser.add_argument('--netG', default='', help=\"path to netG (to continue training)\")\nparser.add_argument('--netD', default='', help=\"path to netD (to continue training)\")\nparser.add_argument('--outf', default='.', help='folder to output images and model checkpoints')\nparser.add_argument('--manualSeed', type=int, help='manual seed')\nparser.add_argument('--classes', default='bedroom', help='comma separated list of classes for the lsun data set')\nparser.add_argument('--accel', action='store_true', default=False, help='enables accelerator')\n\nopt = parser.parse_args()\nprint(opt)\n\ntry:\n    os.makedirs(opt.outf)\nexcept OSError:\n    pass\n\nif opt.manualSeed is None:\n    opt.manualSeed = random.randint(1, 10000)\nprint(\"Random Seed: \", opt.manualSeed)\nrandom.seed(opt.manualSeed)\ntorch.manual_seed(opt.manualSeed)\n\ncudnn.benchmark = True\n\nif opt.accel and torch.accelerator.is_available():\n    device = torch.accelerator.current_accelerator()\nelse:\n    device = torch.device(\"cpu\")\n\nprint(f\"Using device: {device}\")\n\nif opt.dataroot is None and str(opt.dataset).lower() != 'fake':\n    raise ValueError(\"`dataroot` parameter is required for dataset \\\"%s\\\"\" % opt.dataset)\n\nif opt.dataset in ['imagenet', 'folder', 'lfw']:\n    # folder dataset\n    dataset = dset.ImageFolder(root=opt.dataroot,\n                               transform=transforms.Compose([\n                                   transforms.Resize(opt.imageSize),\n                                   transforms.CenterCrop(opt.imageSize),\n                                   transforms.ToTensor(),\n                                   transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n                               ]))\n    nc=3\nelif opt.dataset == 'lsun':\n    classes = [ c + '_train' for c in opt.classes.split(',')]\n    dataset = dset.LSUN(root=opt.dataroot, classes=classes,\n                        transform=transforms.Compose([\n                            transforms.Resize(opt.imageSize),\n                            transforms.CenterCrop(opt.imageSize),\n                            transforms.ToTensor(),\n                            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n                        ]))\n    nc=3\nelif opt.dataset == 'cifar10':\n    dataset = dset.CIFAR10(root=opt.dataroot, download=True,\n                           transform=transforms.Compose([\n                               transforms.Resize(opt.imageSize),\n                               transforms.ToTensor(),\n                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n                           ]))\n    nc=3\n\nelif opt.dataset == 'mnist':\n        dataset = dset.MNIST(root=opt.dataroot, download=True,\n                           transform=transforms.Compose([\n                               transforms.Resize(opt.imageSize),\n                               transforms.ToTensor(),\n                               transforms.Normalize((0.5,), (0.5,)),\n                           ]))\n        nc=1\n\nelif opt.dataset == 'fake':\n    dataset = dset.FakeData(image_size=(3, opt.imageSize, opt.imageSize),\n                            transform=transforms.ToTensor())\n    nc=3\n\nassert dataset\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=opt.batchSize,\n                                         shuffle=True, num_workers=int(opt.workers))\n\nngpu = int(opt.ngpu)\nnz = int(opt.nz)\nngf = int(opt.ngf)\nndf = int(opt.ndf)\n\n\n# custom weights initialization called on netG and netD\ndef weights_init(m):\n    classname = m.__class__.__name__\n    if classname.find('Conv') != -1:\n        torch.nn.init.normal_(m.weight, 0.0, 0.02)\n    elif classname.find('BatchNorm') != -1:\n        torch.nn.init.normal_(m.weight, 1.0, 0.02)\n        torch.nn.init.zeros_(m.bias)\n\n\nclass Generator(nn.Module):\n    def __init__(self, ngpu):\n        super(Generator, self).__init__()\n        self.ngpu = ngpu\n        self.main = nn.Sequential(\n            # input is Z, going into a convolution\n            nn.ConvTranspose2d(     nz, ngf * 8, 4, 1, 0, bias=False),\n            nn.BatchNorm2d(ngf * 8),\n            nn.ReLU(True),\n            # state size. (ngf*8) x 4 x 4\n            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),\n            nn.BatchNorm2d(ngf * 4),\n            nn.ReLU(True),\n            # state size. (ngf*4) x 8 x 8\n            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),\n            nn.BatchNorm2d(ngf * 2),\n            nn.ReLU(True),\n            # state size. (ngf*2) x 16 x 16\n            nn.ConvTranspose2d(ngf * 2,     ngf, 4, 2, 1, bias=False),\n            nn.BatchNorm2d(ngf),\n            nn.ReLU(True),\n            # state size. (ngf) x 32 x 32\n            nn.ConvTranspose2d(    ngf,      nc, 4, 2, 1, bias=False),\n            nn.Tanh()\n            # state size. (nc) x 64 x 64\n        )\n\n    def forward(self, input):\n        \n        if (input.is_cuda or input.is_xpu) and self.ngpu > 1:\n            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))\n        else:\n            output = self.main(input)\n        return output\n\n\nnetG = Generator(ngpu).to(device)\nnetG.apply(weights_init)\nif opt.netG != '':\n    netG.load_state_dict(torch.load(opt.netG))\nprint(netG)\n\n\nclass Discriminator(nn.Module):\n    def __init__(self, ngpu):\n        super(Discriminator, self).__init__()\n        self.ngpu = ngpu\n        self.main = nn.Sequential(\n            # input is (nc) x 64 x 64\n            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),\n            nn.LeakyReLU(0.2, inplace=True),\n            # state size. (ndf) x 32 x 32\n            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),\n            nn.BatchNorm2d(ndf * 2),\n            nn.LeakyReLU(0.2, inplace=True),\n            # state size. (ndf*2) x 16 x 16\n            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),\n            nn.BatchNorm2d(ndf * 4),\n            nn.LeakyReLU(0.2, inplace=True),\n            # state size. (ndf*4) x 8 x 8\n            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),\n            nn.BatchNorm2d(ndf * 8),\n            nn.LeakyReLU(0.2, inplace=True),\n            # state size. (ndf*8) x 4 x 4\n            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),\n            nn.Sigmoid()\n        )\n\n    def forward(self, input):\n        if (input.is_cuda or input.is_xpu) and self.ngpu > 1:\n            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))\n        else:\n            output = self.main(input)\n\n        return output.view(-1, 1).squeeze(1)\n\n\nnetD = Discriminator(ngpu).to(device)\nnetD.apply(weights_init)\nif opt.netD != '':\n    netD.load_state_dict(torch.load(opt.netD))\nprint(netD)\n\ncriterion = nn.BCELoss()\n\nfixed_noise = torch.randn(opt.batchSize, nz, 1, 1, device=device)\nreal_label = 1\nfake_label = 0\n\n# setup optimizer\noptimizerD = optim.Adam(netD.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999))\noptimizerG = optim.Adam(netG.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999))\n\nif opt.dry_run:\n    opt.niter = 1\n\nfor epoch in range(opt.niter):\n    for i, data in enumerate(dataloader, 0):\n        ############################\n        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))\n        ###########################\n        # train with real\n        netD.zero_grad()\n        real_cpu = data[0].to(device)\n        batch_size = real_cpu.size(0)\n        label = torch.full((batch_size,), real_label,\n                           dtype=real_cpu.dtype, device=device)\n\n        output = netD(real_cpu)\n        errD_real = criterion(output, label)\n        errD_real.backward()\n        D_x = output.mean().item()\n\n        # train with fake\n        noise = torch.randn(batch_size, nz, 1, 1, device=device)\n        fake = netG(noise)\n        label.fill_(fake_label)\n        output = netD(fake.detach())\n        errD_fake = criterion(output, label)\n        errD_fake.backward()\n        D_G_z1 = output.mean().item()\n        errD = errD_real + errD_fake\n        optimizerD.step()\n\n        ############################\n        # (2) Update G network: maximize log(D(G(z)))\n        ###########################\n        netG.zero_grad()\n        label.fill_(real_label)  # fake labels are real for generator cost\n        output = netD(fake)\n        errG = criterion(output, label)\n        errG.backward()\n        D_G_z2 = output.mean().item()\n        optimizerG.step()\n\n        print('[%d/%d][%d/%d] Loss_D: %.4f Loss_G: %.4f D(x): %.4f D(G(z)): %.4f / %.4f'\n              % (epoch, opt.niter, i, len(dataloader),\n                 errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))\n        if i % 100 == 0:\n            vutils.save_image(real_cpu,\n                    '%s/real_samples.png' % opt.outf,\n                    normalize=True)\n            fake = netG(fixed_noise)\n            vutils.save_image(fake.detach(),\n                    '%s/fake_samples_epoch_%03d.png' % (opt.outf, epoch),\n                    normalize=True)\n\n        if opt.dry_run:\n            break\n    # do checkpointing\n    torch.save(netG.state_dict(), '%s/netG_epoch_%d.pth' % (opt.outf, epoch))\n    torch.save(netD.state_dict(), '%s/netD_epoch_%d.pth' % (opt.outf, epoch))\n"
  },
  {
    "path": "dcgan/requirements.txt",
    "content": "torch\ntorchvision\nlmdb\n"
  },
  {
    "path": "distributed/FSDP/.gitignore",
    "content": "__pycache__/\n*.pt\n*.csv"
  },
  {
    "path": "distributed/FSDP/README.md",
    "content": "Note: FSDP1 is deprecated. Please follow [FSDP2 tutorial](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html) and [code examples](https://github.com/pytorch/examples/tree/main/distributed/FSDP2).\n\n## FSDP1 T5\n\n\n\nTo run the T5 example with FSDP1 for text summarization:\n\n## Get the wikihow dataset\n```bash\n\nsh download_dataset.sh\n\n```\n\n## Install the requirements:\n~~~\npip install -r requirements.txt\n~~~\n## Ensure you are running a recent version of PyTorch:\nsee https://pytorch.org to install at least 1.12 and ideally a current nightly build. \n\nStart the training with Torchrun (adjust nproc_per_node to your GPU count):\n\n```\ntorchrun --nnodes 1 --nproc_per_node 4  T5_training.py\n\n```\n"
  },
  {
    "path": "distributed/FSDP/T5_training.py",
    "content": "import os\nimport argparse\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom transformers import AutoTokenizer, GPT2TokenizerFast\nfrom transformers import T5Tokenizer, T5ForConditionalGeneration\nimport functools\nfrom torch.optim.lr_scheduler import StepLR\nimport torch.nn.functional as F\nimport torch.distributed as dist\nimport torch.multiprocessing as mp\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom torch.utils.data.distributed import DistributedSampler\nfrom transformers.models.t5.modeling_t5 import T5Block\nfrom nlp import load_dataset\n\nfrom torch.distributed.fsdp import (\n    FullyShardedDataParallel as FSDP,\n    CPUOffload,\n    MixedPrecision,\n    BackwardPrefetch,\n    ShardingStrategy,\n    FullStateDictConfig,\n    StateDictType,\n)\n\nfrom functools import partial\nfrom torch.utils.data import DataLoader\nfrom pathlib import Path\nfrom summarization_dataset import *\nimport policies\nimport model_checkpointing\nfrom configs import fsdp_config, train_config\nfrom utils import (bfloat_support, setup,\n                   cleanup, get_date_of_run,\n                   format_metrics_to_gb,\n                   train,validation,setup_model)\nfrom transformers.models.t5.modeling_t5 import T5Block\nfrom typing import Type\nimport time\nimport tqdm\nfrom datetime import datetime\n\n\ndef get_policies(cfg, rank):\n\n    \"\"\"establish current policies for mixed precision and fsdp wrapping\"\"\"\n\n    mixed_precision_policy = None\n    wrapping_policy = None\n\n    # mixed precision -----\n    if cfg.mixed_precision:\n        bfloat_available = bfloat_support()\n        if bfloat_available and not cfg.use_fp16:\n            mixed_precision_policy = policies.bfSixteen\n            if rank == 0:\n                print(f\"bFloat16 enabled for mixed precision - using bfSixteen policy\")\n        elif cfg.use_fp16:\n            mixed_precision_policy = policies.fpSixteen\n            if rank == 0:\n                print(f\"FP16 enabled. \")\n        else:\n            # mixed_precision_policy = policies.fpSixteen\n            print(\n                f\"bFloat16 support not present. Will use FP32, and not mixed precision\"\n            )\n\n    wrapping_policy = policies.get_t5_wrapper()\n\n    return mixed_precision_policy, wrapping_policy\n\n\ndef fsdp_main(args):\n\n    model, tokenizer = setup_model(train_config.model_name)\n\n    local_rank = int(os.environ['LOCAL_RANK'])\n    rank = int(os.environ['RANK'])\n    world_size = int(os.environ['WORLD_SIZE'])\n\n\n    dataset = load_dataset('wikihow', 'all', data_dir='data/')\n    print(dataset.keys())\n    print(\"Size of train dataset: \", dataset['train'].shape)\n    print(\"Size of Validation dataset: \", dataset['validation'].shape)\n\n\n    #wikihow(tokenizer, type_path, num_samples, input_length, output_length, print_text=False)\n    train_dataset = wikihow(tokenizer, 'train', 1500, 512, 150, False)\n    val_dataset = wikihow(tokenizer, 'validation', 300, 512, 150, False)\n\n    sampler1 = DistributedSampler(train_dataset, rank=rank, num_replicas=world_size, shuffle=True)\n    sampler2 = DistributedSampler(val_dataset, rank=rank, num_replicas=world_size)\n\n    setup()\n\n\n    train_kwargs = {'batch_size': args.batch_size, 'sampler': sampler1}\n    test_kwargs = {'batch_size': args.test_batch_size, 'sampler': sampler2}\n    cuda_kwargs = {'num_workers': 2,\n                    'pin_memory': True,\n                    'shuffle': False}\n    train_kwargs.update(cuda_kwargs)\n    test_kwargs.update(cuda_kwargs)\n\n    train_loader = torch.utils.data.DataLoader(train_dataset,**train_kwargs)\n    val_loader = torch.utils.data.DataLoader(val_dataset, **test_kwargs)\n\n    torch.cuda.set_device(local_rank)\n\n    # Set up FSDP parameters\n    mixed_precision_policy, t5_auto_wrap_policy = get_policies(train_config, rank)\n\n    # Apply FSDP wrapping to the model\n    model = FSDP(model,\n        auto_wrap_policy=t5_auto_wrap_policy,\n        mixed_precision=mixed_precision_policy,\n        sharding_strategy=fsdp_config.sharding_strategy,\n        device_id=torch.cuda.current_device(),\n        limit_all_gathers=fsdp_config.limit_all_gathers)\n\n    # Enabling this causes https://github.com/pytorch/examples/issues/1210\n    if fsdp_config.fsdp_activation_checkpointing:\n        policies.apply_fsdp_checkpointing(model)\n\n    # Set up optimizer and scheduler\n    optimizer = optim.AdamW(model.parameters(), lr=train_config.lr)\n\n    scheduler = StepLR(optimizer, step_size=1, gamma=train_config.gamma)\n    best_val_loss = float(\"inf\")\n    curr_val_loss = float(\"inf\")\n    file_save_name = \"T5-model-\"\n\n    if rank == 0:\n        time_of_run = get_date_of_run()\n        dur = []\n        train_acc_tracking = []\n        val_acc_tracking = []\n        training_start_time = time.time()\n\n    if rank == 0 and args.track_memory:\n        mem_alloc_tracker = []\n        mem_reserved_tracker = []\n\n    for epoch in range(1, args.epochs + 1):\n        t0 = time.time()\n        train_accuracy = train(args, model, rank, world_size, train_loader, optimizer, epoch, sampler=sampler1)\n        if args.run_validation:\n            curr_val_loss = validation(model, rank, world_size, val_loader)\n        scheduler.step()\n\n        if rank == 0:\n\n            print(f\"--> epoch {epoch} completed...entering save and stats zone\")\n\n            dur.append(time.time() - t0)\n            train_acc_tracking.append(train_accuracy.item())\n\n            if args.run_validation:\n                val_acc_tracking.append(curr_val_loss.item())\n\n            if args.track_memory:\n                mem_alloc_tracker.append(\n                    format_metrics_to_gb(torch.cuda.memory_allocated())\n                )\n                mem_reserved_tracker.append(\n                    format_metrics_to_gb(torch.cuda.memory_reserved())\n                )\n\n        if train_config.save_model and curr_val_loss < best_val_loss:\n\n            if fsdp_config.checkpoint_type == StateDictType.FULL_STATE_DICT:\n                model_checkpointing.save_model_checkpoint(\n                    model, optimizer, rank, fsdp_config, epoch=1\n                )\n            elif fsdp_config.checkpoint_type == StateDictType.SHARDED_STATE_DICT:\n                model_checkpointing.save_model_and_optimizer_sharded(model, rank, fsdp_config)\n                if fsdp_config.save_optimizer:\n                    model_checkpointing.save_model_and_optimizer_sharded(model, rank, fsdp_config, optim=optimizer)\n\n            if fsdp_config.save_optimizer:\n                model_checkpointing.save_optimizer_checkpoint(\n                    model, optimizer, rank, fsdp_config, epoch=1\n                )\n        if curr_val_loss < best_val_loss:\n\n            best_val_loss = curr_val_loss\n            if rank==0:\n                print(f\"-->>>> New Val Loss Record: {best_val_loss}\")\n\n    dist.barrier()\n    cleanup()\n\n\nif __name__ == '__main__':\n    # Training settings\n    parser = argparse.ArgumentParser(description='PyTorch T5 FSDP Example')\n    parser.add_argument('--batch-size', type=int, default=4, metavar='N',\n                        help='input batch size for training (default: 4)')\n    parser.add_argument('--test-batch-size', type=int, default=4, metavar='N',\n                        help='input batch size for testing (default: 4)')\n    parser.add_argument('--epochs', type=int, default=2, metavar='N',\n                        help='number of epochs to train (default: 2)')\n    parser.add_argument('--seed', type=int, default=1, metavar='S',\n                        help='random seed (default: 1)')\n    parser.add_argument('--track_memory', action='store_false', default=True,\n                        help='track the gpu memory')\n    parser.add_argument('--run_validation', action='store_false', default=True,\n                        help='running the validation')\n    args = parser.parse_args()\n\n    torch.manual_seed(args.seed)\n\n    fsdp_main(args)\n"
  },
  {
    "path": "distributed/FSDP/configs/__init__.py",
    "content": "from .fsdp import fsdp_config\nfrom .training import train_config\n"
  },
  {
    "path": "distributed/FSDP/configs/fsdp.py",
    "content": "from dataclasses import dataclass, field\nfrom typing import ClassVar\nfrom torch.distributed.fsdp import ShardingStrategy\nfrom torch.distributed.fsdp.fully_sharded_data_parallel import StateDictType\n\n@dataclass\nclass fsdp_config:\n    mixed_precision: bool=True\n    use_fp16: bool=False\n    seed: int=42\n    fsdp_activation_checkpointing: bool=False\n    limit_all_gathers: bool=True\n    sharding_strategy: ShardingStrategy = ShardingStrategy.FULL_SHARD #HYBRID_SHARD, SHARD_GRAD_OP\n    checkpoint_type: StateDictType = StateDictType.FULL_STATE_DICT # alternatively can use SHARDED_STATE_DICT to avoid OOMs\n    save_optimizer: bool=False\n    \n    \n    \n    "
  },
  {
    "path": "distributed/FSDP/configs/training.py",
    "content": "from dataclasses import dataclass\nfrom typing import ClassVar\n\n\n@dataclass\nclass train_config:\n    model_name: str=\"t5-base\"\n    run_validation: bool=True\n    batch_size_training: int=4\n    num_workers_dataloader: int=2\n    lr: float=0.002\n    weight_decay: float=0.0\n    gamma: float= 0.85\n    use_fp16: bool=False\n    mixed_precision: bool=True\n    save_model: bool=False\n    \n    \n    "
  },
  {
    "path": "distributed/FSDP/download_dataset.sh",
    "content": "#!/bin/bash\n\n# Create the \"data\" folder if it doesn't exist\nmkdir -p data\n\n# Download the files into the \"data\" folder\nwget -P data https://public-nlp-datasets.s3.us-west-2.amazonaws.com/wikihowAll.csv\nwget -P data https://public-nlp-datasets.s3.us-west-2.amazonaws.com/wikihowSep.csv\n"
  },
  {
    "path": "distributed/FSDP/model_checkpointing/__init__.py",
    "content": "from .checkpoint_handler import (\n    load_model_checkpoint,\n    save_model_checkpoint,\n    save_distributed_model_checkpoint,\n    load_distributed_model_checkpoint,\n    load_optimizer_checkpoint,\n    save_optimizer_checkpoint,\n    save_model_and_optimizer_sharded,\n    load_model_sharded,\n)\n"
  },
  {
    "path": "distributed/FSDP/model_checkpointing/checkpoint_handler.py",
    "content": "from pathlib import Path\nfrom datetime import datetime\nimport torch\nimport time\n\nfrom torch.distributed.fsdp import (\n    FullyShardedDataParallel as FSDP,\n    StateDictType,\n    FullStateDictConfig,  # general model non-sharded, non-flattened params\n    LocalStateDictConfig,  # flattened params, usable only by FSDP\n    # ShardedStateDictConfig, # un-flattened param but shards, usable by other parallel schemes.\n)\n\nfrom torch.distributed.checkpoint import (\n    FileSystemReader,\n    FileSystemWriter,\n    save_state_dict,\n    load_state_dict,\n)\nfrom torch.distributed.checkpoint.default_planner import (\n    DefaultSavePlanner,\n    DefaultLoadPlanner,\n)\n\n\nfrom torch.distributed.fsdp.fully_sharded_data_parallel import StateDictType\nimport torch.distributed.checkpoint as dist_cp\nimport torch.distributed as dist\n\n\ndef get_date_of_run():\n    \"\"\"create date and time for file save uniqueness\n    example: 2022-05-07-08:31:12_PM'\n    \"\"\"\n    date_of_run = datetime.now().strftime(\"%Y-%m-%d-%I:%M:%S_%p\")\n    print(f\"--> current date and time of run = {date_of_run}\")\n    return date_of_run\n\n\n# create singleton saving policies to avoid making over and over\nfullstate_save_policy = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)\n\n\ndef load_model_sharded(model, rank, cfg, verbose=True):\n    # torch.manual_seed(103)\n    folder_name = (\n        cfg.dist_checkpoint_root_folder\n        + \"/\"\n        + cfg.dist_checkpoint_folder\n        + \"-\"\n        + cfg.model_name\n    )\n\n    load_dir = Path.cwd() / folder_name\n\n    if not load_dir.exists():\n        if rank == 0:\n            print(f\"No sharded_state_dict checkpoint directory found...skipping\")\n        return\n\n    reader = FileSystemReader(load_dir)\n\n    with FSDP.state_dict_type(model, StateDictType.SHARDED_STATE_DICT):\n        checkpoint = model.state_dict()\n        if rank == 0:\n            ck = checkpoint.keys()\n            print(f\" checkpoint key len = {len(ck)} and \\n keys =  {ck}\")\n\n        dist_cp.load_state_dict(\n            state_dict=checkpoint,\n            storage_reader=reader,\n        )\n        if rank == 0:\n            print(f\"checkpoint after load_state_dict()\")\n            ck = checkpoint.keys()\n            print(f\" checkpoint key len = {len(ck)} and \\n keys =  {ck}\")\n        model.load_state_dict(checkpoint)\n    if rank == 0:\n        print(f\"Sharded state checkpoint loaded from {load_dir}\")\n\n\ndef save_model_and_optimizer_sharded(model, rank, cfg,optim=None, verbose=True):\n    \"\"\"save model and optimizer via sharded_state_dict to save_dir\"\"\"\n    folder_name = (\n        cfg.dist_checkpoint_root_folder\n        + \"/\"\n        + cfg.dist_checkpoint_folder\n        + \"-\"\n        + cfg.model_name\n    )\n\n    save_dir = Path.cwd() / folder_name\n    if rank == 0:\n        print(f\"Saving model to {save_dir}\")\n\n    distributed_writer = dist_cp.FileSystemWriter(\n        save_dir,\n    )\n    t0 = time.perf_counter()\n\n    with FSDP.state_dict_type(model, StateDictType.SHARDED_STATE_DICT):\n\n        state_dict = {\"model\": model.state_dict()}\n        if optim is not None:\n            state_dict[\"optim\"] = FSDP.optim_state_dict(model, optim)\n\n        dist_cp.save_state_dict(\n            state_dict=state_dict,\n            storage_writer=distributed_writer,\n            planner=DefaultSavePlanner(),\n\n        )\n    dist.barrier()\n    t1 = time.perf_counter()\n    if rank == 0:\n        print(f\"Sharded state checkpoint saved to {save_dir}\")\n        print(\n            f\"Checkpoint Time = {t1-t0:.4f}\\n using {cfg.save_using_num_threads=} total threads\"\n        )\n\ndef save_model_checkpoint(\n    model,\n    optimizer,\n    rank,\n    cfg,\n    epoch=1,\n):\n    \"\"\"saving model via rank0 cpu streaming and full_state_dict\"\"\"\n\n    # saving with rank0 cpu\n    if not cfg.checkpoint_type == StateDictType.FULL_STATE_DICT:\n        print(f\" unable to handle checkpoint type {cfg.checkpoint_type}, aborting\")\n\n    with FSDP.state_dict_type(\n        model, StateDictType.FULL_STATE_DICT, fullstate_save_policy\n    ):\n        cpu_state = model.state_dict()\n\n    if cfg.verbose:\n        print(f\"saving process: rank {rank}  done w model state_dict\\n\")\n\n\n    if rank == 0:\n        print(f\"--> saving model ...\")\n        # create save path\n        save_dir = Path.cwd() / cfg.checkpoint_folder\n        save_dir.mkdir(parents=True, exist_ok=True)\n        save_name = cfg.model_save_name + \"-\" + str(epoch) + \".pt\"\n        save_full_path = str(save_dir) + \"/\" + save_name\n\n        # save model\n        torch.save(cpu_state, save_full_path)\n\n        if cfg.verbose:\n            print(f\"model checkpoint saved for epoch {epoch} at {save_full_path}\\n\")\n\n\n\ndef load_model_checkpoint(model, rank, cfg, verbose=True):\n    \"\"\"load local checkpoint to rank0 cpu\n    must be called * before * passing to FSDP\"\"\"\n\n    if rank != 0:\n        return\n\n    # where is the checkpoint at...\n    full_state_dict_model_path = (\n        Path.cwd() / cfg.checkpoint_folder / cfg.checkpoint_model_filename\n    )\n    # is it present...\n    if not full_state_dict_model_path.is_file():\n        print(\n            f\"model checkpoint {full_state_dict_model_path} not present. Returning...\"\n        )\n        return\n\n\n    model_checkpoint = torch.load(full_state_dict_model_path)\n    # integrate into loaded model\n    model.load_state_dict(model_checkpoint)\n\n    if cfg.verbose:\n        print(f\"model checkpoint loaded to rank0 cpu\")\n\n\ndef save_optimizer_checkpoint(model, optimizer, rank, cfg, epoch=1):\n    \"\"\"save optimizer state via full state dict\"\"\"\n\n    if cfg.verbose:\n        print(f\"--> optim state call on rank {rank}\\n\")\n\n    # pull all sharded optimizer states to rank0 cpu...\n\n    optim_state = FSDP.full_optim_state_dict(model, optimizer)\n\n    if cfg.verbose:\n        print(f\"optim state dict ready on {rank} and len of {len(optim_state)}\\n\")\n\n    if rank == 0:\n        save_dir = Path.cwd() / cfg.checkpoint_folder\n        save_dir.mkdir(parents=True, exist_ok=True)\n\n        opt_save_name = (\n            cfg.optimizer_name + \"-\" + cfg.model_save_name + \"-\" + str(epoch) + \".pt\"\n        )\n        opt_save_full_path = save_dir / opt_save_name\n\n        print(f\"--> saving optimizer state...\")\n\n        torch.save(optim_state, opt_save_full_path)\n\n        print(f\"--> saved {opt_save_full_path} to disk\")\n\n\ndef load_optimizer_checkpoint(model, optimizer, rank, cfg):\n    \"\"\"load an fdsp optimizer full_state checkpoint using scatter method\n    this ensures only rank 0 loads the optimizer state dict and scatters to other ranks\n    \"\"\"\n\n    opt_file_path = Path.cwd() / cfg.checkpoint_folder / cfg.optimizer_checkpoint_file\n\n    if not opt_file_path.is_file():\n        print(\n            f\"warning - optimizer checkpoint not present {opt_file_path}. Returning. \"\n        )\n        return\n\n    full_osd = None\n\n    if rank == 0:\n        full_osd = torch.load(opt_file_path)\n\n        if cfg.verbose:\n            print(f\"loaded full osd on rank 0\")\n\n    # called from all ranks, though only rank0 has a valid param for full_osd\n    sharded_osd = FSDP.scatter_full_optim_state_dict(full_osd, model)\n\n    if cfg.verbose:\n        print(f\"optimizer shard loaded on rank {rank}\")\n\n\n\ndef load_distributed_model_checkpoint(model, rank, cfg):\n    if cfg.checkpoint_type == StateDictType.LOCAL_STATE_DICT:\n        print(f\"loading distributed checkpoint, rank {rank}...\")\n        folder_name = (\n            cfg.dist_checkpoint_root_folder\n            + \"/\"\n            + cfg.dist_checkpoint_folder\n            + \"-\"\n            + cfg.model_name\n        )\n\n        checkdir = Path.cwd() / folder_name\n\n        if not checkdir.exists():\n            if rank == 0:\n                print(f\"No checkpoint directory found...skipping\")\n            return\n\n\n        reader = FileSystemReader(checkdir)\n\n        with FSDP.state_dict_type(\n            model,\n            StateDictType.LOCAL_STATE_DICT,\n        ):\n            state_dict = model.state_dict()\n            load_state_dict(state_dict, reader)\n            model.load_state_dict(state_dict)\n\n        print(f\"--> local state loaded on rank {rank}\")\n\n        return\n\n\ndef save_distributed_model_checkpoint(model, rank, cfg, epoch=1):\n    # distributed checkpoint saving\n\n    # confirm type of checkpoint and save\n    if cfg.checkpoint_type == StateDictType.LOCAL_STATE_DICT:\n        # create writer to current path\n        folder_name = (\n            cfg.dist_checkpoint_root_folder\n            + \"/\"\n            + cfg.dist_checkpoint_folder\n            + \"-\"\n            + cfg.model_name\n        )\n        save_dir = Path.cwd() / folder_name\n\n        writer = FileSystemWriter(\n            save_dir,\n        )\n\n        with FSDP.state_dict_type(\n            model,\n            StateDictType.LOCAL_STATE_DICT,\n        ):\n            state_dict = model.state_dict()\n\n\n        # write out distributed checkpoint\n        save_state_dict(state_dict, writer)\n\n        return\n"
  },
  {
    "path": "distributed/FSDP/policies/__init__.py",
    "content": "from .mixed_precision import *\nfrom .wrapping import *\nfrom .activation_checkpointing_functions import apply_fsdp_checkpointing\n"
  },
  {
    "path": "distributed/FSDP/policies/activation_checkpointing_functions.py",
    "content": "import torch\nimport os\nimport torch.distributed as dist\nfrom torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (\n    checkpoint_wrapper,\n    CheckpointImpl,\n    apply_activation_checkpointing,\n)\n\nfrom transformers.models.t5.modeling_t5 import T5Block\n\nfrom functools import partial\n\nnon_reentrant_wrapper = partial(\n    checkpoint_wrapper,\n    offload_to_cpu=False,\n    checkpoint_impl=CheckpointImpl.NO_REENTRANT,\n)\n\ncheck_fn = lambda submodule: isinstance(submodule, T5Block)\n\n\ndef apply_fsdp_checkpointing(model):\n    \"\"\"apply activation checkpointing to model\n    returns None as model is updated directly\n    \"\"\"\n    print(f\"--> applying fdsp activation checkpointing...\")\n\n    apply_activation_checkpointing(\n        model, checkpoint_wrapper_fn=non_reentrant_wrapper, check_fn=check_fn\n    )\n"
  },
  {
    "path": "distributed/FSDP/policies/mixed_precision.py",
    "content": "import torch\n\nfrom torch.distributed.fsdp import (\n    # FullyShardedDataParallel as FSDP,\n    # CPUOffload,\n    MixedPrecision,\n    # BackwardPrefetch,\n    # ShardingStrategy,\n)\n\n# requires grad scaler in main loop\nfpSixteen = MixedPrecision(\n    param_dtype=torch.float16,\n    # Gradient communication precision.\n    reduce_dtype=torch.float16,\n    # Buffer precision.\n    buffer_dtype=torch.float16,\n)\n\nbfSixteen = MixedPrecision(\n    param_dtype=torch.bfloat16,\n    # Gradient communication precision.\n    reduce_dtype=torch.bfloat16,\n    # Buffer precision.\n    buffer_dtype=torch.bfloat16,\n)\n\nbfSixteen_working = MixedPrecision(\n    param_dtype=torch.float32,\n    reduce_dtype=torch.bfloat16,\n    buffer_dtype=torch.bfloat16,\n)\n\nfp32_policy = MixedPrecision(\n    param_dtype=torch.float32,\n    reduce_dtype=torch.float32,\n    buffer_dtype=torch.float32,\n)\n"
  },
  {
    "path": "distributed/FSDP/policies/wrapping.py",
    "content": "# holds various wrapping policies for fsdp\n\n\nimport torch.distributed as dist\nimport torch.nn as nn\nimport torch\n\nfrom transformers.models.t5.modeling_t5 import T5Block\n\nfrom torch.distributed.fsdp.fully_sharded_data_parallel import (\n    FullyShardedDataParallel as FSDP,\n    CPUOffload,\n    BackwardPrefetch,\n    MixedPrecision,\n)\nfrom torch.distributed.fsdp.wrap import (\n    transformer_auto_wrap_policy,\n    size_based_auto_wrap_policy,\n    enable_wrap,\n    wrap,\n)\n\nimport functools\nfrom typing import Type\n\n\ndef get_size_policy(min_params=1e8):\n    num_wrap_policy = functools.partial(\n        size_based_auto_wrap_policy, min_num_params=min_params\n    )\n    return num_wrap_policy\n\n\ndef get_t5_wrapper():\n    \"\"\"we register our main layer class and use the fsdp transformer wrapping policy\n    ensures embedding layers are in the root fsdp unit for shared access and that fsdp units map to transformer layers\n    \"\"\"\n    # ====   use new transformer wrapper\n\n    t5_auto_wrap_policy = functools.partial(\n        transformer_auto_wrap_policy,\n        transformer_layer_cls={\n            T5Block,\n        },\n    )\n\n    return t5_auto_wrap_policy\n"
  },
  {
    "path": "distributed/FSDP/requirements.txt",
    "content": "transformers\ndatasets\ntqdm\nprotobuf\nSentencePiece\nnlp\n"
  },
  {
    "path": "distributed/FSDP/summarization_dataset.py",
    "content": "import argparse\nimport glob\nimport os\nimport json\nimport time\nimport logging\nimport random\nimport re\nfrom itertools import chain\nfrom string import punctuation\n\nimport pandas as pd\nimport numpy as np\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\n\nfrom nlp import load_dataset\n\nfrom transformers import (\n    AdamW,\n    T5ForConditionalGeneration,\n    T5Tokenizer,\n    get_linear_schedule_with_warmup\n)\n\nclass wikihow(Dataset):\n    def __init__(self, tokenizer, type_path, num_samples, input_length, output_length, print_text=False):\n        self.dataset =  load_dataset('wikihow', 'all', data_dir='data/', split=type_path)\n        if num_samples:\n            self.dataset = self.dataset.select(list(range(0, num_samples)))\n        self.input_length = input_length\n        self.tokenizer = tokenizer\n        self.output_length = output_length\n        self.print_text = print_text\n\n    def __len__(self):\n        return self.dataset.shape[0]\n\n    def clean_text(self, text):\n        text = text.replace('Example of text:', '')\n        text = text.replace('Example of Summary:', '')\n        text = text.replace('\\n','')\n        text = text.replace('``', '')\n        text = text.replace('\"', '')\n\n        return text\n\n\n    def convert_to_features(self, example_batch):\n        # Tokenize contexts and questions (as pairs of inputs)\n\n        if self.print_text:\n            print(\"Input Text: \", self.clean_text(example_batch['text']))\n#         input_ = self.clean_text(example_batch['text']) + \" </s>\"\n#         target_ = self.clean_text(example_batch['headline']) + \" </s>\"\n\n        input_ = self.clean_text(example_batch['text'])\n        target_ = self.clean_text(example_batch['headline'])\n\n        source = self.tokenizer.batch_encode_plus([input_], max_length=self.input_length,\n                                                     padding='max_length', truncation=True, return_tensors=\"pt\")\n\n        targets = self.tokenizer.batch_encode_plus([target_], max_length=self.output_length,\n                                                     padding='max_length', truncation=True, return_tensors=\"pt\")\n\n\n        return source, targets\n\n    def __getitem__(self, index):\n        source, targets = self.convert_to_features(self.dataset[index])\n\n        source_ids = source[\"input_ids\"].squeeze()\n        target_ids = targets[\"input_ids\"].squeeze()\n\n        src_mask    = source[\"attention_mask\"].squeeze()\n        target_mask = targets[\"attention_mask\"].squeeze()\n\n        return {\"source_ids\": source_ids, \"source_mask\": src_mask, \"target_ids\": target_ids, \"target_mask\": target_mask}\n\ndef get_dataset(tokenizer, type_path, num_samples, args):\n      return wikihow(tokenizer=tokenizer, type_path=type_path, num_samples=num_samples,  input_length=max_input_length,\n                        output_length=max_output_length)\n"
  },
  {
    "path": "distributed/FSDP/utils/__init__.py",
    "content": "from .environment import bfloat_support\nfrom .train_utils import setup, cleanup, get_date_of_run, format_metrics_to_gb, train, validation,setup_model\n                          \n                          "
  },
  {
    "path": "distributed/FSDP/utils/environment.py",
    "content": "# Copyright (c) 2022 Meta Platforms, Inc. and its affiliates.\n# All rights reserved.\n#\n\n# This is a simple check to confirm that your current server has full bfloat support -\n#  both GPU native support, and Network communication support.\n\n# Be warned that if you run on V100 without a check like this, you will be running without native Bfloat16\n# support and will find significant performance degradation (but it will not complain via an error).\n# Hence the reason for a checker!\n\nfrom pkg_resources import packaging\nimport torch\nimport torch.cuda.nccl as nccl\nimport torch.distributed as dist\n\n# global flag that confirms ampere architecture, cuda version and\n# nccl version to verify bfloat16 native support is ready\n\ndef bfloat_support():\n    return (\n        torch.version.cuda\n        and torch.cuda.is_bf16_supported()\n        and packaging.version.parse(torch.version.cuda).release >= (11, 0)\n        and dist.is_nccl_available()\n        and nccl.version() >= (2, 10)\n    )\n"
  },
  {
    "path": "distributed/FSDP/utils/train_utils.py",
    "content": "import os\nimport torch\nimport torch.distributed as dist\nfrom datetime import datetime\nimport tqdm\nfrom transformers import AutoTokenizer, GPT2TokenizerFast\nfrom transformers import T5Tokenizer, T5ForConditionalGeneration\n\ng_gigabyte = 1024**3\n\ndef setup():\n    # initialize the process group\n    dist.init_process_group(\"nccl\")\n\n\ndef cleanup():\n    dist.destroy_process_group()\n\ndef get_date_of_run():\n    \"\"\"create date and time for file save uniqueness\n    example: 2022-05-07-08:31:12_PM'\n    \"\"\"\n    date_of_run = datetime.now().strftime(\"%Y-%m-%d-%I:%M:%S_%p\")\n    print(f\"--> current date and time of run = {date_of_run}\")\n    return date_of_run\n\n\n\ndef format_metrics_to_gb(item):\n    \"\"\"quick function to format numbers to gigabyte and round to 4 digit precision\"\"\"\n    metric_num = item / g_gigabyte\n    metric_num = round(metric_num, ndigits=4)\n    return metric_num\n\ndef train(args, model, rank, world_size, train_loader, optimizer, epoch, sampler=None):\n    model.train()\n    local_rank = int(os.environ['LOCAL_RANK'])\n    fsdp_loss = torch.zeros(2).to(local_rank)\n\n    if sampler:\n        sampler.set_epoch(epoch)\n    if rank==0:\n        inner_pbar = tqdm.tqdm(\n            range(len(train_loader)), colour=\"blue\", desc=\"r0 Training Epoch\"\n        )\n    for batch in train_loader:\n        for key in batch.keys():\n            batch[key] = batch[key].to(local_rank)\n        optimizer.zero_grad()\n        output = model(input_ids=batch[\"source_ids\"],attention_mask=batch[\"source_mask\"],labels=batch[\"target_ids\"] )\n        loss = output[\"loss\"]\n        loss.backward()\n        optimizer.step()\n        fsdp_loss[0] += loss.item()\n        fsdp_loss[1] += len(batch)\n        if rank==0:\n            inner_pbar.update(1)\n\n    dist.all_reduce(fsdp_loss, op=dist.ReduceOp.SUM)\n    train_accuracy = fsdp_loss[0] / fsdp_loss[1]\n\n\n    if rank == 0:\n        inner_pbar.close()\n        print(\n                f\"Train Epoch: \\t{epoch}, Loss: \\t{train_accuracy:.4f}\"\n            )\n    return train_accuracy\n\n\ndef validation(model, rank, world_size, val_loader):\n    model.eval()\n    correct = 0\n    local_rank = int(os.environ['LOCAL_RANK'])\n    fsdp_loss = torch.zeros(2).to(local_rank)\n    if rank == 0:\n        inner_pbar = tqdm.tqdm(\n            range(len(val_loader)), colour=\"green\", desc=\"Validation Epoch\"\n        )\n    with torch.no_grad():\n        for batch in val_loader:\n            for key in batch.keys():\n                batch[key] = batch[key].to(local_rank)\n            output = model(input_ids=batch[\"source_ids\"],attention_mask=batch[\"source_mask\"],labels=batch[\"target_ids\"])\n            fsdp_loss[0] += output[\"loss\"].item()  # sum up batch loss\n            fsdp_loss[1] += len(batch)\n\n            if rank==0:\n                inner_pbar.update(1)\n\n    dist.all_reduce(fsdp_loss, op=dist.ReduceOp.SUM)\n    val_loss = fsdp_loss[0] / fsdp_loss[1]\n    if rank == 0:\n        inner_pbar.close()\n        print(f\"Validation Loss: {val_loss:.4f}\")\n    return val_loss\n\n\ndef setup_model(model_name):\n        model = T5ForConditionalGeneration.from_pretrained(model_name)\n        tokenizer =  T5Tokenizer.from_pretrained(model_name, legacy=False)\n        return model, tokenizer\n"
  },
  {
    "path": "distributed/FSDP2/README.md",
    "content": "## FSDP2\nTo run FSDP2 on transformer model:\n\n```\ncd distributed/FSDP2\npip install -r requirements.txt\ntorchrun --nproc_per_node 2 example.py\n```\n* For 1st time, it creates a \"checkpoints\" folder and saves state dicts there\n* For 2nd time, it loads from previous checkpoints\n\nTo enable explicit prefetching\n```\ntorchrun --nproc_per_node 2 example.py --explicit-prefetch\n```\n\nTo enable mixed precision\n```\ntorchrun --nproc_per_node 2 example.py --mixed-precision\n```\n\nTo showcase DCP API\n```\ntorchrun --nproc_per_node 2 example.py --dcp-api\n```\n\n## Ensure you are running a recent version of PyTorch:\nsee https://pytorch.org/get-started/locally/ to install at least 2.5 and ideally a current nightly build.\n"
  },
  {
    "path": "distributed/FSDP2/checkpoint.py",
    "content": "import os\nimport time\n\nimport torch\nimport torch.nn as nn\nfrom torch.distributed.checkpoint.state_dict import (\n    _init_optim_state,\n    get_model_state_dict,\n    get_optimizer_state_dict,\n    set_model_state_dict,\n    set_optimizer_state_dict,\n    StateDictOptions,\n)\nfrom torch.distributed.fsdp import FSDPModule\nfrom torch.distributed.tensor import distribute_tensor, DTensor\n\n\nMODEL_CHECKPOINT = \"model_state_dict.pt\"\nOPTIM_CHECKPOINT = \"optim_state_dict.pt\"\nPARAMS = \"params\"\n\n\ndef get_latest_checkpoint_folder(path):\n    max_num = None\n    if not os.path.exists(path):\n        return max_num\n    for name in os.listdir(path):\n        folder_path = os.path.join(path, name)\n        if os.path.isdir(folder_path):\n            try:\n                num = int(name)\n                if max_num is None or num > max_num:\n                    max_num = num\n            except ValueError:\n                pass  # Skip non-numeric folder names\n    return max_num\n\n\nclass Checkpointer:\n    def __init__(self, folder: str, dcp_api: bool):\n        self.folder = folder\n        self.dcp_api = dcp_api\n        self.last_training_time = get_latest_checkpoint_folder(\n            f\"{folder}/{'dcp_api' if dcp_api else 'dtensor_api'}\"\n        )\n\n    def is_empty(self):\n        return self.last_training_time is None\n\n    def load_model(self, model: FSDPModule):\n        last_model_checkpoint = (\n            f\"{self.folder}/{'dcp_api' if self.dcp_api else 'dtensor_api'}\"\n            f\"/{self.last_training_time}/{MODEL_CHECKPOINT}\"\n        )\n        full_sd = torch.load(\n            last_model_checkpoint, mmap=True, weights_only=True, map_location=\"cpu\"\n        )\n        if self.dcp_api:\n            set_model_state_dict(\n                model=model,\n                model_state_dict=full_sd,\n                options=StateDictOptions(\n                    full_state_dict=True,\n                    broadcast_from_rank0=True,\n                ),\n            )\n            return\n        meta_sharded_sd = model.state_dict()\n        sharded_sd = {}\n        for param_name, full_tensor in full_sd.items():\n            sharded_meta_param = meta_sharded_sd.get(param_name)\n            sharded_tensor = distribute_tensor(\n                full_tensor,\n                sharded_meta_param.device_mesh,\n                sharded_meta_param.placements,\n            )\n            sharded_sd[param_name] = nn.Parameter(sharded_tensor)\n        # choose `assign=True` since we cannot call `copy_` on meta tensor\n        model.load_state_dict(sharded_sd, strict=False, assign=True)\n\n    def load_optim(self, model: FSDPModule, opt: torch.optim.Optimizer):\n        last_optim_checkpoint = (\n            f\"{self.folder}/{'dcp_api' if self.dcp_api else 'dtensor_api'}\"\n            f\"/{self.last_training_time}/{OPTIM_CHECKPOINT}\"\n        )\n        full_sd = torch.load(\n            last_optim_checkpoint, mmap=True, weights_only=True, map_location=\"cpu\"\n        )\n        if self.dcp_api:\n            set_optimizer_state_dict(\n                model=model,\n                optimizers=opt,\n                optim_state_dict=full_sd,\n                options=StateDictOptions(\n                    full_state_dict=True,\n                    broadcast_from_rank0=True,\n                ),\n            )\n            return\n        _init_optim_state(opt)\n        param_groups = opt.state_dict()[\"param_groups\"]\n        state = opt.state_dict()[\"state\"]\n\n        full_param_groups = full_sd[\"param_groups\"]\n        full_state = full_sd[\"state\"]\n\n        for param_group, full_param_group in zip(param_groups, full_param_groups):\n            for key, value in full_param_group.items():\n                if key == PARAMS:\n                    continue\n                param_group[key] = value\n            for pid, full_pid in zip(param_group[PARAMS], full_param_group[PARAMS]):\n                if pid not in state:\n                    continue\n                param_state = state[pid]\n                full_param_state = full_state[full_pid]\n                for attr, full_tensor in full_param_state.items():\n                    sharded_tensor = param_state[attr]\n                    if isinstance(sharded_tensor, DTensor):\n                        # exp_avg is DTensor\n                        param_state[attr] = distribute_tensor(\n                            full_tensor,\n                            sharded_tensor.device_mesh,\n                            sharded_tensor.placements,\n                        )\n                    else:\n                        # step is plain tensor\n                        param_state[attr] = full_tensor\n        opt.load_state_dict(\n            {\n                \"param_groups\": param_groups,\n                \"state\": state,\n            }\n        )\n\n    def _get_full_model_state_dict(self, model: FSDPModule):\n        if self.dcp_api:\n            return get_model_state_dict(\n                model=model,\n                options=StateDictOptions(\n                    full_state_dict=True,\n                    cpu_offload=True,\n                ),\n            )\n\n        sharded_sd = model.state_dict()\n        cpu_state_dict = {}\n        for param_name, sharded_param in sharded_sd.items():\n            full_param = sharded_param.full_tensor()\n            if torch.distributed.get_rank() == 0:\n                cpu_state_dict[param_name] = full_param.cpu()\n            else:\n                del full_param\n        return cpu_state_dict\n\n    def _get_full_optimizer_state_dict(\n        self,\n        model: FSDPModule,\n        opt: torch.optim.Optimizer,\n    ):\n        if self.dcp_api:\n            return get_optimizer_state_dict(\n                model=model,\n                optimizers=opt,\n                options=StateDictOptions(\n                    full_state_dict=True,\n                    cpu_offload=True,\n                ),\n            )\n        is_rank_zero = torch.distributed.get_rank() == 0\n        sharded_sd = opt.state_dict()\n        sharded_state = sharded_sd[\"state\"]\n        full_state = {}\n        for group_id, sharded_group in sharded_state.items():\n            group_state = {}\n            for attr, sharded_tensor in sharded_group.items():\n                if isinstance(sharded_tensor, DTensor):\n                    # \"exp_avg\" in AdamW is `DTensor`\n                    full_tensor = sharded_tensor.full_tensor()\n                else:\n                    # \"step\" in AdamW is plain tensor\n                    full_tensor = sharded_tensor\n                if is_rank_zero:\n                    group_state[attr] = full_tensor.cpu()\n                else:\n                    del full_tensor\n            if is_rank_zero:\n                full_state[group_id] = group_state\n            else:\n                del group_state\n        if is_rank_zero:\n            return {\n                \"param_groups\": sharded_sd[\"param_groups\"],\n                \"state\": full_state,\n            }\n        else:\n            return {}\n\n    def save(self, model: FSDPModule, optim: torch.optim.Optimizer):\n        model_state_dict = self._get_full_model_state_dict(model)\n        optim_state_dict = self._get_full_optimizer_state_dict(model, optim)\n        if torch.distributed.get_rank() == 0:\n            new_training_time = int(time.time() * 1000)\n            new_checkpoint_folder = f\"{self.folder}/{'dcp_api' if self.dcp_api else 'dtensor_api'}/{new_training_time}\"\n            new_model_checkpoint = f\"{new_checkpoint_folder}/{MODEL_CHECKPOINT}\"\n            new_optim_checkpoint = f\"{new_checkpoint_folder}/{OPTIM_CHECKPOINT}\"\n            os.makedirs(new_checkpoint_folder, exist_ok=True)\n            torch.save(model_state_dict, new_model_checkpoint)\n            torch.save(optim_state_dict, new_optim_checkpoint)\n"
  },
  {
    "path": "distributed/FSDP2/example.py",
    "content": "import argparse\nimport os\n\nimport torch\nfrom checkpoint import Checkpointer\nfrom model import ModelArgs, Transformer\nfrom torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy\nfrom utils import inspect_mixed_precision, inspect_model\n\ndef verify_min_gpu_count(min_gpus: int = 2) -> bool:\n    \"\"\" verification that we have at least 2 gpus to run dist examples \"\"\"\n    has_gpu = torch.accelerator.is_available()\n    gpu_count = torch.accelerator.device_count()\n    return has_gpu and gpu_count >= min_gpus\n\ndef set_modules_to_forward_prefetch(model, num_to_forward_prefetch):\n    for i, layer in enumerate(model.layers):\n        if i >= len(model.layers) - num_to_forward_prefetch:\n            break\n        layers_to_prefetch = [\n            model.layers[i + j] for j in range(1, num_to_forward_prefetch + 1)\n        ]\n        layer.set_modules_to_forward_prefetch(layers_to_prefetch)\n\n\ndef set_modules_to_backward_prefetch(model, num_to_backward_prefetch):\n    for i, layer in enumerate(model.layers):\n        if i < num_to_backward_prefetch:\n            continue\n        layers_to_prefetch = [\n            model.layers[i - j] for j in range(1, num_to_backward_prefetch + 1)\n        ]\n        layer.set_modules_to_backward_prefetch(layers_to_prefetch)\n\n\ndef main(args):\n    _min_gpu_count = 2\n    if not verify_min_gpu_count(min_gpus=_min_gpu_count):\n        print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n        exit()\n    rank = int(os.environ[\"LOCAL_RANK\"])\n    if torch.accelerator.is_available():\n        device_type = torch.accelerator.current_accelerator()\n        device = torch.device(f\"{device_type}:{rank}\")\n        torch.accelerator.device_index(rank)\n        print(f\"Running on rank {rank} on device {device}\")\n    else:\n        device = torch.device(\"cpu\")\n        print(f\"Running on device {device}\")\n\n    backend = torch.distributed.get_default_backend_for_device(device)\n    torch.distributed.init_process_group(backend=backend, device_id=device)\n\n    torch.manual_seed(0)\n    vocab_size = 1024\n    batch_size = 32\n    seq_len = 64\n    model_args = ModelArgs(\n        n_layers=10,\n        n_heads=4,\n        vocab_size=vocab_size,\n        max_seq_len=seq_len,\n        dropout_p=0,\n    )\n    with torch.device(\"meta\"):\n        model = Transformer(model_args)\n    fsdp_kwargs = {}\n    if args.mixed_precision:\n        fsdp_kwargs[\"mp_policy\"] = MixedPrecisionPolicy(\n            param_dtype=torch.bfloat16,\n            reduce_dtype=torch.float32,\n        )\n    for layer in model.layers:\n        fully_shard(layer, **fsdp_kwargs)\n    fully_shard(model, **fsdp_kwargs)\n\n    inspect_model(model)\n\n    if args.explicit_prefetching:\n        set_modules_to_forward_prefetch(model, num_to_forward_prefetch=2)\n        set_modules_to_backward_prefetch(model, num_to_backward_prefetch=2)\n\n    checkpointer = Checkpointer(\"checkpoints\", dcp_api=args.dcp_api)\n    if checkpointer.last_training_time is None:\n        model.to_empty(device=device)\n        model.reset_parameters()\n    else:\n        checkpointer.load_model(model)\n    \n    if args.mixed_precision:\n        inspect_mixed_precision(model)\n\n    optim = torch.optim.Adam(model.parameters(), lr=1e-2)\n    if checkpointer.last_training_time is not None:\n        checkpointer.load_optim(model, optim)\n\n    for _ in range(10):\n        if args.explicit_prefetching:\n            model.unshard()\n        x = torch.randint(0, vocab_size, (batch_size, seq_len), device=device)\n        loss = model(x).sum()\n        loss.backward()\n        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n        optim.step()\n        optim.zero_grad()\n\n    checkpointer.save(model, optim)\n    torch.distributed.destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    parser = argparse.ArgumentParser(description=\"PyTorch FSDP2 example\")\n    parser.add_argument(\"--explicit-prefetching\", action=\"store_true\", default=False)\n    parser.add_argument(\"--mixed-precision\", action=\"store_true\", default=False)\n    parser.add_argument(\"--dcp-api\", action=\"store_true\", default=False)\n    args = parser.parse_args()\n    \n    main(args)\n"
  },
  {
    "path": "distributed/FSDP2/model.py",
    "content": "from dataclasses import dataclass\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n\n@dataclass\nclass ModelArgs:\n    n_layers: int = 2\n    vocab_size: int = 8\n    max_seq_len: int = 16\n    dim: int = 16\n    n_heads: int = 4\n    dropout_p: float = 0.1\n\n\nclass Attention(nn.Module):\n    def __init__(self, args: ModelArgs):\n        super().__init__()\n        assert args.dim % args.n_heads == 0\n        self.head_dim = args.dim // args.n_heads\n        self.n_heads = args.n_heads\n        self.dropout_p = args.dropout_p\n        self.resid_dropout = nn.Dropout(args.dropout_p)\n\n        self.wq = nn.Linear(args.dim, args.dim, bias=False)\n        self.wk = nn.Linear(args.dim, args.dim, bias=False)\n        self.wv = nn.Linear(args.dim, args.dim, bias=False)\n        self.wo = nn.Linear(args.dim, args.dim, bias=False)\n\n    def forward(self, x):\n        bsz, seq_len, _ = x.size()\n        queries, keys, values = self.wq(x), self.wk(x), self.wv(x)\n        queries = queries.view(bsz, seq_len, self.n_heads, self.head_dim)\n        keys = keys.view(bsz, seq_len, self.n_heads, self.head_dim)\n        values = values.view(bsz, seq_len, self.n_heads, self.head_dim)\n\n        queries = queries.transpose(1, 2)  # (bsz, n_heads, seq_len, head_dim)\n        keys = keys.transpose(1, 2)  # (bsz, n_heads, seq_len, head_dim)\n        values = values.transpose(1, 2)  # (bsz, n_heads, seq_len, head_dim)\n\n        output = F.scaled_dot_product_attention(\n            queries,\n            keys,\n            values,\n            None,\n            self.dropout_p if self.training else 0,\n        )\n        output = output.transpose(1, 2).contiguous().view(bsz, seq_len, -1)\n        return self.resid_dropout(self.wo(output))\n\n    def reset_parameters(self):\n        self.wq.reset_parameters()\n        self.wk.reset_parameters()\n        self.wv.reset_parameters()\n        self.wo.reset_parameters()\n\n\nclass FeedForward(nn.Module):\n    def __init__(self, dim, hidden_dim, dropout_p):\n        super().__init__()\n        self.w1 = nn.Linear(dim, hidden_dim)\n        self.gelu = nn.GELU()\n        self.w2 = nn.Linear(hidden_dim, dim)\n        self.resid_dropout = nn.Dropout(dropout_p)\n\n    def forward(self, x):\n        return self.resid_dropout(self.w2(self.gelu(self.w1(x))))\n\n    def reset_parameters(self):\n        self.w1.reset_parameters()\n        self.w2.reset_parameters()\n\n\nclass TransformerBlock(nn.Module):\n    def __init__(self, args: ModelArgs):\n        super().__init__()\n        self.attention_norm = nn.LayerNorm(args.dim)\n        self.attention = Attention(args)\n        self.ffn_norm = nn.LayerNorm(args.dim)\n        self.feed_forward = FeedForward(\n            args.dim, hidden_dim=4 * args.dim, dropout_p=args.dropout_p\n        )\n\n    def forward(self, x):\n        h = x + self.attention(self.attention_norm(x))\n        out = h + self.feed_forward(self.ffn_norm(h))\n        return out\n\n    def reset_parameters(self):\n        self.attention_norm.reset_parameters()\n        self.attention.reset_parameters()\n        self.ffn_norm.reset_parameters()\n        self.feed_forward.reset_parameters()\n\n\n# A toy transformer model, partly inspired by the nanoGPT model:\n# https://github.com/karpathy/nanoGPT.\nclass Transformer(nn.Module):\n    def __init__(self, args: ModelArgs):\n        super().__init__()\n        assert args.vocab_size is not None\n        assert args.max_seq_len is not None\n        self.model_args = args\n        self.max_seq_len = args.max_seq_len\n        self.tok_embeddings = nn.Embedding(args.vocab_size, args.dim)\n        self.pos_embeddings = nn.Embedding(args.max_seq_len, args.dim)\n        self.dropout = nn.Dropout(args.dropout_p)\n        self.layers = nn.ModuleList()\n        for _ in range(args.n_layers):\n            self.layers.append(TransformerBlock(args))\n        self.norm = nn.LayerNorm(args.dim)\n        self.output = nn.Linear(args.dim, args.vocab_size, bias=False)\n\n    def forward(self, tokens):\n        _bsz, seq_len = tokens.size()\n        assert seq_len <= self.max_seq_len\n        h = self.tok_embeddings(tokens)\n        pos = torch.arange(0, seq_len, device=tokens.device)\n        p = self.pos_embeddings(pos)  # positional embeddings of shape (seq_len, dim)\n        h = h + p\n        h = self.dropout(h)\n        for layer in self.layers:\n            h = layer(h)\n        h = self.norm(h)\n        output = self.output(h).float()\n        return output\n\n    def reset_parameters(self):\n        self.tok_embeddings.reset_parameters()\n        self.pos_embeddings.reset_parameters()\n        self.norm.reset_parameters()\n        self.output.reset_parameters()\n"
  },
  {
    "path": "distributed/FSDP2/requirements.txt",
    "content": "torch>=2.7\nnumpy\n"
  },
  {
    "path": "distributed/FSDP2/run_example.sh",
    "content": "# /bin/bash\n# bash run_example.sh {file_to_run.py} {num_gpus}\n# where file_to_run = example to run. Default = 'example.py'\n# num_gpus = num local gpus to use (must be at least 2). Default = 4\n\n# samples to run include:\n# example.py\n\necho \"Launching ${1:-example.py} with ${2:-4} gpus\"\ntorchrun --nnodes=1 --nproc_per_node=${2:-4} ${1:-example.py}\n\n"
  },
  {
    "path": "distributed/FSDP2/utils.py",
    "content": "import torch\nfrom model import Transformer\nfrom torch.distributed.fsdp import FSDPModule\nfrom torch.distributed.tensor import Shard\n\n\ndef inspect_model(model: FSDPModule):\n    assert isinstance(model, Transformer)\n    assert isinstance(model, FSDPModule)\n\n    if torch.distributed.get_rank() == 0:\n        print(model)\n\n    for param in model.parameters():\n        assert param.placements == (Shard(0),)\n        assert param.dtype == torch.float32\n        # print(param.get_local_tensor())\n\n\ndef inspect_mixed_precision(model: FSDPModule):\n    model.unshard()\n    for param in model.parameters(recurse=False):\n        assert param.dtype == torch.bfloat16\n    model.reshard()\n"
  },
  {
    "path": "distributed/ddp/README.md",
    "content": "\n# Distributed Data Parallel (DDP) Applications with PyTorch\n\nThis guide demonstrates how to structure a distributed model training application for convenient multi-node launches using `torchrun`.\n\n---\n\n## Prerequisites\n\nYou should be familiar with:\n\n- [PyTorch basics](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)\n- [Writing distributed applications](https://pytorch.org/tutorials/intermediate/dist_tuto.html)\n- [Distributed model training](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html)\n\nThis tutorial uses the [`torch.nn.parallel.DistributedDataParallel`](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) class for data parallel training: multiple workers train the same global model on different data shards, compute local gradients, and synchronize them using AllReduce. In High-Performance Computing (HCP), this is called _Single Program Multiple Data_ (SPMD).\n\n---\n\n## Application Process Topologies\n\nA Distributed Data Parallel (DDP) application can be executed on\nmultiple nodes where each node can consist of multiple GPU\ndevices. Each node in turn can run multiple copies of the DDP\napplication, each of which processes its models on multiple GPUs.\n\nLet:\n- _N_ = number of nodes\n- _G_ = number of GPUs per node\n- _W_ = **World Size** = total number of processes\n- _L_ = **Local World Size** = processes per node\n\nEach process has:\n- **Local rank**: in `[0, L-1]`\n- **Global rank**: in `[0, W-1]`\n\n**Example:**\nIf you launch a DDP app on 2 nodes, each with 4 GPUs, and want each process to span 2 GPUs, the mapping is as follows:\n\n![ProcessMapping](https://user-images.githubusercontent.com/875518/77676984-4c81e400-6f4c-11ea-87d8-f2ff505a99da.png)\n\nWhile there are quite a few ways to map processes to nodes, a good rule of thumb is to have one process span a single GPU. This enables the DDP application to have as many parallel reader streams as there are GPUs and in practice provides a good balance between I/O and computational costs. In the rest of this tutorial, we assume that the application follows this heuristic.\n\n# Preparing and launching a DDP application\n\nIndependent of how a DDP application is launched, each process needs a mechanism to know its global and local ranks. Once this is known, all processes create a `ProcessGroup` that enables them to participate in collective communication operations such as AllReduce.\n\nA convenient way to start multiple DDP processes and initialize all values needed to create a `ProcessGroup` is to use the [`torchrun`](https://docs.pytorch.org/docs/stable/elastic/run.html) script provided with PyTorch.\n\n---\n\n## Sample Application\n\nThis example is based on the [\"Hello, World\" DDP tutorial](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html).\n\nThe application calls the `spmd_main` entrypoint:\n\n```python\nif __name__ == \"__main__\":\n    spmd_main()\n```\n\nIn `spmd_main`, the process group is initialized using the Accelerator API. The rest of the rendezvous information comes from environment variables set by `torchrun`:\n\n```python\ndef spmd_main():\n    # These are the parameters used to initialize the process group\n    env_dict = {\n        key: os.environ[key]\n        for key in (\"MASTER_ADDR\", \"MASTER_PORT\", \"RANK\", \"WORLD_SIZE\")\n    }\n    rank = int(env_dict['RANK'])\n    local_rank = int(env_dict['LOCAL_RANK'])\n    local_world_size = int(env_dict['LOCAL_WORLD_SIZE'])\n\n    print(f\"[{os.getpid()}] Initializing process group with: {env_dict}\")\n    acc = torch.accelerator.current_accelerator()\n    vendor_backend = torch.distributed.get_default_backend_for_device(acc)\n    torch.accelerator.set_device_index(rank)\n    torch.distributed.init_process_group(backend=vendor_backend)\n\n    demo_basic(rank)\n\n    # Tear down the process group\n    torch.distributed.destroy_process_group()\n```\n\n**Key points:**\n- Each process reads its rank and world size from environment variables.\n- The process group is initialized for distributed communication.\n\nThe training function, `demo_basic`, initializes the DDP model on the appropriate GPU:\n\n```python\ndef demo_basic(rank):\n    print(\n        f\"[{os.getpid()}] rank = {torch.distributed.get_rank()}, \"\n        + f\"world_size = {torch.distributed.get_world_size()}\"\n    )\n\n    model = ToyModel().to(rank)\n    ddp_model = DDP(model, device_ids=[rank])\n\n    loss_fn = nn.MSELoss()\n    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)\n\n    optimizer.zero_grad()\n    outputs = ddp_model(torch.randn(20, 10))\n    labels = torch.randn(20, 5).to(rank)\n    loss_fn(outputs, labels).backward()\n    optimizer.step()\n```\n\n---\n\n## Launching the Application\n\n```sh\ntorchrun --nnodes=1 --nproc_per_node=8 example.py\n```\n\n---\n\n## Example Output\n\nExpected output:\n\n```sh\n*****************************************\nSetting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.\n*****************************************\n[238627] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '0', 'WORLD_SIZE': '8'}\n[238630] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '3', 'WORLD_SIZE': '8'}\n[238628] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '1', 'WORLD_SIZE': '8'}\n[238634] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '7', 'WORLD_SIZE': '8'}\n[238631] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '4', 'WORLD_SIZE': '8'}\n[238632] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '5', 'WORLD_SIZE': '8'}\n[238629] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '2', 'WORLD_SIZE': '8'}\n[238633] Initializing process group with: {'MASTER_ADDR': '127.0.0.1', 'MASTER_PORT': '29500', 'RANK': '6', 'WORLD_SIZE': '8'}\n[238633] world_size = 8, rank = 6, backend=nccl\n[238628] world_size = 8, rank = 1, backend=nccl\n[238629] world_size = 8, rank = 2, backend=nccl\n[238631] world_size = 8, rank = 4, backend=nccl\n[238630] world_size = 8, rank = 3, backend=nccl\n[238632] world_size = 8, rank = 5, backend=nccl\n[238634] world_size = 8, rank = 7, backend=nccl\n[238627] world_size = 8, rank = 0, backend=nccl\n[238633] rank = 6, world_size = 8\n[238628] rank = 1, world_size = 8\n[238632] rank = 5, world_size = 8\n[238634] rank = 7, world_size = 8\n[238629] rank = 2, world_size = 8\n[238630] rank = 3, world_size = 8\n[238631] rank = 4, world_size = 8\n[238627] rank = 0, world_size = 8\n```\n\n# Conclusions\n\nAs the author of a distributed data parallel application, your code needs to be aware of two types of resources: compute nodes and the GPUs within each node. The process of setting up bookkeeping to track how the set of GPUs is mapped to the processes of your application can be tedious and error-prone. We hope that by structuring your application as shown in this example and using `torchrun`, the mechanics of setting up distributed training can be significantly simplified.\n"
  },
  {
    "path": "distributed/ddp/example.py",
    "content": "import argparse\nimport os\nimport sys\nimport tempfile\nfrom urllib.parse import urlparse\n\nimport torch\nimport torch.distributed as dist\nimport torch.nn as nn\nimport torch.optim as optim\n\nfrom torch.nn.parallel import DistributedDataParallel as DDP\n\ndef verify_min_gpu_count(min_gpus: int = 2) -> bool:\n    \"\"\" verification that we have at least 2 gpus to run dist examples \"\"\"\n    has_gpu = torch.accelerator.is_available()\n    gpu_count = torch.accelerator.device_count()\n    return has_gpu and gpu_count >= min_gpus\n\nclass ToyModel(nn.Module):\n    def __init__(self):\n        super(ToyModel, self).__init__()\n        self.net1 = nn.Linear(10, 10)\n        self.relu = nn.ReLU()\n        self.net2 = nn.Linear(10, 5)\n\n    def forward(self, x):\n        return self.net2(self.relu(self.net1(x)))\n\n\ndef demo_basic(rank):\n\n    print(\n        f\"[{os.getpid()}] rank = {dist.get_rank()}, \"\n        + f\"world_size = {dist.get_world_size()}\"\n        )\n\n    model = ToyModel().to(rank)\n    ddp_model = DDP(model, device_ids=[rank])\n\n    loss_fn = nn.MSELoss()\n    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)\n\n    optimizer.zero_grad()\n    outputs = ddp_model(torch.randn(20, 10))\n    labels = torch.randn(20, 5).to(rank)\n    loss_fn(outputs, labels).backward()\n    optimizer.step()\n\n    print(f\"training completed in rank {rank}!\")\n\n\ndef main():\n    # These are the parameters used to initialize the process group\n    env_dict = {\n        key: os.environ[key]\n        for key in (\"MASTER_ADDR\", \"MASTER_PORT\", \"RANK\", \"LOCAL_RANK\", \"WORLD_SIZE\", \"LOCAL_WORLD_SIZE\")\n    }\n    rank = int(env_dict['RANK'])\n    local_rank = int(env_dict['LOCAL_RANK'])\n    local_world_size = int(env_dict['LOCAL_WORLD_SIZE'])\n    \n    if sys.platform == \"win32\":\n        # Distributed package only covers collective communications with Gloo\n        # backend and FileStore on Windows platform. Set init_method parameter\n        # in init_process_group to a local file.\n        if \"INIT_METHOD\" in os.environ.keys():\n            print(f\"init_method is {os.environ['INIT_METHOD']}\")\n            url_obj = urlparse(os.environ[\"INIT_METHOD\"])\n            if url_obj.scheme.lower() != \"file\":\n                raise ValueError(\"Windows only supports FileStore\")\n            else:\n                init_method = os.environ[\"INIT_METHOD\"]\n        else:\n            # It is a example application, For convience, we create a file in temp dir.\n            temp_dir = tempfile.gettempdir()\n            init_method = f\"file:///{os.path.join(temp_dir, 'ddp_example')}\"\n        dist.init_process_group(backend=\"gloo\", init_method=init_method, rank=int(env_dict[\"RANK\"]), world_size=int(env_dict[\"WORLD_SIZE\"]))\n    else:\n        print(f\"[{os.getpid()}] Initializing process group with: {env_dict}\")  \n        acc = torch.accelerator.current_accelerator()\n        backend = torch.distributed.get_default_backend_for_device(acc)\n        torch.accelerator.set_device_index(rank)\n        dist.init_process_group(backend=backend)\n\n    print(\n        f\"[{os.getpid()}]: world_size = {dist.get_world_size()}, \"\n        + f\"rank = {dist.get_rank()}, backend={dist.get_backend()} \\n\", end=''\n    )\n\n    demo_basic(rank)\n\n    # Tear down the process group\n    dist.destroy_process_group()\n\nif __name__ == \"__main__\":\n    _min_gpu_count = 2\n    if not verify_min_gpu_count(min_gpus=_min_gpu_count):\n        print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n        sys.exit()\n    main()\n"
  },
  {
    "path": "distributed/ddp/requirements.txt",
    "content": "torch>=2.7\n"
  },
  {
    "path": "distributed/ddp/run_example.sh",
    "content": "# /bin/bash\n# bash run_example.sh {file_to_run.py} {num_gpus}\n# where file_to_run = example to run. Default = 'example.py'\n# num_gpus = num local gpus to use (must be at least 2). Default = 2\n\n# samples to run include:\n# example.py\n\necho \"Launching ${1:-example.py} with ${2:-2} gpus\"\ntorchrun --nnodes=1 --nproc_per_node=${2:-2} --rdzv_id=101 --rdzv_endpoint=\"localhost:5972\" ${1:-example.py}\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/README.md",
    "content": "# distributed-pytorch\n\nCode for the DDP tutorial series at https://pytorch.org/tutorials/beginner/ddp_series_intro.html\n\nEach code file extends upon the previous one. The series starts with a non-distributed script that runs on a single GPU and incrementally updates to end with multinode training on a Slurm cluster.\n\n## Files\n* [single_gpu.py](single_gpu.py): Non-distributed training script\n\n* [multigpu.py](multigpu.py): DDP on a single node\n\n* [multigpu_torchrun.py](multigpu_torchrun.py): DDP on a single node using Torchrun\n\n* [multinode.py](multinode.py): DDP on multiple nodes using Torchrun (and optionally Slurm)\n    * [slurm/setup_pcluster_slurm.md](slurm/setup_pcluster_slurm.md): instructions to set up an AWS cluster\n    * [slurm/config.yaml.template](slurm/config.yaml.template): configuration to set up an AWS cluster\n    * [slurm/sbatch_run.sh](slurm/sbatch_run.sh): slurm script to launch the training job\n\n\n\n\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/datautils.py",
    "content": "import torch\nfrom torch.utils.data import Dataset\n\nclass MyTrainDataset(Dataset):\n    def __init__(self, size):\n        self.size = size\n        self.data = [(torch.rand(20), torch.rand(1)) for _ in range(size)]\n\n    def __len__(self):\n        return self.size\n    \n    def __getitem__(self, index):\n        return self.data[index]"
  },
  {
    "path": "distributed/ddp-tutorial-series/multigpu.py",
    "content": "import torch\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom datautils import MyTrainDataset\n\nimport torch.multiprocessing as mp\nfrom torch.utils.data.distributed import DistributedSampler\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom torch.distributed import init_process_group, destroy_process_group\nimport os\n\n\ndef ddp_setup(rank, world_size):\n    \"\"\"\n    Args:\n        rank: Unique identifier of each process\n        world_size: Total number of processes\n    \"\"\"\n    os.environ[\"MASTER_ADDR\"] = \"localhost\"\n    os.environ[\"MASTER_PORT\"] = \"12355\"\n    torch.cuda.set_device(rank)\n    init_process_group(backend=\"nccl\", rank=rank, world_size=world_size)\n\nclass Trainer:\n    def __init__(\n        self,\n        model: torch.nn.Module,\n        train_data: DataLoader,\n        optimizer: torch.optim.Optimizer,\n        gpu_id: int,\n        save_every: int,\n    ) -> None:\n        self.gpu_id = gpu_id\n        self.model = model.to(gpu_id)\n        self.train_data = train_data\n        self.optimizer = optimizer\n        self.save_every = save_every\n        self.model = DDP(model, device_ids=[gpu_id])\n\n    def _run_batch(self, source, targets):\n        self.optimizer.zero_grad()\n        output = self.model(source)\n        loss = F.cross_entropy(output, targets)\n        loss.backward()\n        self.optimizer.step()\n\n    def _run_epoch(self, epoch):\n        b_sz = len(next(iter(self.train_data))[0])\n        print(f\"[GPU{self.gpu_id}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(self.train_data)}\")\n        self.train_data.sampler.set_epoch(epoch)\n        for source, targets in self.train_data:\n            source = source.to(self.gpu_id)\n            targets = targets.to(self.gpu_id)\n            self._run_batch(source, targets)\n\n    def _save_checkpoint(self, epoch):\n        ckp = self.model.module.state_dict()\n        PATH = \"checkpoint.pt\"\n        torch.save(ckp, PATH)\n        print(f\"Epoch {epoch} | Training checkpoint saved at {PATH}\")\n\n    def train(self, max_epochs: int):\n        for epoch in range(max_epochs):\n            self._run_epoch(epoch)\n            if self.gpu_id == 0 and epoch % self.save_every == 0:\n                self._save_checkpoint(epoch)\n\n\ndef load_train_objs():\n    train_set = MyTrainDataset(2048)  # load your dataset\n    model = torch.nn.Linear(20, 1)  # load your model\n    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)\n    return train_set, model, optimizer\n\n\ndef prepare_dataloader(dataset: Dataset, batch_size: int):\n    return DataLoader(\n        dataset,\n        batch_size=batch_size,\n        pin_memory=True,\n        shuffle=False,\n        sampler=DistributedSampler(dataset)\n    )\n\n\ndef main(rank: int, world_size: int, save_every: int, total_epochs: int, batch_size: int):\n    ddp_setup(rank, world_size)\n    dataset, model, optimizer = load_train_objs()\n    train_data = prepare_dataloader(dataset, batch_size)\n    trainer = Trainer(model, train_data, optimizer, rank, save_every)\n    trainer.train(total_epochs)\n    destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    import argparse\n    parser = argparse.ArgumentParser(description='simple distributed training job')\n    parser.add_argument('total_epochs', type=int, help='Total epochs to train the model')\n    parser.add_argument('save_every', type=int, help='How often to save a snapshot')\n    parser.add_argument('--batch_size', default=32, type=int, help='Input batch size on each device (default: 32)')\n    args = parser.parse_args()\n\n    world_size = torch.cuda.device_count()\n    mp.spawn(main, args=(world_size, args.save_every, args.total_epochs, args.batch_size), nprocs=world_size)\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/multigpu_torchrun.py",
    "content": "import torch\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom datautils import MyTrainDataset\n\nimport torch.multiprocessing as mp\nfrom torch.utils.data.distributed import DistributedSampler\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom torch.distributed import init_process_group, destroy_process_group\nimport os\n\n\ndef ddp_setup():\n    torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))\n    init_process_group(backend=\"nccl\")\n\nclass Trainer:\n    def __init__(\n        self,\n        model: torch.nn.Module,\n        train_data: DataLoader,\n        optimizer: torch.optim.Optimizer,\n        save_every: int,\n        snapshot_path: str,\n    ) -> None:\n        self.gpu_id = int(os.environ[\"LOCAL_RANK\"])\n        self.model = model.to(self.gpu_id)\n        self.train_data = train_data\n        self.optimizer = optimizer\n        self.save_every = save_every\n        self.epochs_run = 0\n        self.snapshot_path = snapshot_path\n        if os.path.exists(snapshot_path):\n            print(\"Loading snapshot\")\n            self._load_snapshot(snapshot_path)\n\n        self.model = DDP(self.model, device_ids=[self.gpu_id])\n\n    def _load_snapshot(self, snapshot_path):\n        loc = f\"cuda:{self.gpu_id}\"\n        snapshot = torch.load(snapshot_path, map_location=loc)\n        self.model.load_state_dict(snapshot[\"MODEL_STATE\"])\n        self.epochs_run = snapshot[\"EPOCHS_RUN\"]\n        print(f\"Resuming training from snapshot at Epoch {self.epochs_run}\")\n\n    def _run_batch(self, source, targets):\n        self.optimizer.zero_grad()\n        output = self.model(source)\n        loss = F.cross_entropy(output, targets)\n        loss.backward()\n        self.optimizer.step()\n\n    def _run_epoch(self, epoch):\n        b_sz = len(next(iter(self.train_data))[0])\n        print(f\"[GPU{self.gpu_id}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(self.train_data)}\")\n        self.train_data.sampler.set_epoch(epoch)\n        for source, targets in self.train_data:\n            source = source.to(self.gpu_id)\n            targets = targets.to(self.gpu_id)\n            self._run_batch(source, targets)\n\n    def _save_snapshot(self, epoch):\n        snapshot = {\n            \"MODEL_STATE\": self.model.module.state_dict(),\n            \"EPOCHS_RUN\": epoch,\n        }\n        torch.save(snapshot, self.snapshot_path)\n        print(f\"Epoch {epoch} | Training snapshot saved at {self.snapshot_path}\")\n\n    def train(self, max_epochs: int):\n        for epoch in range(self.epochs_run, max_epochs):\n            self._run_epoch(epoch)\n            if self.gpu_id == 0 and epoch % self.save_every == 0:\n                self._save_snapshot(epoch)\n\n\ndef load_train_objs():\n    train_set = MyTrainDataset(2048)  # load your dataset\n    model = torch.nn.Linear(20, 1)  # load your model\n    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)\n    return train_set, model, optimizer\n\n\ndef prepare_dataloader(dataset: Dataset, batch_size: int):\n    return DataLoader(\n        dataset,\n        batch_size=batch_size,\n        pin_memory=True,\n        shuffle=False,\n        sampler=DistributedSampler(dataset)\n    )\n\n\ndef main(save_every: int, total_epochs: int, batch_size: int, snapshot_path: str = \"snapshot.pt\"):\n    ddp_setup()\n    dataset, model, optimizer = load_train_objs()\n    train_data = prepare_dataloader(dataset, batch_size)\n    trainer = Trainer(model, train_data, optimizer, save_every, snapshot_path)\n    trainer.train(total_epochs)\n    destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    import argparse\n    parser = argparse.ArgumentParser(description='simple distributed training job')\n    parser.add_argument('total_epochs', type=int, help='Total epochs to train the model')\n    parser.add_argument('save_every', type=int, help='How often to save a snapshot')\n    parser.add_argument('--batch_size', default=32, type=int, help='Input batch size on each device (default: 32)')\n    args = parser.parse_args()\n\n    main(args.save_every, args.total_epochs, args.batch_size)\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/multinode.py",
    "content": "import torch\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom datautils import MyTrainDataset\n\nimport torch.multiprocessing as mp\nfrom torch.utils.data.distributed import DistributedSampler\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom torch.distributed import init_process_group, destroy_process_group\nimport os\n\n\ndef ddp_setup():\n    torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))\n    init_process_group(backend=\"nccl\")\n\nclass Trainer:\n    def __init__(\n        self,\n        model: torch.nn.Module,\n        train_data: DataLoader,\n        optimizer: torch.optim.Optimizer,\n        save_every: int,\n        snapshot_path: str,\n    ) -> None:\n        self.local_rank = int(os.environ[\"LOCAL_RANK\"])\n        self.global_rank = int(os.environ[\"RANK\"])\n        self.model = model.to(self.local_rank)\n        self.train_data = train_data\n        self.optimizer = optimizer\n        self.save_every = save_every\n        self.epochs_run = 0\n        self.snapshot_path = snapshot_path\n        if os.path.exists(snapshot_path):\n            print(\"Loading snapshot\")\n            self._load_snapshot(snapshot_path)\n\n        self.model = DDP(self.model, device_ids=[self.local_rank])\n\n    def _load_snapshot(self, snapshot_path):\n        loc = f\"cuda:{self.local_rank}\"\n        snapshot = torch.load(snapshot_path, map_location=loc)\n        self.model.load_state_dict(snapshot[\"MODEL_STATE\"])\n        self.epochs_run = snapshot[\"EPOCHS_RUN\"]\n        print(f\"Resuming training from snapshot at Epoch {self.epochs_run}\")\n\n    def _run_batch(self, source, targets):\n        self.optimizer.zero_grad()\n        output = self.model(source)\n        loss = F.cross_entropy(output, targets)\n        loss.backward()\n        self.optimizer.step()\n\n    def _run_epoch(self, epoch):\n        b_sz = len(next(iter(self.train_data))[0])\n        print(f\"[GPU{self.global_rank}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(self.train_data)}\")\n        self.train_data.sampler.set_epoch(epoch)\n        for source, targets in self.train_data:\n            source = source.to(self.local_rank)\n            targets = targets.to(self.local_rank)\n            self._run_batch(source, targets)\n\n    def _save_snapshot(self, epoch):\n        snapshot = {\n            \"MODEL_STATE\": self.model.module.state_dict(),\n            \"EPOCHS_RUN\": epoch,\n        }\n        torch.save(snapshot, self.snapshot_path)\n        print(f\"Epoch {epoch} | Training snapshot saved at {self.snapshot_path}\")\n\n    def train(self, max_epochs: int):\n        for epoch in range(self.epochs_run, max_epochs):\n            self._run_epoch(epoch)\n            if self.global_rank == 0 and epoch % self.save_every == 0:\n                self._save_snapshot(epoch)\n\n\ndef load_train_objs():\n    train_set = MyTrainDataset(2048)  # load your dataset\n    model = torch.nn.Linear(20, 1)  # load your model\n    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)\n    return train_set, model, optimizer\n\n\ndef prepare_dataloader(dataset: Dataset, batch_size: int):\n    return DataLoader(\n        dataset,\n        batch_size=batch_size,\n        pin_memory=True,\n        shuffle=False,\n        sampler=DistributedSampler(dataset)\n    )\n\n\ndef main(save_every: int, total_epochs: int, batch_size: int, snapshot_path: str = \"snapshot.pt\"):\n    ddp_setup()\n    dataset, model, optimizer = load_train_objs()\n    train_data = prepare_dataloader(dataset, batch_size)\n    trainer = Trainer(model, train_data, optimizer, save_every, snapshot_path)\n    trainer.train(total_epochs)\n    destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    import argparse\n    parser = argparse.ArgumentParser(description='simple distributed training job')\n    parser.add_argument('total_epochs', type=int, help='Total epochs to train the model')\n    parser.add_argument('save_every', type=int, help='How often to save a snapshot')\n    parser.add_argument('--batch_size', default=32, type=int, help='Input batch size on each device (default: 32)')\n    args = parser.parse_args()\n\n    main(args.save_every, args.total_epochs, args.batch_size)\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/requirements.txt",
    "content": "torch>=1.11.0"
  },
  {
    "path": "distributed/ddp-tutorial-series/single_gpu.py",
    "content": "import torch\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom datautils import MyTrainDataset\n\n\nclass Trainer:\n    def __init__(\n        self,\n        model: torch.nn.Module,\n        train_data: DataLoader,\n        optimizer: torch.optim.Optimizer,\n        gpu_id: int,\n        save_every: int, \n    ) -> None:\n        self.gpu_id = gpu_id\n        self.model = model.to(gpu_id)\n        self.train_data = train_data\n        self.optimizer = optimizer\n        self.save_every = save_every\n\n    def _run_batch(self, source, targets):\n        self.optimizer.zero_grad()\n        output = self.model(source)\n        loss = F.cross_entropy(output, targets)\n        loss.backward()\n        self.optimizer.step()\n\n    def _run_epoch(self, epoch):\n        b_sz = len(next(iter(self.train_data))[0])\n        print(f\"[GPU{self.gpu_id}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(self.train_data)}\")\n        for source, targets in self.train_data:\n            source = source.to(self.gpu_id)\n            targets = targets.to(self.gpu_id)\n            self._run_batch(source, targets)\n\n    def _save_checkpoint(self, epoch):\n        ckp = self.model.state_dict()\n        PATH = \"checkpoint.pt\"\n        torch.save(ckp, PATH)\n        print(f\"Epoch {epoch} | Training checkpoint saved at {PATH}\")\n\n    def train(self, max_epochs: int):\n        for epoch in range(max_epochs):\n            self._run_epoch(epoch)\n            if epoch % self.save_every == 0:\n                self._save_checkpoint(epoch)\n\n\ndef load_train_objs():\n    train_set = MyTrainDataset(2048)  # load your dataset\n    model = torch.nn.Linear(20, 1)  # load your model\n    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)\n    return train_set, model, optimizer\n\n\ndef prepare_dataloader(dataset: Dataset, batch_size: int):\n    return DataLoader(\n        dataset,\n        batch_size=batch_size,\n        pin_memory=True,\n        shuffle=True\n    )\n\n\ndef main(device, total_epochs, save_every, batch_size):\n    dataset, model, optimizer = load_train_objs()\n    train_data = prepare_dataloader(dataset, batch_size)\n    trainer = Trainer(model, train_data, optimizer, device, save_every)\n    trainer.train(total_epochs)\n\n\nif __name__ == \"__main__\":\n    import argparse\n    parser = argparse.ArgumentParser(description='simple distributed training job')\n    parser.add_argument('total_epochs', type=int, help='Total epochs to train the model')\n    parser.add_argument('save_every', type=int, help='How often to save a snapshot')\n    parser.add_argument('--batch_size', default=32, type=int, help='Input batch size on each device (default: 32)')\n    args = parser.parse_args()\n    \n    device = 0  # shorthand for cuda:0\n    main(device, args.total_epochs, args.save_every, args.batch_size)\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/slurm/config.yaml.template",
    "content": "Region: us-east-1\n\nImage:\n  Os: ubuntu1804\n\nSharedStorage:\n  - MountDir: /shared\n    Name: shared-fs\n    StorageType: FsxLustre\n    FsxLustreSettings:\n      StorageCapacity: 1200\n      DeploymentType: SCRATCH_1\n      StorageType: SSD\n\nHeadNode:\n  InstanceType: c5.xlarge\n  Networking:\n    SubnetId: subnet-xxxxxxx\n  Ssh:\n    KeyName: your-keyname-file\n\nScheduling:\n  Scheduler: slurm\n  SlurmQueues:\n  - Name: train\n    ComputeResources:\n    - Name: p32xlarge\n      InstanceType: p3.2xlarge\n      MinCount: 0\n      MaxCount: 5\n    Networking:\n      SubnetIds:\n      - subnet-xxxxxxx\n"
  },
  {
    "path": "distributed/ddp-tutorial-series/slurm/sbatch_run.sh",
    "content": "#!/bin/bash\n\n#SBATCH --job-name=multinode-example\n#SBATCH --nodes=4\n#SBATCH --ntasks=4\n#SBATCH --gpus-per-task=1\n#SBATCH --cpus-per-task=4\n\nnodes=( $( scontrol show hostnames $SLURM_JOB_NODELIST ) )\nnodes_array=($nodes)\nhead_node=${nodes_array[0]}\nhead_node_ip=$(srun --nodes=1 --ntasks=1 -w \"$head_node\" hostname --ip-address)\n\necho Node IP: $head_node_ip\nexport LOGLEVEL=INFO\n\nsrun torchrun \\\n--nnodes 4 \\\n--nproc_per_node 1 \\\n--rdzv_id $RANDOM \\\n--rdzv_backend c10d \\\n--rdzv_endpoint $head_node_ip:29500 \\\n/shared/examples/multinode_torchrun.py 50 10"
  },
  {
    "path": "distributed/ddp-tutorial-series/slurm/setup_pcluster_slurm.md",
    "content": "# Setup AWS cluster with pcluster\n\n## 1. Sign in to an AWS instance\n\n## 2. Install pcluster\n```\npip3 install awscli -U --user\npip3 install \"aws-parallelcluster\" --upgrade --user\n```\n\n## 3. Create a cluster config file\n```\npcluster configure --config config.yaml\n```\nSee config.yaml.template for an example\n\n\n## 4. Create the cluster\n```\npcluster create-cluster --cluster-name dist-ml --cluster-configuration config.yaml\n```\n\n### 4a. Track progress\n```\npcluster list-clusters\n```\n\n## 5. Login to cluster headnode\n```\npcluster ssh --cluster-name dist-ml -i your-keyname-file\n```\n\n## 6. Install dependencies\n```\nsudo apt-get update\nsudo apt-get install -y python3-venv\npython3 -m venv /shared/venv/\nsource /shared/venv/bin/activate\npip install wheel\necho 'source /shared/venv/bin/activate' >> ~/.bashrc\n```\n\n## 7. Download training code and install requirements\n```\ncd /shared\ngit clone --depth 1 https://github.com/pytorch/examples;\ncd /shared/examples\ngit filter-branch --prune-empty --subdirectory-filter distributed/ddp-tutorial-series\npython3 -m pip install setuptools==59.5.0\npip install -r requirements.txt\n```"
  },
  {
    "path": "distributed/minGPT-ddp/README.md",
    "content": "# minGPT-DDP\n\nCode accompanying the tutorial at https://pytorch.org/tutorials/intermediate/ddp_series_minGPT.html for training a GPT-like model with Distributed Data Parallel (DDP) in PyTorch. \n\nFiles marked with an asterisk (*) are adapted from the minGPT repo (https://github.com/karpathy/minGPT). \n\n- [trainer.py](mingpt/trainer.py) includes the Trainer class that runs the distributed training iterations on the model with the provided dataset.\n- [model.py *](mingpt/model.py) defines the model architecture.\n- [char_dataset.py *](mingpt/char_dataset.py) contains the `Dataset`class for a character-level dataset.\n- [gpt2_train_cfg.yaml](mingpt/gpt2_train_cfg.yaml) contains the configurations for data, model, optimizer and training run.\n- [main.py](mingpt/main.py) is the entry point to the trainig job. It sets up the DDP process group, reads all the configurations and runs the training job.\n- [slurm/](mingpt/slurm) contains files for setting up an AWS cluster and the slurm script to run multinode training."
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/char_dataset.py",
    "content": "import torch\nfrom torch.utils.data import Dataset\nimport fsspec\nfrom dataclasses import dataclass\n\n\"\"\"\nAdapted from https://github.com/karpathy/minGPT/blob/master/projects/chargpt/chargpt.py\n\"\"\"\n\n@dataclass\nclass DataConfig:\n    path: str = None\n    block_size: int = None\n    train_split: float = None\n    truncate: float = 1.0\n\nclass CharDataset(Dataset):\n\n    def __init__(self, data_cfg: DataConfig): #data_path: str, block_size):\n        data = fsspec.open(data_cfg.path).open().read().decode('utf-8')\n        data = data[ : int(len(data) * data_cfg.truncate)]\n\n        chars = sorted(list(set(data)))\n        data_size, vocab_size = len(data), len(chars)\n        print('Data has %d characters, %d unique.' % (data_size, vocab_size))\n\n        self.stoi = {ch: i for i, ch in enumerate(chars)}\n        self.itos = {i: ch for i, ch in enumerate(chars)}\n        self.block_size = data_cfg.block_size\n        self.vocab_size = vocab_size\n        self.data = data\n\n    def __len__(self):\n        return len(self.data) - self.block_size\n\n    def __getitem__(self, idx):\n        # grab a chunk of (block_size + 1) characters from the data\n        chunk = self.data[idx:idx + self.block_size + 1]\n        # encode every character to an integer\n        dix = [self.stoi[s] for s in chunk]\n        x = torch.tensor(dix[:-1], dtype=torch.long)\n        y = torch.tensor(dix[1:], dtype=torch.long)\n        return x, y\n"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/gpt2_train_cfg.yaml",
    "content": "data_config:\n  path: https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n  block_size: 128\n  train_split: 0.9\n  truncate: 0.05\ngpt_config:\n  n_layer: 8\n  n_head: 8\n  n_embd: 512\ntrainer_config:\n  max_epochs: 10\n  batch_size: 216\n  data_loader_workers: 4\n  grad_norm_clip: 1.0\n  snapshot_path: gpt_snapshot.pt\n  save_every: 3\n  use_amp: True\noptimizer_config:\n  weight_decay: 0.1\n  learning_rate: 0.0003\n\nhydra:\n  run:\n    dir: ./\n"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/main.py",
    "content": "import os\nimport sys\nimport torch\nfrom torch.utils.data import random_split\nfrom torch.distributed import init_process_group, destroy_process_group\nfrom model import GPT, GPTConfig, OptimizerConfig, create_optimizer\nfrom trainer import Trainer, TrainerConfig\nfrom char_dataset import CharDataset, DataConfig\nfrom omegaconf import DictConfig\nimport hydra\n\ndef verify_min_gpu_count(min_gpus: int = 2) -> bool:\n    has_gpu = torch.accelerator.is_available()\n    gpu_count = torch.accelerator.device_count()\n    return has_gpu and gpu_count >= min_gpus\n\ndef ddp_setup():\n    acc = torch.accelerator.current_accelerator()\n    rank = int(os.environ[\"LOCAL_RANK\"])\n    device: torch.device = torch.device(f\"{acc}:{rank}\")\n    backend = torch.distributed.get_default_backend_for_device(device)\n    init_process_group(backend=backend)\n    torch.accelerator.set_device_index(rank)\n\ndef get_train_objs(gpt_cfg: GPTConfig, opt_cfg: OptimizerConfig, data_cfg: DataConfig):\n    dataset = CharDataset(data_cfg)\n    train_len = int(len(dataset) * data_cfg.train_split)\n    train_set, test_set = random_split(dataset, [train_len, len(dataset) - train_len])\n\n    gpt_cfg.vocab_size = dataset.vocab_size\n    gpt_cfg.block_size = dataset.block_size\n    model = GPT(gpt_cfg)\n    optimizer = create_optimizer(model, opt_cfg)\n    \n    return model, optimizer, train_set, test_set\n\n@hydra.main(version_base=None, config_path=\".\", config_name=\"gpt2_train_cfg\")\ndef main(cfg: DictConfig):\n    ddp_setup()\n\n    gpt_cfg = GPTConfig(**cfg['gpt_config'])\n    opt_cfg = OptimizerConfig(**cfg['optimizer_config'])\n    data_cfg = DataConfig(**cfg['data_config'])\n    trainer_cfg = TrainerConfig(**cfg['trainer_config'])\n\n    model, optimizer, train_data, test_data = get_train_objs(gpt_cfg, opt_cfg, data_cfg)\n    trainer = Trainer(trainer_cfg, model, optimizer, train_data, test_data)\n    trainer.train()\n\n    destroy_process_group()\n\nif __name__ == \"__main__\":\n    _min_gpu_count = 2\n    if not verify_min_gpu_count(min_gpus=_min_gpu_count):\n        print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n        sys.exit()\n    main()\n"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/model.py",
    "content": "\"\"\"\nFull definition of a GPT Language Model, all of it in this single file.\nAdapted from https://github.com/karpathy/minGPT/blob/master/mingpt/model.py\n\"\"\"\n\nfrom dataclasses import dataclass\nimport math\n\nimport torch\nimport torch.nn as nn\nfrom torch.nn import functional as F\n\n\n@dataclass\nclass GPTConfig:\n    model_type: str = 'gpt2'\n    # model configurations\n    n_layer: int = None\n    n_head: int = None\n    n_embd: int =  None\n    # openai's values for gpt2\n    vocab_size: int = 50257 \n    block_size: int = 1024\n    # dropout hyperparameters\n    embd_pdrop: float = 0.1\n    resid_pdrop: float = 0.1\n    attn_pdrop: float = 0.1\n\n@dataclass\nclass OptimizerConfig:\n    learning_rate: float = 3e-4\n    weight_decay: float = 0.1\n\nclass MultiheadAttentionLayer(nn.Module):\n    \"\"\"\n    A multi-head masked self-attention layer with a projection at the end.\n    \"\"\"\n\n    def __init__(self, config, device=\"cpu\", dtype=torch.float32):\n        super().__init__()\n        assert config.n_embd % config.n_head == 0\n        self.resid_drop = nn.Dropout(config.resid_pdrop)\n        self.c_proj = nn.Linear(config.n_embd, config.n_embd, device=device, dtype=dtype)\n        self.register_buffer(\"mask\", torch.tril(torch.ones(config.block_size, config.block_size))\n                             .view(1, 1, config.block_size, config.block_size))\n        self.attn = torch.nn.MultiheadAttention(\n            embed_dim=config.n_embd,\n            num_heads=config.n_head,\n            dropout=config.attn_pdrop,\n            batch_first=True,\n            device=device,\n            dtype=dtype\n        )\n\n    def forward(self, x):\n        _, seq_size, _ = x.size()\n        y = self.attn(x, x, x, attn_mask=self.mask[0, 0, :seq_size, :seq_size])[0]\n        y = self.resid_drop(self.c_proj(y))\n        return y\n\nclass Block(nn.Module):\n    \"\"\" an unassuming Transformer block \"\"\"\n    def __init__(self, config: GPTConfig):\n        super().__init__()\n        self.ln1 = nn.LayerNorm(config.n_embd)\n        self.ln2 = nn.LayerNorm(config.n_embd)\n        self.attn = MultiheadAttentionLayer(config)\n        self.mlp = nn.Sequential(\n            nn.Linear(config.n_embd, 4 * config.n_embd),\n            nn.GELU(),\n            nn.Linear(4 * config.n_embd, config.n_embd),\n            nn.Dropout(config.resid_pdrop),\n        )\n\n    def forward(self, x):\n        x = x + self.attn(self.ln1(x))\n        x = x + self.mlp(self.ln2(x))\n        return x\n\nclass EmbeddingStem(nn.Module):\n    def __init__(self, config: GPTConfig, device=\"cpu\", dtype=torch.float32):\n        super().__init__()\n        self.tok_emb = nn.Embedding(config.vocab_size, config.n_embd, device=device, dtype=dtype)\n        self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd, device=device, dtype=dtype))\n        self.drop = nn.Dropout(config.embd_pdrop)\n        self.block_size = config.block_size\n\n    def reset_parameters(self):\n        self.tok_emb.reset_parameters()\n\n    def forward(self, idx):\n        b, t = idx.size()\n        assert t <= self.block_size, f\"Cannot forward sequence of length {t}, block size is only {self.block_size}\"\n\n        token_embeddings = self.tok_emb(idx)  # each index maps to a (learnable) embedding vector\n        position_embeddings = self.pos_emb[:, :t, :]  # each position maps to a (learnable) position vector\n        return self.drop(token_embeddings + position_embeddings)\n        \nclass GPT(nn.Module):\n    \"\"\" GPT Language Model \"\"\"\n\n    def __init__(self, config: GPTConfig):\n        super().__init__()\n        self.block_size = config.block_size\n        config = self._set_model_config(config)\n\n        # input embedding stem\n        self.emb_stem = EmbeddingStem(config)\n        # transformer\n        self.blocks = nn.Sequential(*[Block(config) for _ in range(config.n_layer)])\n        # decoder head\n        self.ln_f = nn.LayerNorm(config.n_embd)\n        self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False)\n\n        # init all weights, and apply a special scaled init to the residual projections, per GPT-2 paper\n        self.apply(self._init_weights)\n        for pn, p in self.named_parameters():\n            if pn.endswith('c_proj.weight'):\n                p.data.normal_(mean=0.0, std=0.02/math.sqrt(2 * config.n_layer))\n\n        # report number of parameters (note we don't count the decoder parameters in lm_head)\n        n_params = sum(p.numel() for p in self.blocks.parameters())\n        print(\"number of parameters: %.2fM\" % (n_params/1e6,))\n\n    def _set_model_config(self, config):\n        type_given = config.model_type is not None\n        params_given = all([config.n_layer is not None, config.n_head is not None, config.n_embd is not None])\n        # assert type_given ^ params_given # exactly one of these (XOR)\n        if type_given and not params_given:\n            # translate from model_type to detailed configuration\n            config.__dict__.update({\n                # names follow the huggingface naming conventions\n                # GPT-1\n                'openai-gpt':   dict(n_layer=12, n_head=12, n_embd=768),  # 117M params\n                # GPT-2 configs\n                'gpt2':         dict(n_layer=12, n_head=12, n_embd=768),  # 124M params\n                'gpt2-medium':  dict(n_layer=24, n_head=16, n_embd=1024), # 350M params\n                'gpt2-large':   dict(n_layer=36, n_head=20, n_embd=1280), # 774M params\n                'gpt2-xl':      dict(n_layer=48, n_head=25, n_embd=1600), # 1558M params\n                # Gophers\n                'gopher-44m':   dict(n_layer=8, n_head=16, n_embd=512),\n                # (there are a number more...)\n                # I made these tiny models up\n                'gpt-mini':     dict(n_layer=6, n_head=6, n_embd=192),\n                'gpt-micro':    dict(n_layer=4, n_head=4, n_embd=128),\n                'gpt-nano':     dict(n_layer=3, n_head=3, n_embd=48),\n            }[config.model_type])\n        return config\n    \n    def _init_weights(self, module):\n        if isinstance(module, (nn.Linear, nn.Embedding)):\n            module.weight.data.normal_(mean=0.0, std=0.02)\n            if isinstance(module, nn.Linear) and module.bias is not None:\n                module.bias.data.zero_()\n        elif isinstance(module, nn.LayerNorm):\n            module.bias.data.zero_()\n            module.weight.data.fill_(1.0)\n\n    def forward(self, idx, targets=None):\n        x = self.emb_stem(idx)\n        x = self.blocks(x)\n        x = self.ln_f(x)\n        logits = self.head(x)\n\n        # if we are given some desired targets also calculate the loss\n        loss = None\n        if targets is not None:\n            loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1)\n\n        return logits, loss\n\n    @torch.no_grad()\n    def generate(self, idx, max_new_tokens, temperature=1.0, do_sample=False, top_k=None):\n        \"\"\"\n        Take a conditioning sequence of indices idx (LongTensor of shape (b,t)) and complete\n        the sequence max_new_tokens times, feeding the predictions back into the model each time.\n        Most likely you'll want to make sure to be in model.eval() mode of operation for this.\n        \"\"\"\n        for _ in range(max_new_tokens):\n            # if the sequence context is growing too long we must crop it at block_size\n            idx_cond = idx if idx.size(1) <= self.block_size else idx[:, -self.block_size:]\n            # forward the model to get the logits for the index in the sequence\n            logits, _ = self(idx_cond)\n            # pluck the logits at the final step and scale by desired temperature\n            logits = logits[:, -1, :] / temperature\n            # optionally crop the logits to only the top k options\n            if top_k is not None:\n                v, _ = torch.topk(logits, top_k)\n                logits[logits < v[:, [-1]]] = -float('Inf')\n            # apply softmax to convert logits to (normalized) probabilities\n            probs = F.softmax(logits, dim=-1)\n            # either sample from the distribution or take the most likely element\n            if do_sample:\n                idx_next = torch.multinomial(probs, num_samples=1)\n            else:\n                _, idx_next = torch.topk(probs, k=1, dim=-1)\n            # append sampled index to the running sequence and continue\n            idx = torch.cat((idx, idx_next), dim=1)\n\n        return idx\n\n\ndef create_optimizer(model: torch.nn.Module, opt_config: OptimizerConfig):\n    \"\"\"\n    This long function is unfortunately doing something very simple and is being very defensive:\n    We are separating out all parameters of the model into two buckets: those that will experience\n    weight decay for regularization and those that won't (biases, and layernorm/embedding weights).\n    We are then returning the PyTorch optimizer object.\n    \"\"\"\n\n    # separate out all parameters to those that will and won't experience regularizing weight decay\n    decay = set()\n    no_decay = set()\n    whitelist_weight_modules = (torch.nn.Linear, )\n    blacklist_weight_modules = (torch.nn.LayerNorm, torch.nn.Embedding)\n    for mn, m in model.named_modules():\n        for pn, p in m.named_parameters():\n            fpn = '%s.%s' % (mn, pn) if mn else pn # full param name\n            # random note: because named_modules and named_parameters are recursive\n            # we will see the same tensors p many many times. but doing it this way\n            # allows us to know which parent module any tensor p belongs to...\n            if pn.endswith('bias'):\n                # all biases will not be decayed\n                no_decay.add(fpn)\n            elif pn.endswith('weight') and isinstance(m, whitelist_weight_modules):\n                # weights of whitelist modules will be weight decayed\n                decay.add(fpn)\n            elif pn.endswith('in_proj_weight'):\n                # MHA projection layer\n                decay.add(fpn)\n            elif pn.endswith('weight') and isinstance(m, blacklist_weight_modules):\n                # weights of blacklist modules will NOT be weight decayed\n                no_decay.add(fpn)\n            elif pn.endswith('pos_emb'):\n                # positional embedding shouldn't be decayed\n                no_decay.add(fpn)\n\n    # validate that we considered every parameter\n    param_dict = {pn: p for pn, p in model.named_parameters()}\n    inter_params = decay & no_decay\n    union_params = decay | no_decay\n    assert len(inter_params) == 0, \"parameters %s made it into both decay/no_decay sets!\" % (str(inter_params), )\n    assert len(param_dict.keys() - union_params) == 0, \"parameters %s were not separated into either decay/no_decay set!\" \\\n                                                % (str(param_dict.keys() - union_params), )\n\n    # create the pytorch optimizer object\n    optim_groups = [\n        {\"params\": [param_dict[pn] for pn in sorted(list(decay))], \"weight_decay\": opt_config.weight_decay},\n        {\"params\": [param_dict[pn] for pn in sorted(list(no_decay))], \"weight_decay\": 0.0},\n    ]\n    optimizer = torch.optim.AdamW(optim_groups, lr=opt_config.learning_rate, betas=(0.9, 0.95))\n    return optimizer"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/slurm/config.yaml.template",
    "content": "Region: us-east-1\n\nImage:\n  Os: ubuntu1804\n\nSharedStorage:\n  - MountDir: /shared\n    Name: shared-fs\n    StorageType: FsxLustre\n    FsxLustreSettings:\n      StorageCapacity: 1200\n      DeploymentType: SCRATCH_1\n      StorageType: SSD\n\nHeadNode:\n  InstanceType: c5.xlarge\n  Networking:\n    SubnetId: subnet-xxxxxxx\n  Ssh:\n    KeyName: your-keyname-file\n\nScheduling:\n  Scheduler: slurm\n  SlurmQueues:\n  - Name: train\n    ComputeResources:\n    - Name: p32xlarge\n      InstanceType: p3.2xlarge\n      MinCount: 0\n      MaxCount: 5\n    Networking:\n      SubnetIds:\n      - subnet-xxxxxxx\n"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/slurm/sbatch_run.sh",
    "content": "#!/bin/bash\n\n#SBATCH --job-name=multinode-example\n#SBATCH --nodes=2\n#SBATCH --ntasks=2\n#SBATCH --gpus-per-task=1\n#SBATCH --cpus-per-task=4\n\nnodes=( $( scontrol show hostnames $SLURM_JOB_NODELIST ) )\nnodes_array=($nodes)\nhead_node=${nodes_array[0]}\nhead_node_ip=$(srun --nodes=1 --ntasks=1 -w \"$head_node\" hostname --ip-address)\n\necho Node IP: $head_node_ip\nexport LOGLEVEL=INFO\n\nsrun torchrun \\\n--nnodes 2 \\\n--nproc_per_node 1 \\\n--rdzv_id $RANDOM \\\n--rdzv_backend c10d \\\n--rdzv_endpoint $head_node_ip:29500 \\\n/shared/examples/mingpt/main.py\n\n\n"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/slurm/setup_pcluster_slurm.md",
    "content": "# Setup AWS cluster with pcluster\nRefer https://www.hpcworkshops.com/04-pcluster-cli.html\n\n## 1. Sign in to an AWS instance\n\n## 2. Install pcluster\n```\npip3 install awscli -U --user\npip3 install \"aws-parallelcluster\" --upgrade --user\n```\n\n## 3. Create a cluster config file\n```\npcluster configure --config config.yaml\n```\nSee config.yaml.template for an example. Ensure you have a valid EC2 key-pair file\n\n\n## 4. Create the cluster\n```\npcluster create-cluster --cluster-name dist-ml --cluster-configuration config.yaml\n```\n\n### 4a. Track progress\n```\npcluster list-clusters\n```\n\n## 5. Login to cluster headnode\n```\npcluster ssh --cluster-name dist-ml -i your-keypair-file\n```\n\n## 6. Install dependencies\n```\nsudo apt-get update\nsudo apt-get install -y python3-venv\npython3 -m venv /shared/venv/\nsource /shared/venv/bin/activate\npip install wheel\necho 'source /shared/venv/bin/activate' >> ~/.bashrc\n```\n\n## 7. Download training code and install requirements\n```\ncd /shared\ngit clone --depth 1 https://github.com/pytorch/examples;\ncd /shared/examples\ngit filter-branch --prune-empty --subdirectory-filter distributed/minGPT-ddp\npython3 -m pip install setuptools==59.5.0\npip install -r requirements.txt\n```\n"
  },
  {
    "path": "distributed/minGPT-ddp/mingpt/trainer.py",
    "content": "\"\"\"\nSimple training loop; Boilerplate that could apply to any arbitrary neural network,\nso nothing in this file really has anything to do with GPT specifically.\n\"\"\"\n\nfrom dataclasses import dataclass, asdict\nfrom collections import OrderedDict\nfrom typing import Optional, Any, Dict\nimport os\n\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom torch.utils.data.distributed import DistributedSampler\n\nimport boto3\nfrom urllib.parse import urlparse\nimport fsspec\nimport io\n\n@dataclass\nclass TrainerConfig:\n    max_epochs: int = None\n    batch_size: int = None\n    data_loader_workers: int = None\n    grad_norm_clip: float = None\n    snapshot_path: Optional[str] = None\n    save_every: int = None\n    use_amp: bool = None\n\n@dataclass\nclass Snapshot:\n    model_state: 'OrderedDict[str, torch.Tensor]'\n    optimizer_state: Dict[str, Any]\n    finished_epoch: int\n\ndef upload_to_s3(obj, dst):\n    buffer = io.BytesIO()\n    torch.save(obj, buffer)\n    buffer.seek(0)\n    dst = urlparse(dst, allow_fragments=False)\n    boto3.client('s3').upload_fileobj(buffer, dst.netloc, dst.path.lstrip('/'))\n\nclass Trainer:\n\n    def __init__(self, trainer_config: TrainerConfig, model, optimizer, train_dataset, test_dataset=None):\n        self.config = trainer_config\n        # set torchrun variables\n        self.local_rank = int(os.environ[\"LOCAL_RANK\"])\n        self.global_rank = int(os.environ[\"RANK\"])  \n        # set device\n        self.acc = torch.accelerator.current_accelerator()\n        self.device: torch.device = torch.device(f\"{self.acc}:{self.local_rank}\")\n        self.device_type = self.device.type\n        # data stuff\n        self.train_dataset = train_dataset\n        self.train_loader = self._prepare_dataloader(train_dataset)\n        self.test_loader = self._prepare_dataloader(test_dataset) if test_dataset else None\n        # initialize train states\n        self.epochs_run = 0\n        self.model = model.to(self.local_rank)\n        self.optimizer = optimizer        \n        self.save_every = self.config.save_every\n        if self.config.use_amp:\n            self.scaler = torch.amp.GradScaler(self.device_type)\n        # load snapshot if available. only necessary on the first node.\n        if self.config.snapshot_path is None:\n            self.config.snapshot_path = \"snapshot.pt\"\n        self._load_snapshot()\n        # wrap with DDP. this step will synch model across all the processes.\n        self.model = DDP(self.model, device_ids=[self.local_rank])\n        \n    def _prepare_dataloader(self, dataset: Dataset):\n        return DataLoader(\n            dataset,\n            batch_size=self.config.batch_size,\n            pin_memory=True,\n            shuffle=False,\n            num_workers=self.config.data_loader_workers,\n            sampler=DistributedSampler(dataset)\n        )\n\n    def _load_snapshot(self):\n        try:\n            snapshot = fsspec.open(self.config.snapshot_path)\n            with snapshot as f:\n                snapshot_data = torch.load(f, map_location=\"cpu\")\n        except FileNotFoundError:\n            print(\"Snapshot not found. Training model from scratch\")\n            return \n\n        snapshot = Snapshot(**snapshot_data)\n        self.model.load_state_dict(snapshot.model_state)\n        self.optimizer.load_state_dict(snapshot.optimizer_state)\n        self.epochs_run = snapshot.finished_epoch\n        print(f\"Resuming training from snapshot at Epoch {self.epochs_run}\")\n\n\n    def _run_batch(self, source, targets, train: bool = True) -> float:\n        with torch.set_grad_enabled(train), torch.amp.autocast(device_type=self.device_type, dtype=torch.float16, enabled=(self.config.use_amp)):\n            _, loss = self.model(source, targets)\n        \n        if train:\n            self.optimizer.zero_grad(set_to_none=True)\n            if self.config.use_amp: \n                self.scaler.scale(loss).backward()\n                torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm_clip)\n                self.scaler.step(self.optimizer)\n                self.scaler.update()\n            else:\n                loss.backward()\n                torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm_clip)\n                self.optimizer.step()\n        \n        return loss.item()\n\n    def _run_epoch(self, epoch: int, dataloader: DataLoader, train: bool = True):\n        if train:\n            dataloader.sampler.set_epoch(epoch)\n        for iter, (source, targets) in enumerate(dataloader):\n            step_type = \"Train\" if train else \"Eval\"\n            source = source.to(self.local_rank)\n            targets = targets.to(self.local_rank)\n            batch_loss = self._run_batch(source, targets, train)\n            if iter % 100 == 0:\n                print(f\"[RANK{self.global_rank}] Epoch {epoch} | Iter {iter} | {step_type} Loss {batch_loss:.5f}\")\n\n    def _save_snapshot(self, epoch):\n        # capture snapshot\n        model = self.model\n        raw_model = model.module if hasattr(model, \"module\") else model\n        snapshot = Snapshot(\n            model_state=raw_model.state_dict(),\n            optimizer_state=self.optimizer.state_dict(),\n            finished_epoch=epoch\n        )\n        # save snapshot\n        snapshot = asdict(snapshot)\n        if self.config.snapshot_path.startswith(\"s3://\"):\n            upload_to_s3(snapshot, self.config.snapshot_path)\n        else:\n            torch.save(snapshot, self.config.snapshot_path)\n            \n        print(f\"Snapshot saved at epoch {epoch}\")\n\n    def train(self):\n        for epoch in range(self.epochs_run, self.config.max_epochs):\n            epoch += 1\n            self._run_epoch(epoch, self.train_loader, train=True)\n            if self.local_rank == 0 and epoch % self.save_every == 0:\n                self._save_snapshot(epoch)\n            # eval run\n            if self.test_loader:\n                self._run_epoch(epoch, self.test_loader, train=False)\n"
  },
  {
    "path": "distributed/minGPT-ddp/requirements.txt",
    "content": "torch>=2.7\nboto3\nhydra-core\nrequests\naiohttp\n"
  },
  {
    "path": "distributed/minGPT-ddp/run_example.sh",
    "content": "#!/bin/bash\n# Usage: bash run_example.sh {file_to_run.py} {num_gpus}\n# where file_to_run = example to run. Default = 'main.py'\n# num_gpus = num local gpus to use. Default = 16\n\n# samples to run include:\n# main.py\n\necho \"Launching ${1:-main.py} with ${2:-16} gpus\"\ntorchrun --standalone --nproc_per_node=${2:-16} ${1:-main.py}\n"
  },
  {
    "path": "distributed/rpc/batch/README.md",
    "content": "# Examples For Asynchronous RPC User Functions\n\nThis folder contains two examples for [`@rpc.functions.async_execution`](https://pytorch.org/docs/master/rpc.html#torch.distributed.rpc.functions.async_execution):\n\n1. Synchronized Batch Update Parameter Server: uses `@rpc.functions.async_execution`\n   for parameter update and retrieving. This serves as a simple starter example\n   for batch RPC.\n   ```\n   pip install -r requirements.txt\n   python parameter_server.py\n   ```\n2. Multi-Observer with Batch-Processing Agent: uses `@rpc.functions.async_execution`\n   to run multiple observed states through the policy to get actions.\n   ```\n   pip install -r requirements.txt\n   python reinforce.py\n   ```\n"
  },
  {
    "path": "distributed/rpc/batch/parameter_server.py",
    "content": "import os\nimport threading\nfrom datetime import datetime\nimport warnings\n\nimport torch\nimport torch.distributed as dist\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.nn as nn\nfrom torch import optim\n\nimport torchvision\n\n# Suppress deprecated ProcessGroup warning\nwarnings.filterwarnings(\"ignore\", message=\"You are using a Backend.*ProcessGroup\")\n\n\nbatch_size = 20\nimage_w = 64\nimage_h = 64\nnum_classes = 30\nbatch_update_size = 5\nnum_batches = 6\n\n\ndef timed_log(text):\n    print(f\"{datetime.now().strftime('%H:%M:%S')} {text}\")\n\n\nclass BatchUpdateParameterServer(object):\n\n    def __init__(self, batch_update_size=batch_update_size):\n        self.model = torchvision.models.resnet50(num_classes=num_classes)\n        self.lock = threading.Lock()\n        self.future_model = torch.futures.Future()\n        self.batch_update_size = batch_update_size\n        self.curr_update_size = 0\n        self.optimizer = optim.SGD(self.model.parameters(), lr=0.001, momentum=0.9)\n        for p in self.model.parameters():\n            p.grad = torch.zeros_like(p)\n\n    def get_model(self):\n        return self.model\n\n    @staticmethod\n    @rpc.functions.async_execution\n    def update_and_fetch_model(ps_rref, grads):\n        self = ps_rref.local_value()\n        timed_log(f\"PS got {self.curr_update_size}/{batch_update_size} updates\")\n        for p, g in zip(self.model.parameters(), grads):\n            p.grad += g\n        with self.lock:\n            self.curr_update_size += 1\n            fut = self.future_model\n\n            if self.curr_update_size >= self.batch_update_size:\n                for p in self.model.parameters():\n                    p.grad /= self.batch_update_size\n                self.curr_update_size = 0\n                self.optimizer.step()\n                self.optimizer.zero_grad(set_to_none=False)\n                fut.set_result(self.model)\n                timed_log(\"PS updated model\")\n                self.future_model = torch.futures.Future()\n\n        return fut\n\n\nclass Trainer(object):\n\n    def __init__(self, ps_rref):\n        self.ps_rref = ps_rref\n        self.loss_fn = nn.MSELoss()\n        self.one_hot_indices = torch.LongTensor(batch_size) \\\n                                    .random_(0, num_classes) \\\n                                    .view(batch_size, 1)\n\n    def get_next_batch(self):\n        for _ in range(num_batches):\n            inputs = torch.randn(batch_size, 3, image_w, image_h)\n            labels = torch.zeros(batch_size, num_classes) \\\n                        .scatter_(1, self.one_hot_indices, 1)\n            yield inputs.cuda(), labels.cuda()\n\n    def train(self):\n        name = rpc.get_worker_info().name\n        m = self.ps_rref.rpc_sync().get_model().cuda()\n        for inputs, labels in self.get_next_batch():\n            timed_log(f\"{name} processing one batch\")\n            self.loss_fn(m(inputs), labels).backward()\n            timed_log(f\"{name} reporting grads\")\n            m = rpc.rpc_sync(\n                self.ps_rref.owner(),\n                BatchUpdateParameterServer.update_and_fetch_model,\n                args=(self.ps_rref, [p.grad for p in m.cpu().parameters()]),\n            ).cuda()\n            timed_log(f\"{name} got updated model\")\n\n\ndef run_trainer(ps_rref):\n    trainer = Trainer(ps_rref)\n    trainer.train()\n\n\ndef run_ps(trainers):\n    timed_log(\"Start training\")\n    ps_rref = rpc.RRef(BatchUpdateParameterServer())\n    futs = []\n    for trainer in trainers:\n        futs.append(\n            rpc.rpc_async(trainer, run_trainer, args=(ps_rref,))\n        )\n\n    torch.futures.wait_all(futs)\n    timed_log(\"Finish training\")\n\n\ndef run(rank, world_size):\n    os.environ['MASTER_ADDR'] = 'localhost'\n    os.environ['MASTER_PORT'] = '29500'\n    \n    # Initialize the process group first\n    dist.init_process_group(\n        backend=\"gloo\",\n        rank=rank,\n        world_size=world_size\n    )\n    \n    options=rpc.TensorPipeRpcBackendOptions(\n        num_worker_threads=16,\n        rpc_timeout=60\n     )\n    if rank != 0:\n        rpc.init_rpc(\n            f\"trainer{rank}\",\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=options\n        )\n        # trainer passively waiting for ps to kick off training iterations\n    else:\n        rpc.init_rpc(\n            \"ps\",\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=options\n        )\n        run_ps([f\"trainer{r}\" for r in range(1, world_size)])\n\n    # block until all rpcs finish\n    rpc.shutdown()\n    dist.destroy_process_group()\n\n\nif __name__==\"__main__\":\n    world_size = batch_update_size + 1\n    mp.spawn(run, args=(world_size, ), nprocs=world_size, join=True)\n"
  },
  {
    "path": "distributed/rpc/batch/reinforce.py",
    "content": "import argparse\nimport gymnasium as gym\nimport os\nimport threading\nimport time\nimport warnings\n\n# Suppress deprecated ProcessGroup warning\nwarnings.filterwarnings(\"ignore\", message=\"You are using a Backend.*ProcessGroup\")\n\nimport torch\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.distributed.rpc import RRef, rpc_sync, rpc_async, remote\nfrom torch.distributions import Categorical\n\n# demonstrating using rpc.functions.async_execution to speed up training\n\nNUM_STEPS = 500\nAGENT_NAME = \"agent\"\nOBSERVER_NAME = \"observer{}\"\n\nparser = argparse.ArgumentParser(description='PyTorch RPC Batch RL example')\nparser.add_argument('--gamma', type=float, default=1.0, metavar='G',\n                    help='discount factor (default: 1.0)')\nparser.add_argument('--seed', type=int, default=543, metavar='N',\n                    help='random seed (default: 543)')\nparser.add_argument('--num-episode', type=int, default=10, metavar='E',\n                    help='number of episodes (default: 10)')\nparser.add_argument('--max-world-size', type=int, default=3, metavar='W',\n                    help='maximum world size to test (default: 3)')\nargs = parser.parse_args()\n\ntorch.manual_seed(args.seed)\n\n\nclass Policy(nn.Module):\n    r\"\"\"\n    Borrowing the ``Policy`` class from the Reinforcement Learning example.\n    Copying the code to make these two examples independent.\n    See https://github.com/pytorch/examples/tree/main/reinforcement_learning\n    \"\"\"\n    def __init__(self, batch=True):\n        super(Policy, self).__init__()\n        self.affine1 = nn.Linear(4, 128)\n        self.dropout = nn.Dropout(p=0.6)\n        self.affine2 = nn.Linear(128, 2)\n        self.dim = 2 if batch else 1\n\n    def forward(self, x):\n        x = self.affine1(x)\n        x = self.dropout(x)\n        x = F.relu(x)\n        action_scores = self.affine2(x)\n        return F.softmax(action_scores, dim=self.dim)\n\n\nclass Observer:\n    r\"\"\"\n    An observer has exclusive access to its own environment. Each observer\n    captures the state from its environment, and send the state to the agent to\n    select an action. Then, the observer applies the action to its environment\n    and reports the reward to the agent.\n\n    It is true that CartPole-v1 is a relatively inexpensive environment, and it\n    might be an overkill to use RPC to connect observers and trainers in this\n    specific use case. However, the main goal of this tutorial to how to build\n    an application using the RPC API. Developers can extend the similar idea to\n    other applications with much more expensive environment.\n    \"\"\"\n    def __init__(self, batch=True):\n        self.id = rpc.get_worker_info().id - 1\n        self.env = gym.make('CartPole-v1')\n        self.env.reset(seed=args.seed)\n        self.select_action = Agent.select_action_batch if batch else Agent.select_action\n\n    def run_episode(self, agent_rref, n_steps):\n        r\"\"\"\n        Run one episode of n_steps.\n\n        Args:\n            agent_rref (RRef): an RRef referencing the agent object.\n            n_steps (int): number of steps in this episode\n        \"\"\"\n        state, _ = self.env.reset()\n        ep_reward = NUM_STEPS\n        rewards = torch.zeros(n_steps)\n        start_step = 0\n        for step in range(n_steps):\n            state = torch.from_numpy(state).float().unsqueeze(0)\n            # send the state to the agent to get an action\n            action = rpc.rpc_sync(\n                agent_rref.owner(),\n                self.select_action,\n                args=(agent_rref, self.id, state)\n            )\n\n            # apply the action to the environment, and get the reward\n            state, reward, terminated, truncated, _ = self.env.step(action)\n            rewards[step] = reward\n\n            if terminated or truncated or step + 1 >= n_steps:\n                curr_rewards = rewards[start_step:(step + 1)]\n                R = 0\n                for i in range(curr_rewards.numel() -1, -1, -1):\n                    R = curr_rewards[i] + args.gamma * R\n                    curr_rewards[i] = R\n                state, _ = self.env.reset()\n                if start_step == 0:\n                    ep_reward = min(ep_reward, step - start_step + 1)\n                start_step = step + 1\n\n        return [rewards, ep_reward]\n\n\nclass Agent:\n    def __init__(self, world_size, batch=True):\n        self.ob_rrefs = []\n        self.agent_rref = RRef(self)\n        self.rewards = {}\n        self.policy = Policy(batch).cuda()\n        self.optimizer = optim.Adam(self.policy.parameters(), lr=1e-2)\n        self.running_reward = 0\n\n        for ob_rank in range(1, world_size):\n            ob_info = rpc.get_worker_info(OBSERVER_NAME.format(ob_rank))\n            self.ob_rrefs.append(remote(ob_info, Observer, args=(batch,)))\n            self.rewards[ob_info.id] = []\n\n        self.states = torch.zeros(len(self.ob_rrefs), 1, 4)\n        self.batch = batch\n        # With batching, saved_log_probs contains a list of tensors, where each\n        # tensor contains probs from all observers in one step.\n        # Without batching, saved_log_probs is a dictionary where the key is the\n        # observer id and the value is a list of probs for that observer.\n        self.saved_log_probs = [] if self.batch else {k:[] for k in range(len(self.ob_rrefs))}\n        self.future_actions = torch.futures.Future()\n        self.lock = threading.Lock()\n        self.pending_states = len(self.ob_rrefs)\n\n    @staticmethod\n    @rpc.functions.async_execution\n    def select_action_batch(agent_rref, ob_id, state):\n        r\"\"\"\n        Batching select_action: In each step, the agent waits for states from\n        all observers, and process them together. This helps to reduce the\n        number of CUDA kernels launched and hence speed up amortized inference\n        speed.\n        \"\"\"\n        self = agent_rref.local_value()\n        self.states[ob_id].copy_(state)\n        future_action = self.future_actions.then(\n            lambda future_actions: future_actions.wait()[ob_id].item()\n        )\n\n        with self.lock:\n            self.pending_states -= 1\n            if self.pending_states == 0:\n                self.pending_states = len(self.ob_rrefs)\n                probs = self.policy(self.states.cuda())\n                m = Categorical(probs)\n                actions = m.sample()\n                self.saved_log_probs.append(m.log_prob(actions).t()[0])\n                future_actions = self.future_actions\n                self.future_actions = torch.futures.Future()\n                future_actions.set_result(actions.cpu())\n        return future_action\n\n    @staticmethod\n    def select_action(agent_rref, ob_id, state):\n        r\"\"\"\n        Non-batching select_action, return the action right away.\n        \"\"\"\n        self = agent_rref.local_value()\n        probs = self.policy(state.cuda())\n        m = Categorical(probs)\n        action = m.sample()\n        self.saved_log_probs[ob_id].append(m.log_prob(action))\n        return action.item()\n\n    def run_episode(self, n_steps=0):\n        r\"\"\"\n        Run one episode. The agent will tell each oberser to run one episode\n        with n_steps. Then it collects all actions and rewards, and use those to\n        train the policy.\n        \"\"\"\n        futs = []\n        for ob_rref in self.ob_rrefs:\n            # make async RPC to kick off an episode on all observers\n            futs.append(ob_rref.rpc_async().run_episode(self.agent_rref, n_steps))\n\n        # wait until all obervers have finished this episode\n        rets = torch.futures.wait_all(futs)\n        rewards = torch.stack([ret[0] for ret in rets]).cuda().t()\n        ep_rewards = sum([ret[1] for ret in rets]) / len(rets)\n\n        if self.batch:\n            probs = torch.stack(self.saved_log_probs)\n        else:\n            probs = [torch.stack(self.saved_log_probs[i]) for i in range(len(rets))]\n            probs = torch.stack(probs)\n\n        policy_loss = -probs * rewards / len(rets)\n        policy_loss.sum().backward()\n        self.optimizer.step()\n        self.optimizer.zero_grad()\n\n        # reset variables\n        self.saved_log_probs = [] if self.batch else {k:[] for k in range(len(self.ob_rrefs))}\n        self.states = torch.zeros(len(self.ob_rrefs), 1, 4)\n\n        # calculate running rewards\n        self.running_reward = 0.5 * ep_rewards + 0.5 * self.running_reward\n        return ep_rewards, self.running_reward\n\n\ndef run_worker(rank, world_size, n_episode, batch, print_log=True):\n    r\"\"\"\n    This is the entry point for all processes. The rank 0 is the agent. All\n    other ranks are observers.\n    \"\"\"\n    os.environ['MASTER_ADDR'] = 'localhost'\n    os.environ['MASTER_PORT'] = '29500'\n    if rank == 0:\n        # rank0 is the agent\n        rpc.init_rpc(AGENT_NAME, rank=rank, world_size=world_size)\n\n        agent = Agent(world_size, batch)\n        for i_episode in range(n_episode):\n            last_reward, running_reward = agent.run_episode(n_steps=NUM_STEPS)\n\n            if print_log:\n                print(f'Episode {i_episode}\\tLast reward: {last_reward:.2f}\\tAverage reward: {running_reward:.2f}')\n    else:\n        # other ranks are the observer\n        rpc.init_rpc(OBSERVER_NAME.format(rank), rank=rank, world_size=world_size)\n        # observers passively waiting for instructions from agents\n    rpc.shutdown()\n\n\ndef main():\n    for world_size in range(2, args.max_world_size):\n        delays = []\n        for batch in [True, False]:\n            tik = time.time()\n            mp.spawn(\n                run_worker,\n                args=(world_size, args.num_episode, batch),\n                nprocs=world_size,\n                join=True\n            )\n            tok = time.time()\n            delays.append(tok - tik)\n\n        print(f\"{world_size}, {delays[0]}, {delays[1]}\")\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "distributed/rpc/batch/requirements.txt",
    "content": "torch\ntorchvision\nnumpy\ngymnasium\n"
  },
  {
    "path": "distributed/rpc/ddp_rpc/README.md",
    "content": "Distributed DataParallel + Distributed RPC Framework Example\n\nThis example demonstrates how to combine Distributed DataParallel (DDP) with the Distributed RPC Framework. It requires two trainer nodes (each with a GPU), one master node, and one parameter server.\n\nThe master node initializes an embedding table on the parameter server and orchestrates the training loop across the trainers. The model is composed of a dense component (`nn.Linear`), which is replicated on the trainers using DDP, and a sparse component (`nn.EmbeddingBag`), which resides on the parameter server.\n\nEach trainer performs embedding lookups on the parameter server via RPC, then processes the results through its local `nn.Linear` module. During the backward pass, DDP aggregates gradients for the dense part using allreduce, while the distributed backward pass updates the embedding table parameters on the parameter server.\n\n\n```\npip install -r requirements.txt\npython main.py\n```\n"
  },
  {
    "path": "distributed/rpc/ddp_rpc/main.py",
    "content": "import random\n\nimport torch\nimport torch.distributed as dist\nimport torch.distributed.autograd as dist_autograd\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.optim as optim\nfrom torch.distributed.nn import RemoteModule\nfrom torch.distributed.optim import DistributedOptimizer\nfrom torch.distributed.rpc import RRef\nfrom torch.distributed.rpc import TensorPipeRpcBackendOptions\nfrom torch.nn.parallel import DistributedDataParallel as DDP\n\nNUM_EMBEDDINGS = 100\nEMBEDDING_DIM = 16\n\ndef verify_min_gpu_count(min_gpus: int = 2) -> bool:\n    \"\"\" verification that we have at least 2 gpus to run dist examples \"\"\"\n    has_gpu = torch.accelerator.is_available()\n    gpu_count = torch.accelerator.device_count()\n    return has_gpu and gpu_count >= min_gpus\n\nclass HybridModel(torch.nn.Module):\n    r\"\"\"\n    The model consists of a sparse part and a dense part.\n    1) The dense part is an nn.Linear module that is replicated across all trainers using DistributedDataParallel.\n    2) The sparse part is a Remote Module that holds an nn.EmbeddingBag on the parameter server.\n    This remote model can get a Remote Reference to the embedding table on the parameter server.\n    \"\"\"\n\n    def __init__(self, remote_emb_module, rank):\n        super(HybridModel, self).__init__()\n        self.remote_emb_module = remote_emb_module\n        self.fc = DDP(torch.nn.Linear(16, 8).to(rank))\n        self.rank = rank\n\n    def forward(self, indices, offsets):\n        emb_lookup = self.remote_emb_module.forward(indices, offsets)\n        return self.fc(emb_lookup.to(self.rank))\n\n\ndef _run_trainer(remote_emb_module, rank):\n    r\"\"\"\n    Each trainer runs a forward pass which involves an embedding lookup on the\n    parameter server and running nn.Linear locally. During the backward pass,\n    DDP is responsible for aggregating the gradients for the dense part\n    (nn.Linear) and distributed autograd ensures gradients updates are\n    propagated to the parameter server.\n    \"\"\"\n\n    # Setup the model.\n    model = HybridModel(remote_emb_module, rank)\n\n    # Retrieve all model parameters as rrefs for DistributedOptimizer.\n\n    # Retrieve parameters for embedding table.\n    model_parameter_rrefs = model.remote_emb_module.remote_parameters()\n\n    # model.fc.parameters() only includes local parameters.\n    # NOTE: Cannot call model.parameters() here,\n    # because this will call remote_emb_module.parameters(),\n    # which supports remote_parameters() but not parameters().\n    for param in model.fc.parameters():\n        model_parameter_rrefs.append(RRef(param))\n\n    # Setup distributed optimizer\n    opt = DistributedOptimizer(\n        optim.SGD,\n        model_parameter_rrefs,\n        lr=0.05,\n    )\n\n    criterion = torch.nn.CrossEntropyLoss()\n\n    def get_next_batch(rank):\n        for _ in range(10):\n            num_indices = random.randint(20, 50)\n            indices = torch.LongTensor(num_indices).random_(0, NUM_EMBEDDINGS)\n\n            # Generate offsets.\n            offsets = []\n            start = 0\n            batch_size = 0\n            while start < num_indices:\n                offsets.append(start)\n                start += random.randint(1, 10)\n                batch_size += 1\n\n            offsets_tensor = torch.LongTensor(offsets)\n            target = torch.LongTensor(batch_size).random_(8).to(rank)\n            yield indices, offsets_tensor, target\n\n    # Train for 100 epochs\n    for epoch in range(100):\n        # create distributed autograd context\n        for indices, offsets, target in get_next_batch(rank):\n            with dist_autograd.context() as context_id:\n                output = model(indices, offsets)\n                loss = criterion(output, target)\n\n                # Run distributed backward pass\n                dist_autograd.backward(context_id, [loss])\n\n                # Tun distributed optimizer\n                opt.step(context_id)\n\n                # Not necessary to zero grads as each iteration creates a different\n                # distributed autograd context which hosts different grads\n        print(\"Training done for epoch {}\".format(epoch))\n\n\ndef run_worker(rank, world_size):\n    r\"\"\"\n    A wrapper function that initializes RPC, calls the function, and shuts down\n    RPC.\n    \"\"\"\n\n    # We need to use different port numbers in TCP init_method for init_rpc and\n    # init_process_group to avoid port conflicts.\n    rpc_backend_options = TensorPipeRpcBackendOptions()\n    rpc_backend_options.init_method = \"tcp://localhost:29501\"\n\n    # Rank 2 is master, 3 is ps and 0 and 1 are trainers.\n    if rank == 2:\n        rpc.init_rpc(\n            \"master\",\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=rpc_backend_options,\n        )\n\n        remote_emb_module = RemoteModule(\n            \"ps\",\n            torch.nn.EmbeddingBag,\n            args=(NUM_EMBEDDINGS, EMBEDDING_DIM),\n            kwargs={\"mode\": \"sum\"},\n        )\n\n        # Run the training loop on trainers.\n        futs = []\n        for trainer_rank in [0, 1]:\n            trainer_name = \"trainer{}\".format(trainer_rank)\n            fut = rpc.rpc_async(\n                trainer_name, _run_trainer, args=(remote_emb_module, trainer_rank)\n            )\n            futs.append(fut)\n\n        # Wait for all training to finish.\n        for fut in futs:\n            fut.wait()\n    elif rank <= 1:\n        acc = torch.accelerator.current_accelerator()\n        device = torch.device(acc)\n        backend = torch.distributed.get_default_backend_for_device(device)\n        torch.accelerator.device_index(rank)\n        # Initialize process group for Distributed DataParallel on trainers.\n        dist.init_process_group(\n            backend=backend, rank=rank, world_size=2, init_method=\"tcp://localhost:29500\"\n        )\n\n        # Initialize RPC.\n        trainer_name = \"trainer{}\".format(rank)\n        rpc.init_rpc(\n            trainer_name,\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=rpc_backend_options,\n        )\n\n        # Trainer just waits for RPCs from master.\n    else:\n        rpc.init_rpc(\n            \"ps\",\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=rpc_backend_options,\n        )\n        # parameter server do nothing\n        pass\n\n    # block until all rpcs finish\n    rpc.shutdown()\n    \n    # Clean up process group for trainers to avoid resource leaks\n    if rank <= 1:\n        dist.destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    # 2 trainers, 1 parameter server, 1 master.\n    world_size = 4\n    _min_gpu_count = 2\n    if not verify_min_gpu_count(min_gpus=_min_gpu_count):\n        print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n        exit()\n    mp.spawn(run_worker, args=(world_size,), nprocs=world_size, join=True)\n    print(\"Distributed RPC example completed successfully.\")\n"
  },
  {
    "path": "distributed/rpc/ddp_rpc/requirements.txt",
    "content": "torch>=2.7.0\nnumpy\n"
  },
  {
    "path": "distributed/rpc/parameter_server/README.md",
    "content": "### RPC-based distributed training\n\nThis is a basic example of RPC-based training that uses several trainers remotely train a model hosted on a server.\n\nTo run the example locally, run the following command worker for the server and each worker you wish to spawn, in separate terminal windows:\n`python rpc_parameter_server.py --world_size=WORLD_SIZE --rank=RANK`. For example, for a master node with world size of 2, the command would be `python rpc_parameter_server.py --world_size=2 --rank=0`. The trainer can then be launched with the command `python rpc_parameter_server.py --world_size=2 --rank=1` in a separate window, and this will begin training with one server and a single trainer.\n\nNote that for demonstration purposes, this example supports only between 0-2 GPUs, although the pattern can be extended to make use of additional GPUs. To configure the number of GPUs, pass in `--num_gpus=N` to your training command.\n\nYou can pass in the command line arguments `--master_addr=<address>` and `master_port=PORT` to indicate the address:port that the master worker is listening on. All workers will contact the master for rendezvous during worker discovery. By default, `master_addr` will be `localhost` and `master_port` will be 29500.\n"
  },
  {
    "path": "distributed/rpc/parameter_server/requirements.txt",
    "content": "torch>=2.7.1\nnumpy\n"
  },
  {
    "path": "distributed/rpc/parameter_server/rpc_parameter_server.py",
    "content": "import argparse\nimport os\nfrom threading import Lock\n\nimport torch\nimport torch.distributed.autograd as dist_autograd\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch import optim\nfrom torch.distributed.optim import DistributedOptimizer\nfrom torchvision import datasets, transforms\n\n# --------- MNIST Network to train, from pytorch/examples -----\n\n\nclass Net(nn.Module):\n    def __init__(self, num_gpus=0):\n        super(Net, self).__init__()\n        print(f\"Using {num_gpus} GPUs to train\")\n        self.num_gpus = num_gpus\n        if torch.accelerator.is_available() and self.num_gpus > 0:\n            acc = torch.accelerator.current_accelerator()\n            device = torch.device(f'{acc}:0')\n        else:\n            device = torch.device(\"cpu\")\n        print(f\"Putting first 2 convs on {str(device)}\")\n        # Put conv layers on the first accelerator device\n        self.conv1 = nn.Conv2d(1, 32, 3, 1).to(device)\n        self.conv2 = nn.Conv2d(32, 64, 3, 1).to(device)\n        # Put rest of the network on the 2nd accelerator device, if there is one\n        if torch.accelerator.is_available() and self.num_gpus > 0:\n            acc = torch.accelerator.current_accelerator()\n            device = torch.device(f'{acc}:1')\n\n        print(f\"Putting rest of layers on {str(device)}\")\n        self.dropout1 = nn.Dropout2d(0.25).to(device)\n        self.dropout2 = nn.Dropout2d(0.5).to(device)\n        self.fc1 = nn.Linear(9216, 128).to(device)\n        self.fc2 = nn.Linear(128, 10).to(device)\n\n    def forward(self, x):\n        x = self.conv1(x)\n        x = F.relu(x)\n        x = self.conv2(x)\n        x = F.max_pool2d(x, 2)\n\n        x = self.dropout1(x)\n        x = torch.flatten(x, 1)\n        # Move tensor to next device if necessary\n        next_device = next(self.fc1.parameters()).device\n        x = x.to(next_device)\n\n        x = self.fc1(x)\n        x = F.relu(x)\n        x = self.dropout2(x)\n        x = self.fc2(x)\n        output = F.log_softmax(x, dim=1)\n        return output\n\n\n# --------- Helper Methods --------------------\n\n# On the local node, call a method with first arg as the value held by the\n# RRef. Other args are passed in as arguments to the function called.\n# Useful for calling instance methods.\ndef call_method(method, rref, *args, **kwargs):\n    return method(rref.local_value(), *args, **kwargs)\n\n# Given an RRef, return the result of calling the passed in method on the value\n# held by the RRef. This call is done on the remote node that owns\n# the RRef. args and kwargs are passed into the method.\n# Example: If the value held by the RRef is of type Foo, then\n# remote_method(Foo.bar, rref, arg1, arg2) is equivalent to calling\n# <foo_instance>.bar(arg1, arg2) on the remote node and getting the result\n# back.\n\ndef remote_method(method, rref, *args, **kwargs):\n    args = [method, rref] + list(args)\n    return rpc.rpc_sync(rref.owner(), call_method, args=args, kwargs=kwargs)\n\n# --------- Parameter Server --------------------\nclass ParameterServer(nn.Module):\n    def __init__(self, num_gpus=0):\n        super().__init__()\n        model = Net(num_gpus=num_gpus)\n        self.model = model\n        if torch.accelerator.is_available() and num_gpus > 0:\n            acc = torch.accelerator.current_accelerator()\n            self.input_device = torch.device(f'{acc}:0')\n        else:\n            self.input_device = torch.device(\"cpu\")\n            \n    def forward(self, inp):\n        inp = inp.to(self.input_device)\n        out = self.model(inp)\n        # This output is forwarded over RPC, which as of 1.5.0 only accepts CPU tensors.\n        # Tensors must be moved in and out of GPU memory due to this.\n        out = out.to(\"cpu\")\n        return out\n\n    # Use dist autograd to retrieve gradients accumulated for this model.\n    # Primarily used for verification.\n    def get_dist_gradients(self, cid):\n        grads = dist_autograd.get_gradients(cid)\n        # This output is forwarded over RPC, which as of 1.5.0 only accepts CPU tensors.\n        # Tensors must be moved in and out of GPU memory due to this.\n        cpu_grads = {}\n        for k, v in grads.items():\n            k_cpu, v_cpu = k.to(\"cpu\"), v.to(\"cpu\")\n            cpu_grads[k_cpu] = v_cpu\n        return cpu_grads\n\n    # Wrap local parameters in a RRef. Needed for building the\n    # DistributedOptimizer which optimizes parameters remotely.\n    def get_param_rrefs(self):\n        param_rrefs = [rpc.RRef(param) for param in self.model.parameters()]\n        return param_rrefs\n\nparam_server = None\nglobal_lock = Lock()\n\ndef get_parameter_server(num_gpus=0):\n    global param_server\n    # Ensure that we get only one handle to the ParameterServer.\n    with global_lock:\n        if not param_server:\n            # construct it once\n            param_server = ParameterServer(num_gpus=num_gpus)\n        return param_server\n\n\ndef run_parameter_server(rank, world_size):\n    # The parameter server just acts as a host for the model and responds to\n    # requests from trainers, hence it does not need to run a loop.\n    # rpc.shutdown() will wait for all workers to complete by default, which\n    # in this case means that the parameter server will wait for all trainers\n    # to complete, and then exit.\n    print(\"PS master initializing RPC\")\n    rpc.init_rpc(name=\"parameter_server\", rank=rank, world_size=world_size)\n    print(\"RPC initialized! Running parameter server...\")\n    rpc.shutdown()\n    print(\"RPC shutdown on parameter server.\")\n\n\n# --------- Trainers --------------------\n\n# nn.Module corresponding to the network trained by this trainer. The\n# forward() method simply invokes the network on the given parameter\n# server.\nclass TrainerNet(nn.Module):\n    def __init__(self, num_gpus=0):\n        super().__init__()\n        self.num_gpus = num_gpus\n        self.param_server_rref = rpc.remote(\n            \"parameter_server\", get_parameter_server, args=(num_gpus,))\n\n    def get_global_param_rrefs(self):\n        remote_params = remote_method(\n            ParameterServer.get_param_rrefs,\n            self.param_server_rref)\n        return remote_params\n\n    def forward(self, x):\n        model_output = remote_method(\n            ParameterServer.forward, self.param_server_rref, x)\n        return model_output\n\n\ndef run_training_loop(rank, num_gpus, train_loader, test_loader):\n    # Runs the typical neural network forward + backward + optimizer step, but\n    # in a distributed fashion.\n    net = TrainerNet(num_gpus=num_gpus)\n    # Build DistributedOptimizer.\n    param_rrefs = net.get_global_param_rrefs()\n    opt = DistributedOptimizer(optim.SGD, param_rrefs, lr=0.03)\n    for i, (data, target) in enumerate(train_loader):\n        with dist_autograd.context() as cid:\n            model_output = net(data)\n            target = target.to(model_output.device)\n            loss = F.nll_loss(model_output, target)\n            if i % 5 == 0:\n                print(f\"Rank {rank} training batch {i} loss {loss.item()}\")\n            dist_autograd.backward(cid, [loss])\n            # Ensure that dist autograd ran successfully and gradients were\n            # returned.\n            assert remote_method(\n                ParameterServer.get_dist_gradients,\n                net.param_server_rref,\n                cid) != {}\n            opt.step(cid)\n\n    print(\"Training complete!\")\n    print(\"Getting accuracy....\")\n    get_accuracy(test_loader, net)\n\n\ndef get_accuracy(test_loader, model):\n    model.eval()\n    correct_sum = 0\n    # Use GPU to evaluate if possible\n    if torch.accelerator.is_available() and model.num_gpus > 0:\n        acc = torch.accelerator.current_accelerator()\n        device = torch.device(f'{acc}:0')\n    else:\n        device = torch.device(\"cpu\")\n    with torch.no_grad():\n        for i, (data, target) in enumerate(test_loader):\n            out = model(data)\n            pred = out.argmax(dim=1, keepdim=True)\n            pred, target = pred.to(device), target.to(device)\n            correct = pred.eq(target.view_as(pred)).sum().item()\n            correct_sum += correct\n\n    print(f\"Accuracy {correct_sum / len(test_loader.dataset)}\")\n\n\n# Main loop for trainers.\ndef run_worker(rank, world_size, num_gpus, train_loader, test_loader):\n    print(f\"Worker rank {rank} initializing RPC\")\n    rpc.init_rpc(\n        name=f\"trainer_{rank}\",\n        rank=rank,\n        world_size=world_size)\n\n    print(f\"Worker {rank} done initializing RPC\")\n\n    run_training_loop(rank, num_gpus, train_loader, test_loader)\n    rpc.shutdown()\n\n# --------- Launcher --------------------\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser(\n        description=\"Parameter-Server RPC based training\")\n    parser.add_argument(\n        \"--world_size\",\n        type=int,\n        default=4,\n        help=\"\"\"Total number of participating processes. Should be the sum of\n        master node and all training nodes.\"\"\")\n    parser.add_argument(\n        \"--rank\",\n        type=int,\n        default=None,\n        help=\"Global rank of this process. Pass in 0 for master.\")\n    parser.add_argument(\n        \"--num_gpus\",\n        type=int,\n        default=0,\n        help=\"\"\"Number of GPUs to use for training, currently supports between 0\n         and 2 GPUs. Note that this argument will be passed to the parameter servers.\"\"\")\n    parser.add_argument(\n        \"--master_addr\",\n        type=str,\n        default=\"localhost\",\n        help=\"\"\"Address of master, will default to localhost if not provided.\n        Master must be able to accept network traffic on the address + port.\"\"\")\n    parser.add_argument(\n        \"--master_port\",\n        type=str,\n        default=\"29500\",\n        help=\"\"\"Port that master is listening on, will default to 29500 if not\n        provided. Master must be able to accept network traffic on the host and port.\"\"\")\n\n    args = parser.parse_args()\n    assert args.rank is not None, \"must provide rank argument.\"\n    assert args.num_gpus <= 3, f\"Only 0-2 GPUs currently supported (got {args.num_gpus}).\"\n    os.environ['MASTER_ADDR'] = args.master_addr\n    os.environ['MASTER_PORT'] = args.master_port\n    processes = []\n    world_size = args.world_size\n\n    # Note that Linux uses \"fork\" by default, which may cause deadlock.\n    # Besides, cuda doesn't support \"fork\" and Windows only supports \"spawn\"\n    mp.set_start_method(\"spawn\")\n\n    if args.rank == 0:\n        p = mp.Process(target=run_parameter_server, args=(0, world_size))\n        p.start()\n        processes.append(p)\n    else:\n        # Get data to train on\n        train_loader = torch.utils.data.DataLoader(\n            datasets.MNIST('../data', train=True, download=True,\n                           transform=transforms.Compose([\n                               transforms.ToTensor(),\n                               transforms.Normalize((0.1307,), (0.3081,))\n                           ])),\n            batch_size=32, shuffle=True)\n        test_loader = torch.utils.data.DataLoader(\n            datasets.MNIST('../data', train=False,\n                           transform=transforms.Compose([\n                               transforms.ToTensor(),\n                               transforms.Normalize((0.1307,), (0.3081,))\n                           ])),\n            batch_size=32, shuffle=True)\n        # start training worker on this node\n        p = mp.Process(\n            target=run_worker,\n            args=(\n                args.rank,\n                world_size, args.num_gpus,\n                train_loader,\n                test_loader))\n        p.start()\n        processes.append(p)\n\n    for p in processes:\n        p.join()\n"
  },
  {
    "path": "distributed/rpc/pipeline/README.md",
    "content": "Distributed Pipeline Parallel Example\n\nThis example shows how to distribute a ResNet50 model on two RPC workers and\nthen implement distributed pipeline parallelism using RPC. With pipeline\nparallelism, every input batch is divided into micro-batches and thse\nmicro-batches are feed into the model in a pipelined fashion to increase the\namortized device utilization. Note that this example only parallelizes the\nforward pass which can be viewed as the distributed counterpart of the\n[single machine pipeline parallel](https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html#speed-up-by-pipelining-inputs)\nexample.\n\n```\npip install -r requirements.txt\npython main.py\n```\n"
  },
  {
    "path": "distributed/rpc/pipeline/main.py",
    "content": "import os\nimport threading\nimport time\nimport warnings\nfrom functools import wraps\n\nimport torch\nimport torch.nn as nn\nimport torch.distributed.autograd as dist_autograd\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.optim as optim\nfrom torch.distributed.rpc import RRef\n\nfrom torchvision.models.resnet import Bottleneck\n\n# Suppress warnings that can't be fixed from user code\nwarnings.filterwarnings(\"ignore\", \n    message=\"You are using a Backend .* as a ProcessGroup. This usage is deprecated\", \n    category=UserWarning)\nwarnings.filterwarnings(\"ignore\", \n    message=\"networkx backend defined more than once: nx-loopback\", \n    category=RuntimeWarning)\n\n\n#########################################################\n#           Define Model Parallel ResNet50              #\n#########################################################\n\n# In order to split the ResNet50 and place it on two different workers, we\n# implement it in two model shards. The ResNetBase class defines common\n# attributes and methods shared by two shards. ResNetShard1 and ResNetShard2\n# contain two partitions of the model layers respectively.\n\n\nnum_classes = 1000\n\n\ndef conv1x1(in_planes, out_planes, stride=1):\n    \"\"\"1x1 convolution\"\"\"\n    return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)\n\nclass ResNetBase(nn.Module):\n    def __init__(self, block, inplanes, num_classes=1000,\n                 groups=1, width_per_group=64, norm_layer=None):\n        super(ResNetBase, self).__init__()\n\n        self._lock = threading.Lock()\n        self._block = block\n        self._norm_layer = nn.BatchNorm2d\n        self.inplanes = inplanes\n        self.dilation = 1\n        self.groups = groups\n        self.base_width = width_per_group\n\n    def _make_layer(self, planes, blocks, stride=1):\n        norm_layer = self._norm_layer\n        downsample = None\n        previous_dilation = self.dilation\n        if stride != 1 or self.inplanes != planes * self._block.expansion:\n            downsample = nn.Sequential(\n                conv1x1(self.inplanes, planes * self._block.expansion, stride),\n                norm_layer(planes * self._block.expansion),\n            )\n\n        layers = []\n        layers.append(self._block(self.inplanes, planes, stride, downsample, self.groups,\n                                  self.base_width, previous_dilation, norm_layer))\n        self.inplanes = planes * self._block.expansion\n        for _ in range(1, blocks):\n            layers.append(self._block(self.inplanes, planes, groups=self.groups,\n                                      base_width=self.base_width, dilation=self.dilation,\n                                      norm_layer=norm_layer))\n\n        return nn.Sequential(*layers)\n\n    def parameter_rrefs(self):\n        r\"\"\"\n        Create one RRef for each parameter in the given local module, and return a\n        list of RRefs.\n        \"\"\"\n        return [RRef(p) for p in self.parameters()]\n\n\nclass ResNetShard1(ResNetBase):\n    \"\"\"\n    The first part of ResNet.\n    \"\"\"\n    def __init__(self, device, *args, **kwargs):\n        super(ResNetShard1, self).__init__(\n            Bottleneck, 64, num_classes=num_classes, *args, **kwargs)\n\n        self.device = device\n        self.seq = nn.Sequential(\n            nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3, bias=False),\n            self._norm_layer(self.inplanes),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(kernel_size=3, stride=2, padding=1),\n            self._make_layer(64, 3),\n            self._make_layer(128, 4, stride=2)\n        ).to(self.device)\n\n        for m in self.modules():\n            if isinstance(m, nn.Conv2d):\n                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')\n            elif isinstance(m, nn.BatchNorm2d):\n                nn.init.ones_(m.weight)\n                nn.init.zeros_(m.bias)\n\n    def forward(self, x_rref):\n        x = x_rref.to_here().to(self.device)\n        with self._lock:\n            out =  self.seq(x)\n        return out.cpu()\n\n\nclass ResNetShard2(ResNetBase):\n    \"\"\"\n    The second part of ResNet.\n    \"\"\"\n    def __init__(self, device, *args, **kwargs):\n        super(ResNetShard2, self).__init__(\n            Bottleneck, 512, num_classes=num_classes, *args, **kwargs)\n\n        self.device = device\n        self.seq = nn.Sequential(\n            self._make_layer(256, 6, stride=2),\n            self._make_layer(512, 3, stride=2),\n            nn.AdaptiveAvgPool2d((1, 1)),\n        ).to(self.device)\n\n        self.fc =  nn.Linear(512 * self._block.expansion, num_classes).to(self.device)\n\n    def forward(self, x_rref):\n        x = x_rref.to_here().to(self.device)\n        with self._lock:\n            out = self.fc(torch.flatten(self.seq(x), 1))\n        return out.cpu()\n\n\nclass DistResNet50(nn.Module):\n    \"\"\"\n    Assemble two parts as an nn.Module and define pipelining logic\n    \"\"\"\n    def __init__(self, split_size, workers, *args, **kwargs):\n        super(DistResNet50, self).__init__()\n\n        self.split_size = split_size\n\n        # Put the first part of the ResNet50 on workers[0]\n        self.p1_rref = rpc.remote(\n            workers[0],\n            ResNetShard1,\n            args = (\"cuda:0\",) + args,\n            kwargs = kwargs\n        )\n\n        # Put the second part of the ResNet50 on workers[1]\n        self.p2_rref = rpc.remote(\n            workers[1],\n            ResNetShard2,\n            args = (\"cuda:1\",) + args,\n            kwargs = kwargs\n        )\n\n    def forward(self, xs):\n        # Split the input batch xs into micro-batches, and collect async RPC\n        # futures into a list\n        out_futures = []\n        for x in iter(xs.split(self.split_size, dim=0)):\n            x_rref = RRef(x)\n            y_rref = self.p1_rref.remote().forward(x_rref)\n            z_fut = self.p2_rref.rpc_async().forward(y_rref)\n            out_futures.append(z_fut)\n\n        # collect and cat all output tensors into one tensor.\n        return torch.cat(torch.futures.wait_all(out_futures))\n\n    def parameter_rrefs(self):\n        remote_params = []\n        remote_params.extend(self.p1_rref.remote().parameter_rrefs().to_here())\n        remote_params.extend(self.p2_rref.remote().parameter_rrefs().to_here())\n        return remote_params\n\n\n#########################################################\n#                   Run RPC Processes                   #\n#########################################################\n\nnum_batches = 3\nbatch_size = 120\nimage_w = 128\nimage_h = 128\n\n\ndef create_optimizer_for_remote_params(worker_name, param_rrefs, lr=0.05):\n    \"\"\"Create torch.compiled optimizers on  each worker\"\"\"\n    params = [p.to_here() for p in param_rrefs]\n    opt = optim.SGD(params, lr=lr)\n    opt.step = torch.compile(opt.step)\n    return opt\n\n\ndef run_master(split_size):\n\n    # put the two model parts on worker1 and worker2 respectively\n    model = DistResNet50(split_size, [\"worker1\", \"worker2\"])\n    loss_fn = nn.MSELoss()\n    \n    # Get parameter RRefs for each model shard\n    p1_param_rrefs = model.p1_rref.remote().parameter_rrefs().to_here()\n    p2_param_rrefs = model.p2_rref.remote().parameter_rrefs().to_here()\n    \n    # Create optimizers on remote workers\n    opt1_rref = rpc.remote(\n        \"worker1\",\n        create_optimizer_for_remote_params,\n        args=(\"worker1\", p1_param_rrefs)\n    )\n    opt2_rref = rpc.remote(\n        \"worker2\",\n        create_optimizer_for_remote_params,\n        args=(\"worker2\", p2_param_rrefs)\n    )\n\n    one_hot_indices = torch.LongTensor(batch_size) \\\n                           .random_(0, num_classes) \\\n                           .view(batch_size, 1)\n\n    for i in range(num_batches):\n        print(f\"Processing batch {i}\")\n        # generate random inputs and labels\n        inputs = torch.randn(batch_size, 3, image_w, image_h)\n        labels = torch.zeros(batch_size, num_classes) \\\n                      .scatter_(1, one_hot_indices, 1)\n\n        # The distributed autograd context is the dedicated scope for the\n        # distributed backward pass to store gradients, which can later be\n        # retrieved using the context_id by the distributed optimizer.\n        with dist_autograd.context() as context_id:\n            outputs = model(inputs)\n            dist_autograd.backward(context_id, [loss_fn(outputs, labels)])\n            \n            opt1_rref.rpc_sync().step()\n            opt2_rref.rpc_sync().step()\n            \n            opt1_rref.rpc_sync().zero_grad()\n            opt2_rref.rpc_sync().zero_grad()\n\n\ndef run_worker(rank, world_size, num_split):\n    os.environ['MASTER_ADDR'] = 'localhost'\n    os.environ['MASTER_PORT'] = '29500'\n\n    # Higher timeout is added to accommodate for kernel compilation time in case of ROCm.\n    options = rpc.TensorPipeRpcBackendOptions(num_worker_threads=256, rpc_timeout=300)\n\n    if rank == 0:\n        rpc.init_rpc(\n            \"master\",\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=options\n        )\n        run_master(num_split)\n    else:\n        rpc.init_rpc(\n            f\"worker{rank}\",\n            rank=rank,\n            world_size=world_size,\n            rpc_backend_options=options\n        )\n        pass\n\n    # block until all rpcs finish\n    rpc.shutdown()\n\n\nif __name__==\"__main__\":\n    # Suppress torch compile profiler warnings\n    os.environ['TORCH_LOGS'] = '-dynamo'\n    \n    world_size = 3\n    for num_split in [1, 2, 4, 8]:\n        tik = time.time()\n        mp.spawn(run_worker, args=(world_size, num_split), nprocs=world_size, join=True)\n        tok = time.time()\n        print(f\"number of splits = {num_split}, execution time = {tok - tik}\")\n"
  },
  {
    "path": "distributed/rpc/pipeline/requirements.txt",
    "content": "torch\ntorchvision"
  },
  {
    "path": "distributed/rpc/rl/README.md",
    "content": "Distributed Multi-Observer Single-Agent Reinforcement Learning Example\n\nThis example demonstrates `torch.distributed.rpc` API using an CartPole\nreinforcement learning example. Please note that the goal is to present the RPC\nAPI instead of building the best CartPole solver.\n\n```\npip install -r requirements.txt\npython main.py\n```\n"
  },
  {
    "path": "distributed/rpc/rl/main.py",
    "content": "import argparse\nimport gymnasium as gym\nimport numpy as np\nimport os\nfrom itertools import count\n\nimport torch\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.distributed.rpc import RRef, rpc_sync, rpc_async, remote\nfrom torch.distributions import Categorical\n\nTOTAL_EPISODE_STEP = 5000\nAGENT_NAME = \"agent\"\nOBSERVER_NAME = \"observer{}\"\n\nparser = argparse.ArgumentParser(description='PyTorch RPC RL example')\nparser.add_argument('--world-size', type=int, default=2, metavar='W',\n                    help='world size for RPC, rank 0 is the agent, others are observers')\nparser.add_argument('--gamma', type=float, default=0.99, metavar='G',\n                    help='discount factor (default: 0.99)')\nparser.add_argument('--seed', type=int, default=543, metavar='N',\n                    help='random seed (default: 543)')\nparser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                    help='interval between training status logs (default: 10)')\nargs = parser.parse_args()\n\ntorch.manual_seed(args.seed)\n\n\ndef _call_method(method, rref, *args, **kwargs):\n    r\"\"\"\n    a helper function to call a method on the given RRef\n    \"\"\"\n    return method(rref.local_value(), *args, **kwargs)\n\n\ndef _remote_method(method, rref, *args, **kwargs):\n    r\"\"\"\n    a helper function to run method on the owner of rref and fetch back the\n    result using RPC\n    \"\"\"\n    args = [method, rref] + list(args)\n    return rpc_sync(rref.owner(), _call_method, args=args, kwargs=kwargs)\n\n\nclass Policy(nn.Module):\n    r\"\"\"\n    Borrowing the ``Policy`` class from the Reinforcement Learning example.\n    Copying the code to make these two examples independent.\n    See https://github.com/pytorch/examples/tree/main/reinforcement_learning\n    \"\"\"\n    def __init__(self):\n        super(Policy, self).__init__()\n        self.affine1 = nn.Linear(4, 128)\n        self.dropout = nn.Dropout(p=0.6)\n        self.affine2 = nn.Linear(128, 2)\n\n        self.saved_log_probs = []\n        self.rewards = []\n\n    def forward(self, x):\n        x = self.affine1(x)\n        x = self.dropout(x)\n        x = F.relu(x)\n        action_scores = self.affine2(x)\n        return F.softmax(action_scores, dim=1)\n\nclass Observer:\n    r\"\"\"\n    An observer has exclusive access to its own environment. Each observer\n    captures the state from its environment, and send the state to the agent to\n    select an action. Then, the observer applies the action to its environment\n    and reports the reward to the agent.\n\n    It is true that CartPole-v1 is a relatively inexpensive environment, and it\n    might be an overkill to use RPC to connect observers and trainers in this\n    specific use case. However, the main goal of this tutorial to how to build\n    an application using the RPC API. Developers can extend the similar idea to\n    other applications with much more expensive environment.\n    \"\"\"\n    def __init__(self):\n        self.id = rpc.get_worker_info().id\n        self.env = gym.make('CartPole-v1')\n        self.env.reset(seed=args.seed)\n\n    def run_episode(self, agent_rref, n_steps):\n        r\"\"\"\n        Run one episode of n_steps.\n\n        Args:\n            agent_rref (RRef): an RRef referencing the agent object.\n            n_steps (int): number of steps in this episode\n        \"\"\"\n        state, ep_reward = self.env.reset()[0], 0\n        for step in range(n_steps):\n            # send the state to the agent to get an action\n            action = _remote_method(Agent.select_action, agent_rref, self.id, state)\n\n            # apply the action to the environment, and get the reward\n            state, reward, terminated, truncated, _ = self.env.step(action)\n\n            # report the reward to the agent for training purpose\n            _remote_method(Agent.report_reward, agent_rref, self.id, reward)\n\n            if terminated or truncated:\n                break\n\nclass Agent:\n    def __init__(self, world_size):\n        self.ob_rrefs = []\n        self.agent_rref = RRef(self)\n        self.rewards = {}\n        self.saved_log_probs = {}\n        self.policy = Policy()\n        self.optimizer = optim.Adam(self.policy.parameters(), lr=1e-2)\n        self.eps = np.finfo(np.float32).eps.item()\n        self.running_reward = 0\n        self.reward_threshold = gym.make('CartPole-v1').spec.reward_threshold\n        for ob_rank in range(1, world_size):\n            ob_info = rpc.get_worker_info(OBSERVER_NAME.format(ob_rank))\n            self.ob_rrefs.append(remote(ob_info, Observer))\n            self.rewards[ob_info.id] = []\n            self.saved_log_probs[ob_info.id] = []\n\n    def select_action(self, ob_id, state):\n        r\"\"\"\n        This function is mostly borrowed from the Reinforcement Learning example.\n        See https://github.com/pytorch/examples/tree/main/reinforcement_learning\n        The main difference is that instead of keeping all probs in one list,\n        the agent keeps probs in a dictionary, one key per observer.\n\n        NB: no need to enforce thread-safety here as GIL will serialize\n        executions.\n        \"\"\"\n        state = torch.from_numpy(state).float().unsqueeze(0)\n        probs = self.policy(state)\n        m = Categorical(probs)\n        action = m.sample()\n        self.saved_log_probs[ob_id].append(m.log_prob(action))\n        return action.item()\n\n    def report_reward(self, ob_id, reward):\n        r\"\"\"\n        Observers call this function to report rewards.\n        \"\"\"\n        self.rewards[ob_id].append(reward)\n\n    def run_episode(self, n_steps=0):\n        r\"\"\"\n        Run one episode. The agent will tell each oberser to run n_steps.\n        \"\"\"\n        futs = []\n        for ob_rref in self.ob_rrefs:\n            # make async RPC to kick off an episode on all observers\n            futs.append(\n                rpc_async(\n                    ob_rref.owner(),\n                    _call_method,\n                    args=(Observer.run_episode, ob_rref, self.agent_rref, n_steps)\n                )\n            )\n\n        # wait until all obervers have finished this episode\n        for fut in futs:\n            fut.wait()\n\n    def finish_episode(self):\n        r\"\"\"\n        This function is mostly borrowed from the Reinforcement Learning example.\n        See https://github.com/pytorch/examples/tree/main/reinforcement_learning\n        The main difference is that it joins all probs and rewards from\n        different observers into one list, and uses the minimum observer rewards\n        as the reward of the current episode.\n        \"\"\"\n\n        # joins probs and rewards from different observers into lists\n        R, probs, rewards = 0, [], []\n        for ob_id in self.rewards:\n            probs.extend(self.saved_log_probs[ob_id])\n            rewards.extend(self.rewards[ob_id])\n\n        # use the minimum observer reward to calculate the running reward\n        min_reward = min([sum(self.rewards[ob_id]) for ob_id in self.rewards])\n        self.running_reward = 0.05 * min_reward + (1 - 0.05) * self.running_reward\n\n        # clear saved probs and rewards\n        for ob_id in self.rewards:\n            self.rewards[ob_id] = []\n            self.saved_log_probs[ob_id] = []\n\n        policy_loss, returns = [], []\n        for r in rewards[::-1]:\n            R = r + args.gamma * R\n            returns.insert(0, R)\n        returns = torch.tensor(returns)\n        returns = (returns - returns.mean()) / (returns.std() + self.eps)\n        for log_prob, R in zip(probs, returns):\n            policy_loss.append(-log_prob * R)\n        self.optimizer.zero_grad()\n        policy_loss = torch.cat(policy_loss).sum()\n        policy_loss.backward()\n        self.optimizer.step()\n        return min_reward\n\n\ndef run_worker(rank, world_size):\n    r\"\"\"\n    This is the entry point for all processes. The rank 0 is the agent. All\n    other ranks are observers.\n    \"\"\"\n    os.environ['MASTER_ADDR'] = 'localhost'\n    os.environ['MASTER_PORT'] = '29500'\n    if rank == 0:\n        # rank0 is the agent\n        rpc.init_rpc(AGENT_NAME, rank=rank, world_size=world_size)\n\n        agent = Agent(world_size)\n        for i_episode in count(1):\n            n_steps = int(TOTAL_EPISODE_STEP / (args.world_size - 1))\n            agent.run_episode(n_steps=n_steps)\n            last_reward = agent.finish_episode()\n\n            if i_episode % args.log_interval == 0:\n                print('Episode {}\\tLast reward: {:.2f}\\tAverage reward: {:.2f}'.format(\n                      i_episode, last_reward, agent.running_reward))\n\n            if agent.running_reward > agent.reward_threshold:\n                print(\"Solved! Running reward is now {}!\".format(agent.running_reward))\n                break\n    else:\n        # other ranks are the observer\n        rpc.init_rpc(OBSERVER_NAME.format(rank), rank=rank, world_size=world_size)\n        # observers passively waiting for instructions from agents\n    rpc.shutdown()\n\n\ndef main():\n    mp.spawn(\n        run_worker,\n        args=(args.world_size, ),\n        nprocs=args.world_size,\n        join=True\n    )\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "distributed/rpc/rl/requirements.txt",
    "content": "torch\nnumpy\ngymnasium\n"
  },
  {
    "path": "distributed/rpc/rnn/README.md",
    "content": "Distributed RNN Model Parallel Example\n\nThis example shows how to build an RNN model using RPC where different\ncomponents of the RNN model can be placed on different workers.\n\n```\npip install -r requirements.txt\npython main.py\n```\n"
  },
  {
    "path": "distributed/rpc/rnn/main.py",
    "content": "import os\n\nimport torch\nimport torch.distributed.autograd as dist_autograd\nimport torch.distributed.rpc as rpc\nimport torch.multiprocessing as mp\nimport torch.optim as optim\nfrom torch.distributed.optim import DistributedOptimizer\n\nimport rnn\n\n\ndef _run_trainer():\n    r\"\"\"\n    The trainer creates a distributed RNNModel and a DistributedOptimizer. Then,\n    it performs training using random input data.\n    \"\"\"\n    batch = 5\n    ntoken = 7\n    ninp = 2\n\n    nhid = 3\n    nindices = 6\n    nlayers = 4\n    hidden = (\n        torch.randn(nlayers, nindices, nhid),\n        torch.randn(nlayers, nindices, nhid)\n    )\n\n    model = rnn.RNNModel('ps', ntoken, ninp, nhid, nlayers)\n\n    # setup distributed optimizer\n    opt = DistributedOptimizer(\n        optim.SGD,\n        model.parameter_rrefs(),\n        lr=0.05,\n    )\n\n    criterion = torch.nn.CrossEntropyLoss()\n\n    def get_next_batch():\n        for _ in range(5):\n            data = torch.LongTensor(batch, nindices) % ntoken\n            target = torch.LongTensor(batch, ntoken) % nindices\n            yield data, target\n\n    # train for 10 iterations\n    for epoch in range(10):\n        # create distributed autograd context\n        for data, target in get_next_batch():\n            with dist_autograd.context() as context_id:\n                hidden[0].detach_()\n                hidden[1].detach_()\n                output, hidden = model(data, hidden)\n                loss = criterion(output, target)\n                # run distributed backward pass\n                dist_autograd.backward(context_id, [loss])\n                # run distributed optimizer\n                opt.step(context_id)\n                # not necessary to zero grads as each iteration creates a different\n                # distributed autograd context which hosts different grads\n        print(\"Training epoch {}\".format(epoch))\n\n\ndef run_worker(rank, world_size):\n    r\"\"\"\n    A wrapper function that initializes RPC, calls the function, and shuts down\n    RPC.\n    \"\"\"\n    os.environ['MASTER_ADDR'] = 'localhost'\n    os.environ['MASTER_PORT'] = '29500'\n    if rank == 1:\n        rpc.init_rpc(\"trainer\", rank=rank, world_size=world_size)\n        _run_trainer()\n    else:\n        rpc.init_rpc(\"ps\", rank=rank, world_size=world_size)\n        # parameter server does nothing\n        pass\n\n    # block until all rpcs finish\n    rpc.shutdown()\n\n\nif __name__ == \"__main__\":\n    world_size = 2\n    mp.spawn(run_worker, args=(world_size, ), nprocs=world_size, join=True)\n"
  },
  {
    "path": "distributed/rpc/rnn/requirements.txt",
    "content": "torch>=2.7.1\nnumpy\n"
  },
  {
    "path": "distributed/rpc/rnn/rnn.py",
    "content": "import torch\nimport torch.nn as nn\nimport torch.distributed.rpc as rpc\nfrom torch.distributed.rpc import RRef\n\n\ndef _call_method(method, rref, *args, **kwargs):\n    r\"\"\"\n    a helper function to call a method on the given RRef\n    \"\"\"\n    return method(rref.local_value(), *args, **kwargs)\n\n\ndef _remote_method(method, rref, *args, **kwargs):\n    r\"\"\"\n    a helper function to run method on the owner of rref and fetch back the\n    result using RPC\n    \"\"\"\n    return rpc.rpc_sync(\n        rref.owner(),\n        _call_method,\n        args=[method, rref] + list(args),\n        kwargs=kwargs\n    )\n\n\ndef _parameter_rrefs(module):\n    r\"\"\"\n    Create one RRef for each parameter in the given local module, and return a\n    list of RRefs.\n    \"\"\"\n    param_rrefs = []\n    for param in module.parameters():\n        param_rrefs.append(RRef(param))\n    return param_rrefs\n\n\nclass EmbeddingTable(nn.Module):\n    r\"\"\"\n    Encoding layers of the RNNModel\n    \"\"\"\n    def __init__(self, ntoken, ninp, dropout):\n        super(EmbeddingTable, self).__init__()\n        self.drop = nn.Dropout(dropout)\n        self.encoder = nn.Embedding(ntoken, ninp)\n        if torch.accelerator.is_available():\n            device = torch.accelerator.current_accelerator()\n            self.encoder = self.encoder.to(device)\n        nn.init.uniform_(self.encoder.weight, -0.1, 0.1)\n\n    def forward(self, input):\n        if torch.accelerator.is_available():\n            device = torch.accelerator.current_accelerator()\n            input = input.to(device)\n        return self.drop(self.encoder(input)).cpu()\n\n\nclass Decoder(nn.Module):\n    r\"\"\"\n    Decoding layers of the RNNModel\n    \"\"\"\n    def __init__(self, ntoken, nhid, dropout):\n        super(Decoder, self).__init__()\n        self.drop = nn.Dropout(dropout)\n        self.decoder = nn.Linear(nhid, ntoken)\n        nn.init.zeros_(self.decoder.bias)\n        nn.init.uniform_(self.decoder.weight, -0.1, 0.1)\n\n    def forward(self, output):\n        return self.decoder(self.drop(output))\n\n\nclass RNNModel(nn.Module):\n    r\"\"\"\n    A distributed RNN model which puts embedding table and decoder parameters on\n    a remote parameter server, and locally holds parameters for the LSTM module.\n    The structure of the RNN model is borrowed from the word language model\n    example. See https://github.com/pytorch/examples/blob/main/word_language_model/model.py\n    \"\"\"\n    def __init__(self, ps, ntoken, ninp, nhid, nlayers, dropout=0.5):\n        super(RNNModel, self).__init__()\n\n        # setup embedding table remotely\n        self.emb_table_rref = rpc.remote(ps, EmbeddingTable, args=(ntoken, ninp, dropout))\n        # setup LSTM locally\n        self.rnn = nn.LSTM(ninp, nhid, nlayers, dropout=dropout)\n        # setup decoder remotely\n        self.decoder_rref = rpc.remote(ps, Decoder, args=(ntoken, nhid, dropout))\n\n    def forward(self, input, hidden):\n        # pass input to the remote embedding table and fetch emb tensor back\n        emb = _remote_method(EmbeddingTable.forward, self.emb_table_rref, input)\n        output, hidden = self.rnn(emb, hidden)\n        # pass output to the remote decoder and get the decoded output back\n        decoded = _remote_method(Decoder.forward, self.decoder_rref, output)\n        return decoded, hidden\n\n    def parameter_rrefs(self):\n        remote_params = []\n        # get RRefs of embedding table\n        remote_params.extend(_remote_method(_parameter_rrefs, self.emb_table_rref))\n        # create RRefs for local parameters\n        remote_params.extend(_parameter_rrefs(self.rnn))\n        # get RRefs of decoder\n        remote_params.extend(_remote_method(_parameter_rrefs, self.decoder_rref))\n        return remote_params\n"
  },
  {
    "path": "distributed/tensor_parallelism/README.md",
    "content": "# PyTorch native Tensor Parallel for distributed training\n\nThis example demonstrates SPMD Megatron-LM style Tensor Parallel by using\nPyTorch native Tensor Parallel APIs, which include:\n\n1. Simple module-level Tensor Parallelism on a dummy MLP model.\n2. Simple module-level Tensor Parallelism with Sequence Parallel inputs/outputs on a dummy MLP model.\n3. A E2E demo of Fully Sharded Data Parallel + Tensor Parallel (with Sequence Parallel) on a example Llama2 model.\n\nMore details about the PyTorch native Tensor Parallel APIs, please see PyTorch docs:\nhttps://pytorch.org/docs/stable/distributed.tensor.parallel.html\n\n## Installation\n\n```bash\npip install -r requirements.txt\n```\n\n## Running Examples\n\nYou can run the examples using `torchrun` to launch distributed training:\n\n```bash\n# Simple Tensor Parallel example\ntorchrun --nnodes=1 --nproc_per_node=4 tensor_parallel_example.py\n\n# Tensor Parallel with Sequence Parallel\ntorchrun --nnodes=1 --nproc_per_node=4 sequence_parallel_example.py\n\n# FSDP + Tensor Parallel with Llama2 model\ntorchrun --nnodes=1 --nproc_per_node=4 fsdp_tp_example.py\n```\n\nFor more details, check the `run_examples.sh` script.\n"
  },
  {
    "path": "distributed/tensor_parallelism/fsdp_tp_example.py",
    "content": "\"\"\"\nThis is the script to test 2D Parallel which combines Tensor/Sequence\nparallel with Fully Sharded Data Parallel (TP/SP + FSDP) on a example\nLlama2 model. We show an E2E working flow from forward, backward\nand optimization.\n\nWe enabled Fully Sharded Data Parallel + Tensor Parallel in\nseparate parallel dimensions:\n    Data Parallel (\"dp\") across hosts\n    Tensor Parallel (\"tp\") within each host\n\n We use a simple diagram to illustrate below:\n\n======================================================================\n------------       ------------       ------------       ------------\n| Host 1   |       | Host 2   |       |          |       | Host N   |\n| 8 GPUs   |       | 8 GPUs   |       |          |       | 8 GPUs   |\n|          |       |          |       |    ...   |       |          |\n| (TP)     |       | (TP)     |       |          |       | (TP)     |\n|[0,1,..,7]|       |[8,9..,15]|       |          |       |[8N-8,8N-7|\n|          |       |          |       |          |       | .., 8N-1]|\n|          |       |          |       |          |       |          |\n------------       ------------       ------------       ------------\nFSDP:\n[0, 8, ..., 8N-8], [1, 9, ..., 8N-7], ..., [7, 15, ..., 8N-1]\n======================================================================\n\nMore details can be seen in the PyTorch tutorials:\nhttps://pytorch.org/tutorials/intermediate/TP_tutorial.html\n\"\"\"\n\nimport sys\nimport os\nimport torch\nimport torch.distributed as dist\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom log_utils import rank_log, get_logger, verify_min_gpu_count\n\n# ---- GPU check ------------\n_min_gpu_count = 4\n\nif not verify_min_gpu_count(min_gpus=_min_gpu_count):\n    print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n    sys.exit()\n# ---------------------------\n\nfrom llama2_model import Transformer, ModelArgs\n\nfrom torch.distributed.device_mesh import init_device_mesh\nfrom torch.distributed.fsdp import fully_shard\nfrom torch.distributed._tensor import Shard, Replicate\nfrom torch.distributed.tensor.parallel import (\n    parallelize_module,\n    ColwiseParallel,\n    RowwiseParallel,\n    PrepareModuleInput,\n    SequenceParallel\n)\n\ntp_size = 2\nlogger = get_logger()\n\n# understand world topology\n_rank = int(os.environ[\"RANK\"])\n_world_size = int(os.environ[\"WORLD_SIZE\"])\n\n\nprint(f\"Starting PyTorch 2D (FSDP + TP) example on rank {_rank}.\")\nassert (\n    _world_size % tp_size == 0\n), f\"World size {_world_size} needs to be divisible by TP size {tp_size}\"\n\n\n# create a sharding plan based on the given world_size.\ndp_size = _world_size // tp_size\n\ndevice_type = torch.accelerator.current_accelerator().type\n# Create a device mesh with 2 dimensions.\n# First dim is the data parallel dimension\n# Second dim is the tensor parallel dimension.\ndevice_mesh = init_device_mesh(device_type, (dp_size, tp_size), mesh_dim_names=(\"dp\", \"tp\"))\n\nrank_log(_rank, logger, f\"Device Mesh created: {device_mesh=}\")\ntp_mesh = device_mesh[\"tp\"]\ndp_mesh = device_mesh[\"dp\"]\n\n# For TP, input needs to be same across all TP ranks.\n# while for SP, input can be different across all ranks.\n# We will use dp_rank for setting the random seed\n# to mimic the behavior of the dataloader.\ndp_rank = dp_mesh.get_local_rank()\n\n# create model and move it to GPU - initdevice_type_mesh has already mapped GPU ids.\nsimple_llama2_config = ModelArgs(dim=256, n_layers=2, n_heads=16, vocab_size=32000)\n\nmodel = Transformer.from_model_args(simple_llama2_config).to(device_type)\n\n# init model weights\nmodel.init_weights()\n\n# parallelize the first embedding and the last linear out projection\nmodel = parallelize_module(\n    model,\n    tp_mesh,\n    {\n        \"tok_embeddings\": RowwiseParallel(\n            input_layouts=Replicate(),\n            output_layouts=Shard(1),\n        ),\n        \"norm\": SequenceParallel(),\n        \"output\": ColwiseParallel(\n            input_layouts=Shard(1),\n            output_layouts=Replicate()\n        ),\n    }\n)\n\nfor layer_id, transformer_block in enumerate(model.layers):\n    layer_tp_plan = {\n        \"attention_norm\": SequenceParallel(),\n        \"attention\": PrepareModuleInput(\n            input_layouts=(Shard(1), Replicate()),\n            desired_input_layouts=(Replicate(), Replicate()),\n        ),\n        \"attention.wq\": ColwiseParallel(use_local_output=False),\n        \"attention.wk\": ColwiseParallel(use_local_output=False),\n        \"attention.wv\": ColwiseParallel(use_local_output=False),\n        \"attention.wo\": RowwiseParallel(output_layouts=Shard(1)),\n        \"ffn_norm\": SequenceParallel(),\n        \"feed_forward\": PrepareModuleInput(\n            input_layouts=(Shard(1),),\n            desired_input_layouts=(Replicate(),),\n        ),\n        \"feed_forward.w1\": ColwiseParallel(),\n        \"feed_forward.w2\": RowwiseParallel(output_layouts=Shard(1)),\n        \"feed_forward.w3\": ColwiseParallel(),\n    }\n\n    # Custom parallelization plan for the model\n    parallelize_module(\n        module=transformer_block,\n        device_mesh=tp_mesh,\n        parallelize_plan=layer_tp_plan\n    )\n\n# Init FSDP using the dp device mesh\nsharded_model = fully_shard(model, mesh=dp_mesh)\n\nrank_log(_rank, logger, f\"Model after parallelization {sharded_model=}\\n\")\n\n# Create an optimizer for the parallelized and sharded model.\nlr = 3e-3\nrank_log(_rank, logger, f\"Creating AdamW optimizer with learning rate {lr}\")\noptimizer = torch.optim.AdamW(sharded_model.parameters(), lr=lr, foreach=True)\n\n# Training loop:\n# Perform a num of iterations of forward/backward\n# and optimizations for the sharded module.\nrank_log(_rank, logger, \"\\nStarting 2D training...\")\nnum_iterations = 10\nbatch_size = 2\n\nfor i in range(num_iterations):\n    # seeding with dp_rank to ensure identical inputs for TP groups\n    torch.manual_seed(i + dp_rank)\n    inp = torch.randint(32000, (8, 256), device=device_type)\n\n    output = sharded_model(inp)\n    output.sum().backward()\n    optimizer.step()\n    rank_log(_rank, logger, f\"2D iter {i} complete\")\n\nrank_log(_rank, logger, \"2D training successfully completed!\")\n\nif dist.is_initialized():\n    dist.destroy_process_group()\n"
  },
  {
    "path": "distributed/tensor_parallelism/llama2_model.py",
    "content": "# Copyright (c) Meta Platforms, Inc. and affiliates.\n# This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.\n\nfrom dataclasses import dataclass\nfrom typing import Optional, Tuple\n\nimport torch\nimport torch.nn.functional as F\nfrom torch import nn\n\n\n@dataclass\nclass ModelArgs:\n    dim: int = 4096\n    n_layers: int = 32\n    n_heads: int = 32\n    n_kv_heads: Optional[int] = None\n    vocab_size: int = -1  # defined later by tokenizer\n    multiple_of: int = 256  # make SwiGLU hidden layer size multiple of large power of 2\n    ffn_dim_multiplier: Optional[float] = None\n    norm_eps: float = 1e-5\n\n    max_batch_size: int = 32\n    max_seq_len: int = 32768\n    # If `True`, then each transformer block init uses its layer ID, and if\n    # `False`, each uses the total number of transformer blocks\n    depth_init: bool = True\n\n\ndef precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):\n    \"\"\"\n    Precompute the frequency tensor for complex exponentials (cis) with given dimensions.\n\n    This function calculates a frequency tensor with complex exponentials using the given dimension 'dim'\n    and the end index 'end'. The 'theta' parameter scales the frequencies.\n    The returned tensor contains complex values in complex64 data type.\n\n    Args:\n        dim (int): Dimension of the frequency tensor.\n        end (int): End index for precomputing frequencies.\n        theta (float, optional): Scaling factor for frequency computation. Defaults to 10000.0.\n\n    Returns:\n        torch.Tensor: Precomputed frequency tensor with complex exponentials.\n    \"\"\"\n    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))\n    t = torch.arange(end, device=freqs.device)  # type: ignore\n    freqs = torch.outer(t, freqs).float()  # type: ignore\n    freqs_cis = torch.polar(torch.ones_like(freqs), freqs)  # complex64\n    return freqs_cis\n\n\ndef reshape_for_broadcast(freqs_cis: torch.Tensor, x: torch.Tensor):\n    \"\"\"\n    Reshape frequency tensor for broadcasting it with another tensor.\n\n    This function reshapes the frequency tensor to have the same shape as the target tensor 'x'\n    for the purpose of broadcasting the frequency tensor during element-wise operations.\n\n    Args:\n        freqs_cis (torch.Tensor): Frequency tensor to be reshaped.\n        x (torch.Tensor): Target tensor for broadcasting compatibility.\n\n    Returns:\n        torch.Tensor: Reshaped frequency tensor.\n    \"\"\"\n    ndim = x.ndim\n    assert 0 <= 1 < ndim\n    assert freqs_cis.shape == (x.shape[1], x.shape[-1])\n    shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]\n    return freqs_cis.view(*shape)\n\n\ndef apply_rotary_emb(\n    xq: torch.Tensor,\n    xk: torch.Tensor,\n    freqs_cis: torch.Tensor,\n) -> Tuple[torch.Tensor, torch.Tensor]:\n    \"\"\"\n    Apply rotary embeddings to input tensors using the given frequency tensor.\n\n    This function applies rotary embeddings to the given query 'xq' and key 'xk' tensors using the provided\n    frequency tensor 'freqs_cis'. The input tensors are reshaped as complex numbers, and the frequency tensor\n    is reshaped for broadcasting compatibility. The resulting tensors contain rotary embeddings and are\n    returned as real tensors.\n\n    Args:\n        xq (torch.Tensor): Query tensor to apply rotary embeddings.\n        xk (torch.Tensor): Key tensor to apply rotary embeddings.\n        freqs_cis (torch.Tensor): Precomputed frequency tensor for complex exponentials.\n\n    Returns:\n        Tuple[torch.Tensor, torch.Tensor]: Tuple of modified query tensor and key tensor with rotary embeddings.\n    \"\"\"\n    xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))\n    xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))\n    freqs_cis = reshape_for_broadcast(freqs_cis, xq_)\n    xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3)\n    xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3)\n    return xq_out.type_as(xq), xk_out.type_as(xk)\n\n\ndef repeat_kv(x: torch.Tensor, n_rep: int) -> torch.Tensor:\n    \"\"\"torch.repeat_interleave(x, dim=2, repeats=n_rep)\"\"\"\n    bs, slen, n_kv_heads, head_dim = x.shape\n    if n_rep == 1:\n        return x\n    return (\n        x[:, :, :, None, :]\n        .expand(bs, slen, n_kv_heads, n_rep, head_dim)\n        .reshape(bs, slen, n_kv_heads * n_rep, head_dim)\n    )\n\n\nclass RMSNorm(nn.Module):\n    \"\"\"\n    Initialize the RMSNorm normalization layer.\n\n    Args:\n        dim (int): The dimension of the input tensor.\n        eps (float, optional): A small value added to the denominator for numerical stability. Default is 1e-6.\n\n    Attributes:\n        eps (float): A small value added to the denominator for numerical stability.\n        weight (nn.Parameter): Learnable scaling parameter.\n\n    \"\"\"\n\n    def __init__(self, dim: int, eps: float = 1e-6):\n        super().__init__()\n        self.eps = eps\n        self.weight = nn.Parameter(torch.ones(dim))\n\n    def _norm(self, x: torch.Tensor):\n        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)\n\n    def forward(self, x: torch.Tensor):\n        output = self._norm(x.float()).type_as(x)\n        return output * self.weight\n\n    def reset_parameters(self):\n        torch.nn.init.ones_(self.weight)  # type: ignore\n\n\nclass Attention(nn.Module):\n    \"\"\"\n    Multi-head attention module.\n\n    Args:\n        model_args (ModelArgs): Model configuration arguments.\n\n    Attributes:\n        n_kv_heads (int): Number of key and value heads.\n        n_heads (int): Number of query heads.\n        n_local_kv_heads (int): Number of local key and value heads.\n        n_rep (int): Number of repetitions for local heads.\n        head_dim (int): Dimension size of each attention head.\n        wq (Linear): Linear transformation for queries.\n        wk (Linear): Linear transformation for keys.\n        wv (Linear): Linear transformation for values.\n        wo (Linear): Linear transformation for output.\n\n    \"\"\"\n\n    def __init__(self, model_args: ModelArgs):\n        super().__init__()\n        self.n_heads = model_args.n_heads\n        self.n_kv_heads = (\n            model_args.n_heads\n            if model_args.n_kv_heads is None\n            else model_args.n_kv_heads\n        )\n        self.n_rep = self.n_heads // self.n_kv_heads\n        self.head_dim = model_args.dim // model_args.n_heads\n\n        self.wq = nn.Linear(\n            model_args.dim, model_args.n_heads * self.head_dim, bias=False\n        )\n        self.wk = nn.Linear(model_args.dim, self.n_kv_heads * self.head_dim, bias=False)\n        self.wv = nn.Linear(model_args.dim, self.n_kv_heads * self.head_dim, bias=False)\n        self.wo = nn.Linear(\n            model_args.n_heads * self.head_dim, model_args.dim, bias=False\n        )\n\n    def init_weights(self, init_std: float):\n        for linear in (self.wq, self.wk, self.wv):\n            nn.init.trunc_normal_(linear.weight, mean=0.0, std=0.02)\n        nn.init.trunc_normal_(self.wo.weight, mean=0.0, std=init_std)\n\n    def forward(\n        self,\n        x: torch.Tensor,\n        freqs_cis: torch.Tensor,\n    ):\n        \"\"\"\n        Forward pass of the attention module.\n\n        Args:\n            x (torch.Tensor): Input tensor.\n            freqs_cis (torch.Tensor): Precomputed frequency tensor.\n\n        Returns:\n            torch.Tensor: Output tensor after attention.\n\n        \"\"\"\n        bsz, seqlen, _ = x.shape\n        xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)\n\n        xq = xq.view(bsz, seqlen, self.n_heads, self.head_dim)\n        xk = xk.view(bsz, seqlen, self.n_kv_heads, self.head_dim)\n        xv = xv.view(bsz, seqlen, self.n_kv_heads, self.head_dim)\n\n        xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)\n\n        keys = repeat_kv(xk, self.n_rep)  # (bs, seqlen, n_local_heads, head_dim)\n        values = repeat_kv(xv, self.n_rep)  # (bs, seqlen, n_local_heads, head_dim)\n\n        xq = xq.transpose(1, 2)  # (bs, n_local_heads, seqlen, head_dim)\n        xk = keys.transpose(1, 2)  # (bs, n_local_heads, seqlen, head_dim)\n        xv = values.transpose(1, 2)  # (bs, n_local_heads, seqlen, head_dim)\n\n        # we use casual mask for training\n        output = F.scaled_dot_product_attention(xq, xk, xv, is_causal=True)\n        output = output.transpose(\n            1, 2\n        ).contiguous()  # (bs, seqlen, n_local_heads, head_dim)\n        output = output.view(bsz, seqlen, -1)\n        return self.wo(output)\n\n\nclass FeedForward(nn.Module):\n    \"\"\"\n    FeedForward module\n\n    Args:\n        dim (int): Input dimension.\n        hidden_dim (int): Hidden dimension of the feedforward layer.\n        multiple_of (int): Value to ensure hidden dimension is a multiple of this value.\n        ffn_dim_multiplier (Optional[float]): Custom multiplier for hidden dimension. Defaults to None.\n\n    Attributes:\n        w1 (Linear): Linear transformation for the first layer.\n        w2 (Linear): Linear transformation for the second layer.\n        w3 (Linear): Linear transformation for the third layer.\n\n    \"\"\"\n\n    def __init__(\n        self,\n        dim: int,\n        hidden_dim: int,\n        multiple_of: int,\n        ffn_dim_multiplier: Optional[float],\n    ):\n        super().__init__()\n        hidden_dim = int(2 * hidden_dim / 3)\n        # custom dim factor multiplier\n        if ffn_dim_multiplier is not None:\n            hidden_dim = int(ffn_dim_multiplier * hidden_dim)\n        hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)\n\n        self.w1 = nn.Linear(dim, hidden_dim, bias=False)\n        self.w2 = nn.Linear(hidden_dim, dim, bias=False)\n        self.w3 = nn.Linear(dim, hidden_dim, bias=False)\n\n    def forward(self, x):\n        return self.w2(F.silu(self.w1(x)) * self.w3(x))\n\n    def init_weights(self, init_std: float):\n        nn.init.trunc_normal_(self.w1.weight, mean=0.0, std=0.02)\n        for linear in (self.w2, self.w3):\n            nn.init.trunc_normal_(linear.weight, mean=0.0, std=init_std)\n\n\nclass TransformerBlock(nn.Module):\n    \"\"\"\n    TransformerBlock Module\n\n    Args:\n        layer_id (int): Identifier for the layer.\n        model_args (ModelArgs): Model configuration arguments.\n\n    Attributes:\n        n_heads (int): Number of attention heads.\n        dim (int): Dimension size of the model.\n        head_dim (int): Dimension size of each attention head.\n        attention (Attention): Attention module.\n        feed_forward (FeedForward): FeedForward module.\n        layer_id (int): Identifier for the layer.\n        attention_norm (RMSNorm): Layer normalization for attention output.\n        ffn_norm (RMSNorm): Layer normalization for feedforward output.\n\n    \"\"\"\n\n    def __init__(self, layer_id: int, model_args: ModelArgs):\n        super().__init__()\n        self.n_heads = model_args.n_heads\n        self.dim = model_args.dim\n        self.attention = Attention(model_args)\n        self.feed_forward = FeedForward(\n            dim=model_args.dim,\n            hidden_dim=4 * model_args.dim,\n            multiple_of=model_args.multiple_of,\n            ffn_dim_multiplier=model_args.ffn_dim_multiplier,\n        )\n        self.layer_id = layer_id\n        self.num_layers = model_args.n_layers\n\n        self.attention_norm = RMSNorm(\n            dim=model_args.dim, eps=model_args.norm_eps\n        )\n        self.ffn_norm = RMSNorm(\n            dim=model_args.dim, eps=model_args.norm_eps\n        )\n\n        if model_args.depth_init:\n            self.weight_init_std = 0.02 / (2 * (self.layer_id + 1)) ** 0.5\n        else:\n            self.weight_init_std = 0.02 / (2 * self.num_layers) ** 0.5\n\n    def forward(\n        self,\n        x: torch.Tensor,\n        freqs_cis: torch.Tensor,\n    ):\n        \"\"\"\n        Perform a forward pass through the TransformerBlock.\n\n        Args:\n            x (torch.Tensor): Input tensor.\n            freqs_cis (torch.Tensor): Precomputed cosine and sine frequencies.\n\n        Returns:\n            torch.Tensor: Output tensor after applying attention and feedforward layers.\n\n        \"\"\"\n        h = x + self.attention(self.attention_norm(x), freqs_cis)\n        out = h + self.feed_forward(self.ffn_norm(h))\n        return out\n\n    def init_weights(self):\n        for norm in (self.attention_norm, self.ffn_norm):\n            norm.reset_parameters()\n        self.attention.init_weights(self.weight_init_std)\n        self.feed_forward.init_weights(self.weight_init_std)\n\n\nclass Transformer(nn.Module):\n    \"\"\"\n    Transformer Module\n\n    Args:\n        model_args (ModelArgs): Model configuration arguments.\n\n    Attributes:\n        model_args (ModelArgs): Model configuration arguments.\n        vocab_size (int): Vocabulary size.\n        n_layers (int): Number of layers in the model.\n        tok_embeddings (ParallelEmbedding): Token embeddings.\n        layers (torch.nn.ModuleList): List of Transformer blocks.\n        norm (RMSNorm): Layer normalization for the model output.\n        output (ColumnParallelLinear): Linear layer for final output.\n        freqs_cis (torch.Tensor): Precomputed cosine and sine frequencies.\n\n    \"\"\"\n\n    def __init__(self, model_args: ModelArgs):\n        super().__init__()\n        self.model_args = model_args\n        self.vocab_size = model_args.vocab_size\n        self.n_layers = model_args.n_layers\n        self.model_dim = model_args.dim\n\n        self.tok_embeddings = nn.Embedding(model_args.vocab_size, model_args.dim)\n        self.register_buffer(\n            \"freqs_cis\",\n            precompute_freqs_cis(\n                model_args.dim // model_args.n_heads,\n                # Need to compute until at least the max token limit for generation\n                # (use 2x max sequence length to be safe)\n                model_args.max_seq_len * 2,\n            ),\n        )\n        self.layers = torch.nn.ModuleList()\n        for layer_id in range(model_args.n_layers):\n            self.layers.append(TransformerBlock(layer_id, model_args))\n\n        self.norm = RMSNorm(\n            dim=model_args.dim, eps=model_args.norm_eps\n        )\n\n        self.output = nn.Linear(model_args.dim, model_args.vocab_size, bias=False)\n        self.init_weights()\n\n    def init_weights(self):\n        \"\"\"\n        [Note: On ``init_weights`` vs. ``reset_parameters``]\n        Modules may define ``reset_parameters`` to initialize parameter values.\n        ``reset_parameters`` is meant to only initialize directly owned\n        parameters/buffers, not those of their child modules, and it can be\n        used to give the initial values for these tensors.\n        Separately, users may want custom initialization for their modules,\n        different from that in ``reset_parameters``. For this, we define\n        ``init_weights``. We only call it in the constructor of this\n        ``Transformer`` root module to avoid reinitializing tensors.\n        \"\"\"\n        with torch.device(self.freqs_cis.device):\n            self.freqs_cis = precompute_freqs_cis(\n                self.model_args.dim // self.model_args.n_heads,\n                # Need to compute until at least the max token limit for generation\n                # (use 2x max sequence length to be safe)\n                self.model_args.max_seq_len * 2,\n            )\n        nn.init.normal_(self.tok_embeddings.weight)\n        for layer in self.layers:\n            layer.init_weights()\n        self.norm.reset_parameters()\n        final_out_std = self.model_args.dim**-0.5\n        cutoff_factor = 3\n        nn.init.trunc_normal_(\n            self.output.weight,\n            mean=0.0,\n            std=final_out_std,\n            a=-cutoff_factor * final_out_std,\n            b=cutoff_factor * final_out_std,\n        )\n\n    def forward(self, tokens: torch.Tensor):\n        \"\"\"\n        Perform a forward pass through the Transformer model.\n\n        Args:\n            tokens (torch.Tensor): Input token indices.\n\n        Returns:\n            torch.Tensor: Output logits after applying the Transformer model.\n\n        \"\"\"\n        _bsz, seqlen = tokens.shape\n        h = self.tok_embeddings(tokens)\n        self.freqs_cis = self.freqs_cis.to(h.device)\n        freqs_cis = self.freqs_cis[0:seqlen]\n\n        for layer in self.layers:\n            h = layer(h, freqs_cis)\n        h = self.norm(h)\n        output = self.output(h).float()\n        return output\n\n    @classmethod\n    def from_model_args(cls, model_args: ModelArgs) -> \"Transformer\":\n        \"\"\"\n        Initialize a Transformer model from a ModelArgs object.\n\n        Args:\n            model_args (ModelArgs): Model configuration arguments.\n\n        Returns:\n            Transformer: Transformer model.\n\n        \"\"\"\n        return cls(model_args)\n"
  },
  {
    "path": "distributed/tensor_parallelism/log_utils.py",
    "content": "import logging\nimport torch\n\nlogging.basicConfig(\n    format=\"%(asctime)s %(message)s\", datefmt=\"%m/%d/%Y %I:%M:%S %p\", level=logging.INFO\n)\n\ndef get_logger():\n    return logging.getLogger(__name__)\n\n\ndef rank_log(_rank, logger, msg):\n    \"\"\"helper function to log only on global rank 0\"\"\"\n    if _rank == 0:\n        logger.info(f\" {msg}\")\n\n\ndef verify_min_gpu_count(min_gpus: int = 2) -> bool:\n    \"\"\" verification that we have at least 2 gpus to run dist examples \"\"\"\n    has_gpu = torch.accelerator.is_available()\n    gpu_count = torch.accelerator.device_count()\n    return has_gpu and gpu_count >= min_gpus\n"
  },
  {
    "path": "distributed/tensor_parallelism/requirements.txt",
    "content": "# Python dependencies required for running the example\n\ntorch >= 2.7.1; sys_platform == \"linux\"\n"
  },
  {
    "path": "distributed/tensor_parallelism/run_example.sh",
    "content": "\n# To run samples:\n# bash run_example.sh {file_to_run.py} {num_gpus}\n# where file_to_run = example to launch.  Default = 'fsdp_tp_example.py'\n# num_gpus = num local gpus to use (must be at least 2). Default = 4\n\n# samples to run include:\n# sequence_parallel_example.py\n# tensor_parallel_example.py\n# fsdp_tp_example.py\n\necho \"Launching ${1:-fsdp_tp_example.py} with ${2:-4} gpus\"\ntorchrun --nnodes=1 --nproc_per_node=${2:-4} --rdzv_id=101 --rdzv_endpoint=\"localhost:5972\" ${1:-fsdp_tp_example.py}\n"
  },
  {
    "path": "distributed/tensor_parallelism/sequence_parallel_example.py",
    "content": "\"\"\"\nThis is the script to test Sequence Parallel(SP) on a toy model in a\nMegetron-LM SPMD style. We show an E2E working flow from forward,\nbackward and optimization.\n\nWe use the example of two `nn.Linear` layers with an element-wise `nn.RELU`\nin between to show an example of sequence parallel, which was proposed in paper:\n\nhttps://arxiv.org/pdf/2205.05198.pdf.\n\nLike tensor parallel, we parallelize the first linear layer by column\nand also parallelize the second linear layer by row. But the input in each rank\nnow is different so that we need one all-gather for input and one reduce-scatter\nin the end of the second linear layer.\n\nThe following is an example command to run this code\n    torchrun --nnodes 1 --nproc-per-node 4 sequence_parallel_example.py\n\"\"\"\n\nimport os\nimport sys\nimport torch\nimport torch.nn as nn\n\nimport torch.distributed as dist\nfrom torch.distributed._tensor import Shard\n\nfrom torch.distributed.tensor.parallel import (\n    parallelize_module,\n    ColwiseParallel,\n    RowwiseParallel,\n)\n\nfrom log_utils import rank_log, get_logger, verify_min_gpu_count\n\n\n# ---- GPU check ------------\n_min_gpu_count = 2\n\nif not verify_min_gpu_count(min_gpus=_min_gpu_count):\n    print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n    sys.exit()\n# ---------------------------\n\nfrom torch.distributed._tensor.device_mesh import init_device_mesh\n\nclass ToyModel(nn.Module):\n    \"\"\"MLP based model\"\"\"\n\n    def __init__(self):\n        super().__init__()\n        self.in_proj = nn.Linear(10, 32)\n        self.relu = nn.ReLU()\n        self.out_proj = nn.Linear(32, 5)\n\n    def forward(self, x):\n        return self.out_proj(self.relu(self.in_proj(x)))\n\n\n\"\"\"\nMain body of the demo of a basic version of sequence parallel by using\nPyTorch native APIs.\n\"\"\"\nlogger = get_logger()\n\ndevice_type = torch.accelerator.current_accelerator().type\n# create a device mesh based on the given world_size.\ndevice_mesh = init_device_mesh(\n    device_type=device_type, mesh_shape=(int(os.environ[\"WORLD_SIZE\"]),)\n)\n\n_rank = device_mesh.get_rank()\n\nprint(f\"Starting PyTorch Sequence Parallel example on rank {_rank}.\")\n\nrank_log(_rank, logger, f\"Device Mesh created: {device_mesh=}\")\n\n# create model and move it to GPU.  Init_device_mesh has already assigned gpu ids...\nmodel = ToyModel().to(device_type)\n\n# Custom parallelization plan for the model\nsp_model = parallelize_module(\n    module=model,\n    device_mesh=device_mesh,\n    parallelize_plan={\n        \"in_proj\": ColwiseParallel(input_layouts=Shard(0)),\n        \"out_proj\": RowwiseParallel(output_layouts=Shard(0)),\n    },\n)\n\n\n# Create a optimizer for the parallelized module.\nlr = 0.25\noptimizer = torch.optim.AdamW(sp_model.parameters(), lr=lr, foreach=True)\n\n\n# Perform a num of iterations of forward/backward\n# and optimizations for the sharded module.\nnum_iters = 10\nrank_log(_rank, logger, \"Sequence Parallel training starting...\")\n\nfor i in range(num_iters):\n    # For SP, input can be different across all ranks.\n    inp = torch.rand(20, 10, device=device_type)\n    output = sp_model(inp)\n    output.sum().backward()\n    optimizer.step()\n    rank_log(_rank, logger, f\"Sequence Parallel iter {i} completed\")\n\nrank_log(_rank, logger, \"Sequence Parallel training completed!\")\n\nif dist.is_initialized():\n    dist.destroy_process_group()\n"
  },
  {
    "path": "distributed/tensor_parallelism/tensor_parallel_example.py",
    "content": "\"\"\"\nThis is the script to test Tensor Parallel(TP) on a toy model in a\nMegetron-LM SPMD style. We show an E2E working flow from forward,\nbackward and optimization.\n\nMore context about API designs can be found in the design:\n\nhttps://github.com/pytorch/pytorch/issues/89884.\n\nAnd it is built on top of Distributed Tensor which is proposed in:\n\nhttps://github.com/pytorch/pytorch/issues/88838.\n\nWe use the example of two `nn.Linear` layers with an element-wise `nn.RELU`\nin between to show an example of Megatron-LM, which was proposed in paper:\n\nhttps://arxiv.org/abs/1909.08053.\n\nThe basic idea is that we parallelize the first linear layer by column\nand also parallelize the second linear layer by row so that we only need\none all reduce in the end of the second linear layer.\n\nWe can speed up the model training by avoiding communications between\ntwo layers.\n\nTo parallelize a nn module, we need to specify what parallel style we want\nto use and our `parallelize_module` API will parse and parallelize the modules\nbased on the given `ParallelStyle`. We are using this PyTorch native Tensor\nParallelism APIs in this example to show users how to use them.\n\nThe following is an example command to run this code\n    torchrun --nnodes 1 --nproc-per-node 4 tensor_parallel_example.py\n\"\"\"\n\nimport os\nimport sys\nimport torch\nimport torch.nn as nn\nimport torch.distributed as dist\nfrom torch.distributed.tensor.parallel import (\n    parallelize_module,\n    ColwiseParallel,\n    RowwiseParallel,\n)\nfrom log_utils import rank_log, get_logger, verify_min_gpu_count\n\n# ---- GPU check ------------\n_min_gpu_count = 2\n\nif not verify_min_gpu_count(min_gpus=_min_gpu_count):\n    print(f\"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.\")\n    sys.exit()\n# ---------------------------\n\nfrom torch.distributed._tensor.device_mesh import init_device_mesh\n\nclass ToyModel(nn.Module):\n    \"\"\"MLP based model\"\"\"\n\n    def __init__(self):\n        super(ToyModel, self).__init__()\n        self.in_proj = nn.Linear(10, 32)\n        self.relu = nn.ReLU()\n        self.out_proj = nn.Linear(32, 5)\n\n    def forward(self, x):\n        return self.out_proj(self.relu(self.in_proj(x)))\n\n\n\"\"\"\nMain body of the demo of a basic version of tensor parallel by using\nPyTorch native APIs.\n\"\"\"\nlogger = get_logger()\n\n# create a device mesh based on the given world_size.\n_world_size = int(os.environ[\"WORLD_SIZE\"])\ndevice_type = torch.accelerator.current_accelerator().type\ndevice_mesh = init_device_mesh(device_type=device_type, mesh_shape=(_world_size,))\n_rank = device_mesh.get_rank()\n\n\nprint(f\"Starting PyTorch TP example on rank {_rank}.\")\nassert (\n    _world_size % 2 == 0\n), f\"TP examples require even number of GPUs, but got {_world_size} gpus\"\n\nrank_log(_rank, logger, f\"Device Mesh created: {device_mesh=}\")\n\n# create model and move it to GPU - initdevice_type_mesh has already mapped GPU ids.\ntp_model = ToyModel().to(device_type)\n\n\n# Custom parallelization plan for the model\ntp_model = parallelize_module(\n    module=tp_model,\n    device_mesh=device_mesh,\n    parallelize_plan={\n        \"in_proj\": ColwiseParallel(),\n        \"out_proj\": RowwiseParallel(),\n    },\n)\n\n# Create an optimizer for the parallelized module.\nlr = 0.25\noptimizer = torch.optim.AdamW(tp_model.parameters(), lr=lr, foreach=True)\n\n\n# Perform a num of iterations of forward/backward\n# and optimizations for the sharded module.\nnum_iters = 10\nrank_log(_rank, logger, \"Tensor Parallel training starting...\")\n\nfor i in range(num_iters):\n    # For TP, input needs to be same across all TP ranks.\n    # Setting the random seed is to mimic the behavior of dataloader.\n    torch.manual_seed(i)\n    inp = torch.rand(20, 10, device=device_type)\n    output = tp_model(inp)\n    output.sum().backward()\n    optimizer.step()\n    rank_log(_rank, logger, f\"Tensor Parallel iter {i} completed\")\n\nrank_log(_rank, logger, \"Tensor Parallel training completed!\")\n\nif dist.is_initialized():\n    dist.destroy_process_group()\n"
  },
  {
    "path": "docs/Makefile",
    "content": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the environment for the first two.\nSPHINXOPTS    ?=\nSPHINXBUILD   ?= sphinx-build\nSOURCEDIR     = source\nBUILDDIR      = build\n\n# Put it first so that \"make\" without argument is like \"make help\".\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n.PHONY: help Makefile\n\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n"
  },
  {
    "path": "docs/make.bat",
    "content": "@ECHO OFF\r\n\r\npushd %~dp0\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sphinx-build\r\n)\r\nset SOURCEDIR=source\r\nset BUILDDIR=build\r\n\r\nif \"%1\" == \"\" goto help\r\n\r\n%SPHINXBUILD% >NUL 2>NUL\r\nif errorlevel 9009 (\r\n\techo.\r\n\techo.The 'sphinx-build' command was not found. Make sure you have Sphinx\r\n\techo.installed, then set the SPHINXBUILD environment variable to point\r\n\techo.to the full path of the 'sphinx-build' executable. Alternatively you\r\n\techo.may add the Sphinx directory to PATH.\r\n\techo.\r\n\techo.If you don't have Sphinx installed, grab it from\r\n\techo.https://www.sphinx-doc.org/\r\n\texit /b 1\r\n)\r\n\r\n%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\ngoto end\r\n\r\n:help\r\n%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\n\r\n:end\r\npopd\r\n"
  },
  {
    "path": "docs/requirements.txt",
    "content": "\nsphinx==5.3.0\n#Pinned versions: 5.3.0\n-e git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme\nsphinx-panels\n"
  },
  {
    "path": "docs/source/_static/.gitkeep",
    "content": ""
  },
  {
    "path": "docs/source/conf.py",
    "content": "#!/usr/bin/env python3\n# Copyright (c) Meta Platforms, Inc. and affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the BSD-style license found in the\n# LICENSE file in the root directory of this source tree.\n\n# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common options. For a full\n# list see the documentation:\n# https://www.sphinx-doc.org/en/master/usage/configuration.html\n\n# -- Path setup --------------------------------------------------------------\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n#\nimport os\nimport sys\n\nimport pytorch_sphinx_theme\n\ncurrent_dir = os.path.dirname(__file__)\ntarget_dir = os.path.abspath(os.path.join(current_dir, \"../..\"))\nsys.path.insert(0, target_dir)\nprint(target_dir)\n\n# -- Project information -----------------------------------------------------\n\nproject = \"PyTorchExamples\"\ncopyright = \"2022, Meta\"\nauthor = \"Meta\"\n\n# The full version, including alpha/beta/rc tags\nrelease = \"1.11\"\n\n# -- General configuration ---------------------------------------------------\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\"sphinx.ext.napoleon\", \"sphinx.ext.autodoc\", \"sphinx_panels\"]\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = [\"_templates\"]\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path.\nexclude_patterns = []\n\n# -- Options for HTML output -------------------------------------------------\n\n# The theme to use for HTML and HTML Help pages.  See the documentation for\n# a list of builtin themes.\n#\n# html_theme = 'alabaster'\nhtml_theme = \"pytorch_sphinx_theme\"\nhtml_theme_path = [pytorch_sphinx_theme.get_html_theme_path()]\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\n\nhtml_static_path = [\"_static\"]\npanels_add_fontawesome_latex = True\n\nhtml_theme_options = {\n    'pytorch_project': 'examples',\n    'collapse_navigation': False,\n    'display_version': True,\n    'logo_only': False,\n    'analytics_id': 'UA-117752657-2',\n}\n"
  },
  {
    "path": "docs/source/index.rst",
    "content": "PyTorch Examples\n================\n\nThis pages lists various PyTorch examples that you can use to learn and\nexperiment with PyTorch.\n\n.. panels::\n\n    Image Classification Using ConvNets\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \n    This example demonstrates how to run image classification\n    with `Convolutional Neural Networks ConvNets <https://cs231n.github.io/convolutional-networks/>`__\n    on the `MNIST <https://en.wikipedia.org/wiki/MNIST_database>`__ database.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/mnist>`__ :opticon:`link-external` \n\n    ---\n\n    Measuring Similarity using Siamese Network\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \n    This example demonstrates how to measure similarity between two images\n    using `Siamese network <https://en.wikipedia.org/wiki/Siamese_neural_network>`__\n    on the `MNIST <https://en.wikipedia.org/wiki/MNIST_database>`__ database.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/siamese_network>`__ :opticon:`link-external` \n\n    ---\n\n    Word-level Language Modeling using RNN and Transformer\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example demonstrates how to train a multi-layer `recurrent neural\n    network (RNN) <https://en.wikipedia.org/wiki/Recurrent_neural_network>`__,\n    such as Elman, GRU, or LSTM, or Transformer on a language\n    modeling task by using the Wikitext-2 dataset. \n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/word_language_model>`__ :opticon:`link-external`\n    ---\n\n    Training ImageNet Classifiers\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example demonstrates how you can train some of the most popular\n    model architectures, including `ResNet <https://en.wikipedia.org/wiki/Residual_neural_network>`__, \n    `AlexNet <https://en.wikipedia.org/wiki/AlexNet>`__, and `VGG <https://arxiv.org/pdf/1409.1556.pdf>`__\n    on the `ImageNet <https://image-net.org/>`__ dataset.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/imagenet>`__ :opticon:`link-external`\n    ---\n\n    Generative Adversarial Networks (DCGAN)\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example implements the `Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks  <https://arxiv.org/abs/1511.06434>`__ paper.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/dcgan>`__ :opticon:`link-external`\n    ---\n     \n    Variational Auto-Encoders\n    ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example implements the `Auto-Encoding Variational Bayes <https://arxiv.org/abs/1312.6114>`__ paper\n    with `ReLUs <https://en.wikipedia.org/wiki/Rectifier_(neural_networks)>`__ and the Adam optimizer.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/vae>`__ :opticon:`link-external`\n    ---\n   \n    Super-resolution Using an Efficient Sub-Pixel CNN\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example demonstrates how to use the sub-pixel convolution layer\n    described in `Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network <https://arxiv.org/abs/1609.05158>`__ paper. This example trains a super-resolution\n    network on the `BSD300 dataset <https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/>`__. \n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/super_resolution>`__ :opticon:`link-external`\n\n    ---\n    HOGWILD! Training of Shared ConvNets\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    `HOGWILD! <https://arxiv.org/abs/1106.5730>`__ is a scheme that allows\n    Stochastic Gradient Descent (SGD)\n    parallelization without memory locking. This example demonstrates how\n    to perform HOGWILD! training of shared ConvNets on MNIST.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/mnist_hogwild>`__ :opticon:`link-external`\n\n    ---\n    Training a CartPole to balance with actor-critic\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This reinforcement learning tutorial demonstrates how to train a\n    CartPole to balance\n    in the `Gymnasium <https://gymnasium.farama.org/>`__ toolkit by using the\n    `Actor-Critic <https://proceedings.neurips.cc/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf>`__ method.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/reinforcement_learning>`__ :opticon:`link-external`\n    ---\n\n    Time Sequence Prediction\n    ^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This beginner example demonstrates how to use LSTMCell to\n    learn sine wave signals to predict the signal values in the future.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/tree/main/time_sequence_prediction>`__ :opticon:`link-external`\n\n    ---\n\n    Implement the Neural Style Transfer algorithm on images\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This tutorial demonstrates how you can use PyTorch's implementation\n    of the `Neural Style Transfer (NST) <https://en.wikipedia.org/wiki/Neural_style_transfer>`__\n    algorithm on images.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/fast_neural_style>`__ :opticon:`link-external`\n    ---\n\n    PyTorch Module Transformations using fx\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This set of examples demonstrates the torch.fx toolkit. For more\n    information about `torch.fx`, see\n    `torch.fx Overview <https://pytorch.org/docs/master/fx.html>`__.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/fx>`__ :opticon:`link-external`\n    ---\n\n    Distributed PyTorch\n    ^^^^^^^^^^^^^^^^^^^\n\n    This set of examples demonstrates `Distributed Data Parallel (DDP) <https://pytorch.org/tutorials/intermediate/ddp_tutorial.html>`__ and `Distributed RPC framework <https://pytorch.org/docs/stable/rpc.html>`__. \n    Includes the code used in the `DDP tutorial series <https://pytorch.org/tutorials/beginner/ddp_series_intro.html>`__.\n\n    `GO TO EXAMPLES <https://github.com/pytorch/examples/tree/main/distributed>`__ :opticon:`link-external`\n    \n    ---\n\n    C++ Frontend\n    ^^^^^^^^^^^^\n\n    The PyTorch C++ frontend is a C++14 library for CPU and GPU tensor computation.\n    This set of examples includes a linear regression, autograd, image recognition\n    (MNIST), and other useful examples using PyTorch C++ frontend.\n\n    `GO TO EXAMPLES <https://github.com/pytorch/examples/tree/main/cpp>`__ :opticon:`link-external`\n\n    ---\n\n    Image Classification Using Forward-Forward Algorithm\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example implements the paper `The Forward-Forward Algorithm: Some Preliminary Investigations <https://arxiv.org/pdf/2212.13345.pdf>`__ by Geoffrey Hinton.\n    on the `MNIST <https://en.wikipedia.org/wiki/MNIST_database>`__ database.\n    It is an introductory example to the Forward-Forward algorithm.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/tree/main/mnist_forward_forward>`__ :opticon:`link-external` \n\n    ---\n\n    Graph Convolutional Network\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n    This example implements the `Semi-Supervised Classification with Graph Convolutional Networks <https://arxiv.org/pdf/1609.02907.pdf>`__ paper on the CORA database.\n\n    `GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/gcn>`__ :opticon:`link-external` \n"
  },
  {
    "path": "fast_neural_style/README.md",
    "content": "# fast-neural-style :city_sunrise: :rocket:\n\nThis repository contains a pytorch implementation of an algorithm for artistic style transfer. The algorithm can be used to mix the content of an image with the style of another image. For example, here is a photograph of a door arch rendered in the style of a stained glass painting.\n\nThe model uses the method described in [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://arxiv.org/abs/1603.08155) along with [Instance Normalization](https://arxiv.org/pdf/1607.08022.pdf). The saved-models for examples shown in the README can be downloaded from [here](https://www.dropbox.com/s/lrvwfehqdcxoza8/saved_models.zip?dl=0).\n\n<p align=\"center\">\n    <img src=\"images/style-images/mosaic.jpg\" height=\"200px\">\n    <img src=\"images/content-images/amber.jpg\" height=\"200px\">\n    <img src=\"images/output-images/amber-mosaic.jpg\" height=\"440px\">\n</p>\n\n## Requirements\n\nThe program is written in Python, and uses [pytorch](http://pytorch.org/), [scipy](https://www.scipy.org). A GPU is not necessary, but can provide a significant speed up especially for training a new model. Regular sized images can be styled on a laptop or desktop using saved models.\n\n## Usage\n\nStylize image\n\n```\npython neural_style/neural_style.py eval --content-image </path/to/content/image> --model </path/to/saved/model> --output-image </path/to/output/image> --accel\n```\n\n- `--content-image`: path to content image you want to stylize.\n- `--model`: saved model to be used for stylizing the image (eg: `mosaic.pth`)\n- `--output-image`: path for saving the output image.\n- `--content-scale`: factor for scaling down the content image if memory is an issue (eg: value of 2 will halve the height and width of content-image)\n- `--accel`: use accelerator\n\nTrain model\n\n```bash\npython neural_style/neural_style.py train --dataset </path/to/train-dataset> --style-image </path/to/style/image> --save-model-dir </path/to/save-model/folder> --epochs 2 --accel\n```\n\nThere are several command line arguments, the important ones are listed below\n\n- `--dataset`: path to training dataset, the path should point to a folder containing another folder with all the training images. I used COCO 2014 Training images dataset [80K/13GB] [(download)](https://cocodataset.org/#download).\n- `--style-image`: path to style-image.\n- `--save-model-dir`: path to folder where trained model will be saved.\n- `--accel`: use accelerator.\n\nIf `--accel` argument is given, pytorch will search for available hardware acceleration device and attempt to use it. This example is known to work on CUDA, MPS and XPU devices.\n\nRefer to `neural_style/neural_style.py` for other command line arguments. For training new models you might have to tune the values of `--content-weight` and `--style-weight`. The mosaic style model shown above was trained with `--content-weight 1e5` and `--style-weight 1e10`. The remaining 3 models were also trained with similar order of weight parameters with slight variation in the `--style-weight` (`5e10` or `1e11`).\n\n## Models\n\nModels for the examples shown below can be downloaded from [here](https://www.dropbox.com/s/lrvwfehqdcxoza8/saved_models.zip?dl=0) or by running the script `download_saved_models.py`.\n\n<div align='center'>\n  <img src='images/content-images/amber.jpg' height=\"174px\">\t\t\n</div>\n\n<div align='center'>\n  <img src='images/style-images/mosaic.jpg' height=\"174px\">\n  <img src='images/output-images/amber-mosaic.jpg' height=\"174px\">\n  <img src='images/output-images/amber-candy.jpg' height=\"174px\">\n  <img src='images/style-images/candy.jpg' height=\"174px\">\n  <br>\n  <img src='images/style-images/rain-princess-cropped.jpg' height=\"174px\">\n  <img src='images/output-images/amber-rain-princess.jpg' height=\"174px\">\n  <img src='images/output-images/amber-udnie.jpg' height=\"174px\">\n  <img src='images/style-images/udnie.jpg' height=\"174px\">\n</div>\n"
  },
  {
    "path": "fast_neural_style/download_saved_models.py",
    "content": "import os\nimport zipfile\n\n# PyTorch 1.1 moves _download_url_to_file\n#   from torch.utils.model_zoo to torch.hub\n# PyTorch 1.0 exists another _download_url_to_file\n#   2 argument\n# TODO: If you remove support PyTorch 1.0 or older,\n#       You should remove torch.utils.model_zoo\n#       Ref. PyTorch #18758\n#         https://github.com/pytorch/pytorch/pull/18758/commits\ntry:\n    from torch.utils.model_zoo import _download_url_to_file\nexcept ImportError:\n    try:\n        from torch.hub import download_url_to_file as _download_url_to_file\n    except ImportError:\n        from torch.hub import _download_url_to_file\n\n\ndef unzip(source_filename, dest_dir):\n    with zipfile.ZipFile(source_filename) as zf:\n        zf.extractall(path=dest_dir)\n\n\nif __name__ == '__main__':\n    _download_url_to_file('https://www.dropbox.com/s/lrvwfehqdcxoza8/saved_models.zip?dl=1', 'saved_models.zip', None, True)\n    unzip('saved_models.zip', '.')\n"
  },
  {
    "path": "fast_neural_style/neural_style/__init__.py",
    "content": ""
  },
  {
    "path": "fast_neural_style/neural_style/neural_style.py",
    "content": "import argparse\nimport os\nimport sys\nimport time\nimport re\n\nimport numpy as np\nimport torch\nfrom torch.optim import Adam\nfrom torch.utils.data import DataLoader\nfrom torchvision import datasets\nfrom torchvision import transforms\nimport torch.onnx\n\nimport utils\nfrom transformer_net import TransformerNet\nfrom vgg import Vgg16\n\n\ndef check_paths(args):\n    try:\n        if not os.path.exists(args.save_model_dir):\n            os.makedirs(args.save_model_dir)\n        if args.checkpoint_model_dir is not None and not (os.path.exists(args.checkpoint_model_dir)):\n            os.makedirs(args.checkpoint_model_dir)\n    except OSError as e:\n        print(e)\n        sys.exit(1)\n\n\ndef train(args):\n    if args.accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n\n    print(f\"Using device: {device}\")\n\n    np.random.seed(args.seed)\n    torch.manual_seed(args.seed)\n\n    transform = transforms.Compose([\n        transforms.Resize(args.image_size),\n        transforms.CenterCrop(args.image_size),\n        transforms.ToTensor(),\n        transforms.Lambda(lambda x: x.mul(255))\n    ])\n    train_dataset = datasets.ImageFolder(args.dataset, transform)\n    train_loader = DataLoader(train_dataset, batch_size=args.batch_size)\n\n    transformer = TransformerNet().to(device)\n    optimizer = Adam(transformer.parameters(), args.lr)\n    mse_loss = torch.nn.MSELoss()\n\n    vgg = Vgg16(requires_grad=False).to(device)\n    style_transform = transforms.Compose([\n        transforms.ToTensor(),\n        transforms.Lambda(lambda x: x.mul(255))\n    ])\n    style = utils.load_image(args.style_image, size=args.style_size)\n    style = style_transform(style)\n    style = style.repeat(args.batch_size, 1, 1, 1).to(device)\n\n    features_style = vgg(utils.normalize_batch(style))\n    gram_style = [utils.gram_matrix(y) for y in features_style]\n\n    for e in range(args.epochs):\n        transformer.train()\n        agg_content_loss = 0.\n        agg_style_loss = 0.\n        count = 0\n        for batch_id, (x, _) in enumerate(train_loader):\n            n_batch = len(x)\n            count += n_batch\n            optimizer.zero_grad()\n\n            x = x.to(device)\n            y = transformer(x)\n\n            y = utils.normalize_batch(y)\n            x = utils.normalize_batch(x)\n\n            features_y = vgg(y)\n            features_x = vgg(x)\n\n            content_loss = args.content_weight * mse_loss(features_y.relu2_2, features_x.relu2_2)\n\n            style_loss = 0.\n            for ft_y, gm_s in zip(features_y, gram_style):\n                gm_y = utils.gram_matrix(ft_y)\n                style_loss += mse_loss(gm_y, gm_s[:n_batch, :, :])\n            style_loss *= args.style_weight\n\n            total_loss = content_loss + style_loss\n            total_loss.backward()\n            optimizer.step()\n\n            agg_content_loss += content_loss.item()\n            agg_style_loss += style_loss.item()\n\n            if (batch_id + 1) % args.log_interval == 0:\n                mesg = \"{}\\tEpoch {}:\\t[{}/{}]\\tcontent: {:.6f}\\tstyle: {:.6f}\\ttotal: {:.6f}\".format(\n                    time.ctime(), e + 1, count, len(train_dataset),\n                                  agg_content_loss / (batch_id + 1),\n                                  agg_style_loss / (batch_id + 1),\n                                  (agg_content_loss + agg_style_loss) / (batch_id + 1)\n                )\n                print(mesg)\n\n            if args.checkpoint_model_dir is not None and (batch_id + 1) % args.checkpoint_interval == 0:\n                transformer.eval().cpu()\n                ckpt_model_filename = \"ckpt_epoch_\" + str(e) + \"_batch_id_\" + str(batch_id + 1) + \".pth\"\n                ckpt_model_path = os.path.join(args.checkpoint_model_dir, ckpt_model_filename)\n                torch.save(transformer.state_dict(), ckpt_model_path)\n                transformer.to(device).train()\n\n    # save model\n    transformer.eval().cpu()\n    timestamp = time.strftime(\"%Y-%m-%d_%H-%M-%S\")\n    save_model_filename = f\"epoch_{args.epochs}_{timestamp}_{args.content_weight}_{args.style_weight}.model\"\n    save_model_path = os.path.join(args.save_model_dir, save_model_filename)\n    torch.save(transformer.state_dict(), save_model_path)\n\n    print(\"\\nDone, trained model saved at\", save_model_path)\n\n\ndef stylize(args):\n    if args.accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n    \n    print(f\"Using device: {device}\")\n\n    content_image = utils.load_image(args.content_image, scale=args.content_scale)\n    content_transform = transforms.Compose([\n        transforms.ToTensor(),\n        transforms.Lambda(lambda x: x.mul(255))\n    ])\n    content_image = content_transform(content_image)\n    content_image = content_image.unsqueeze(0).to(device)\n\n    if args.model.endswith(\".onnx\"):\n        output = stylize_onnx(content_image, args)\n    else:\n        with torch.no_grad():\n            style_model = TransformerNet()\n            state_dict = torch.load(args.model)\n            # remove saved deprecated running_* keys in InstanceNorm from the checkpoint\n            for k in list(state_dict.keys()):\n                if re.search(r'in\\d+\\.running_(mean|var)$', k):\n                    del state_dict[k]\n            style_model.load_state_dict(state_dict)\n            style_model.to(device)\n            style_model.eval()\n            if args.export_onnx:\n                assert args.export_onnx.endswith(\".onnx\"), \"Export model file should end with .onnx\"\n                output = torch.onnx._export(\n                    style_model, content_image, args.export_onnx, opset_version=11,\n                ).cpu()            \n            else:\n                output = style_model(content_image).cpu()\n    utils.save_image(args.output_image, output[0])\n\n\ndef stylize_onnx(content_image, args):\n    \"\"\"\n    Read ONNX model and run it using onnxruntime\n    \"\"\"\n\n    assert not args.export_onnx\n\n    import onnxruntime\n\n    ort_session = onnxruntime.InferenceSession(args.model)\n\n    def to_numpy(tensor):\n        return (\n            tensor.detach().cpu().numpy()\n            if tensor.requires_grad\n            else tensor.cpu().numpy()\n        )\n\n    ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(content_image)}\n    ort_outs = ort_session.run(None, ort_inputs)\n    img_out_y = ort_outs[0]\n\n    return torch.from_numpy(img_out_y)\n\n\ndef main():\n    main_arg_parser = argparse.ArgumentParser(description=\"parser for fast-neural-style\")\n    subparsers = main_arg_parser.add_subparsers(title=\"subcommands\", dest=\"subcommand\")\n\n    train_arg_parser = subparsers.add_parser(\"train\", help=\"parser for training arguments\")\n    train_arg_parser.add_argument(\"--epochs\", type=int, default=2,\n                                  help=\"number of training epochs, default is 2\")\n    train_arg_parser.add_argument(\"--batch-size\", type=int, default=4,\n                                  help=\"batch size for training, default is 4\")\n    train_arg_parser.add_argument(\"--dataset\", type=str, required=True,\n                                  help=\"path to training dataset, the path should point to a folder \"\n                                       \"containing another folder with all the training images\")\n    train_arg_parser.add_argument(\"--style-image\", type=str, default=\"images/style-images/mosaic.jpg\",\n                                  help=\"path to style-image\")\n    train_arg_parser.add_argument(\"--save-model-dir\", type=str, required=True,\n                                  help=\"path to folder where trained model will be saved.\")\n    train_arg_parser.add_argument(\"--checkpoint-model-dir\", type=str, default=None,\n                                  help=\"path to folder where checkpoints of trained models will be saved\")\n    train_arg_parser.add_argument(\"--image-size\", type=int, default=256,\n                                  help=\"size of training images, default is 256 X 256\")\n    train_arg_parser.add_argument(\"--style-size\", type=int, default=None,\n                                  help=\"size of style-image, default is the original size of style image\")\n    train_arg_parser.add_argument('--accel', action='store_true',\n                                  help='use accelerator')\n    train_arg_parser.add_argument(\"--seed\", type=int, default=42,\n                                  help=\"random seed for training\")\n    train_arg_parser.add_argument(\"--content-weight\", type=float, default=1e5,\n                                  help=\"weight for content-loss, default is 1e5\")\n    train_arg_parser.add_argument(\"--style-weight\", type=float, default=1e10,\n                                  help=\"weight for style-loss, default is 1e10\")\n    train_arg_parser.add_argument(\"--lr\", type=float, default=1e-3,\n                                  help=\"learning rate, default is 1e-3\")\n    train_arg_parser.add_argument(\"--log-interval\", type=int, default=500,\n                                  help=\"number of images after which the training loss is logged, default is 500\")\n    train_arg_parser.add_argument(\"--checkpoint-interval\", type=int, default=2000,\n                                  help=\"number of batches after which a checkpoint of the trained model will be created\")\n\n    eval_arg_parser = subparsers.add_parser(\"eval\", help=\"parser for evaluation/stylizing arguments\")\n    eval_arg_parser.add_argument(\"--content-image\", type=str, required=True,\n                                 help=\"path to content image you want to stylize\")\n    eval_arg_parser.add_argument(\"--content-scale\", type=float, default=None,\n                                 help=\"factor for scaling down the content image\")\n    eval_arg_parser.add_argument(\"--output-image\", type=str, required=True,\n                                 help=\"path for saving the output image\")\n    eval_arg_parser.add_argument(\"--model\", type=str, required=True,\n                                 help=\"saved model to be used for stylizing the image. If file ends in .pth - PyTorch path is used, if in .onnx - Caffe2 path\")\n    eval_arg_parser.add_argument(\"--export_onnx\", type=str,\n                                 help=\"export ONNX model to a given file\")\n    eval_arg_parser.add_argument('--accel', action='store_true',\n                                 help='use accelerator')\n\n    args = main_arg_parser.parse_args()\n\n    if args.subcommand is None:\n        print(\"ERROR: specify either train or eval\")\n        sys.exit(1)\n    if args.accel and not torch.accelerator.is_available():\n        print(\"ERROR: accelerator is not available, try running on CPU\")\n        sys.exit(1)\n    if not args.accel and torch.accelerator.is_available():\n        print(\"WARNING: accelerator is available, run with --accel to enable it\")\n\n    if args.subcommand == \"train\":\n        check_paths(args)\n        train(args)\n    else:\n        stylize(args)\n\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "fast_neural_style/neural_style/transformer_net.py",
    "content": "import torch\n\n\nclass TransformerNet(torch.nn.Module):\n    def __init__(self):\n        super(TransformerNet, self).__init__()\n        # Initial convolution layers\n        self.conv1 = ConvLayer(3, 32, kernel_size=9, stride=1)\n        self.in1 = torch.nn.InstanceNorm2d(32, affine=True)\n        self.conv2 = ConvLayer(32, 64, kernel_size=3, stride=2)\n        self.in2 = torch.nn.InstanceNorm2d(64, affine=True)\n        self.conv3 = ConvLayer(64, 128, kernel_size=3, stride=2)\n        self.in3 = torch.nn.InstanceNorm2d(128, affine=True)\n        # Residual layers\n        self.res1 = ResidualBlock(128)\n        self.res2 = ResidualBlock(128)\n        self.res3 = ResidualBlock(128)\n        self.res4 = ResidualBlock(128)\n        self.res5 = ResidualBlock(128)\n        # Upsampling Layers\n        self.deconv1 = UpsampleConvLayer(128, 64, kernel_size=3, stride=1, upsample=2)\n        self.in4 = torch.nn.InstanceNorm2d(64, affine=True)\n        self.deconv2 = UpsampleConvLayer(64, 32, kernel_size=3, stride=1, upsample=2)\n        self.in5 = torch.nn.InstanceNorm2d(32, affine=True)\n        self.deconv3 = ConvLayer(32, 3, kernel_size=9, stride=1)\n        # Non-linearities\n        self.relu = torch.nn.ReLU()\n\n    def forward(self, X):\n        y = self.relu(self.in1(self.conv1(X)))\n        y = self.relu(self.in2(self.conv2(y)))\n        y = self.relu(self.in3(self.conv3(y)))\n        y = self.res1(y)\n        y = self.res2(y)\n        y = self.res3(y)\n        y = self.res4(y)\n        y = self.res5(y)\n        y = self.relu(self.in4(self.deconv1(y)))\n        y = self.relu(self.in5(self.deconv2(y)))\n        y = self.deconv3(y)\n        return y\n\n\nclass ConvLayer(torch.nn.Module):\n    def __init__(self, in_channels, out_channels, kernel_size, stride):\n        super(ConvLayer, self).__init__()\n        reflection_padding = kernel_size // 2\n        self.reflection_pad = torch.nn.ReflectionPad2d(reflection_padding)\n        self.conv2d = torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride)\n\n    def forward(self, x):\n        out = self.reflection_pad(x)\n        out = self.conv2d(out)\n        return out\n\n\nclass ResidualBlock(torch.nn.Module):\n    \"\"\"ResidualBlock\n    introduced in: https://arxiv.org/abs/1512.03385\n    recommended architecture: http://torch.ch/blog/2016/02/04/resnets.html\n    \"\"\"\n\n    def __init__(self, channels):\n        super(ResidualBlock, self).__init__()\n        self.conv1 = ConvLayer(channels, channels, kernel_size=3, stride=1)\n        self.in1 = torch.nn.InstanceNorm2d(channels, affine=True)\n        self.conv2 = ConvLayer(channels, channels, kernel_size=3, stride=1)\n        self.in2 = torch.nn.InstanceNorm2d(channels, affine=True)\n        self.relu = torch.nn.ReLU()\n\n    def forward(self, x):\n        residual = x\n        out = self.relu(self.in1(self.conv1(x)))\n        out = self.in2(self.conv2(out))\n        out = out + residual\n        return out\n\n\nclass UpsampleConvLayer(torch.nn.Module):\n    \"\"\"UpsampleConvLayer\n    Upsamples the input and then does a convolution. This method gives better results\n    compared to ConvTranspose2d.\n    ref: http://distill.pub/2016/deconv-checkerboard/\n    \"\"\"\n\n    def __init__(self, in_channels, out_channels, kernel_size, stride, upsample=None):\n        super(UpsampleConvLayer, self).__init__()\n        self.upsample = upsample\n        reflection_padding = kernel_size // 2\n        self.reflection_pad = torch.nn.ReflectionPad2d(reflection_padding)\n        self.conv2d = torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride)\n\n    def forward(self, x):\n        x_in = x\n        if self.upsample:\n            x_in = torch.nn.functional.interpolate(x_in, mode='nearest', scale_factor=self.upsample)\n        out = self.reflection_pad(x_in)\n        out = self.conv2d(out)\n        return out\n"
  },
  {
    "path": "fast_neural_style/neural_style/utils.py",
    "content": "import torch\nfrom PIL import Image\n\n\ndef load_image(filename, size=None, scale=None):\n    img = Image.open(filename).convert('RGB')\n    if size is not None:\n        img = img.resize((size, size), Image.ANTIALIAS)\n    elif scale is not None:\n        img = img.resize((int(img.size[0] / scale), int(img.size[1] / scale)), Image.ANTIALIAS)\n    return img\n\n\ndef save_image(filename, data):\n    img = data.clone().clamp(0, 255).numpy()\n    img = img.transpose(1, 2, 0).astype(\"uint8\")\n    img = Image.fromarray(img)\n    img.save(filename)\n\n\ndef gram_matrix(y):\n    (b, ch, h, w) = y.size()\n    features = y.view(b, ch, w * h)\n    features_t = features.transpose(1, 2)\n    gram = features.bmm(features_t) / (ch * h * w)\n    return gram\n\n\ndef normalize_batch(batch):\n    # normalize using imagenet mean and std\n    mean = batch.new_tensor([0.485, 0.456, 0.406]).view(-1, 1, 1)\n    std = batch.new_tensor([0.229, 0.224, 0.225]).view(-1, 1, 1)\n    batch = batch.div_(255.0)\n    return (batch - mean) / std\n"
  },
  {
    "path": "fast_neural_style/neural_style/vgg.py",
    "content": "from collections import namedtuple\n\nimport torch\nfrom torchvision import models\n\n\nclass Vgg16(torch.nn.Module):\n    def __init__(self, requires_grad=False):\n        super(Vgg16, self).__init__()\n        vgg_pretrained_features = models.vgg16(weights=models.VGG16_Weights.IMAGENET1K_V1).features\n        self.slice1 = torch.nn.Sequential()\n        self.slice2 = torch.nn.Sequential()\n        self.slice3 = torch.nn.Sequential()\n        self.slice4 = torch.nn.Sequential()\n        for x in range(4):\n            self.slice1.add_module(str(x), vgg_pretrained_features[x])\n        for x in range(4, 9):\n            self.slice2.add_module(str(x), vgg_pretrained_features[x])\n        for x in range(9, 16):\n            self.slice3.add_module(str(x), vgg_pretrained_features[x])\n        for x in range(16, 23):\n            self.slice4.add_module(str(x), vgg_pretrained_features[x])\n        if not requires_grad:\n            for param in self.parameters():\n                param.requires_grad = False\n\n    def forward(self, X):\n        h = self.slice1(X)\n        h_relu1_2 = h\n        h = self.slice2(h)\n        h_relu2_2 = h\n        h = self.slice3(h)\n        h_relu3_3 = h\n        h = self.slice4(h)\n        h_relu4_3 = h\n        vgg_outputs = namedtuple(\"VggOutputs\", ['relu1_2', 'relu2_2', 'relu3_3', 'relu4_3'])\n        out = vgg_outputs(h_relu1_2, h_relu2_2, h_relu3_3, h_relu4_3)\n        return out\n"
  },
  {
    "path": "fast_neural_style/requirements.txt",
    "content": "numpy\ntorch>=2.6\ntorchvision\n"
  },
  {
    "path": "fx/README.md",
    "content": "# FX Examples\n\nThis folder contains several examples of program transformations implemented using `torch.fx`. More information about FX can be found in the [documentation](https://pytorch.org/docs/master/fx.html).\n\nNote that all examples should be runnable as standalone Python files. In the case of an exception, the example will appear in a subfolder with a `README.md` file explaining how to run the example.\n\nAs FX is currently in a Beta release, the API or these examples are subject to change.\n"
  },
  {
    "path": "fx/custom_tracer.py",
    "content": "import torch\nfrom torch.fx import symbolic_trace, Tracer, Graph, GraphModule, Node\nfrom typing import Any, Callable, Dict, Optional, Tuple, Union\n\n\n\n\"\"\"\nHow to Create and Use Custom Tracers\n\n`Tracer`--the class that implements the symbolic tracing functionality\nof `torch.fx.symbolic_trace`--can be subclassed to override various\nbehaviors of the tracing process. In this tutorial, we'll demonstrate\nhow to customize the symbolic tracing process using some handwritten\nTracers. Each example will show that, by simply overriding a few methods\nin the `Tracer` class, you can alter the Graph produced by symbolic\ntracing. For a complete description of the methods that can be changed,\nrefer to the docstrings of the methods in the Tracer class. Information\ncan be found at: https://pytorch.org/docs/master/fx.html#torch.fx.Tracer\n\nIf you want a real-world example of a custom tracer, check out FX's AST\nRewriter in `rewriter.py`. `RewritingTracer` inherits from Tracer but\noverrides the `trace` function so that we can rewrite all calls to\n`assert` to the more FX-friendly `torch.assert`.\n\nNote that a call to `symbolic_trace(m)` is equivalent to\n`GraphModule(m, Tracer().trace(m))`. (`Tracer` is the default\nimplementation of Tracer as defined in `symbolic_trace.py`.)\n\"\"\"\n\n\n\n\"\"\"\nCustom Tracer #1: Trace Through All `torch.nn.ReLU` Submodules\n\nDuring symbolic tracing, some submodules are traced through and their\nconstituent ops are recorded; other submodules appear as an\natomic \"call_module\" Node in the IR. A module in this latter category\nis called a \"leaf module\". By default, all modules in the PyTorch\nstandard library (`torch.nn`) are leaf modules. We can change this\nby creating a custom Tracer and overriding `is_leaf_module`. In this\ncase, we'll keep the default behavior for all `torch.nn` Modules except\nfor `ReLU`.\n\"\"\"\n\nclass M1(torch.nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.relu = torch.nn.ReLU()\n\n    def forward(self, x):\n        return self.relu(x)\n\ndefault_traced: GraphModule = symbolic_trace(M1())\n\"\"\"\nTracing with the default tracer and calling `print_tabular` produces:\n\n    opcode       name    target    args       kwargs\n    -----------  ------  --------  ---------  --------\n    placeholder  x       x         ()         {}\n    call_module  relu_1  relu      (x,)       {}\n    output       output  output    (relu_1,)  {}\n\n\"\"\"\ndefault_traced.graph.print_tabular()\n\nclass LowerReluTracer(Tracer):\n    def is_leaf_module(self, m : torch.nn.Module, qualname : str):\n        if isinstance(m, torch.nn.ReLU):\n            return False\n        return super().is_leaf_module(m, qualname)\n\n\"\"\"\nTracing with our custom tracer and calling `print_tabular` produces:\n\n    opcode         name    target                             args       kwargs\n    -------------  ------  ---------------------------------  ---------  ------------------\n    placeholder    x       x                                  ()         {}\n    call_function  relu_1  <function relu at 0x7f66f7170b80>  (x,)       {'inplace': False}\n    output         output  output                             (relu_1,)  {}\n\"\"\"\nlower_relu_tracer = LowerReluTracer()\ncustom_traced_graph: Graph = lower_relu_tracer.trace(M1())\ncustom_traced_graph.print_tabular()\n\n\n\n\"\"\"\nCustom Tracer #2: Add an Extra Attribute to Each Node\n\nHere, we'll override `create_node` so that we can add a new attribute to\neach Node during its creation\n\"\"\"\n\nclass M2(torch.nn.Module):\n    def forward(self, a, b):\n        return a + b\n\nclass TaggingTracer(Tracer):\n    def create_node(self, kind : str, target : Union[str, Callable],\n                    args : Tuple[Any], kwargs : Dict[str, Any], name : Optional[str] = None,\n                    type_expr : Optional[Any] = None) -> Node:\n        n = super().create_node(kind, target, args, kwargs, name)\n        n.tag = \"foo\"\n        return n\n\ncustom_traced_graph: Graph = TaggingTracer().trace(M2())\n\ndef assert_all_nodes_have_tags(g: Graph) -> bool:\n    for n in g.nodes:\n        if not hasattr(n, \"tag\") or not n.tag == \"foo\":\n            return False\n    return True\n\n# Prints \"True\"\nprint(assert_all_nodes_have_tags(custom_traced_graph))\n\n"
  },
  {
    "path": "fx/inline_function.py",
    "content": "import torch\nfrom torch.fx import Proxy, symbolic_trace\nfrom torch.fx.node import map_arg\n\n\n'''\nHow to Inline a Function Into an Existing Graph\n\nOne reason you might want to inline a function is to get around FX's\ndefault tracing behavior. For example, unless you've defined a custom\nTracer, the out-of-the-box implementation of ``symbolic_trace`` causes\nreferences to ``torch.nn`` module instances to appear as\n``call_module`` calls rather than being traced through. Let's say this\nbehavior is almost what you need; the only problem is that there's a\nsingle module call that you want to replace with an inlined trace of the\nfunction. Creating a custom Tracer would be too much. Instead, you can\naccomplish this using Proxies.\n\nThe following code demonstrates how to trace a module and inline it\ninto an existing Graph using Proxy. We'll trace our Graph, then iterate\nthrough its Nodes until we find the right place to swap out the\n``call_module`` Node with an inlined trace. At that point, we'll create\nProxies from the Node's args and kwargs. Finally, we'll call the\nfunction we want to replace with those Proxies--which will, in essence,\n\"trace\" that function. Finally, we'll insert the result of that call\ninto our Graph. (This last step will automatically inline the function.)\n'''\n\n\n# Sample module\nclass M(torch.nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.relu = torch.nn.ReLU()\n\n    def forward(self, x):\n        return self.relu(x) + 1.0\n\n# Symbolically trace an instance of `M`. After tracing, `self.relu` is\n# represented as a `call_module` Node. The full operation in the\n# generated `forward` function's code will appear as `self.relu(x)`\nm = symbolic_trace(M())\n\n# Insert nodes from the ReLU graph in place of the original call to\n# `self.relu`\n# create a graph-appending tracer pointing to the original graph\ntracer = torch.fx.proxy.GraphAppendingTracer(m.graph)\nfor node in m.graph.nodes:\n    # Find `call_module` Node in `m` that corresponds to `self.relu`.\n    # This is the Node we want to swap out for an inlined version of the\n    # same call\n    if (node.op, node.target) == (\"call_module\", \"relu\"):\n        with m.graph.inserting_before(node):\n            # Create a Proxy from each Node in the current Node's\n            # args/kwargs\n            proxy_args = map_arg(node.args, lambda n: Proxy(n, tracer))\n            proxy_kwargs = map_arg(node.kwargs, lambda n: Proxy(n, tracer))\n            # Call `m.relu` with the newly-created Proxy arguments.\n            # `m.relu` is the generic version of the function; by\n            # calling it with Proxies created from Nodes in `m`, we're\n            # emitting Nodes that reference exiting values in the IR.\n            # The result of this call is another Proxy, which we can\n            # hook into our existing Graph to complete the function\n            # inlining.\n            proxy_output = m.relu(*proxy_args, **proxy_kwargs)\n            # Replace the relu `call_module` node with the inlined\n            # version of the function\n            node.replace_all_uses_with(proxy_output.node)\n            # Make sure that the old relu Node is erased\n            m.graph.erase_node(node)\n"
  },
  {
    "path": "fx/invert.py",
    "content": "import torch\nimport torch.fx as fx\n\n# An inverse mapping is one that takes a function f(x) and returns a function g\n# such that f(g(x)) == x. For example,since log(exp(x)) == x, exp and log are\n# inverses.\n\ninvert_mapping = {}\ndef add_inverse(a, b):\n    invert_mapping[a] = b\n    invert_mapping[b] = a\ninverses = [\n    (torch.sin, torch.arcsin),\n    (torch.cos, torch.arccos),\n    (torch.tan, torch.arctan),\n    (torch.exp, torch.log),\n]\nfor a, b in inverses:\n    add_inverse(a, b)\n\n# The general strategy is that we walk the graph backwards, transforming each\n# node into its inverse. To do so, we swap the outputs and inputs of the\n# functions, and then we look up its inverse in `invert_mapping`. Note that\n# this transform assumes that all operations take in only one input and return\n# one output.\ndef invert(model: torch.nn.Module) -> torch.nn.Module:\n    fx_model = fx.symbolic_trace(model)\n    new_graph = fx.Graph()  # As we're building up a new graph\n    env = {}\n    for node in reversed(fx_model.graph.nodes):\n        if node.op == 'call_function':\n            # This creates a node in the new graph with the inverse function,\n            # and passes `env[node.name]` (i.e. the previous output node) as\n            # input.\n            new_node = new_graph.call_function(invert_mapping[node.target], (env[node.name],))\n            env[node.args[0].name] = new_node\n        elif node.op == 'output':\n            # We turn the output into an input placeholder\n            new_node = new_graph.placeholder(node.name)\n            env[node.args[0].name] = new_node\n        elif node.op == 'placeholder':\n            # We turn the input placeholder into an output\n            new_graph.output(env[node.name])\n        else:\n            raise RuntimeError(\"Not implemented\")\n\n    new_graph.lint()\n    return fx.GraphModule(fx_model, new_graph)\n\n\ndef f(x):\n    return torch.exp(torch.tan(x))\n\nres = invert(f)\nprint(res.code)\n\"\"\"\ndef forward(self, output):\n    log_1 = torch.log(output);  output = None\n    arctan_1 = torch.arctan(log_1);  log_1 = None\n    return arctan_1\n\"\"\"\nprint(f(res((torch.arange(5) + 1))))  # [1., 2., 3., 4, 5.]\n"
  },
  {
    "path": "fx/module_tracer.py",
    "content": "\"\"\"\nRecording Module Hierarchy With a Custom Tracer\n\nIn this example, we are going to define a custom `fx.Tracer` instance that--\nfor each recorded operation--also notes down the qualified name of the module\nfrom which that operation originated. The _qualified name_ is the path to the\nModule from the root module. More information about this concept can be\nfound in the documentation for `Module.get_submodule`:\nhttps://github.com/pytorch/pytorch/blob/9f2aea7b88f69fc74ad90b1418663802f80c1863/torch/nn/modules/module.py#L385\n\"\"\"\nimport torch\nimport torch.fx\nfrom typing import Any, Callable, Dict, Optional, Tuple\n\nclass ModulePathTracer(torch.fx.Tracer):\n    \"\"\"\n    ModulePathTracer is an FX tracer that--for each operation--also records\n    the qualified name of the Module from which the operation originated.\n    \"\"\"\n\n    # The current qualified name of the Module being traced. The top-level\n    # module is signified by empty string. This is updated when entering\n    # call_module and restored when exiting call_module\n    current_module_qualified_name : str = ''\n    # A map from FX Node to the qualname of the Module from which it\n    # originated. This is recorded by `create_proxy` when recording an\n    # operation\n    node_to_originating_module : Dict[torch.fx.Node, str] = {}\n\n    def call_module(self, m: torch.nn.Module, forward: Callable[..., Any],\n                    args : Tuple[Any, ...], kwargs : Dict[str, Any]) -> Any:\n        \"\"\"\n        Override of Tracer.call_module (see\n        https://pytorch.org/docs/stable/fx.html#torch.fx.Tracer.call_module).\n\n        This override:\n        1) Stores away the qualified name of the caller for restoration later\n        2) Installs the qualified name of the caller in `current_module_qualified_name`\n           for retrieval by `create_proxy`\n        3) Delegates into the normal Tracer.call_module method\n        4) Restores the caller's qualified name into current_module_qualified_name\n        \"\"\"\n        old_qualname = self.current_module_qualified_name\n        try:\n            self.current_module_qualified_name = self.path_of_module(m)\n            return super().call_module(m, forward, args, kwargs)\n        finally:\n            self.current_module_qualified_name = old_qualname\n\n    def create_proxy(self, kind: str, target: torch.fx.node.Target, args: Tuple[Any, ...],\n                     kwargs: Dict[str, Any], name: Optional[str] = None, type_expr: Optional[Any] = None):\n        \"\"\"\n        Override of `Tracer.create_proxy`. This override intercepts the recording\n        of every operation and stores away the current traced module's qualified\n        name in `node_to_originating_module`\n        \"\"\"\n        proxy = super().create_proxy(kind, target, args, kwargs, name, type_expr)\n        self.node_to_originating_module[proxy.node] = self.current_module_qualified_name\n        return proxy\n\n\n# Testing: let's see how this works on a torchvision ResNet18 model\nimport torchvision.models as models\n\n# Model under test\nrn18 = models.resnet18()\n\n# Instantiate our ModulePathTracer and use that to trace our ResNet18\ntracer = ModulePathTracer()\ntraced_rn18 = tracer.trace(rn18)\n\n# Print (node, module qualified name) for every node in the Graph\nfor node in traced_rn18.nodes:\n    module_qualname = tracer.node_to_originating_module.get(node)\n    print('Node', node, 'is from module', module_qualname)\n\"\"\"\nNode x is from module \nNode conv1 is from module conv1\nNode bn1 is from module bn1\nNode relu is from module relu\nNode maxpool is from module maxpool\nNode layer1_0_conv1 is from module layer1.0.conv1\nNode layer1_0_bn1 is from module layer1.0.bn1\nNode layer1_0_relu is from module layer1.0.relu\nNode layer1_0_conv2 is from module layer1.0.conv2\nNode layer1_0_bn2 is from module layer1.0.bn2\nNode add is from module layer1.0\nNode layer1_0_relu_1 is from module layer1.0.relu\nNode layer1_1_conv1 is from module layer1.1.conv1\nNode layer1_1_bn1 is from module layer1.1.bn1\nNode layer1_1_relu is from module layer1.1.relu\nNode layer1_1_conv2 is from module layer1.1.conv2\nNode layer1_1_bn2 is from module layer1.1.bn2\nNode add_1 is from module layer1.1\nNode layer1_1_relu_1 is from module layer1.1.relu\nNode layer2_0_conv1 is from module layer2.0.conv1\nNode layer2_0_bn1 is from module layer2.0.bn1\nNode layer2_0_relu is from module layer2.0.relu\nNode layer2_0_conv2 is from module layer2.0.conv2\nNode layer2_0_bn2 is from module layer2.0.bn2\nNode layer2_0_downsample_0 is from module layer2.0.downsample.0\nNode layer2_0_downsample_1 is from module layer2.0.downsample.1\nNode add_2 is from module layer2.0\nNode layer2_0_relu_1 is from module layer2.0.relu\nNode layer2_1_conv1 is from module layer2.1.conv1\nNode layer2_1_bn1 is from module layer2.1.bn1\nNode layer2_1_relu is from module layer2.1.relu\nNode layer2_1_conv2 is from module layer2.1.conv2\nNode layer2_1_bn2 is from module layer2.1.bn2\nNode add_3 is from module layer2.1\nNode layer2_1_relu_1 is from module layer2.1.relu\nNode layer3_0_conv1 is from module layer3.0.conv1\nNode layer3_0_bn1 is from module layer3.0.bn1\nNode layer3_0_relu is from module layer3.0.relu\nNode layer3_0_conv2 is from module layer3.0.conv2\nNode layer3_0_bn2 is from module layer3.0.bn2\nNode layer3_0_downsample_0 is from module layer3.0.downsample.0\nNode layer3_0_downsample_1 is from module layer3.0.downsample.1\nNode add_4 is from module layer3.0\nNode layer3_0_relu_1 is from module layer3.0.relu\nNode layer3_1_conv1 is from module layer3.1.conv1\nNode layer3_1_bn1 is from module layer3.1.bn1\nNode layer3_1_relu is from module layer3.1.relu\nNode layer3_1_conv2 is from module layer3.1.conv2\nNode layer3_1_bn2 is from module layer3.1.bn2\nNode add_5 is from module layer3.1\nNode layer3_1_relu_1 is from module layer3.1.relu\nNode layer4_0_conv1 is from module layer4.0.conv1\nNode layer4_0_bn1 is from module layer4.0.bn1\nNode layer4_0_relu is from module layer4.0.relu\nNode layer4_0_conv2 is from module layer4.0.conv2\nNode layer4_0_bn2 is from module layer4.0.bn2\nNode layer4_0_downsample_0 is from module layer4.0.downsample.0\nNode layer4_0_downsample_1 is from module layer4.0.downsample.1\nNode add_6 is from module layer4.0\nNode layer4_0_relu_1 is from module layer4.0.relu\nNode layer4_1_conv1 is from module layer4.1.conv1\nNode layer4_1_bn1 is from module layer4.1.bn1\nNode layer4_1_relu is from module layer4.1.relu\nNode layer4_1_conv2 is from module layer4.1.conv2\nNode layer4_1_bn2 is from module layer4.1.bn2\nNode add_7 is from module layer4.1\nNode layer4_1_relu_1 is from module layer4.1.relu\nNode avgpool is from module avgpool\nNode flatten is from module \nNode fc is from module fc\nNode output is from module None\n\"\"\"\n"
  },
  {
    "path": "fx/native_interpreter/CMakeLists.txt",
    "content": "cmake_minimum_required(VERSION 3.1 FATAL_ERROR)\nproject(interpreter)\n\nfind_package(Torch REQUIRED)\n\n# Define our library target\nadd_library(interpreter SHARED interpreter.cpp)\nset(CMAKE_CXX_STANDARD 17)\n# Link against LibTorch\ntarget_link_libraries(interpreter \"${TORCH_LIBRARIES}\")\n"
  },
  {
    "path": "fx/native_interpreter/README.md",
    "content": "# Converting PyTorch Code to a Native Runtime With FX and TorchScript Custom Classes\n\nIn this example, we are going to build a pipeline that does the following things:\n\n1. Converts (or “lowers”) code in a PyTorch module into another representation (we will define the representation within the example)\n2. Registers an interpreter for that code representation that can be used in TorchScript or Python\n3. Wrap the converted code into a format that can still be used in TorchScript compilation.\n\nWe are going to build up a trivial interpreter for this example, but you can imagine extending the same process to work with more sophisticated backends, ones which may do code optimization or offloading to an accelerator.\n\nWe will be using [TorchScript custom classes](https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html) to expose this Interpreter to Python and TorchScript. You may want to review that tutorial and documentation before reading this example project.\n\n### Defining the Interpreter\n\nWe define the interpreter in `interpreter.cpp`. This interpreter is very limited: it only supports two element-wise operations (`add` and `mul`) and it only supports `Tensor` values. When this interpreter runs code, it iterates through the list of instructions and simply calls the appropriate PyTorch operator from C++.\n\nTo build the interpreter into a shared-object file to be loaded in for use, use the following commands from this example’s root:\n\n\n```\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_PREFIX_PATH=\"$(python -c 'import torch.utils; print(torch.utils.cmake_prefix_path)')\" ..\n$ make -j\n```\n\nAfter the build finishes, you should see `build/libinterpreter.so` (or with a different extension depending on your OS). We will use this dynamic library next when we load it up into a process to be used in execution.\n\n### Defining the Transformation\n\nWe define the code that transforms a `PyTorch` module to the format the interpreter understands in `use_interpreter.py`. Note that that file loads in the shared object we built in the previous step via a `torch.classes.load_library` call. `use_interpreter.py` contains driver code and the end that can be directly run to test the lowering transformation.\n\n### Questions, Comments, Feedback\n\nPlease direct questions and discussion to the [PyTorch forums](https://discuss.pytorch.org/). To report any issues with PyTorch (including FX and custom classes), please use the [issue tracker](https://github.com/pytorch/pytorch/issues).\n"
  },
  {
    "path": "fx/native_interpreter/interpreter.cpp",
    "content": "#include <torch/script.h>\n\n#include <sstream>\n\n// ElementwiseInterpreter is a class that takes in a list of Instructions\n// (represented as triples (op, inputs, outputs)) and executes them in a\n// C++-based interpreter loop. For brevity, this interpreter only supports\n// two operations: element-wise add and element-wise mul.\nstruct ElementwiseInterpreter : torch::CustomClassHolder {\n  using InstructionType =\n      std::tuple<std::string /*op*/, std::vector<std::string> /*inputs*/,\n                 std::string /*output*/>;\n\n  ElementwiseInterpreter() {}\n\n  // Load a list of instructions into the interpreter. As specified above,\n  // instructions specify the operation (currently support \"add\" and \"mul\"),\n  // the names of the input values, and the name of the single output value\n  // from this instruction\n  void setInstructions(std::vector<InstructionType> instructions) {\n    instructions_ = std::move(instructions);\n  }\n\n  // Add a constant. The interpreter maintains a set of constants across\n  // calls. They are keyed by name, and constants can be referenced in\n  // Instructions by the name specified\n  void addConstant(const std::string &name, at::Tensor value) {\n    constants_.insert_or_assign(name, std::move(value));\n  }\n\n  // Set the string names for the positional inputs to the function this\n  // interpreter represents. When invoked, the interpreter will assign\n  // the positional inputs to the names in the corresponding position in\n  // input_names.\n  void setInputNames(std::vector<std::string> input_names) {\n    input_names_ = std::move(input_names);\n  }\n\n  // Specify the output name for the function this interpreter represents. This\n  // should match the \"output\" field of one of the instructions in the\n  // instruction list, typically the last instruction.\n  void setOutputName(std::string output_name) {\n    output_name_ = std::move(output_name);\n  }\n\n  // Invoke this interpreter. This takes a list of positional inputs and returns\n  // a single output. Currently, inputs and outputs must all be Tensors.\n  at::Tensor __call__(std::vector<at::Tensor> inputs) {\n    // Environment to hold local variables\n    std::unordered_map<std::string, at::Tensor> environment;\n\n    // Load inputs according to the specified names\n    if (inputs.size() != input_names_.size()) {\n      std::stringstream err;\n      err << \"Expected \" << input_names_.size() << \" inputs, but got \"\n          << inputs.size() << \"!\";\n      throw std::runtime_error(err.str());\n    }\n    for (size_t i = 0; i < inputs.size(); ++i) {\n      environment[input_names_[i]] = inputs[i];\n    }\n\n    if (!output_name_) {\n      throw std::runtime_error(\"Output name not specified!\");\n    }\n\n    for (InstructionType &instr : instructions_) {\n      // Retrieve all input values for this op\n      std::vector<at::Tensor> inputs;\n      for (const auto &input_name : std::get<1>(instr)) {\n        // Operator output values shadow constants.\n        // Imagine all constants are defined in statements at the beginning\n        // of a function (a la K&R C). Any definition of an output value must\n        // necessarily come after constant definition in textual order. Thus,\n        // We look up values in the environment first then the constant table\n        // second to implement this shadowing behavior\n        if (environment.find(input_name) != environment.end()) {\n          inputs.push_back(environment.at(input_name));\n        } else if (constants_.find(input_name) != constants_.end()) {\n          inputs.push_back(constants_.at(input_name));\n        } else {\n          std::stringstream err;\n          err << \"Instruction referenced unknown value \" << input_name << \"!\";\n          throw std::runtime_error(err.str());\n        }\n      }\n\n      // Run the specified operation\n      at::Tensor result;\n      const auto &op = std::get<0>(instr);\n      if (op == \"add\") {\n        if (inputs.size() != 2) {\n          throw std::runtime_error(\"Unexpected number of inputs for add op!\");\n        }\n        result = inputs[0] + inputs[1];\n      } else if (op == \"mul\") {\n        if (inputs.size() != 2) {\n          throw std::runtime_error(\"Unexpected number of inputs for mul op!\");\n        }\n        result = inputs[0] * inputs[1];\n      } else {\n        std::stringstream err;\n        err << \"Unknown operator \" << op << \"!\";\n        throw std::runtime_error(err.str());\n      }\n\n      // Write back result into environment\n      const auto &output_name = std::get<2>(instr);\n      environment[output_name] = std::move(result);\n    }\n\n    if (!environment.count(*output_name_)) {\n      std::stringstream err;\n      err << \"Execution expected an output value with name \";\n      err << *output_name_;\n      err << \" but no instruction produced a value with that name!\";\n      throw std::runtime_error(err.str());\n    }\n\n    return environment.at(*output_name_);\n  }\n\n  // Ser/De infrastructure. See\n  // https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html#defining-serialization-deserialization-methods-for-custom-c-classes\n  // for more info.\n\n  // This is the type we will use to marshall information on disk during\n  // ser/de. It is a simple tuple composed of primitive types and simple\n  // collection types like vector, optional, and dict.\n  using SerializationType =\n      std::tuple<std::vector<std::string> /*input_names_*/,\n                 c10::optional<std::string> /*output_name_*/,\n                 c10::Dict<std::string, at::Tensor> /*constants_*/,\n                 std::vector<InstructionType> /*instructions_*/\n                 >;\n\n  // This function yields the SerializationType instance for `this`.\n  SerializationType __getstate__() const {\n    return SerializationType{input_names_, output_name_, constants_,\n                             instructions_};\n  }\n\n  // This function will create an instance of `ElementwiseInterpreter` given\n  // an instance of `SerializationType`.\n  static c10::intrusive_ptr<ElementwiseInterpreter>\n  __setstate__(SerializationType state) {\n    auto instance = c10::make_intrusive<ElementwiseInterpreter>();\n    std::tie(instance->input_names_, instance->output_name_,\n             instance->constants_, instance->instructions_) = std::move(state);\n    return instance;\n  }\n\n  // Class members\n  std::vector<std::string> input_names_;\n  c10::optional<std::string> output_name_;\n  c10::Dict<std::string, at::Tensor> constants_;\n  std::vector<InstructionType> instructions_;\n};\n\n// Register ElementwiseInterpreter as callable from Python\n// and TorchScript.\nTORCH_LIBRARY(NativeInterpretation, m) {\n  m.class_<ElementwiseInterpreter>(\"ElementwiseInterpreter\")\n      .def(torch::init<>())\n      .def(\"set_instructions\", &ElementwiseInterpreter::setInstructions)\n      .def(\"add_constant\", &ElementwiseInterpreter::addConstant)\n      .def(\"set_input_names\", &ElementwiseInterpreter::setInputNames)\n      .def(\"set_output_name\", &ElementwiseInterpreter::setOutputName)\n      .def(\"__call__\", &ElementwiseInterpreter::__call__)\n      .def_pickle(\n          /* __getstate__ */\n          [](const c10::intrusive_ptr<ElementwiseInterpreter> &self) {\n            return self->__getstate__();\n          },\n          /* __setstate__ */\n          [](ElementwiseInterpreter::SerializationType state) {\n            return ElementwiseInterpreter::__setstate__(std::move(state));\n          });\n}\n"
  },
  {
    "path": "fx/native_interpreter/use_interpreter.py",
    "content": "import torch\nimport torch.fx\nimport operator\n\n# Does this path not exist? Check that you've done the following:\n# 1) Read README.md and follow the instructions to build libinterpreter.\n# 2) If this file still does not exist after you've followed those instructions,\n#    check if it is under a different extension (e.g. `dylib` on mac or `dll` on\n#    windows).\ntorch.classes.load_library('build/libinterpreter.so')\n\n# This is what a lowering pass should look like: a function that takes\n# a valid nn.Module, symbolically traces it, lowers the Module to some\n# representation, and wraps that representation up into another\n# nn.Module instance that handles dispatch to the compiled/lowered code.\n# This will ensure that this lowering transformation still fits into the\n# PyTorch programming model and enables features like composing with other\n# transformations and TorchScript compilation.\ndef lower_to_elementwise_interpreter(orig_mod : torch.nn.Module) -> torch.nn.Module:\n    # ===== Stage 1: Symbolic trace the module =====\n    mod = torch.fx.symbolic_trace(orig_mod)\n\n    # ===== Stage 2: Lower GraphModule representation to the C++\n    #       interpreter's instruction format ======\n    instructions = []\n    constant_idx = 0\n    constants = {}\n    fn_input_names = []\n\n    target_to_name = {\n        operator.add : \"add\",\n        operator.mul : \"mul\"\n    }\n\n    output_node : Optional[torch.fx.Node] = None\n    # For each instruction, create a triple\n    # (instruction_name : str, inputs : List[str], output : str)\n    # to feed into the C++ interpreter\n    for n in mod.graph.nodes:\n        target, args, out_name = n.target, n.args, n.name\n        assert len(n.kwargs) == 0, \"kwargs currently not supported\"\n\n        if n.op == 'placeholder':\n            # Placeholders specify function argument names. Save these\n            # for later when we generate the wrapper GraphModule\n            fn_input_names.append(target)\n        elif n.op == 'call_function':\n            assert target in target_to_name, \"Unsupported call target \" + target\n            arg_names = []\n            for arg in args:\n                if not isinstance(arg, torch.fx.Node):\n                    # Pull out constants. These constants will later be\n                    # fed to the interpreter C++ object via add_constant()\n                    arg_name = f'constant_{constant_idx}'\n                    constants[arg_name] = torch.Tensor(\n                        [arg] if isinstance(arg, numbers.Number) else arg)\n                    arg_names.append(arg_name)\n                    constant_idx += 1\n                else:\n                    arg_names.append(arg.name)\n            instructions.append((target_to_name[target], arg_names, out_name))\n        elif n.op == 'output':\n            if output_node is not None:\n                raise RuntimeError('Multiple output nodes!')\n            output_node = n\n        else:\n            raise RuntimeError('Unsupported opcode ' + n.op)\n\n    interpreter = torch.classes.NativeInterpretation.ElementwiseInterpreter()\n    # Load constants\n    for k, v in constants.items():\n        interpreter.add_constant(k, v)\n    # Specify names for positional input arguments\n    interpreter.set_input_names(fn_input_names)\n    # Load instructions\n    interpreter.set_instructions(instructions)\n    # Specify name for single output\n    assert isinstance(output_node.args[0], torch.fx.Node)\n    interpreter.set_output_name(output_node.args[0].name)\n\n    # ===== Stage 3: Create a wrapper GraphModule around the interpreter =====\n    class WrapperModule(torch.nn.Module):\n        def __init__(self, interpreter):\n            super().__init__()\n            self.interpreter = interpreter\n\n    wrapper = WrapperModule(interpreter)\n\n    # Create a forward() function that is compatible with TorchScript compilation.\n    # Create a graph that: 1) Takes function arguments 2) Invokes the interpreter\n    # 3) Returns the specified return value\n\n    graph = torch.fx.Graph()\n    # Add placeholders for fn inputs\n    placeholder_nodes = []\n    for name in fn_input_names:\n        placeholder_nodes.append(graph.create_node('placeholder', name))\n\n    # Get the interpreter object\n    interpreter_node = graph.create_node('get_attr', 'interpreter')\n\n    # Add a node to call the interpreter instance\n    output_node = graph.create_node(\n        op='call_method', target='__call__', args=(interpreter_node, placeholder_nodes))\n\n    # Register output\n    graph.output(output_node)\n\n    graph.lint(wrapper)\n\n    # Return final GraphModule!!!\n    return torch.fx.GraphModule(wrapper, graph)\n\nclass MyElementwiseModule(torch.nn.Module):\n    def forward(self, x, y):\n        return x * y + y\n\nmem = MyElementwiseModule()\nlowered = lower_to_elementwise_interpreter(mem)\nprint(lowered.code)\n# The lowered module can also be compiled into TorchScript\nscripted = torch.jit.script(lowered)\nprint(scripted.graph)\n\n# Stress test correctness\nfor _ in range(50):\n    x, y = torch.randn(10, 20, 30), torch.randn(10, 20, 30)\n    torch.testing.assert_allclose(lowered(x, y), mem(x, y))\n    torch.testing.assert_allclose(scripted(x, y), mem(x, y))\n"
  },
  {
    "path": "fx/primitive_library.py",
    "content": "import torch\nimport torch.fx\n\"\"\"\nIn this example we are going do define a library of\n\"composite\" operations. Composite operations are those\nthat are defined as callable functions that are composed\nof several other operations in their implementation.\n\nComposite operations allow you to choose at what level\nof abstraction you want to interpret/manipulate the\ncode. We show that we can provide a function to inline\nthese functions as well as use a custom Tracer to auto-\nmatically inline such functions.\n\nComposite operations can be useful for exposing higher-\nlevel context to a backend/transform while still\nmaintaining the ability to examine things at a more\nfine-grained level.\n\"\"\"\n\n\ndef sigmoid_lowp(x : torch.Tensor):\n    x = x.float()\n    x = x.sigmoid()\n    return x.half()\n\n# wrap() indicates that the passed-in function should always\n# be recorded as a call_function node rather than being traced\n# through. Later, we will see how we can:\n# a. Inline the implementation of such a function and\n# b. Define a tracer that automatically traces through such a function\ntorch.fx.wrap(sigmoid_lowp)\n\ndef add_lowp(a : torch.Tensor, b : torch.Tensor):\n    a, b = a.float(), b.float()\n    c = a + b\n    return c.half()\n\ntorch.fx.wrap(add_lowp)\n\n\n# Let's see what happens when we symbolically trace through some code\n# that uses these functions\n\nclass Foo(torch.nn.Module):\n    def forward(self, x, y):\n        x = sigmoid_lowp(x)\n        y = sigmoid_lowp(y)\n        return add_lowp(x, y)\n\n\ntraced = torch.fx.symbolic_trace(Foo())\nprint(traced.code)\n\"\"\"\ndef forward(self, x, y):\n    sigmoid_lowp = __main___sigmoid_lowp(x);  x = None\n    sigmoid_lowp_1 = __main___sigmoid_lowp(y);  y = None\n    add_lowp = __main___add_lowp(sigmoid_lowp, sigmoid_lowp_1);  sigmoid_lowp = sigmoid_lowp_1 = None\n    return add_lowp\n\"\"\"\n\n# Notice that the calls to `sigmoid_lowp` and `add_lowp`\n# appear literally in the trace; they are not traced through\n\n\n# ***** Inlining calls *****\n# Now let's define a function that allows for inlining these calls\n# during graph manipulation.\n\ndef inline_lowp_func(n : torch.fx.Node):\n    # If we find a call to a function in our \"lowp\" module, inline it\n    if n.op == 'call_function' and n.target.__module__ == inline_lowp_func.__module__:\n        # We want to insert the operations comprising the implementation of the\n        # function before the function itself. Then, we can swap the output value\n        # of the function call with the output value for its implementation nodes\n        tracer = torch.fx.proxy.GraphAppendingTracer(n.graph)\n        with n.graph.inserting_before(n):\n            # We can inline code by using `fx.Proxy` instances.\n            # map_arg traverses all aggregate types and applies the given function\n            # to Node instances in the data structure. In this case, we are applying\n            # the fx.Proxy constructor.\n            proxy_args = torch.fx.node.map_arg(n.args, lambda x: torch.fx.Proxy(x, tracer))\n            proxy_kwargs = torch.fx.node.map_arg(n.kwargs, lambda x: torch.fx.Proxy(x, tracer))\n            # Call the function itself with proxy arguments. This will emit\n            # nodes in the graph corresponding to the operations in the im-\n            # plementation of the function\n            output_proxy = n.target(*proxy_args, **proxy_kwargs)\n            # Now replace the original node's uses with the output node of\n            # the implementation.\n            node.replace_all_uses_with(output_proxy.node)\n            # Delete the old node\n            node.graph.erase_node(node)\n\nfor node in traced.graph.nodes:\n    if node.op == 'call_function' and node.target is sigmoid_lowp:\n        inline_lowp_func(node)\n\n# Don't forget to recompile after graph manipulation\ntraced.recompile()\n\nprint(traced.code)\n\"\"\"\ndef forward(self, x, y):\n    float_1 = x.float();  x = None\n    sigmoid = float_1.sigmoid();  float_1 = None\n    half = sigmoid.half();  sigmoid = None\n    float_2 = y.float();  y = None\n    sigmoid_1 = float_2.sigmoid();  float_2 = None\n    half_1 = sigmoid_1.half();  sigmoid_1 = None\n    add_lowp = __main___add_lowp(half, half_1);  half = half_1 = None\n    return add_lowp\n\"\"\"\n\n# At this point, the implementation of `sigmoid_lowp` has been substituted\n# in for all of the calls to that function.\n\n# ***** Inlining calls during tracing *****\n# Now we are going to define a custom tracer that can selectively inline\n# calls to certain composite operations on-the-fly.\n\n# New instance of our module\nf = Foo()\n\nclass InliningTracer(torch.fx.Tracer):\n    FNS_TO_INLINE = [add_lowp]\n\n    def create_node(self, kind, target, args, kwargs, name=None, type_expr=None):\n        if kind == 'call_function' and target in self.FNS_TO_INLINE:\n            tracer = torch.fx.proxy.GraphAppendingTracer(self.graph)\n            # Trace through the implementation of the function rather than\n            # create a node\n            proxy_args = torch.fx.node.map_arg(args, lambda x: torch.fx.Proxy(x, tracer))\n            proxy_kwargs = torch.fx.node.map_arg(kwargs, lambda x: torch.fx.Proxy(x, tracer))\n            return target(*proxy_args, **proxy_kwargs).node\n        else:\n            return super().create_node(kind, target, args, kwargs, name, type_expr)\n\n\ntracer = InliningTracer()\ngraph = tracer.trace(f)\nmodule = torch.fx.GraphModule(f, graph)\nprint(module.code)\n\"\"\"\ndef forward(self, x, y):\n    sigmoid_lowp = __main___sigmoid_lowp(x);  x = None\n    sigmoid_lowp_1 = __main___sigmoid_lowp(y);  y = None\n    float_1 = sigmoid_lowp.float();  sigmoid_lowp = None\n    float_2 = sigmoid_lowp_1.float();  sigmoid_lowp_1 = None\n    add = float_1 + float_2;  float_1 = float_2 = None\n    half = add.half();  add = None\n    return half\n\"\"\"\n\n# As you can see, the implementation for `add_lowp` has been\n# inlined in the course of tracing with our InliningTracer.\n# Such functionality can be used to, for example, implement\n# a backend that wants to see the lowered form of some operations\n# but the high-level form of another.\n\n# ***** Future direction *****\n#\n# We may define an API, such as `Tracer.is_leaf_function`, that\n# Tracer implementers can use to more easily specify the inlining\n# behavior implemented in InliningTracer. Such a method would return\n# True by default, but a Tracer can override it and return `False` for\n# functions the Tracer wants to be traced through.\n"
  },
  {
    "path": "fx/profiling_tracer.py",
    "content": "\"\"\"\nThis file demonstrates using a custom FX Tracer to override\nthe behavior of `torch.autograd.profiler.record_function` and\nmake profiler ranges appear in FX-traced code. This is done\nwith Python dynamic patching magic, allowing us to explicitly\nemit calls to\n`torch.ops.profiler._record_function_enter/_record_function_exit`.\n\nPlease note that before https://github.com/pytorch/pytorch/pull/65180 lands,\nthese ranges may be elimineated by `Graph.eliminate_dead_code`\n\"\"\"\nimport torch\nimport torch.fx\n\n# Setup: a module with `record_function`\nclass Foo(torch.nn.Module):\n  def forward(self, x):\n    with torch.profiler.record_function('foo'):\n      return torch.relu(x)\n\nf = Foo()\nx = torch.randn(5, 3, 2)\nwith torch.autograd.profiler.profile() as prof:\n  f(x)\n\nprint(prof)\n# \"foo\" range is correctly recorded with normal execution\n\"\"\"\n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  \n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n        aten::zeros         6.10%      10.298us        10.04%      16.943us      16.943us             1  \n        aten::empty         2.88%       4.857us         2.88%       4.857us       4.857us             1  \n        aten::zero_         1.06%       1.788us         1.06%       1.788us       1.788us             1  \n                foo        21.28%      35.925us        89.96%     151.888us     151.888us             1  \n        aten::empty        11.59%      19.572us        11.59%      19.572us      19.572us             1  \n         aten::relu        23.81%      40.203us        57.09%      96.391us      96.391us             1  \n    aten::clamp_min         3.87%       6.539us        33.28%      56.188us      56.188us             1  \n        aten::empty         1.09%       1.847us         1.09%       1.847us       1.847us             1  \n    aten::clamp_min        28.31%      47.802us        28.31%      47.802us      47.802us             1  \n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \nSelf CPU time total: 168.831us\n\"\"\"\n\n\ntraced = torch.fx.symbolic_trace(f)\nwith torch.autograd.profiler.profile() as prof:\n  traced(x)\n\nprint(prof)\n# \"foo\" range is not recorded with FX tracing\n\"\"\"\n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  \n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n         aten::relu        23.50%      10.618us       100.00%      45.186us      45.186us             1  \n    aten::clamp_min        18.05%       8.154us        76.50%      34.568us      34.568us             1  \n        aten::empty        11.77%       5.317us        11.77%       5.317us       5.317us             1  \n    aten::clamp_min        46.69%      21.097us        46.69%      21.097us      21.097us             1  \n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \nSelf CPU time total: 45.186us\n\"\"\"\n\n\nclass ProfilerTracer(torch.fx.Tracer):\n  def trace(self, root, concrete_args=None):\n    orig_record_function_enter = torch.autograd.profiler.record_function.__enter__\n    orig_record_function_exit = torch.autograd.profiler.record_function.__exit__\n\n    def fake_profiler_enter(_self):\n      nonlocal self\n      handle_proxy = self.create_proxy(\n          kind='call_function',\n          target=torch.ops.profiler._record_function_enter,\n          args=(_self.name,),\n          kwargs={})\n      \n      assert getattr(_self, '_fx_profiler_ctx', None) is None\n      setattr(_self, '_fx_profiler_ctx', handle_proxy)\n      return handle_proxy\n\n    def fake_profiler_exit(_self, exc_type, exc_value, traceback):\n      assert hasattr(_self, '_fx_profiler_ctx')\n      handle_proxy = _self._fx_profiler_ctx\n      torch.ops.profiler._record_function_exit(handle_proxy)\n      setattr(_self, '_fx_profiler_ctx', None)\n\n    torch.autograd.profiler.record_function.__enter__ = fake_profiler_enter\n    torch.autograd.profiler.record_function.__exit__ = fake_profiler_exit\n\n    try:\n      return super().trace(root, concrete_args)\n    finally:\n      torch.autograd.profiler.record_function.__enter__ = orig_record_function_enter\n      torch.autograd.profiler.record_function.__exit__ = orig_record_function_exit\n\npt = ProfilerTracer()\n\ngraph_with_profiler = pt.trace(f)\ntraced_with_profiler = torch.fx.GraphModule(pt.root, graph_with_profiler)\n\nwith torch.autograd.profiler.profile() as prof:\n  traced_with_profiler(x)\n\nprint(prof)\n# \"foo\" range is recorded with special tracer behavior\n\"\"\"\n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  \n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n                foo        19.76%      39.928us       100.00%     202.055us     202.055us             1  \n        aten::empty         3.93%       7.950us         3.93%       7.950us       7.950us             1  \n         aten::relu        33.79%      68.282us        76.30%     154.177us     154.177us             1  \n    aten::clamp_min        27.32%      55.198us        42.51%      85.895us      85.895us             1  \n        aten::empty         1.28%       2.585us         1.28%       2.585us       2.585us             1  \n    aten::clamp_min        13.91%      28.112us        13.91%      28.112us      28.112us             1  \n-------------------  ------------  ------------  ------------  ------------  ------------  ------------  \nSelf CPU time total: 202.055us\n\"\"\"\n"
  },
  {
    "path": "fx/proxy_based_graph_creation.py",
    "content": "import torch\nfrom torch.fx import Proxy, Graph, GraphModule\n\n\n'''\nHow to Create a Graph Using Proxy Objects Instead of Tracing\n\nIt's possible to directly create a Proxy object around a raw Node. This\ncan be used to create a Graph independently of symbolic tracing.\n\nThe following code demonstrates how to use Proxy with a raw Node to\nappend operations to a fresh Graph. We'll create two parameters (``x``\nand ``y``), perform some operations on those parameters, then add\neverything we created to the new Graph. We'll then wrap that Graph in\na GraphModule. Doing so creates a runnable instance of ``nn.Module``\nwhere previously-created operations are represented in the Module's\n``forward`` function.\n\nBy the end of the tutorial, we'll have added the following method to an\nempty ``nn.Module`` class.\n\n.. code-block:: python\n\n    def forward(self, x, y):\n        cat_1 = torch.cat([x, y]);  x = y = None\n        tanh_1 = torch.tanh(cat_1);  cat_1 = None\n        neg_1 = torch.neg(tanh_1);  tanh_1 = None\n        return neg_1\n\n'''\n\n\n# Create a graph independently of symbolic tracing\ngraph = Graph()\ntracer = torch.fx.proxy.GraphAppendingTracer(graph)\n\n# Create raw Nodes\nraw1 = graph.placeholder('x')\nraw2 = graph.placeholder('y')\n\n# Initialize Proxies using the raw Nodes and graph's default tracer\ny = Proxy(raw1, tracer)\nz = Proxy(raw2, tracer)\n# y = Proxy(raw1)\n# z = Proxy(raw2)\n\n# Create other operations using the Proxies `y` and `z`\na = torch.cat([y, z])\nb = torch.tanh(a)\nc = torch.neg(b)\n# By using the graph's own appending tracer to create Proxies,\n# notice we can now use n-ary operators on operations without\n# multiple tracers being created at run-time (line 52) which leads\n# to errors # To try this out for yourself, replace lines 42, 43\n# with 44, 45\nz = torch.add(b, c)\n\n# Create a new output Node and add it to the Graph. By doing this, the\n# Graph will contain all the Nodes we just created (since they're all\n# linked to the output Node)\ngraph.output(c.node)\n\n# Wrap our created Graph in a GraphModule to get a final, runnable\n# `nn.Module` instance\nmod = GraphModule(torch.nn.Module(), graph)\n"
  },
  {
    "path": "fx/replace_op.py",
    "content": "import torch\nfrom torch.fx import symbolic_trace\nimport operator\n\n\"\"\"\nHow to Replace One Op With Another\n\n1. Iterate through all Nodes in your GraphModule's Graph.\n2. Determine if the current Node should be replaced. (Suggested: match\non the Node's ``target`` attribute).\n3. Create a replacement Node and add it to the Graph.\n4. Use the FX built-in ``replace_all_uses_with`` to replace all uses of\nthe current Node with the replacement.\n5. Delete the old Node from the graph.\n6. Call ``recompile`` on the GraphModule. This updates the generated\nPython code to reflect the new Graph state.\n\nCurrently, FX does not provide any way to guarantee that replaced\noperators are syntactically valid. It's up to the user to confirm that\nany new operators will work with the existing operands.\n\nThe following code demonstrates an example of replacing any instance of\naddition with a bitwise AND.\n\nTo examine how the Graph evolves during op replacement, add the\nstatement `print(traced.graph)` after the line you want to inspect.\nAlternatively, call `traced.graph.print_tabular()` to see the IR in a\ntabular format.\n\"\"\"\n\n# Sample module\nclass M(torch.nn.Module):\n    def forward(self, x, y):\n        return x + y, torch.add(x, y), x.add(y)\n\n# Symbolically trace an instance of the module\ntraced = symbolic_trace(M())\n\n# As demonstrated in the above example, there are several different ways\n# to denote addition. The possible cases are:\n#     1. `x + y` - A `call_function` Node with target `operator.add`.\n#         We can match for equality on that `operator.add` directly.\n#     2. `torch.add(x, y)` - A `call_function` Node with target\n#         `torch.add`. Similarly, we can match this function directly.\n#     3. `x.add(y)` - The Tensor method call, whose target we can match\n#         as a string.\n\npatterns = set([operator.add, torch.add, \"add\"])\n\n# Go through all the nodes in the Graph\nfor n in traced.graph.nodes:\n    # If the target matches one of the patterns\n    if any(n.target == pattern for pattern in patterns):\n        # Set the insert point, add the new node, and replace all uses\n        # of `n` with the new node\n        with traced.graph.inserting_after(n):\n            new_node = traced.graph.call_function(torch.bitwise_and, n.args, n.kwargs)\n            n.replace_all_uses_with(new_node)\n        # Remove the old node from the graph\n        traced.graph.erase_node(n)\n\n# Don't forget to recompile!\ntraced.recompile()\n"
  },
  {
    "path": "fx/requirements.txt",
    "content": "torch\ntorchvision\n"
  },
  {
    "path": "fx/subgraph_rewriter_basic_use.py",
    "content": "import torch\nfrom torch.fx import symbolic_trace, replace_pattern\n\n\n'''\nHow to Use the FX Subgraph Rewriter\n\nFor easy subgraph rewriting, FX exposes the utility function:\n\n    replace_pattern(gm : GraphModule,\n                    pattern : Callable,\n                    replacement : Callable)\n                    -> None\n\n`replace_pattern` matches all possible non-overlapping sets of operators\nand their data dependencies (`pattern`) in the Graph of a GraphModule\n(`gm`), then replaces each of these matched subgraphs with another\nsubgraph (`replacement).\n\nThe docstring for `replace_pattern` (located in `subgraph_rewriter.py`)\ngives an in-depth explanation as to how `pattern` and `replacement`\nshould be specified, what happens during pattern matching, and other\nimportant technical details. This tutorial, therefore, is only meant to\ngive an overview as to the FX Subgraph Rewriter's basic functionality.\nLet's go rewrite a Graph!\n'''\n\n# Sample module\nclass M(torch.nn.Module):\n    def __init__(self):\n        super().__init__()\n\n    def forward(self, x, w1, w2):\n        val1 = torch.neg(w1)\n        m1 = torch.cat([val1, w2]).sum()\n        val2 = torch.neg(w1)\n        m2 = torch.cat([val2, w2]).sum()\n        return x + torch.max(m1) + torch.max(m2)\n\n# Symbolically trace an instance of `M`\ntraced = symbolic_trace(M())\n\n# Define the pattern. The FX Subgraph Rewriter will match all\n# non-overlapping instances of the pattern in the larger graph.\n# Note that Pattern-matching is done based on data dependencies,\n# not Node names. Even though we're operating on Nodes named `a1` and\n# `a2` instead of `w1` and `w2`, the pattern is still a valid match\n# for the two instances of `torch.cat([w1, w2]).sum()` above. Only\n# operations that contribute to the single output value of the pattern\n# are considered\ndef pattern(a1, a2):\n    val1 = torch.neg(a1)\n    return torch.cat([val1, a2]).sum()\n\n# Define the replacement (same rules as the pattern)\ndef replacement(w1, w2):\n    return torch.stack([w1, w2])\n\n# Replace `pattern` with `replacement` in `traced`\nreplace_pattern(traced, pattern, replacement)\n\n# After calling `replace_pattern`, the generated code is:\n'''\ndef forward(self, x, w1, w2):\n    stack = torch.stack([w1, w2])\n    max_1 = torch.max(stack);  stack = None\n    add = x + max_1;  x = max_1 = None\n    stack_1 = torch.stack([w1, w2]);  w1 = w2 = None\n    max_2 = torch.max(stack_1);  stack_1 = None\n    add_1 = add + max_2;  add = max_2 = None\n    return add_1\n'''\n"
  },
  {
    "path": "fx/wrap_output_dynamically.py",
    "content": "\nfrom enum import Enum, auto\n\nimport torch\nfrom torch.fx import GraphModule, Node, Proxy, symbolic_trace\n\n'''\nWrap Graph Output Dynamically\n\nThe following code demonstrates how change an existing Graph based on\nparameters specified at runtime. We'll let the user specify an\nactivation function from a predefined Enum list, then we'll symbolically\ntrace it. Next, we'll create a Proxy from the last operation in the\nGraph. We'll call our traced activation function with this Proxy and\ninsert the ``output`` Node from that call into our Graph. (This final\nstep will automatically inline the entire traced function.)\n'''\n\n\n# Sample module\nclass M(torch.nn.Module):\n    def __init__(self):\n        super().__init__()\n\n    def forward(self, x, y):\n        y = torch.cat([x, y])\n        return y\n\n# Symbolically trace an instance of `M`\ntraced = symbolic_trace(M())\n\n# Selected activation functions\nclass ActivationFunction(Enum):\n    RELU = auto()\n    LEAKY_RELU = auto()\n    PRELU = auto()\n\n# Map activation function names to their implementation\nactivation_functions = {\n    ActivationFunction.RELU: torch.nn.ReLU(),\n    ActivationFunction.LEAKY_RELU: torch.nn.LeakyReLU(),\n    ActivationFunction.PRELU: torch.nn.PReLU(),\n}\n\ndef wrap_in_activation_function(m: GraphModule, fn: ActivationFunction) -> GraphModule:\n    # Get output node\n    output_node: Optional[Node] = None\n    for n in reversed(m.graph.nodes):\n        if n.op == \"output\":\n            output_node = n\n            break\n    assert output_node\n\n    # Get the actual output (the \"input\" of the output node). This is\n    # the Node we want to wrap in a user-specified activation function\n    assert len(output_node.all_input_nodes) == 1\n    wrap_node = output_node.all_input_nodes[0]\n\n    # Wrap the actual output in a Proxy\n    wrap_proxy = Proxy(wrap_node)\n\n    # Get the implementation of the specified activation function and\n    # symbolically trace it\n    fn_impl = activation_functions[fn]\n    fn_impl_traced = symbolic_trace(fn_impl)\n\n    # Call the specified activation function using the Proxy wrapper for\n    # `output_op`. The result of this call is another Proxy, which we\n    # can hook into our existing Graph.\n    with traced.graph.inserting_after(wrap_node):\n        fn_impl_output_node = fn_impl_traced(wrap_proxy)\n        new_args = (fn_impl_output_node.node,)\n        output_node.args = new_args\n\n    m.recompile()\n\n\n# Example call\nx, y = torch.randn(5, 3), torch.randn(5, 3)\norig_output = traced(x, y)\n\nwrap_in_activation_function(traced, ActivationFunction.LEAKY_RELU)\nnew_output = traced(x, y)\n\ntorch.testing.assert_close(new_output, torch.nn.LeakyReLU()(orig_output))\n"
  },
  {
    "path": "gat/README.md",
    "content": "# Graph Attention Network\n\nThis repository contains a PyTorch implementation of the **Graph Attention Networks (GAT)** based on the paper [\"Graph Attention Network\" by Velickovic et al](https://arxiv.org/abs/1710.10903v3). \n\nThe Graph Attention Network is a powerful graph neural network model for learning represtations on graph-structured data, which has shown excellent performance in various tasks such as node classification, link prediction, and graph classification.\n\n\n## Overview\nThe Graph Attention Network (GAT) is a graph neural network architecture designed specifically for handling graph-structured data. It leverages multi-head attention mechanism to capture the information of neighboring nodes in an attentive manner to learn represtations for each node. This attention mechanism allows the model to focus on relevant nodes and adaptively weight their contributions during message passing.\n\nCheck out the following resources for more ino on GATs:\n- [Blog post by the main auther, Petar Velickovic](https://petar-v.com/GAT/)\n- [Main paper](https://doi.org/10.48550/arXiv.1710.10903)\n\nThis repository provides a clean and short implementation of the official GAT model using PyTorch. The code is well-documented and easy to understand, making it a valuable resource for researchers and practitioners interested in graph deep learning.\n\n\n## Key Features\n\n- **GAT Model**: Implementation of the Graph Attention Network model with multi-head attention based on the paper \"Graph Attention Network\" by Velickovic et al.\n- **Graph Attention Layers**: Implementation of graph convolutional layers that aggregate information from neighboring nodes using a self-attention mechanisms to learn node importance weights.\n- **Training and Evaluation**: Code for training GAT models on graph-structured data and evaluating their performance on node classification tasks on the *Cora* benchmark dataset.\n\n---\n\n# Requirements\n- Python 3.7 or higher\n- PyTorch 2.0 or higher\n- Requests 2.31 or higher\n- NumPy 1.24 or higher\n\n\n\n# Dataset\nThe implementation includes support for the Cora dataset, a standard benchmark dataset for graph-based machine learning tasks. The Cora dataset consists of scientific publications, where nodes represent papers and edges represent citation relationships. Each paper is associated with a binary label indicating one of seven classes. The dataset is downloaded, preprocessed and ready to use.\n\n# Model Architecture\nThe official architecture (used in this project) proposed in the paper \"Graph Attention Network\" by Velickovic et al. consists of two graph attention layers which incorporates the multi-head attention mechanisms during its message trasformation and aggregation. Each graph attention layer applies a shared self-attention mechanism to every node in the graph, allowing them to learn different representations based on the importance of their neighbors.\n\nIn terms of activation functions, the GAT model employs both the **Exponential Linear Unit (ELU)** and the **Leaky Rectified Linear Unit (LeakyReLU)** activations, which introduce non-linearity to the model. ELU is used as the activation function for the **hidden layers**, while LeakyReLU is applied to the **attention coefficients** to ensure non-zero gradients for negative values.\n\nFollowing the official implementation, the first GAT layer consists of **K = 8 attention heads** computing **F' = 8 features** each (for a **total of 64 features**) followed by an exponential linear unit (ELU) activation on the layer outputs. The second GAT layer is used for classification: a **single attention head** that computes C features (where C is the number of classes), followed by a softmax activation for probablisitic outputs. (we use log-softmax instead for computational convenience with using NLLLoss)\n\n*Note that due to being an educational example, this implementation uses the full dense form of the adjacency matrix of the graph, and not the sparse form of the matrix. Thus all the operations in the model implemeation is done in a non-sparse from. This will not affect the model's performance accuracy-wise. However an sparse-friendly implementation will help with the efficiency in the use of resources, storage, and speed.*\n \n\n# Usage\nTraining and evaluating the GAT model on the Cora dataset can be done through running the `main.py` script as follows:\n\n1. Clone the PyTorch examples repository:\n\n```\ngit clone https://github.com/pytorch/examples.git\ncd examples/gat\n```\n\n2. Install the required dependencies:\n\n```\npip install -r requirements.txt\n```\n\n3. Train the GAT model by running the `main.py` script as follows:: (Example using the default parameters)\n\n```bash\npython main.py --epochs 300 --lr 0.005 --l2 5e-4 --dropout-p 0.6 --num-heads 8 --hidden-dim 64 --val-every 20\n```\n\nIn more detail, the `main.py` script recieves following arguments:\n```\nusage: main.py [-h] [--epochs EPOCHS] [--lr LR] [--l2 L2] [--dropout-p DROPOUT_P] [--hidden-dim HIDDEN_DIM] [--num-heads NUM_HEADS] [--concat-heads] [--val-every VAL_EVERY]\n               [--no-cuda] [--no-mps] [--dry-run] [--seed S]\n\nPyTorch Graph Attention Network\n\noptions:\n  -h, --help            show this help message and exit\n  --epochs EPOCHS       number of epochs to train (default: 300)\n  --lr LR               learning rate (default: 0.005)\n  --l2 L2               weight decay (default: 6e-4)\n  --dropout-p DROPOUT_P\n                        dropout probability (default: 0.6)\n  --hidden-dim HIDDEN_DIM\n                        dimension of the hidden representation (default: 64)\n  --num-heads NUM_HEADS\n                        number of the attention heads (default: 4)\n  --concat-heads        wether to concatinate attention heads, or average over them (default: False)\n  --val-every VAL_EVERY\n                        epochs to wait for print training and validation evaluation (default: 20)\n  --no-accel            disables accelerator\n  --dry-run             quickly check a single pass\n  --seed S              random seed (default: 13)\n```\n\n\n\n# Results\nAfter training for **300 epochs** with default hyperparameters on random train/val/test data splits, the GAT model achieves around **%81.25** classification accuracy on the test split. This result is comparable to the performance reported in the original paper. However, the results can vary due to the randomness of the train/val/test split.\n\n# Reference\n\n``` \n@article{\n  velickovic2018graph,\n  title=\"{Graph Attention Networks}\",\n  author={Veli{\\v{c}}kovi{\\'{c}}, Petar and Cucurull, Guillem and Casanova, Arantxa and Romero, Adriana and Li{\\`{o}}, Pietro and Bengio, Yoshua},\n  journal={International Conference on Learning Representations},\n  year={2018},\n  url={https://openreview.net/forum?id=rJXMpikCZ},\n}\n```\n- Paper on arxiv: [arXiv:1710.10903v3](https://doi.org/10.48550/arXiv.1710.10903)\n- Original paper repository: [https://github.com/PetarV-/GAT](https://github.com/PetarV-/GAT)\n"
  },
  {
    "path": "gat/main.py",
    "content": "import os\nimport time\nimport requests\nimport tarfile\nimport numpy as np\nimport argparse\n\nimport torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom torch.optim import Adam\n\n\n################################\n###  GAT LAYER DEFINITION    ###\n################################\n\nclass GraphAttentionLayer(nn.Module):\n    \"\"\"\n    Graph Attention Layer (GAT) as described in the paper `\"Graph Attention Networks\" <https://arxiv.org/pdf/1710.10903.pdf>`.\n\n        This operation can be mathematically described as:\n\n            e_ij = a(W h_i, W h_j)\n            α_ij = softmax_j(e_ij) = exp(e_ij) / Σ_k(exp(e_ik))     \n            h_i' = σ(Σ_j(α_ij W h_j))\n            \n            where h_i and h_j are the feature vectors of nodes i and j respectively, W is a learnable weight matrix,\n            a is an attention mechanism that computes the attention coefficients e_ij, and σ is an activation function.\n\n    \"\"\"\n    def __init__(self, in_features: int, out_features: int, n_heads: int, concat: bool = False, dropout: float = 0.4, leaky_relu_slope: float = 0.2):\n        super(GraphAttentionLayer, self).__init__()\n\n        self.n_heads = n_heads # Number of attention heads\n        self.concat = concat # wether to concatenate the final attention heads\n        self.dropout = dropout # Dropout rate\n\n        if concat: # concatenating the attention heads\n            self.out_features = out_features # Number of output features per node\n            assert out_features % n_heads == 0 # Ensure that out_features is a multiple of n_heads\n            self.n_hidden = out_features // n_heads\n        else: # averaging output over the attention heads (Used in the main paper)\n            self.n_hidden = out_features\n\n        #  A shared linear transformation, parametrized by a weight matrix W is applied to every node\n        #  Initialize the weight matrix W \n        self.W = nn.Parameter(torch.empty(size=(in_features, self.n_hidden * n_heads)))\n\n        # Initialize the attention weights a\n        self.a = nn.Parameter(torch.empty(size=(n_heads, 2 * self.n_hidden, 1)))\n\n        self.leakyrelu = nn.LeakyReLU(leaky_relu_slope) # LeakyReLU activation function\n        self.softmax = nn.Softmax(dim=1) # softmax activation function to the attention coefficients\n\n        self.reset_parameters() # Reset the parameters\n\n\n    def reset_parameters(self):\n        \"\"\"\n        Reinitialize learnable parameters.\n        \"\"\"\n        nn.init.xavier_normal_(self.W)\n        nn.init.xavier_normal_(self.a)\n    \n\n    def _get_attention_scores(self, h_transformed: torch.Tensor):\n        \"\"\"calculates the attention scores e_ij for all pairs of nodes (i, j) in the graph\n        in vectorized parallel form. for each pair of source and target nodes (i, j),\n        the attention score e_ij is computed as follows:\n\n            e_ij = LeakyReLU(a^T [Wh_i || Wh_j]) \n\n            where || denotes the concatenation operation, and a and W are the learnable parameters.\n\n        Args:\n            h_transformed (torch.Tensor): Transformed feature matrix with shape (n_nodes, n_heads, n_hidden),\n                where n_nodes is the number of nodes and out_features is the number of output features per node.\n\n        Returns:\n            torch.Tensor: Attention score matrix with shape (n_heads, n_nodes, n_nodes), where n_nodes is the number of nodes.\n        \"\"\"\n        \n        source_scores = torch.matmul(h_transformed, self.a[:, :self.n_hidden, :])\n        target_scores = torch.matmul(h_transformed, self.a[:, self.n_hidden:, :])\n\n        # broadcast add \n        # (n_heads, n_nodes, 1) + (n_heads, 1, n_nodes) = (n_heads, n_nodes, n_nodes)\n        e = source_scores + target_scores.mT\n        return self.leakyrelu(e)\n\n    def forward(self,  h: torch.Tensor, adj_mat: torch.Tensor):\n        \"\"\"\n        Performs a graph attention layer operation.\n\n        Args:\n            h (torch.Tensor): Input tensor representing node features.\n            adj_mat (torch.Tensor): Adjacency matrix representing graph structure.\n\n        Returns:\n            torch.Tensor: Output tensor after the graph convolution operation.\n        \"\"\"\n        n_nodes = h.shape[0]\n\n        # Apply linear transformation to node feature -> W h\n        # output shape (n_nodes, n_hidden * n_heads)\n        h_transformed = torch.mm(h, self.W)\n        h_transformed = F.dropout(h_transformed, self.dropout, training=self.training)\n\n        # splitting the heads by reshaping the tensor and putting heads dim first\n        # output shape (n_heads, n_nodes, n_hidden)\n        h_transformed = h_transformed.view(n_nodes, self.n_heads, self.n_hidden).permute(1, 0, 2)\n        \n        # getting the attention scores\n        # output shape (n_heads, n_nodes, n_nodes)\n        e = self._get_attention_scores(h_transformed)\n\n        # Set the attention score for non-existent edges to -9e15 (MASKING NON-EXISTENT EDGES)\n        connectivity_mask = -9e16 * torch.ones_like(e)\n        e = torch.where(adj_mat > 0, e, connectivity_mask) # masked attention scores\n        \n        # attention coefficients are computed as a softmax over the rows\n        # for each column j in the attention score matrix e\n        attention = F.softmax(e, dim=-1)\n        attention = F.dropout(attention, self.dropout, training=self.training)\n\n        # final node embeddings are computed as a weighted average of the features of its neighbors\n        h_prime = torch.matmul(attention, h_transformed)\n\n        # concatenating/averaging the attention heads\n        # output shape (n_nodes, out_features)\n        if self.concat:\n            h_prime = h_prime.permute(1, 0, 2).contiguous().view(n_nodes, self.out_features)\n        else:\n            h_prime = h_prime.mean(dim=0)\n\n        return h_prime\n\n################################\n### MAIN GAT NETWORK MODULE  ###\n################################\n\nclass GAT(nn.Module):\n    \"\"\"\n    Graph Attention Network (GAT) as described in the paper `\"Graph Attention Networks\" <https://arxiv.org/pdf/1710.10903.pdf>`.\n    Consists of a 2-layer stack of Graph Attention Layers (GATs). The fist GAT Layer is followed by an ELU activation.\n    And the second (final) layer is a GAT layer with a single attention head and softmax activation function. \n    \"\"\"\n    def __init__(self,\n        in_features,\n        n_hidden,\n        n_heads,\n        num_classes,\n        concat=False,\n        dropout=0.4,\n        leaky_relu_slope=0.2):\n        \"\"\" Initializes the GAT model. \n\n        Args:\n            in_features (int): number of input features per node.\n            n_hidden (int): output size of the first Graph Attention Layer.\n            n_heads (int): number of attention heads in the first Graph Attention Layer.\n            num_classes (int): number of classes to predict for each node.\n            concat (bool, optional): Wether to concatinate attention heads or take an average over them for the\n                output of the first Graph Attention Layer. Defaults to False.\n            dropout (float, optional): dropout rate. Defaults to 0.4.\n            leaky_relu_slope (float, optional): alpha (slope) of the leaky relu activation. Defaults to 0.2.\n        \"\"\"\n\n        super(GAT, self).__init__()\n\n        # Define the Graph Attention layers\n        self.gat1 = GraphAttentionLayer(\n            in_features=in_features, out_features=n_hidden, n_heads=n_heads,\n            concat=concat, dropout=dropout, leaky_relu_slope=leaky_relu_slope\n            )\n        \n        self.gat2 = GraphAttentionLayer(\n            in_features=n_hidden, out_features=num_classes, n_heads=1,\n            concat=False, dropout=dropout, leaky_relu_slope=leaky_relu_slope\n            )\n        \n\n    def forward(self, input_tensor: torch.Tensor , adj_mat: torch.Tensor):\n        \"\"\"\n        Performs a forward pass through the network.\n\n        Args:\n            input_tensor (torch.Tensor): Input tensor representing node features.\n            adj_mat (torch.Tensor): Adjacency matrix representing graph structure.\n\n        Returns:\n            torch.Tensor: Output tensor after the forward pass.\n        \"\"\"\n\n        # Apply the first Graph Attention layer\n        x = self.gat1(input_tensor, adj_mat)\n        x = F.elu(x) # Apply ELU activation function to the output of the first layer\n\n        # Apply the second Graph Attention layer\n        x = self.gat2(x, adj_mat)\n\n        return F.log_softmax(x, dim=1) # Apply log softmax activation function\n\n################################\n### LOADING THE CORA DATASET ###\n################################\n\ndef load_cora(path='./cora', device='cpu'):\n    \"\"\"\n    Loads the Cora dataset. The dataset is downloaded from https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgz.\n\n    \"\"\"\n\n    # Set the paths to the data files\n    content_path = os.path.join(path, 'cora.content')\n    cites_path = os.path.join(path, 'cora.cites')\n\n    # Load data from files\n    content_tensor = np.genfromtxt(content_path, dtype=np.dtype(str))\n    cites_tensor = np.genfromtxt(cites_path, dtype=np.int32)\n\n    # Process features\n    features = torch.FloatTensor(content_tensor[:, 1:-1].astype(np.int32)) # Extract feature values\n    scale_vector = torch.sum(features, dim=1) # Compute sum of features for each node\n    scale_vector = 1 / scale_vector # Compute reciprocal of the sums\n    scale_vector[scale_vector == float('inf')] = 0 # Handle division by zero cases\n    scale_vector = torch.diag(scale_vector).to_sparse() # Convert the scale vector to a sparse diagonal matrix\n    features = scale_vector @ features # Scale the features using the scale vector\n\n    # Process labels\n    classes, labels = np.unique(content_tensor[:, -1], return_inverse=True) # Extract unique classes and map labels to indices\n    labels = torch.LongTensor(labels) # Convert labels to a tensor\n\n    # Process adjacency matrix\n    idx = content_tensor[:, 0].astype(np.int32) # Extract node indices\n    idx_map = {id: pos for pos, id in enumerate(idx)} # Create a dictionary to map indices to positions\n\n    # Map node indices to positions in the adjacency matrix\n    edges = np.array(\n        list(map(lambda edge: [idx_map[edge[0]], idx_map[edge[1]]], \n            cites_tensor)), dtype=np.int32)\n\n    V = len(idx) # Number of nodes\n    E = edges.shape[0] # Number of edges\n    adj_mat = torch.sparse_coo_tensor(edges.T, torch.ones(E), (V, V), dtype=torch.int64) # Create the initial adjacency matrix as a sparse tensor\n    adj_mat = torch.eye(V) + adj_mat # Add self-loops to the adjacency matrix\n\n    # return features.to_sparse().to(device), labels.to(device), adj_mat.to_sparse().to(device)\n    return features.to(device), labels.to(device), adj_mat.to(device)\n\n#################################\n### TRAIN AND TEST FUNCTIONS  ###\n#################################\n\ndef train_iter(epoch, model, optimizer, criterion, input, target, mask_train, mask_val, print_every=10):\n    start_t = time.time()\n    model.train()\n    optimizer.zero_grad()\n\n    # Forward pass\n    output = model(*input) \n    loss = criterion(output[mask_train], target[mask_train]) # Compute the loss using the training mask\n\n    loss.backward()\n    optimizer.step()\n\n    # Evaluate the model performance on training and validation sets\n    loss_train, acc_train = test(model, criterion, input, target, mask_train)\n    loss_val, acc_val = test(model, criterion, input, target, mask_val)\n\n    if epoch % print_every == 0:\n        # Print the training progress at specified intervals\n        print(f'Epoch: {epoch:04d} ({(time.time() - start_t):.4f}s) loss_train: {loss_train:.4f} acc_train: {acc_train:.4f} loss_val: {loss_val:.4f} acc_val: {acc_val:.4f}')\n\n\ndef test(model, criterion, input, target, mask):\n    model.eval()\n    with torch.no_grad():\n        output = model(*input)\n        output, target = output[mask], target[mask]\n\n        loss = criterion(output, target)\n        acc = (output.argmax(dim=1) == target).float().sum() / len(target)\n    return loss.item(), acc.item()\n\n\nif __name__ == '__main__':\n\n    # Training settings\n    # All defalut values are the same as in the config used in the main paper\n\n    parser = argparse.ArgumentParser(description='PyTorch Graph Attention Network')\n    parser.add_argument('--epochs', type=int, default=300,\n                        help='number of epochs to train (default: 300)')\n    parser.add_argument('--lr', type=float, default=0.005,\n                        help='learning rate (default: 0.005)')\n    parser.add_argument('--l2', type=float, default=5e-4,\n                        help='weight decay (default: 6e-4)')\n    parser.add_argument('--dropout-p', type=float, default=0.6,\n                        help='dropout probability (default: 0.6)')\n    parser.add_argument('--hidden-dim', type=int, default=64,\n                        help='dimension of the hidden representation (default: 64)')\n    parser.add_argument('--num-heads', type=int, default=8,\n                        help='number of the attention heads (default: 4)')\n    parser.add_argument('--concat-heads', action='store_true',\n                        help='wether to concatinate attention heads, or average over them (default: False)')\n    parser.add_argument('--val-every', type=int, default=20,\n                        help='epochs to wait for print training and validation evaluation (default: 20)')\n    parser.add_argument('--no-accel', action='store_true',\n                        help='disables CUDA training')\n    parser.add_argument('--dry-run', action='store_true', \n                        help='quickly check a single pass')\n    parser.add_argument('--seed', type=int, default=13, metavar='S',\n                        help='random seed (default: 13)')\n    args = parser.parse_args()\n\n    torch.manual_seed(args.seed)\n\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    # Set the device to run on\n    if use_accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device('cpu')\n    print(f'Using {device} device')\n\n    # Load the dataset\n    cora_url = 'https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgz'\n    path = './cora'\n\n    if os.path.isfile(os.path.join(path, 'cora.content')) and os.path.isfile(os.path.join(path, 'cora.cites')):\n        print('Dataset already downloaded...')\n    else:\n        print('Downloading dataset...')\n        with requests.get(cora_url, stream=True) as tgz_file:\n            with tarfile.open(fileobj=tgz_file.raw, mode='r:gz') as tgz_object:\n                tgz_object.extractall()\n\n    print('Loading dataset...')\n    # Load the dataset\n    features, labels, adj_mat = load_cora(device=device)\n    # Split the dataset into training, validation, and test sets\n    idx = torch.randperm(len(labels)).to(device)\n    idx_test, idx_val, idx_train = idx[:1200], idx[1200:1600], idx[1600:]\n\n\n    # Create the model\n    # The model consists of a 2-layer stack of Graph Attention Layers (GATs).\n    gat_net = GAT(\n        in_features=features.shape[1],          # Number of input features per node  \n        n_hidden=args.hidden_dim,               # Output size of the first Graph Attention Layer\n        n_heads=args.num_heads,                 # Number of attention heads in the first Graph Attention Layer\n        num_classes=labels.max().item() + 1,    # Number of classes to predict for each node\n        concat=args.concat_heads,               # Wether to concatinate attention heads\n        dropout=args.dropout_p,                 # Dropout rate\n        leaky_relu_slope=0.2                    # Alpha (slope) of the leaky relu activation\n    ).to(device)\n\n    # configure the optimizer and loss function\n    optimizer = Adam(gat_net.parameters(), lr=args.lr, weight_decay=args.l2)\n    criterion = nn.NLLLoss()\n\n    # Train and evaluate the model\n    for epoch in range(args.epochs):\n        train_iter(epoch + 1, gat_net, optimizer, criterion, (features, adj_mat), labels, idx_train, idx_val, args.val_every)\n        if args.dry_run:\n            break\n    loss_test, acc_test = test(gat_net, criterion, (features, adj_mat), labels, idx_test)\n    print(f'Test set results: loss {loss_test:.4f} accuracy {acc_test:.4f}')"
  },
  {
    "path": "gat/requirements.txt",
    "content": "torch\nrequests\nnumpy\n"
  },
  {
    "path": "gcn/README.md",
    "content": "# Graph Convolutional Network\n\nThis repository contains an implementation of Graph Convolutional Networks (GCN) based on the paper \"Semi-Supervised Classification with Graph Convolutional Networks\" by Thomas N. Kipf and Max Welling.\n\n## Overview\nThis project implements the GCN model proposed in the paper for semi-supervised node classification on graph-structured data. GCN leverages graph convolutions to aggregate information from neighboring nodes and learn node representations for downstream tasks. The implementation provides a flexible and efficient GCN model for graph-based machine learning tasks.\n\n## Requirements\n```bash\npip install -r requirements.txt\n```\n\n# Usage\n```bash\npython main.py --epochs 200 --lr 0.01 --l2 5e-4 --dropout-p 0.5 --hidden-dim 16 --val-every 20 --include-bias\n```\n\n# Dataset\nThe implementation includes support for the Cora dataset, a standard benchmark dataset for graph-based machine learning tasks. The Cora dataset consists of scientific publications, where nodes represent papers and edges represent citation relationships. Each paper is associated with a binary label indicating one of seven classes. The dataset is downloaded, preprocessed and ready to use.\n\n## Model Architecture\nThe GCN model architecture follows the details provided in the paper. It consists of multiple graph convolutional layers with ReLU activation, followed by a final softmax layer for classification. The implementation supports customizable hyperparameters such as the number of hidden units, the number of layers, and dropout rate.\n\n# Results\nThe model achieves a classification accuracy of 82.5% on the test set of the Cora dataset after 200 epochs of training. This result is comparable to the performance reported in the original paper. However, the results can vary due to the randomness of the train/val/test split.\n\nReferences\nThomas N. Kipf and Max Welling. \"Semi-Supervised Classification with Graph Convolutional Networks.\" Link to the paper\n\nOriginal paper repository: [https://github.com/tkipf/gcn](https://github.com/tkipf/gcn)"
  },
  {
    "path": "gcn/main.py",
    "content": "import os\nimport time\nimport requests\nimport tarfile\nimport numpy as np\nimport argparse\n\nimport torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom torch.optim import Adam\n\n\nclass GraphConv(nn.Module):\n    \"\"\"\n        Graph Convolutional Layer described in \"Semi-Supervised Classification with Graph Convolutional Networks\".\n\n        Given an input feature representation for each node in a graph, the Graph Convolutional Layer aims to aggregate\n        information from the node's neighborhood to update its own representation. This is achieved by applying a graph\n        convolutional operation that combines the features of a node with the features of its neighboring nodes.\n\n        Mathematically, the Graph Convolutional Layer can be described as follows:\n\n            H' = f(D^(-1/2) * A * D^(-1/2) * H * W)\n\n        where:\n            H: Input feature matrix with shape (N, F_in), where N is the number of nodes and F_in is the number of \n                input features per node.\n            A: Adjacency matrix of the graph with shape (N, N), representing the relationships between nodes.\n            W: Learnable weight matrix with shape (F_in, F_out), where F_out is the number of output features per node.\n            D: The degree matrix.\n    \"\"\"\n    def __init__(self, input_dim, output_dim, use_bias=False):\n        super(GraphConv, self).__init__()\n\n        # Initialize the weight matrix W (in this case called `kernel`)\n        self.kernel = nn.Parameter(torch.Tensor(input_dim, output_dim))\n        nn.init.xavier_normal_(self.kernel) # Initialize the weights using Xavier initialization\n\n        # Initialize the bias (if use_bias is True)\n        self.bias = None\n        if use_bias:\n            self.bias = nn.Parameter(torch.Tensor(output_dim))\n            nn.init.zeros_(self.bias) # Initialize the bias to zeros\n\n    def forward(self, input_tensor, adj_mat):\n        \"\"\"\n        Performs a graph convolution operation.\n\n        Args:\n            input_tensor (torch.Tensor): Input tensor representing node features.\n            adj_mat (torch.Tensor): Normalized adjacency matrix representing graph structure.\n\n        Returns:\n            torch.Tensor: Output tensor after the graph convolution operation.\n        \"\"\"\n\n        support = torch.mm(input_tensor, self.kernel) # Matrix multiplication between input and weight matrix\n        output = torch.spmm(adj_mat, support) # Sparse matrix multiplication between adjacency matrix and support\n        # Add the bias (if bias is not None)\n        if self.bias is not None:\n            output = output + self.bias\n\n        return output\n\n\nclass GCN(nn.Module):\n    \"\"\"\n    Graph Convolutional Network (GCN) as described in the paper `\"Semi-Supervised Classification with Graph \n    Convolutional Networks\" <https://arxiv.org/pdf/1609.02907.pdf>`.\n\n    The Graph Convolutional Network is a deep learning architecture designed for semi-supervised node \n    classification tasks on graph-structured data. It leverages the graph structure to learn node representations \n    by propagating information through the graph using graph convolutional layers.\n\n    The original implementation consists of two stacked graph convolutional layers. The ReLU activation function is \n    applied to the hidden representations, and the Softmax activation function is applied to the output representations.\n    \"\"\"\n    def __init__(self, input_dim, hidden_dim, output_dim, use_bias=True, dropout_p=0.1):\n        super(GCN, self).__init__()\n\n        # Define the Graph Convolution layers\n        self.gc1 = GraphConv(input_dim, hidden_dim, use_bias=use_bias)\n        self.gc2 = GraphConv(hidden_dim, output_dim, use_bias=use_bias)\n\n        # Define the dropout layer\n        self.dropout = nn.Dropout(dropout_p)\n\n    def forward(self, input_tensor, adj_mat):\n        \"\"\"\n        Performs forward pass of the Graph Convolutional Network (GCN).\n\n        Args:\n            input_tensor (torch.Tensor): Input node feature matrix with shape (N, input_dim), where N is the number of nodes\n                and input_dim is the number of input features per node.\n            adj_mat (torch.Tensor): Normalized adjacency matrix of the graph with shape (N, N), representing the relationships between\n                nodes.\n\n        Returns:\n            torch.Tensor: Output tensor with shape (N, output_dim), representing the predicted class probabilities for each node.\n        \"\"\"\n\n        # Perform the first graph convolutional layer\n        x = self.gc1(input_tensor, adj_mat)\n        x = F.relu(x) # Apply ReLU activation function\n        x = self.dropout(x) # Apply dropout regularization\n\n        # Perform the second graph convolutional layer\n        x = self.gc2(x, adj_mat)\n\n        # Apply log-softmax activation function for classification\n        return F.log_softmax(x, dim=1)\n\n\ndef load_cora(path='./cora', device='cpu'):\n    \"\"\"\n    The graph convolutional operation rquires the normalized adjacency matrix: D^(-1/2) * A * D^(-1/2). This step \n    scales the adjacency matrix such that the features of neighboring nodes are weighted appropriately during \n    aggregation. The steps involved in the renormalization trick are as follows:\n        - Compute the degree matrix.\n        - Compute the inverse square root of the degree matrix.\n        - Multiply the inverse square root of the degree matrix with the adjacency matrix.\n    \"\"\"\n\n    # Set the paths to the data files\n    content_path = os.path.join(path, 'cora.content')\n    cites_path = os.path.join(path, 'cora.cites')\n\n    # Load data from files\n    content_tensor = np.genfromtxt(content_path, dtype=np.dtype(str))\n    cites_tensor = np.genfromtxt(cites_path, dtype=np.int32)\n\n    # Process features\n    features = torch.FloatTensor(content_tensor[:, 1:-1].astype(np.int32)) # Extract feature values\n    scale_vector = torch.sum(features, dim=1) # Compute sum of features for each node\n    scale_vector = 1 / scale_vector # Compute reciprocal of the sums\n    scale_vector[scale_vector == float('inf')] = 0 # Handle division by zero cases\n    scale_vector = torch.diag(scale_vector).to_sparse() # Convert the scale vector to a sparse diagonal matrix\n    features = scale_vector @ features # Scale the features using the scale vector\n\n    # Process labels\n    classes, labels = np.unique(content_tensor[:, -1], return_inverse=True) # Extract unique classes and map labels to indices\n    labels = torch.LongTensor(labels) # Convert labels to a tensor\n\n    # Process adjacency matrix\n    idx = content_tensor[:, 0].astype(np.int32) # Extract node indices\n    idx_map = {id: pos for pos, id in enumerate(idx)} # Create a dictionary to map indices to positions\n\n    # Map node indices to positions in the adjacency matrix\n    edges = np.array(\n        list(map(lambda edge: [idx_map[edge[0]], idx_map[edge[1]]], \n            cites_tensor)), dtype=np.int32)\n\n    V = len(idx) # Number of nodes\n    E = edges.shape[0] # Number of edges\n    adj_mat = torch.sparse_coo_tensor(edges.T, torch.ones(E), (V, V), dtype=torch.int64) # Create the initial adjacency matrix as a sparse tensor\n    adj_mat = torch.eye(V) + adj_mat # Add self-loops to the adjacency matrix\n\n    degree_mat = torch.sum(adj_mat, dim=1) # Compute the sum of each row in the adjacency matrix (degree matrix)\n    degree_mat = torch.sqrt(1 / degree_mat) # Compute the reciprocal square root of the degrees\n    degree_mat[degree_mat == float('inf')] = 0 # Handle division by zero cases\n    degree_mat = torch.diag(degree_mat).to_sparse() # Convert the degree matrix to a sparse diagonal matrix\n\n    adj_mat = degree_mat @ adj_mat @ degree_mat # Apply the renormalization trick\n\n    return features.to_sparse().to(device), labels.to(device), adj_mat.to_sparse().to(device)\n\ndef train_iter(epoch, model, optimizer, criterion, input, target, mask_train, mask_val, print_every=10):\n    start_t = time.time()\n    model.train()\n    optimizer.zero_grad()\n\n    # Forward pass\n    output = model(*input) \n    loss = criterion(output[mask_train], target[mask_train]) # Compute the loss using the training mask\n\n    loss.backward()\n    optimizer.step()\n\n    # Evaluate the model performance on training and validation sets\n    loss_train, acc_train = test(model, criterion, input, target, mask_train)\n    loss_val, acc_val = test(model, criterion, input, target, mask_val)\n\n    if epoch % print_every == 0:\n        # Print the training progress at specified intervals\n        print(f'Epoch: {epoch:04d} ({(time.time() - start_t):.4f}s) loss_train: {loss_train:.4f} acc_train: {acc_train:.4f} loss_val: {loss_val:.4f} acc_val: {acc_val:.4f}')\n\n\ndef test(model, criterion, input, target, mask):\n    model.eval()\n    with torch.no_grad():\n        output = model(*input)\n        output, target = output[mask], target[mask]\n\n        loss = criterion(output, target)\n        acc = (output.argmax(dim=1) == target).float().sum() / len(target)\n    return loss.item(), acc.item()\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser(description='PyTorch Graph Convolutional Network')\n    parser.add_argument('--epochs', type=int, default=200,\n                        help='number of epochs to train (default: 200)')\n    parser.add_argument('--lr', type=float, default=0.01,\n                        help='learning rate (default: 0.01)')\n    parser.add_argument('--l2', type=float, default=5e-4,\n                        help='weight decay (default: 5e-4)')\n    parser.add_argument('--dropout-p', type=float, default=0.5,\n                        help='dropout probability (default: 0.5)')\n    parser.add_argument('--hidden-dim', type=int, default=16,\n                        help='dimension of the hidden representation (default: 16)')\n    parser.add_argument('--val-every', type=int, default=20,\n                        help='epochs to wait for print training and validation evaluation (default: 20)')\n    parser.add_argument('--include-bias', action='store_true',\n                        help='use bias term in convolutions (default: False)')\n    parser.add_argument('--no-accel', action='store_true',\n                        help='disables accelerator')\n    parser.add_argument('--dry-run', action='store_true',\n                        help='quickly check a single pass')\n    parser.add_argument('--seed', type=int, default=42, metavar='S',\n                        help='random seed (default: 42)')\n    args = parser.parse_args()\n\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    torch.manual_seed(args.seed)\n\n    if use_accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device('cpu')\n\n    print(f'Using {device} device')\n\n    cora_url = 'https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgz'\n    print('Downloading dataset...')\n    with requests.get(cora_url, stream=True) as tgz_file:\n        with tarfile.open(fileobj=tgz_file.raw, mode='r:gz') as tgz_object:\n            tgz_object.extractall()\n\n    print('Loading dataset...')\n    features, labels, adj_mat = load_cora(device=device)\n    idx = torch.randperm(len(labels)).to(device)\n    idx_test, idx_val, idx_train = idx[:1000], idx[1000:1500], idx[1500:]\n\n    gcn = GCN(features.shape[1], args.hidden_dim, labels.max().item() + 1, args.include_bias, args.dropout_p).to(device)\n    optimizer = Adam(gcn.parameters(), lr=args.lr, weight_decay=args.l2)\n    criterion = nn.NLLLoss()\n\n    for epoch in range(args.epochs):\n        train_iter(epoch + 1, gcn, optimizer, criterion, (features, adj_mat), labels, idx_train, idx_val, args.val_every)\n        if args.dry_run:\n            break\n\n    loss_test, acc_test = test(gcn, criterion, (features, adj_mat), labels, idx_test)\n    print(f'Test set results: loss {loss_test:.4f} accuracy {acc_test:.4f}')"
  },
  {
    "path": "gcn/requirements.txt",
    "content": "torch>=2.6\ntorchvision\nrequests\nnumpy\n"
  },
  {
    "path": "imagenet/README.md",
    "content": "# ImageNet training in PyTorch\n\nThis implements training of popular model architectures, such as ResNet, AlexNet, and VGG on the ImageNet dataset.\n\n## Requirements\n\n- Install PyTorch ([pytorch.org](http://pytorch.org))\n- `pip install -r requirements.txt`\n- Download the ImageNet dataset from http://www.image-net.org/\n  - Then, move and extract the training and validation images to labeled subfolders, using [the following shell script](extract_ILSVRC.sh)\n\n## Training\n\nTo train a model, run `main.py` with the desired model architecture and the path to the ImageNet dataset:\n\n```bash\npython main.py -a resnet18 [imagenet-folder with train and val folders]\n```\n\nThe default learning rate schedule starts at 0.1 and decays by a factor of 10 every 30 epochs. This is appropriate for ResNet and models with batch normalization, but too high for AlexNet and VGG. Use 0.01 as the initial learning rate for AlexNet or VGG:\n\n```bash\npython main.py -a alexnet --lr 0.01 [imagenet-folder with train and val folders]\n```\n\n## Use Dummy Data\n\nImageNet dataset is large and time-consuming to download. To get started quickly, run `main.py` using dummy data by \"--dummy\". It's also useful for training speed benchmark. Note that the loss or accuracy is useless in this case.\n\n```bash\npython main.py -a resnet18 --dummy\n```\n\n## Multi-processing Distributed Data Parallel Training\n\nIf running on CUDA, you should always use the NCCL backend for multi-processing distributed training since it currently provides the best distributed training performance.\n\nFor XPU multiprocessing is not supported as of PyTorch 2.6.\n\n### Single node, multiple GPUs:\n\n```bash\npython main.py -a resnet50 --dist-url 'tcp://127.0.0.1:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0 [imagenet-folder with train and val folders]\n```\n\n### Multiple nodes:\n\nNode 0:\n\n```bash\npython main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 0 [imagenet-folder with train and val folders]\n```\n\nNode 1:\n\n```bash\npython main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 1 [imagenet-folder with train and val folders]\n```\n\n## Usage\n\n```bash\nusage: main.py [-h] [-a ARCH] [-j N] [--epochs N] [--start-epoch N] [-b N] [--lr LR] [--momentum M] [--wd W] [-p N] [--resume PATH] [-e] [--pretrained] [--world-size WORLD_SIZE] [--rank RANK]\n               [--dist-url DIST_URL] [--dist-backend DIST_BACKEND] [--seed SEED] [--gpu GPU] [--no-accel][--multiprocessing-distributed] [--dummy]\n               [DIR]\n\nPyTorch ImageNet Training\n\npositional arguments:\n  DIR                   path to dataset (default: imagenet)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -a ARCH, --arch ARCH  model architecture: alexnet | convnext_base | convnext_large | convnext_small | convnext_tiny | densenet121 | densenet161 | densenet169 | densenet201 | efficientnet_b0 |\n                        efficientnet_b1 | efficientnet_b2 | efficientnet_b3 | efficientnet_b4 | efficientnet_b5 | efficientnet_b6 | efficientnet_b7 | googlenet | inception_v3 | mnasnet0_5 | mnasnet0_75 |\n                        mnasnet1_0 | mnasnet1_3 | mobilenet_v2 | mobilenet_v3_large | mobilenet_v3_small | regnet_x_16gf | regnet_x_1_6gf | regnet_x_32gf | regnet_x_3_2gf | regnet_x_400mf | regnet_x_800mf |\n                        regnet_x_8gf | regnet_y_128gf | regnet_y_16gf | regnet_y_1_6gf | regnet_y_32gf | regnet_y_3_2gf | regnet_y_400mf | regnet_y_800mf | regnet_y_8gf | resnet101 | resnet152 | resnet18 |\n                        resnet34 | resnet50 | resnext101_32x8d | resnext50_32x4d | shufflenet_v2_x0_5 | shufflenet_v2_x1_0 | shufflenet_v2_x1_5 | shufflenet_v2_x2_0 | squeezenet1_0 | squeezenet1_1 | vgg11 |\n                        vgg11_bn | vgg13 | vgg13_bn | vgg16 | vgg16_bn | vgg19 | vgg19_bn | vit_b_16 | vit_b_32 | vit_l_16 | vit_l_32 | wide_resnet101_2 | wide_resnet50_2 (default: resnet18)\n  -j N, --workers N     number of data loading workers (default: 4)\n  --epochs N            number of total epochs to run\n  --start-epoch N       manual epoch number (useful on restarts)\n  -b N, --batch-size N  mini-batch size (default: 256), this is the total batch size of all GPUs on the current node when using Data Parallel or Distributed Data Parallel\n  --lr LR, --learning-rate LR\n                        initial learning rate\n  --momentum M          momentum\n  --wd W, --weight-decay W\n                        weight decay (default: 1e-4)\n  -p N, --print-freq N  print frequency (default: 10)\n  --resume PATH         path to latest checkpoint (default: none)\n  -e, --evaluate        evaluate model on validation set\n  --pretrained          use pre-trained model\n  --world-size WORLD_SIZE\n                        number of nodes for distributed training\n  --rank RANK           node rank for distributed training\n  --dist-url DIST_URL   url used to set up distributed training\n  --dist-backend DIST_BACKEND\n                        distributed backend\n  --seed SEED           seed for initializing training.\n  --gpu GPU             GPU id to use.\n  --no-accel            disables accelerator\n  --multiprocessing-distributed\n                        Use multi-processing distributed training to launch N processes per node, which has N GPUs. This is the fastest way to use PyTorch for either single node or multi node data parallel\n                        training\n  --dummy               use fake data to benchmark\n\n```\n"
  },
  {
    "path": "imagenet/extract_ILSVRC.sh",
    "content": "#!/bin/bash\n#\n# script to extract ImageNet dataset\n# ILSVRC2012_img_train.tar (about 138 GB)\n# ILSVRC2012_img_val.tar (about 6.3 GB)\n# make sure ILSVRC2012_img_train.tar & ILSVRC2012_img_val.tar in your current directory\n#\n#  Adapted from:\n#  https://github.com/facebook/fb.resnet.torch/blob/master/INSTALL.md\n#  https://gist.github.com/BIGBALLON/8a71d225eff18d88e469e6ea9b39cef4\n# \n#  imagenet/train/\n#  ├── n01440764\n#  │   ├── n01440764_10026.JPEG\n#  │   ├── n01440764_10027.JPEG\n#  │   ├── ......\n#  ├── ......\n#  imagenet/val/\n#  ├── n01440764\n#  │   ├── ILSVRC2012_val_00000293.JPEG\n#  │   ├── ILSVRC2012_val_00002138.JPEG\n#  │   ├── ......\n#  ├── ......\n#\n#\n# Make imagnet directory\n#\nmkdir imagenet\n#\n# Extract the training data:\n#\n# Create train directory; move .tar file; change directory\nmkdir imagenet/train && mv ILSVRC2012_img_train.tar imagenet/train/ && cd imagenet/train\n# Extract training set; remove compressed file\ntar -xvf ILSVRC2012_img_train.tar && rm -f ILSVRC2012_img_train.tar\n#\n# At this stage imagenet/train will contain 1000 compressed .tar files, one for each category\n#\n# For each .tar file: \n#   1. create directory with same name as .tar file\n#   2. extract and copy contents of .tar file into directory\n#   3. remove .tar file\nfind . -name \"*.tar\" | while read NAME ; do mkdir -p \"${NAME%.tar}\"; tar -xvf \"${NAME}\" -C \"${NAME%.tar}\"; rm -f \"${NAME}\"; done\n#\n# This results in a training directory like so:\n#\n#  imagenet/train/\n#  ├── n01440764\n#  │   ├── n01440764_10026.JPEG\n#  │   ├── n01440764_10027.JPEG\n#  │   ├── ......\n#  ├── ......\n#\n# Change back to original directory\ncd ../..\n#\n# Extract the validation data and move images to subfolders:\n#\n# Create validation directory; move .tar file; change directory; extract validation .tar; remove compressed file\nmkdir imagenet/val && mv ILSVRC2012_img_val.tar imagenet/val/ && cd imagenet/val && tar -xvf ILSVRC2012_img_val.tar && rm -f ILSVRC2012_img_val.tar\n# get script from soumith and run; this script creates all class directories and moves images into corresponding directories\nwget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash\n#\n# This results in a validation directory like so:\n#\n#  imagenet/val/\n#  ├── n01440764\n#  │   ├── ILSVRC2012_val_00000293.JPEG\n#  │   ├── ILSVRC2012_val_00002138.JPEG\n#  │   ├── ......\n#  ├── ......\n#\n#\n# Check total files after extract\n#\n#  $ find train/ -name \"*.JPEG\" | wc -l\n#  1281167\n#  $ find val/ -name \"*.JPEG\" | wc -l\n#  50000\n#\n"
  },
  {
    "path": "imagenet/main.py",
    "content": "import argparse\nimport os\nimport random\nimport shutil\nimport time\nimport warnings\nfrom enum import Enum\n\nimport torch\nimport torch.backends.cudnn as cudnn\nimport torch.distributed as dist\nimport torch.multiprocessing as mp\nimport torch.nn as nn\nimport torch.nn.parallel\nimport torch.optim\nimport torch.utils.data\nimport torch.utils.data.distributed\nimport torchvision.datasets as datasets\nimport torchvision.models as models\nimport torchvision.transforms as transforms\nfrom torch.optim.lr_scheduler import StepLR\nfrom torch.utils.data import Subset\n\nmodel_names = sorted(name for name in models.__dict__\n    if name.islower() and not name.startswith(\"__\")\n    and callable(models.__dict__[name]))\n\nparser = argparse.ArgumentParser(description='PyTorch ImageNet Training')\nparser.add_argument('data', metavar='DIR', nargs='?', default='imagenet',\n                    help='path to dataset (default: imagenet)')\nparser.add_argument('-a', '--arch', metavar='ARCH', default='resnet18',\n                    choices=model_names,\n                    help='model architecture: ' +\n                        ' | '.join(model_names) +\n                        ' (default: resnet18)')\nparser.add_argument('-j', '--workers', default=4, type=int, metavar='N',\n                    help='number of data loading workers (default: 4)')\nparser.add_argument('--epochs', default=90, type=int, metavar='N',\n                    help='number of total epochs to run')\nparser.add_argument('--start-epoch', default=0, type=int, metavar='N',\n                    help='manual epoch number (useful on restarts)')\nparser.add_argument('-b', '--batch-size', default=256, type=int,\n                    metavar='N',\n                    help='mini-batch size (default: 256), this is the total '\n                         'batch size of all GPUs on the current node when '\n                         'using Data Parallel or Distributed Data Parallel')\nparser.add_argument('--lr', '--learning-rate', default=0.1, type=float,\n                    metavar='LR', help='initial learning rate', dest='lr')\nparser.add_argument('--momentum', default=0.9, type=float, metavar='M',\n                    help='momentum')\nparser.add_argument('--wd', '--weight-decay', default=1e-4, type=float,\n                    metavar='W', help='weight decay (default: 1e-4)',\n                    dest='weight_decay')\nparser.add_argument('-p', '--print-freq', default=10, type=int,\n                    metavar='N', help='print frequency (default: 10)')\nparser.add_argument('--resume', default='', type=str, metavar='PATH',\n                    help='path to latest checkpoint (default: none)')\nparser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',\n                    help='evaluate model on validation set')\nparser.add_argument('--pretrained', dest='pretrained', action='store_true',\n                    help='use pre-trained model')\nparser.add_argument('--world-size', default=-1, type=int,\n                    help='number of nodes for distributed training')\nparser.add_argument('--rank', default=-1, type=int,\n                    help='node rank for distributed training')\nparser.add_argument('--dist-url', default='tcp://224.66.41.62:23456', type=str,\n                    help='url used to set up distributed training')\nparser.add_argument('--dist-backend', default='nccl', type=str,\n                    help='distributed backend')\nparser.add_argument('--seed', default=None, type=int,\n                    help='seed for initializing training. ')\nparser.add_argument('--gpu', default=None, type=int,\n                    help='GPU id to use.')\nparser.add_argument('--no-accel', action='store_true',\n                    help='disables accelerator')\nparser.add_argument('--multiprocessing-distributed', action='store_true',\n                    help='Use multi-processing distributed training to launch '\n                         'N processes per node, which has N GPUs. This is the '\n                         'fastest way to use PyTorch for either single node or '\n                         'multi node data parallel training')\nparser.add_argument('--dummy', action='store_true', help=\"use fake data to benchmark\")\n\nbest_acc1 = 0\n\n\ndef main():\n    args = parser.parse_args()\n\n    if args.seed is not None:\n        random.seed(args.seed)\n        torch.manual_seed(args.seed)\n        cudnn.deterministic = True\n        cudnn.benchmark = False\n        warnings.warn('You have chosen to seed training. '\n                      'This will turn on the CUDNN deterministic setting, '\n                      'which can slow down your training considerably! '\n                      'You may see unexpected behavior when restarting '\n                      'from checkpoints.')\n\n    if args.gpu is not None:\n        warnings.warn('You have chosen a specific GPU. This will completely '\n                      'disable data parallelism.')\n\n    if args.dist_url == \"env://\" and args.world_size == -1:\n        args.world_size = int(os.environ[\"WORLD_SIZE\"])\n\n    args.distributed = args.world_size > 1 or args.multiprocessing_distributed\n\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    if use_accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n\n    print(f\"Using device: {device}\")\n\n    if device.type =='cuda':\n        ngpus_per_node = torch.accelerator.device_count()\n        if ngpus_per_node == 1 and args.dist_backend == \"nccl\":\n            warnings.warn(\"nccl backend >=2.5 requires GPU count>1, see https://github.com/NVIDIA/nccl/issues/103 perhaps use 'gloo'\")\n    else:\n        ngpus_per_node = 1\n\n    if args.multiprocessing_distributed:\n        # Since we have ngpus_per_node processes per node, the total world_size\n        # needs to be adjusted accordingly\n        args.world_size = ngpus_per_node * args.world_size\n        # Use torch.multiprocessing.spawn to launch distributed processes: the\n        # main_worker process function\n        mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))\n    else:\n        # Simply call main_worker function\n        main_worker(args.gpu, ngpus_per_node, args)\n\n\ndef main_worker(gpu, ngpus_per_node, args):\n    global best_acc1\n    args.gpu = gpu\n\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    if use_accel:\n        if args.gpu is not None:\n            torch.accelerator.set_device_index(args.gpu)\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n\n    if args.distributed:\n        if args.dist_url == \"env://\" and args.rank == -1:\n            args.rank = int(os.environ[\"RANK\"])\n        if args.multiprocessing_distributed:\n            # For multiprocessing distributed training, rank needs to be the\n            # global rank among all the processes\n            args.rank = args.rank * ngpus_per_node + gpu\n        dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url,\n                                world_size=args.world_size, rank=args.rank)\n    # create model\n    if args.pretrained:\n        print(\"=> using pre-trained model '{}'\".format(args.arch))\n        model = models.__dict__[args.arch](pretrained=True)\n    else:\n        print(\"=> creating model '{}'\".format(args.arch))\n        model = models.__dict__[args.arch]()\n\n    if not use_accel:\n        print('using CPU, this will be slow')\n    elif args.distributed:\n        # For multiprocessing distributed, DistributedDataParallel constructor\n        # should always set the single device scope, otherwise,\n        # DistributedDataParallel will use all available devices.\n        if device.type == 'cuda':\n            if args.gpu is not None:\n                torch.cuda.set_device(args.gpu)\n                model.cuda(device)\n                # When using a single GPU per process and per\n                # DistributedDataParallel, we need to divide the batch size\n                # ourselves based on the total number of GPUs of the current node.\n                args.batch_size = int(args.batch_size / ngpus_per_node)\n                args.workers = int((args.workers + ngpus_per_node - 1) / ngpus_per_node)\n                model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])\n            else:\n                model.cuda()\n                # DistributedDataParallel will divide and allocate batch_size to all\n                # available GPUs if device_ids are not set\n                model = torch.nn.parallel.DistributedDataParallel(model)\n    elif device.type == 'cuda':\n        # DataParallel will divide and allocate batch_size to all available GPUs\n        if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):\n            model.features = torch.nn.DataParallel(model.features)\n            model.cuda()\n        else:\n            model = torch.nn.DataParallel(model).cuda()\n    else:\n        model.to(device)\n\n\n    # define loss function (criterion), optimizer, and learning rate scheduler\n    criterion = nn.CrossEntropyLoss().to(device)\n\n    optimizer = torch.optim.SGD(model.parameters(), args.lr,\n                                momentum=args.momentum,\n                                weight_decay=args.weight_decay)\n    \n    \"\"\"Sets the learning rate to the initial LR decayed by 10 every 30 epochs\"\"\"\n    scheduler = StepLR(optimizer, step_size=30, gamma=0.1)\n    \n    # optionally resume from a checkpoint\n    if args.resume:\n        if os.path.isfile(args.resume):\n            print(\"=> loading checkpoint '{}'\".format(args.resume))\n            if args.gpu is None:\n                checkpoint = torch.load(args.resume)\n            else:\n                # Map model to be loaded to specified single gpu.\n                loc = f'{device.type}:{args.gpu}'\n                checkpoint = torch.load(args.resume, map_location=loc)\n            args.start_epoch = checkpoint['epoch']\n            best_acc1 = checkpoint['best_acc1']\n            if args.gpu is not None:\n                # best_acc1 may be from a checkpoint from a different GPU\n                best_acc1 = best_acc1.to(args.gpu)\n            model.load_state_dict(checkpoint['state_dict'])\n            optimizer.load_state_dict(checkpoint['optimizer'])\n            scheduler.load_state_dict(checkpoint['scheduler'])\n            print(\"=> loaded checkpoint '{}' (epoch {})\"\n                  .format(args.resume, checkpoint['epoch']))\n        else:\n            print(\"=> no checkpoint found at '{}'\".format(args.resume))\n\n\n    # Data loading code\n    if args.dummy:\n        print(\"=> Dummy data is used!\")\n        train_dataset = datasets.FakeData(1281167, (3, 224, 224), 1000, transforms.ToTensor())\n        val_dataset = datasets.FakeData(50000, (3, 224, 224), 1000, transforms.ToTensor())\n    else:\n        traindir = os.path.join(args.data, 'train')\n        valdir = os.path.join(args.data, 'val')\n        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                     std=[0.229, 0.224, 0.225])\n\n        train_dataset = datasets.ImageFolder(\n            traindir,\n            transforms.Compose([\n                transforms.RandomResizedCrop(224),\n                transforms.RandomHorizontalFlip(),\n                transforms.ToTensor(),\n                normalize,\n            ]))\n\n        val_dataset = datasets.ImageFolder(\n            valdir,\n            transforms.Compose([\n                transforms.Resize(256),\n                transforms.CenterCrop(224),\n                transforms.ToTensor(),\n                normalize,\n            ]))\n\n    if args.distributed:\n        train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)\n        val_sampler = torch.utils.data.distributed.DistributedSampler(val_dataset, shuffle=False, drop_last=True)\n    else:\n        train_sampler = None\n        val_sampler = None\n\n    train_loader = torch.utils.data.DataLoader(\n        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),\n        num_workers=args.workers, pin_memory=True, sampler=train_sampler)\n\n    val_loader = torch.utils.data.DataLoader(\n        val_dataset, batch_size=args.batch_size, shuffle=False,\n        num_workers=args.workers, pin_memory=True, sampler=val_sampler)\n\n    if args.evaluate:\n        validate(val_loader, model, criterion, args)\n        return\n\n    for epoch in range(args.start_epoch, args.epochs):\n        if args.distributed:\n            train_sampler.set_epoch(epoch)\n\n        # train for one epoch\n        train(train_loader, model, criterion, optimizer, epoch, device, args)\n\n        # evaluate on validation set\n        acc1 = validate(val_loader, model, criterion, args)\n        \n        scheduler.step()\n        \n        # remember best acc@1 and save checkpoint\n        is_best = acc1 > best_acc1\n        best_acc1 = max(acc1, best_acc1)\n\n        if not args.multiprocessing_distributed or (args.multiprocessing_distributed\n                and args.rank % ngpus_per_node == 0):\n            save_checkpoint({\n                'epoch': epoch + 1,\n                'arch': args.arch,\n                'state_dict': model.state_dict(),\n                'best_acc1': best_acc1,\n                'optimizer' : optimizer.state_dict(),\n                'scheduler' : scheduler.state_dict()\n            }, is_best)\n\n\ndef train(train_loader, model, criterion, optimizer, epoch, device, args):\n    \n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    batch_time = AverageMeter('Time', use_accel, ':6.3f', Summary.NONE)\n    data_time = AverageMeter('Data', use_accel, ':6.3f', Summary.NONE)\n    losses = AverageMeter('Loss', use_accel, ':.4e', Summary.NONE)\n    top1 = AverageMeter('Acc@1', use_accel, ':6.2f', Summary.NONE)\n    top5 = AverageMeter('Acc@5', use_accel, ':6.2f', Summary.NONE)\n    progress = ProgressMeter(\n        len(train_loader),\n        [batch_time, data_time, losses, top1, top5],\n        prefix=\"Epoch: [{}]\".format(epoch))\n\n    # switch to train mode\n    model.train()\n\n    end = time.time()\n    for i, (images, target) in enumerate(train_loader):\n        # measure data loading time\n        data_time.update(time.time() - end)\n\n        # move data to the same device as model\n        images = images.to(device, non_blocking=True)\n        target = target.to(device, non_blocking=True)\n\n        # compute output\n        output = model(images)\n        loss = criterion(output, target)\n\n        # measure accuracy and record loss\n        acc1, acc5 = accuracy(output, target, topk=(1, 5))\n        losses.update(loss.item(), images.size(0))\n        top1.update(acc1[0], images.size(0))\n        top5.update(acc5[0], images.size(0))\n\n        # compute gradient and do SGD step\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n\n        # measure elapsed time\n        batch_time.update(time.time() - end)\n        end = time.time()\n\n        if i % args.print_freq == 0:\n            progress.display(i + 1)\n\n\ndef validate(val_loader, model, criterion, args):\n\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    def run_validate(loader, base_progress=0):\n\n        if use_accel:\n            device = torch.accelerator.current_accelerator()\n        else:\n            device = torch.device(\"cpu\")\n\n        with torch.no_grad():\n            end = time.time()\n            for i, (images, target) in enumerate(loader):\n                i = base_progress + i\n                if use_accel:\n                    if args.gpu is not None and device.type=='cuda':\n                        torch.accelerator.set_device_index(args.gpu)\n                        images = images.cuda(args.gpu, non_blocking=True)\n                        target = target.cuda(args.gpu, non_blocking=True)\n                    else:\n                        images = images.to(device)\n                        target = target.to(device)\n\n                # compute output\n                output = model(images)\n                loss = criterion(output, target)\n\n                # measure accuracy and record loss\n                acc1, acc5 = accuracy(output, target, topk=(1, 5))\n                losses.update(loss.item(), images.size(0))\n                top1.update(acc1[0], images.size(0))\n                top5.update(acc5[0], images.size(0))\n\n                # measure elapsed time\n                batch_time.update(time.time() - end)\n                end = time.time()\n\n                if i % args.print_freq == 0:\n                    progress.display(i + 1)\n\n    batch_time = AverageMeter('Time', use_accel, ':6.3f', Summary.NONE)\n    losses = AverageMeter('Loss', use_accel, ':.4e', Summary.NONE)\n    top1 = AverageMeter('Acc@1', use_accel, ':6.2f', Summary.AVERAGE)\n    top5 = AverageMeter('Acc@5', use_accel, ':6.2f', Summary.AVERAGE)\n    progress = ProgressMeter(\n        len(val_loader) + (args.distributed and (len(val_loader.sampler) * args.world_size < len(val_loader.dataset))),\n        [batch_time, losses, top1, top5],\n        prefix='Test: ')\n\n    # switch to evaluate mode\n    model.eval()\n\n    run_validate(val_loader)\n    if args.distributed:\n        top1.all_reduce()\n        top5.all_reduce()\n\n    if args.distributed and (len(val_loader.sampler) * args.world_size < len(val_loader.dataset)):\n        aux_val_dataset = Subset(val_loader.dataset,\n                                 range(len(val_loader.sampler) * args.world_size, len(val_loader.dataset)))\n        aux_val_loader = torch.utils.data.DataLoader(\n            aux_val_dataset, batch_size=args.batch_size, shuffle=False,\n            num_workers=args.workers, pin_memory=True)\n        run_validate(aux_val_loader, len(val_loader))\n\n    progress.display_summary()\n\n    return top1.avg\n\n\ndef save_checkpoint(state, is_best, filename='checkpoint.pth.tar'):\n    torch.save(state, filename)\n    if is_best:\n        shutil.copyfile(filename, 'model_best.pth.tar')\n\nclass Summary(Enum):\n    NONE = 0\n    AVERAGE = 1\n    SUM = 2\n    COUNT = 3\n\nclass AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self, name, use_accel, fmt=':f', summary_type=Summary.AVERAGE):\n        self.name = name\n        self.use_accel = use_accel\n        self.fmt = fmt\n        self.summary_type = summary_type\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count\n\n    def all_reduce(self):    \n        if self.use_accel:\n            device = torch.accelerator.current_accelerator()\n        else:\n            device = torch.device(\"cpu\")\n        total = torch.tensor([self.sum, self.count], dtype=torch.float32, device=device)\n        dist.all_reduce(total, dist.ReduceOp.SUM, async_op=False)\n        self.sum, self.count = total.tolist()\n        self.avg = self.sum / self.count\n\n    def __str__(self):\n        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'\n        return fmtstr.format(**self.__dict__)\n    \n    def summary(self):\n        fmtstr = ''\n        if self.summary_type is Summary.NONE:\n            fmtstr = ''\n        elif self.summary_type is Summary.AVERAGE:\n            fmtstr = '{name} {avg:.3f}'\n        elif self.summary_type is Summary.SUM:\n            fmtstr = '{name} {sum:.3f}'\n        elif self.summary_type is Summary.COUNT:\n            fmtstr = '{name} {count:.3f}'\n        else:\n            raise ValueError('invalid summary type %r' % self.summary_type)\n        \n        return fmtstr.format(**self.__dict__)\n\n\nclass ProgressMeter(object):\n    def __init__(self, num_batches, meters, prefix=\"\"):\n        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)\n        self.meters = meters\n        self.prefix = prefix\n\n    def display(self, batch):\n        entries = [self.prefix + self.batch_fmtstr.format(batch)]\n        entries += [str(meter) for meter in self.meters]\n        print('\\t'.join(entries))\n        \n    def display_summary(self):\n        entries = [\" *\"]\n        entries += [meter.summary() for meter in self.meters]\n        print(' '.join(entries))\n\n    def _get_batch_fmtstr(self, num_batches):\n        num_digits = len(str(num_batches // 1))\n        fmt = '{:' + str(num_digits) + 'd}'\n        return '[' + fmt + '/' + fmt.format(num_batches) + ']'\n\ndef accuracy(output, target, topk=(1,)):\n    \"\"\"Computes the accuracy over the k top predictions for the specified values of k\"\"\"\n    with torch.no_grad():\n        maxk = max(topk)\n        batch_size = target.size(0)\n\n        _, pred = output.topk(maxk, 1, True, True)\n        pred = pred.t()\n        correct = pred.eq(target.view(1, -1).expand_as(pred))\n\n        res = []\n        for k in topk:\n            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)\n            res.append(correct_k.mul_(100.0 / batch_size))\n        return res\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "imagenet/requirements.txt",
    "content": "torch>=2.6\ntorchvision\n"
  },
  {
    "path": "language_translation/README.md",
    "content": "# Language Translation\n\nThis example shows how one might use transformers for language translation. In particular, this implementation is loosely based on the [Attention is All You Need paper](https://arxiv.org/abs/1706.03762).\n\n## Requirements\n\nWe will need a tokenizer for our languages. Torchtext does include a tokenizer for English, but unfortunately, we will need more languages then that. We can get these tokenizers via ```spacy```\n\n```bash\npython3 -m spacy download <language>\npython3 -m spacy download en\npython3 -m spacy download de\n```\n\nSpacy supports many languages. For a full accounting of supported languages, please look [here](https://spacy.io/usage/models). This example will default from German to English.\n\nTorchtext is also required:\n```bash\npip install torchtext\n```\n\nJust running these commands will get you started:\n```bash\npip install -r requirements.txt\npython3 -m spacy download <language-you-want>\n```\n\n## Usage\n\nThis example contains a lot of flags that you can set to change the behavior / training of the module. You can see all of them by running:\n\n```bash\npython3 main.py -h\n```\n\nBut in general, all of the settings have \"sensible\" defaults; however, the default translation is to translate from German to English. To *train* the model, you only need to run the following command, but there is also an example for how to use any language you want:\n\n```bash\npython3 main.py\npython3 main.py --src en --tgt fr # For english to french translation\n```\n\nFor model inference, you can use this command:\n\n```bash\npython3 main.py --inference --model_path <path-to-model>\n```\n\nAfter some loading time, this will open an interactive interface where you can type in whatever sentence you are interested in translating.\n"
  },
  {
    "path": "language_translation/main.py",
    "content": "from time import time # Track how long an epoch takes\nimport os # Creating and finding files/directories\nimport logging # Logging tools\nfrom datetime import date # Logging the date for model versioning\n\nimport torch # For ML\nfrom tqdm import tqdm # For fancy progress bars\n\nfrom src.model import Translator # Our model\nfrom src.data import get_data, create_mask, generate_square_subsequent_mask # Loading data and data preprocessing\nfrom argparse import ArgumentParser # For args\n\n# Train on the GPU if possible\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# Function to generate output sequence using greedy algorithm\ndef greedy_decode(model, src, src_mask, max_len, start_symbol, end_symbol):\n\n    # Move to device\n    src = src.to(DEVICE)\n    src_mask = src_mask.to(DEVICE)\n\n    # Encode input\n    memory = model.encode(src, src_mask)\n\n    # Output will be stored here\n    ys = torch.ones(1, 1).fill_(start_symbol).type(torch.long).to(DEVICE)\n\n    # For each element in our translation (which could range up to the maximum translation length)\n    for _ in range(max_len-1):\n\n        # Decode the encoded representation of the input\n        memory = memory.to(DEVICE)\n        tgt_mask = (generate_square_subsequent_mask(ys.size(0), DEVICE).type(torch.bool)).to(DEVICE)\n        out = model.decode(ys, memory, tgt_mask)\n\n        # Reshape\n        out = out.transpose(0, 1)\n\n        # Covert to probabilities and take the max of these probabilities\n        prob = model.ff(out[:, -1])\n        _, next_word = torch.max(prob, dim=1)\n        next_word = next_word.item()\n\n        # Now we have an output which is the vector representation of the translation\n        ys = torch.cat([ys, torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=0)\n        if next_word == end_symbol:\n            break\n\n    return ys\n\n# Opens an user interface where users can translate an arbitrary sentence\ndef inference(opts):\n\n    # Get training data, tokenizer and vocab\n    # objects as well as any special symbols we added to our dataset\n    _, _, src_vocab, tgt_vocab, src_transform, _, special_symbols = get_data(opts)\n\n    src_vocab_size = len(src_vocab)\n    tgt_vocab_size = len(tgt_vocab)\n\n    # Create model\n    model = Translator(\n        num_encoder_layers=opts.enc_layers,\n        num_decoder_layers=opts.dec_layers,\n        embed_size=opts.embed_size,\n        num_heads=opts.attn_heads,\n        src_vocab_size=src_vocab_size,\n        tgt_vocab_size=tgt_vocab_size,\n        dim_feedforward=opts.dim_feedforward,\n        dropout=opts.dropout\n    ).to(DEVICE)\n\n    # Load in weights\n    model.load_state_dict(torch.load(opts.model_path))\n\n    # Set to inference\n    model.eval()\n\n    # Accept input and keep translating until they quit\n    while True:\n        print(\"> \", end=\"\")\n\n        sentence = input()\n\n        # Convert to tokens\n        src = src_transform(sentence).view(-1, 1)\n        num_tokens = src.shape[0]\n\n        src_mask = (torch.zeros(num_tokens, num_tokens)).type(torch.bool)\n\n        # Decode\n        tgt_tokens = greedy_decode(\n            model, src, src_mask, max_len=num_tokens+5, start_symbol=special_symbols[\"<bos>\"], end_symbol=special_symbols[\"<eos>\"]\n        ).flatten()\n\n        # Convert to list of tokens\n        output_as_list = list(tgt_tokens.cpu().numpy())\n\n        # Convert tokens to words\n        output_list_words = tgt_vocab.lookup_tokens(output_as_list)\n\n        # Remove special tokens and convert to string\n        translation = \" \".join(output_list_words).replace(\"<bos>\", \"\").replace(\"<eos>\", \"\")\n\n        print(translation)\n\n# Train the model for 1 epoch\ndef train(model, train_dl, loss_fn, optim, special_symbols, opts):\n\n    # Object for accumulating losses\n    losses = 0\n\n    # Put model into training mode\n    model.train()\n    for src, tgt in tqdm(train_dl, ascii=True):\n\n        src = src.to(DEVICE)\n        tgt = tgt.to(DEVICE)\n\n        # We need to reshape the input slightly to fit into the transformer\n        tgt_input = tgt[:-1, :]\n\n        # Create masks\n        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input, special_symbols[\"<pad>\"], DEVICE)\n\n        # Pass into model, get probability over the vocab out\n        logits = model(src, tgt_input, src_mask, tgt_mask,src_padding_mask, tgt_padding_mask, src_padding_mask)\n\n        # Reset gradients before we try to compute the gradients over the loss\n        optim.zero_grad()\n\n        # Get original shape back\n        tgt_out = tgt[1:, :]\n\n        # Compute loss and gradient over that loss\n        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n        loss.backward()\n\n        # Step weights\n        optim.step()\n\n        # Accumulate a running loss for reporting\n        losses += loss.item()\n\n        if opts.dry_run:\n            break\n\n    # Return the average loss\n    return losses / len(list(train_dl))\n\n# Check the model accuracy on the validation dataset\ndef validate(model, valid_dl, loss_fn, special_symbols):\n    \n    # Object for accumulating losses\n    losses = 0\n\n    # Turn off gradients a moment\n    model.eval()\n\n    for src, tgt in tqdm(valid_dl):\n\n        src = src.to(DEVICE)\n        tgt = tgt.to(DEVICE)\n\n        # We need to reshape the input slightly to fit into the transformer\n        tgt_input = tgt[:-1, :]\n\n        # Create masks\n        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input, special_symbols[\"<pad>\"], DEVICE)\n\n        # Pass into model, get probability over the vocab out\n        logits = model(src, tgt_input, src_mask, tgt_mask,src_padding_mask, tgt_padding_mask, src_padding_mask)\n\n        # Get original shape back, compute loss, accumulate that loss\n        tgt_out = tgt[1:, :]\n        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n        losses += loss.item()\n\n    # Return the average loss\n    return losses / len(list(valid_dl))\n\n# Train the model\ndef main(opts):\n\n    # Set up logging\n    os.makedirs(opts.logging_dir, exist_ok=True)\n    logger = logging.getLogger(__name__)\n    logging.basicConfig(filename=opts.logging_dir + \"log.txt\", level=logging.INFO)\n\n    # This prints it to the screen as well\n    console = logging.StreamHandler()\n    console.setLevel(logging.INFO)\n    logging.getLogger().addHandler(console)\n\n    logging.info(f\"Translation task: {opts.src} -> {opts.tgt}\")\n    logging.info(f\"Using device: {DEVICE}\")\n\n    # Get training data, tokenizer and vocab\n    # objects as well as any special symbols we added to our dataset\n    train_dl, valid_dl, src_vocab, tgt_vocab, _, _, special_symbols = get_data(opts)\n\n    logging.info(\"Loaded data\")\n\n    src_vocab_size = len(src_vocab)\n    tgt_vocab_size = len(tgt_vocab)\n\n    logging.info(f\"{opts.src} vocab size: {src_vocab_size}\")\n    logging.info(f\"{opts.tgt} vocab size: {tgt_vocab_size}\")\n\n    # Create model\n    model = Translator(\n        num_encoder_layers=opts.enc_layers,\n        num_decoder_layers=opts.dec_layers,\n        embed_size=opts.embed_size,\n        num_heads=opts.attn_heads,\n        src_vocab_size=src_vocab_size,\n        tgt_vocab_size=tgt_vocab_size,\n        dim_feedforward=opts.dim_feedforward,\n        dropout=opts.dropout\n    ).to(DEVICE)\n\n    logging.info(\"Model created... starting training!\")\n\n    # Set up our learning tools\n    loss_fn = torch.nn.CrossEntropyLoss(ignore_index=special_symbols[\"<pad>\"])\n\n    # These special values are from the \"Attention is all you need\" paper\n    optim = torch.optim.Adam(model.parameters(), lr=opts.lr, betas=(0.9, 0.98), eps=1e-9)\n\n    best_val_loss = 1e6\n    \n    for idx, epoch in enumerate(range(1, opts.epochs+1)):\n\n        start_time = time()\n        train_loss = train(model, train_dl, loss_fn, optim, special_symbols, opts)\n        epoch_time = time() - start_time\n        val_loss   = validate(model, valid_dl, loss_fn, special_symbols)\n\n        # Once training is done, we want to save out the model\n        if val_loss < best_val_loss:\n            best_val_loss = val_loss\n            logging.info(\"New best model, saving...\")\n            torch.save(model.state_dict(), opts.logging_dir + \"best.pt\")\n\n        torch.save(model.state_dict(), opts.logging_dir + \"last.pt\")\n\n        logger.info(f\"Epoch: {epoch}\\n\\tTrain loss: {train_loss:.3f}\\n\\tVal loss: {val_loss:.3f}\\n\\tEpoch time = {epoch_time:.1f} seconds\\n\\tETA = {epoch_time*(opts.epochs-idx-1):.1f} seconds\")\n\nif __name__ == \"__main__\":\n\n    parser = ArgumentParser(\n        prog=\"Machine Translator training and inference\",\n    )\n\n    # Inference mode\n    parser.add_argument(\"--inference\", action=\"store_true\",\n                        help=\"Set true to run inference\")\n    parser.add_argument(\"--model_path\", type=str,\n                        help=\"Path to the model to run inference on\")\n\n    # Translation settings\n    parser.add_argument(\"--src\", type=str, default=\"de\",\n                        help=\"Source language (translating FROM this language)\")\n    parser.add_argument(\"--tgt\", type=str, default=\"en\",\n                        help=\"Target language (translating TO this language)\")\n\n    # Training settings \n    parser.add_argument(\"-e\", \"--epochs\", type=int, default=30,\n                        help=\"Epochs\")\n    parser.add_argument(\"--lr\", type=float, default=1e-4,\n                        help=\"Default learning rate\")\n    parser.add_argument(\"--batch\", type=int, default=128,\n                        help=\"Batch size\")\n    parser.add_argument(\"--backend\", type=str, default=\"cpu\",\n                        help=\"Batch size\")\n    \n    # Transformer settings\n    parser.add_argument(\"--attn_heads\", type=int, default=8,\n                        help=\"Number of attention heads\")\n    parser.add_argument(\"--enc_layers\", type=int, default=5,\n                        help=\"Number of encoder layers\")\n    parser.add_argument(\"--dec_layers\", type=int, default=5,\n                        help=\"Number of decoder layers\")\n    parser.add_argument(\"--embed_size\", type=int, default=512,\n                        help=\"Size of the language embedding\")\n    parser.add_argument(\"--dim_feedforward\", type=int, default=512,\n                        help=\"Feedforward dimensionality\")\n    parser.add_argument(\"--dropout\", type=float, default=0.1,\n                        help=\"Transformer dropout\")\n\n    # Logging settings\n    parser.add_argument(\"--logging_dir\", type=str, default=\"./\" + str(date.today()) + \"/\",\n                        help=\"Where the output of this program should be placed\")\n\n    # Just for continuous integration\n    parser.add_argument(\"--dry_run\", action=\"store_true\")\n\n    args = parser.parse_args()\n\n    DEVICE = torch.device(\"cuda\" if args.backend == \"gpu\" and torch.cuda.is_available() else \"cpu\")\n\n    if args.inference:\n        inference(args)\n    else:\n        main(args)\n"
  },
  {
    "path": "language_translation/requirements.txt",
    "content": "torch\ntorchtext\ntorchdata==0.9.0\nspacy\nportalocker\n"
  },
  {
    "path": "language_translation/src/data.py",
    "content": "import torch\nfrom torch.nn.utils.rnn import pad_sequence\nfrom torch.utils.data import DataLoader\nfrom torchtext.data.utils import get_tokenizer\nfrom torchtext.vocab import build_vocab_from_iterator\nfrom torchtext.datasets import Multi30k, multi30k\n\n# Turns an iterable into a generator\ndef _yield_tokens(iterable_data, tokenizer, src):\n\n    # Iterable data stores the samples as (src, tgt) so this will help us select just one language or the other\n    index = 0 if src else 1\n\n    for data in iterable_data:\n        yield tokenizer(data[index])\n\n# Get data, tokenizer, text transform, vocab objs, etc. Everything we\n# need to start training the model\ndef get_data(opts):\n\n    src_lang = opts.src\n    tgt_lang = opts.tgt\n\n    multi30k.URL[\"train\"] = \"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz\"\n    multi30k.URL[\"valid\"] = \"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz\"\n\n    # Define a token \"unkown\", \"padding\", \"beginning of sentence\", and \"end of sentence\"\n    special_symbols = {\n        \"<unk>\":0,\n        \"<pad>\":1,\n        \"<bos>\":2,\n        \"<eos>\":3\n    }\n\n    # Get training examples from torchtext (the multi30k dataset)\n    train_iterator = Multi30k(split=\"train\", language_pair=(src_lang, tgt_lang))\n    valid_iterator = Multi30k(split=\"valid\", language_pair=(src_lang, tgt_lang))\n\n    # Grab a tokenizer for these languages\n    src_tokenizer = get_tokenizer(\"spacy\", src_lang)\n    tgt_tokenizer = get_tokenizer(\"spacy\", tgt_lang)\n\n    # Build a vocabulary object for these languages\n    src_vocab = build_vocab_from_iterator(\n        _yield_tokens(train_iterator, src_tokenizer, True),\n        min_freq=1,\n        specials=list(special_symbols.keys()),\n        special_first=True\n    )\n\n    tgt_vocab = build_vocab_from_iterator(\n        _yield_tokens(train_iterator, tgt_tokenizer, False),\n        min_freq=1,\n        specials=list(special_symbols.keys()),\n        special_first=True\n    )\n\n    src_vocab.set_default_index(special_symbols[\"<unk>\"])\n    tgt_vocab.set_default_index(special_symbols[\"<unk>\"])\n\n    # Helper function to sequentially apply transformations\n    def _seq_transform(*transforms):\n        def func(txt_input):\n            for transform in transforms:\n                txt_input = transform(txt_input)\n            return txt_input\n        return func\n\n    # Function to add BOS/EOS and create tensor for input sequence indices\n    def _tensor_transform(token_ids):\n        return torch.cat(\n            (torch.tensor([special_symbols[\"<bos>\"]]),\n             torch.tensor(token_ids),\n             torch.tensor([special_symbols[\"<eos>\"]]))\n        )\n\n    src_lang_transform = _seq_transform(src_tokenizer, src_vocab, _tensor_transform)\n    tgt_lang_transform = _seq_transform(tgt_tokenizer, tgt_vocab, _tensor_transform)\n\n    # Now we want to convert the torchtext data pipeline to a dataloader. We\n    # will need to collate batches\n    def _collate_fn(batch):\n        src_batch, tgt_batch = [], []\n        for src_sample, tgt_sample in batch:\n            src_batch.append(src_lang_transform(src_sample.rstrip(\"\\n\")))\n            tgt_batch.append(tgt_lang_transform(tgt_sample.rstrip(\"\\n\")))\n\n        src_batch = pad_sequence(src_batch, padding_value=special_symbols[\"<pad>\"])\n        tgt_batch = pad_sequence(tgt_batch, padding_value=special_symbols[\"<pad>\"])\n        return src_batch, tgt_batch\n\n    # Create the dataloader\n    train_dataloader = DataLoader(train_iterator, batch_size=opts.batch, collate_fn=_collate_fn)\n    valid_dataloader = DataLoader(valid_iterator, batch_size=opts.batch, collate_fn=_collate_fn)\n\n    return train_dataloader, valid_dataloader, src_vocab, tgt_vocab, src_lang_transform, tgt_lang_transform, special_symbols\n\ndef generate_square_subsequent_mask(size, device):\n    mask = (torch.triu(torch.ones((size, size), device=device)) == 1).transpose(0, 1)\n    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n    return mask\n\n# Create masks for input into model\ndef create_mask(src, tgt, pad_idx, device):\n\n    # Get sequence length\n    src_seq_len = src.shape[0]\n    tgt_seq_len = tgt.shape[0]\n\n    # Generate the mask\n    tgt_mask = generate_square_subsequent_mask(tgt_seq_len, device)\n    src_mask = torch.zeros((src_seq_len, src_seq_len),device=device).type(torch.bool)\n\n    # Overlay the mask over the original input\n    src_padding_mask = (src == pad_idx).transpose(0, 1)\n    tgt_padding_mask = (tgt == pad_idx).transpose(0, 1)\n    return src_mask, tgt_mask, src_padding_mask, tgt_padding_mask\n\n# A small test to make sure our data loasd in correctly\nif __name__==\"__main__\":\n\n    class Opts:\n        def __init__(self):\n            self.src = \"en\",\n            self.tgt = \"de\"\n            self.batch = 128\n\n    opts = Opts()\n    \n    train_dl, valid_dl, src_vocab, tgt_vocab, src_lang_transform, tgt_lang_transform, special_symbols = get_data(opts)\n\n    print(f\"{opts.src} vocab size: {len(src_vocab)}\")\n    print(f\"{opts.src} vocab size: {len(tgt_vocab)}\")\n\n"
  },
  {
    "path": "language_translation/src/model.py",
    "content": "import math\n\nimport torch\nfrom torch.nn import functional as F\nfrom torch import nn\n\nclass PositionalEncoding(nn.Module):\n    def __init__(\n        self,\n        emb_size,\n        dropout,\n        maxlen=5000\n    ):\n        super(PositionalEncoding, self).__init__()\n        den = torch.exp(- torch.arange(0, emb_size, 2)* math.log(10000) / emb_size)\n        pos = torch.arange(0, maxlen).reshape(maxlen, 1)\n        pos_embedding = torch.zeros((maxlen, emb_size))\n        pos_embedding[:, 0::2] = torch.sin(pos * den)\n        pos_embedding[:, 1::2] = torch.cos(pos * den)\n        pos_embedding = pos_embedding.unsqueeze(-2)\n\n        self.dropout = nn.Dropout(dropout)\n        self.register_buffer('pos_embedding', pos_embedding)\n\n    def forward(self, token_embedding):\n        return self.dropout(token_embedding + self.pos_embedding[:token_embedding.size(0), :])\n\nclass Translator(nn.Module):\n    def __init__(\n            self,\n            num_encoder_layers,\n            num_decoder_layers,\n            embed_size,\n            num_heads,\n            src_vocab_size,\n            tgt_vocab_size,\n            dim_feedforward,\n            dropout\n        ):\n        super(Translator, self).__init__()\n\n        # Output of embedding must be equal (embed_size)\n        self.src_embedding = nn.Embedding(src_vocab_size, embed_size)\n        self.tgt_embedding = nn.Embedding(tgt_vocab_size, embed_size)\n\n        self.pos_enc = PositionalEncoding(embed_size, dropout)\n\n        self.transformer = nn.Transformer(\n            d_model=embed_size,\n            nhead=num_heads,\n            num_encoder_layers=num_encoder_layers,\n            num_decoder_layers=num_decoder_layers,\n            dim_feedforward=dim_feedforward,\n            dropout=dropout\n        )\n\n        self.ff = nn.Linear(embed_size, tgt_vocab_size)\n\n        self._init_weights()\n\n    def _init_weights(self):\n        for p in self.parameters():\n            if p.dim() > 1:\n                nn.init.xavier_uniform_(p)\n\n    def forward(self, src, trg, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, memory_key_padding_mask):\n\n        src_emb = self.pos_enc(self.src_embedding(src))\n        tgt_emb = self.pos_enc(self.tgt_embedding(trg))\n\n        outs = self.transformer(\n            src_emb,\n            tgt_emb,\n            src_mask,\n            tgt_mask,\n            None,\n            src_padding_mask,\n            tgt_padding_mask,\n            memory_key_padding_mask\n        )\n\n        return self.ff(outs)\n\n    def encode(self, src, src_mask):\n\n        embed = self.src_embedding(src)\n\n        pos_enc = self.pos_enc(embed)\n\n        return self.transformer.encoder(pos_enc, src_mask)\n\n    def decode(self, tgt, memory, tgt_mask):\n        \n        embed = self.tgt_embedding(tgt)\n\n        pos_enc = self.pos_enc(embed)\n\n        return self.transformer.decoder(pos_enc, memory, tgt_mask)\n"
  },
  {
    "path": "legacy/snli/README.md",
    "content": "# PyTorch-based NLI Training with SNLI\n\n## 📝 Overview\n\nThis repository contains Python scripts to train a Natural Language Inference (NLI) model, specifically the `SNLIClassifier`, using the Stanford Natural Language Inference (SNLI) corpus. The trained model predicts textual entailment, identifying if a statement is entailed, contradicted, or neither by another statement.\n\n## ⚙️ Dependencies\n\nInstall the necessary Python libraries with:\n\n```bash\npip install -r requirements.txt\n```\n\nThe `requirements.txt` file includes:\n\n```\ntorch\ntorchtext\nspacy\n```\n\n## 💻 Usage\n\nStart the training process with:\n\n```bash\npython train.py --lower --word-vectors [PATH_TO_WORD_VECTORS] --vector-cache [PATH_TO_VECTOR_CACHE] --epochs [NUMBER_OF_EPOCHS] --batch-size [BATCH_SIZE] --save-path [PATH_TO_SAVE_MODEL] --gpu [GPU_NUMBER]\n```\n\n## 🏋️‍♀️ Training\n\nThe script trains the model on mini-batches of data across a specified number of epochs. It saves the best-performing model on the validation set as a `.pt` file in the specified directory.\n\n## 📚 Scripts\n\n- `model.py`: Defines the `SNLIClassifier` model and auxiliary classes.\n- `util.py`: Contains utility functions for directory creation and command-line argument parsing.\n\n## 📣 Note\n\nEnsure the `model.py` and `util.py` scripts are available in your working directory."
  },
  {
    "path": "legacy/snli/model.py",
    "content": "import torch\nimport torch.nn as nn\n\n\nclass Bottle(nn.Module):\n\n    def forward(self, input):\n        if len(input.size()) <= 2:\n            return super(Bottle, self).forward(input)\n        size = input.size()[:2]\n        out = super(Bottle, self).forward(input.view(size[0]*size[1], -1))\n        return out.view(size[0], size[1], -1)\n\n\nclass Linear(Bottle, nn.Linear):\n    pass\n\n\nclass Encoder(nn.Module):\n\n    def __init__(self, config):\n        super(Encoder, self).__init__()\n        self.config = config\n        input_size = config.d_proj if config.projection else config.d_embed\n        dropout = 0 if config.n_layers == 1 else config.dp_ratio\n        self.rnn = nn.LSTM(input_size=input_size, hidden_size=config.d_hidden,\n                        num_layers=config.n_layers, dropout=dropout,\n                        bidirectional=config.birnn)\n\n    def forward(self, inputs):\n        batch_size = inputs.size()[1]\n        state_shape = self.config.n_cells, batch_size, self.config.d_hidden\n        h0 = c0 = inputs.new_zeros(state_shape)\n        outputs, (ht, ct) = self.rnn(inputs, (h0, c0))\n        return ht[-1] if not self.config.birnn else ht[-2:].transpose(0, 1).contiguous().view(batch_size, -1)\n\n\nclass SNLIClassifier(nn.Module):\n\n    def __init__(self, config):\n        super(SNLIClassifier, self).__init__()\n        self.config = config\n        self.embed = nn.Embedding(config.n_embed, config.d_embed)\n        self.projection = Linear(config.d_embed, config.d_proj)\n        self.encoder = Encoder(config)\n        self.dropout = nn.Dropout(p=config.dp_ratio)\n        self.relu = nn.ReLU()\n        seq_in_size = 2*config.d_hidden\n        if self.config.birnn:\n            seq_in_size *= 2\n        lin_config = [seq_in_size]*2\n        self.out = nn.Sequential(\n            Linear(*lin_config),\n            self.relu,\n            self.dropout,\n            Linear(*lin_config),\n            self.relu,\n            self.dropout,\n            Linear(*lin_config),\n            self.relu,\n            self.dropout,\n            Linear(seq_in_size, config.d_out))\n\n    def forward(self, batch):\n        prem_embed = self.embed(batch.premise)\n        hypo_embed = self.embed(batch.hypothesis)\n        if self.config.fix_emb:\n            prem_embed = prem_embed.detach()\n            hypo_embed = hypo_embed.detach()\n        if self.config.projection:\n            prem_embed = self.relu(self.projection(prem_embed))\n            hypo_embed = self.relu(self.projection(hypo_embed))\n        premise = self.encoder(prem_embed)\n        hypothesis = self.encoder(hypo_embed)\n        scores = self.out(torch.cat([premise, hypothesis], 1))\n        return scores\n"
  },
  {
    "path": "legacy/snli/requirements.txt",
    "content": "torch\ntorchtext\nspacy\n"
  },
  {
    "path": "legacy/snli/train.py",
    "content": "import os\nimport time\nimport glob\n\nimport torch\nimport torch.optim as O\nimport torch.nn as nn\n\nfrom torchtext.legacy import data\nfrom torchtext.legacy import datasets\n\nfrom model import SNLIClassifier\nfrom util import get_args, makedirs\n\n\nargs = get_args()\nif torch.cuda.is_available():\n    torch.cuda.set_device(args.gpu)\n    device = torch.device('cuda:{}'.format(args.gpu))\nelif torch.backends.mps.is_available():\n    device = torch.device('mps')\nelse:\n    device = torch.device('cpu')\n\ninputs = data.Field(lower=args.lower, tokenize='spacy')\nanswers = data.Field(sequential=False)\n\ntrain, dev, test = datasets.SNLI.splits(inputs, answers)\n\ninputs.build_vocab(train, dev, test)\nif args.word_vectors:\n    if os.path.isfile(args.vector_cache):\n        inputs.vocab.vectors = torch.load(args.vector_cache)\n    else:\n        inputs.vocab.load_vectors(args.word_vectors)\n        makedirs(os.path.dirname(args.vector_cache))\n        torch.save(inputs.vocab.vectors, args.vector_cache)\nanswers.build_vocab(train)\n\ntrain_iter, dev_iter, test_iter = data.BucketIterator.splits(\n            (train, dev, test), batch_size=args.batch_size, device=device)\n\nconfig = args\nconfig.n_embed = len(inputs.vocab)\nconfig.d_out = len(answers.vocab)\nconfig.n_cells = config.n_layers\n\n# double the number of cells for bidirectional networks\nif config.birnn:\n    config.n_cells *= 2\n\nif args.resume_snapshot:\n    model = torch.load(args.resume_snapshot, map_location=device)\nelse:\n    model = SNLIClassifier(config)\n    if args.word_vectors:\n        model.embed.weight.data.copy_(inputs.vocab.vectors)\n        model.to(device)\n\ncriterion = nn.CrossEntropyLoss()\nopt = O.Adam(model.parameters(), lr=args.lr)\n\niterations = 0\nstart = time.time()\nbest_dev_acc = -1\nheader = '  Time Epoch Iteration Progress    (%Epoch)   Loss   Dev/Loss     Accuracy  Dev/Accuracy'\ndev_log_template = ' '.join('{:>6.0f},{:>5.0f},{:>9.0f},{:>5.0f}/{:<5.0f} {:>7.0f}%,{:>8.6f},{:8.6f},{:12.4f},{:12.4f}'.split(','))\nlog_template =     ' '.join('{:>6.0f},{:>5.0f},{:>9.0f},{:>5.0f}/{:<5.0f} {:>7.0f}%,{:>8.6f},{},{:12.4f},{}'.split(','))\nmakedirs(args.save_path)\nprint(header)\n\nfor epoch in range(args.epochs):\n    train_iter.init_epoch()\n    n_correct, n_total = 0, 0\n    for batch_idx, batch in enumerate(train_iter):\n\n        # switch model to training mode, clear gradient accumulators\n        model.train(); opt.zero_grad()\n\n        iterations += 1\n\n        # forward pass\n        answer = model(batch)\n\n        # calculate accuracy of predictions in the current batch\n        n_correct += (torch.max(answer, 1)[1].view(batch.label.size()) == batch.label).sum().item()\n        n_total += batch.batch_size\n        train_acc = 100. * n_correct/n_total\n\n        # calculate loss of the network output with respect to training labels\n        loss = criterion(answer, batch.label)\n\n        # backpropagate and update optimizer learning rate\n        loss.backward(); opt.step()\n\n        # checkpoint model periodically\n        if iterations % args.save_every == 0:\n            snapshot_prefix = os.path.join(args.save_path, 'snapshot')\n            snapshot_path = snapshot_prefix + '_acc_{:.4f}_loss_{:.6f}_iter_{}_model.pt'.format(train_acc, loss.item(), iterations)\n            torch.save(model, snapshot_path)\n            for f in glob.glob(snapshot_prefix + '*'):\n                if f != snapshot_path:\n                    os.remove(f)\n\n        # evaluate performance on validation set periodically\n        if iterations % args.dev_every == 0:\n\n            # switch model to evaluation mode\n            model.eval(); dev_iter.init_epoch()\n\n            # calculate accuracy on validation set\n            n_dev_correct, dev_loss = 0, 0\n            with torch.no_grad():\n                for dev_batch_idx, dev_batch in enumerate(dev_iter):\n                     answer = model(dev_batch)\n                     n_dev_correct += (torch.max(answer, 1)[1].view(dev_batch.label.size()) == dev_batch.label).sum().item()\n                     dev_loss = criterion(answer, dev_batch.label)\n            dev_acc = 100. * n_dev_correct / len(dev)\n\n            print(dev_log_template.format(time.time()-start,\n                epoch, iterations, 1+batch_idx, len(train_iter),\n                100. * (1+batch_idx) / len(train_iter), loss.item(), dev_loss.item(), train_acc, dev_acc))\n\n            # update best validation set accuracy\n            if dev_acc > best_dev_acc:\n\n                # found a model with better validation set accuracy\n\n                best_dev_acc = dev_acc\n                snapshot_prefix = os.path.join(args.save_path, 'best_snapshot')\n                snapshot_path = snapshot_prefix + '_devacc_{}_devloss_{}__iter_{}_model.pt'.format(dev_acc, dev_loss.item(), iterations)\n\n                # save model, delete previous 'best_snapshot' files\n                torch.save(model, snapshot_path)\n                for f in glob.glob(snapshot_prefix + '*'):\n                    if f != snapshot_path:\n                        os.remove(f)\n\n        elif iterations % args.log_every == 0:\n\n            # print progress message\n            print(log_template.format(time.time()-start,\n                epoch, iterations, 1+batch_idx, len(train_iter),\n                100. * (1+batch_idx) / len(train_iter), loss.item(), ' '*8, n_correct/n_total*100, ' '*12))\n        if args.dry_run:\n            break\n"
  },
  {
    "path": "legacy/snli/util.py",
    "content": "import os\nfrom argparse import ArgumentParser\n\ndef makedirs(name):\n    \"\"\"helper function for python 2 and 3 to call os.makedirs()\n       avoiding an error if the directory to be created already exists\"\"\"\n\n    import os, errno\n\n    try:\n        os.makedirs(name)\n    except OSError as ex:\n        if ex.errno == errno.EEXIST and os.path.isdir(name):\n            # ignore existing directory\n            pass\n        else:\n            # a different error happened\n            raise\n\n\ndef get_args():\n    parser = ArgumentParser(description='PyTorch/torchtext SNLI example')\n    parser.add_argument('--epochs', type=int, default=50,\n                        help='the number of total epochs to run.')\n    parser.add_argument('--batch_size', type=int, default=128,\n                        help='batch size. (default: 128)')\n    parser.add_argument('--d_embed', type=int, default=100,\n                        help='the size of each embedding vector.')\n    parser.add_argument('--d_proj', type=int, default=300,\n                        help='the size of each projection layer.')\n    parser.add_argument('--d_hidden', type=int, default=300,\n                        help='the number of features in the hidden state.')\n    parser.add_argument('--n_layers', type=int, default=1,\n                        help='the number of recurrent layers. (default: 50)')\n    parser.add_argument('--log_every', type=int, default=50,\n                        help='iteration period to output log.')\n    parser.add_argument('--lr',type=float, default=.001,\n                        help='initial learning rate.')\n    parser.add_argument('--dev_every', type=int, default=1000,\n                        help='log period of validation results.')\n    parser.add_argument('--save_every', type=int, default=1000,\n                        help='model checkpoint period.')\n    parser.add_argument('--dp_ratio', type=int, default=0.2,\n                        help='probability of an element to be zeroed.')\n    parser.add_argument('--no-bidirectional', action='store_false', dest='birnn',\n                        help='disable bidirectional LSTM.')\n    parser.add_argument('--preserve-case', action='store_false', dest='lower',\n                        help='case-sensitivity.')\n    parser.add_argument('--no-projection', action='store_false', dest='projection',\n                        help='disable projection layer.')\n    parser.add_argument('--train_embed', action='store_false', dest='fix_emb',\n                        help='enable embedding word training.')\n    parser.add_argument('--gpu', type=int, default=0,\n                        help='gpu id to use. (default: 0)')\n    parser.add_argument('--save_path', type=str, default='results',\n                        help='save path of results.')\n    parser.add_argument('--vector_cache', type=str, default=os.path.join(os.getcwd(), '.vector_cache/input_vectors.pt'),\n                        help='name of vector cache directory, which saved input word-vectors.')\n    parser.add_argument('--word_vectors', type=str, default='glove.6B.100d',\n                        help='one of or a list containing instantiations of the GloVe, CharNGram, or Vectors classes.'\n                        'Alternatively, one of or a list of available pretrained vectors: '\n                        'charngram.100d fasttext.en.300d fasttext.simple.300d'\n                        'glove.42B.300d glove.840B.300d glove.twitter.27B.25d'\n                        'glove.twitter.27B.50d glove.twitter.27B.100d glove.twitter.27B.200d'\n                        'glove.6B.50d glove.6B.100d glove.6B.200d glove.6B.300d')\n    parser.add_argument('--resume_snapshot', type=str, default='',\n                        help='model snapshot to resume.')\n    parser.add_argument('--dry-run', action='store_true',\n                        help='run only a few iterations')\n    args = parser.parse_args()\n    return args\n"
  },
  {
    "path": "mnist/README.md",
    "content": "# Basic MNIST Example\n\n```bash\npip install -r requirements.txt\npython main.py\n# CUDA_VISIBLE_DEVICES=2 python main.py  # to specify GPU id to ex. 2\n```\n"
  },
  {
    "path": "mnist/main.py",
    "content": "import argparse\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torchvision import datasets, transforms\nfrom torch.optim.lr_scheduler import StepLR\n\n\nclass Net(nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n        self.dropout1 = nn.Dropout(0.25)\n        self.dropout2 = nn.Dropout(0.5)\n        self.fc1 = nn.Linear(9216, 128)\n        self.fc2 = nn.Linear(128, 10)\n\n    def forward(self, x):\n        x = self.conv1(x)\n        x = F.relu(x)\n        x = self.conv2(x)\n        x = F.relu(x)\n        x = F.max_pool2d(x, 2)\n        x = self.dropout1(x)\n        x = torch.flatten(x, 1)\n        x = self.fc1(x)\n        x = F.relu(x)\n        x = self.dropout2(x)\n        x = self.fc2(x)\n        output = F.log_softmax(x, dim=1)\n        return output\n\n\ndef train(args, model, device, train_loader, optimizer, epoch):\n    model.train()\n    for batch_idx, (data, target) in enumerate(train_loader):\n        data, target = data.to(device), target.to(device)\n        optimizer.zero_grad()\n        output = model(data)\n        loss = F.nll_loss(output, target)\n        loss.backward()\n        optimizer.step()\n        if batch_idx % args.log_interval == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, batch_idx * len(data), len(train_loader.dataset),\n                100. * batch_idx / len(train_loader), loss.item()))\n            if args.dry_run:\n                break\n\n\ndef test(model, device, test_loader):\n    model.eval()\n    test_loss = 0\n    correct = 0\n    with torch.no_grad():\n        for data, target in test_loader:\n            data, target = data.to(device), target.to(device)\n            output = model(data)\n            test_loss += F.nll_loss(output, target, reduction='sum').item()  # sum up batch loss\n            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability\n            correct += pred.eq(target.view_as(pred)).sum().item()\n\n    test_loss /= len(test_loader.dataset)\n\n    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n        test_loss, correct, len(test_loader.dataset),\n        100. * correct / len(test_loader.dataset)))\n\n\ndef main():\n    # Training settings\n    parser = argparse.ArgumentParser(description='PyTorch MNIST Example')\n    parser.add_argument('--batch-size', type=int, default=64, metavar='N',\n                        help='input batch size for training (default: 64)')\n    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n                        help='input batch size for testing (default: 1000)')\n    parser.add_argument('--epochs', type=int, default=14, metavar='N',\n                        help='number of epochs to train (default: 14)')\n    parser.add_argument('--lr', type=float, default=1.0, metavar='LR',\n                        help='learning rate (default: 1.0)')\n    parser.add_argument('--gamma', type=float, default=0.7, metavar='M',\n                        help='Learning rate step gamma (default: 0.7)')\n    parser.add_argument('--no-accel', action='store_true',\n                        help='disables accelerator')\n    parser.add_argument('--dry-run', action='store_true',\n                        help='quickly check a single pass')\n    parser.add_argument('--seed', type=int, default=1, metavar='S',\n                        help='random seed (default: 1)')\n    parser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                        help='how many batches to wait before logging training status')\n    parser.add_argument('--save-model', action='store_true', \n                        help='For Saving the current Model')\n    args = parser.parse_args()\n\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    torch.manual_seed(args.seed)\n\n    if use_accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n\n    train_kwargs = {'batch_size': args.batch_size}\n    test_kwargs = {'batch_size': args.test_batch_size}\n    if use_accel:\n        accel_kwargs = {'num_workers': 1,\n                        'persistent_workers': True,\n                       'pin_memory': True,\n                       'shuffle': True}\n        train_kwargs.update(accel_kwargs)\n        test_kwargs.update(accel_kwargs)\n\n    transform=transforms.Compose([\n        transforms.ToTensor(),\n        transforms.Normalize((0.1307,), (0.3081,))\n        ])\n    dataset1 = datasets.MNIST('../data', train=True, download=True,\n                       transform=transform)\n    dataset2 = datasets.MNIST('../data', train=False,\n                       transform=transform)\n    train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)\n    test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)\n\n    model = Net().to(device)\n    optimizer = optim.Adadelta(model.parameters(), lr=args.lr)\n\n    scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)\n    for epoch in range(1, args.epochs + 1):\n        train(args, model, device, train_loader, optimizer, epoch)\n        test(model, device, test_loader)\n        scheduler.step()\n\n    if args.save_model:\n        torch.save(model.state_dict(), \"mnist_cnn.pt\")\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "mnist/requirements.txt",
    "content": "torch\ntorchvision\n"
  },
  {
    "path": "mnist_forward_forward/README.md",
    "content": "# Basic Forward-Forward Example\n\nThis example implements the paper [The Forward-Forward Algorithm: Some Preliminary Investigations](https://arxiv.org/abs/2212.13345) by Geoffrey Hinton.\n\nthe aim of this paper is to introduce a new learning procedure for neural networks. the forward and backward passes of backpropagation by two forward passes.\n\n```bash\npip install -r requirements.txt\npython main.py\n```\n\nThe main.py script accepts the following arguments:\n\n```bash\noptional arguments:\n  -h, --help            show this help message and exit\n  --epochs EPOCHS       number of epochs to train (default: 1000)\n  --lr LR               learning rate (default: 0.03)\n  --no_accel            disables accelerator\n  --seed SEED           random seed (default: 1)\n  --save_model          For saving the current Model\n  --train_size TRAIN_SIZE\n                        size of training set\n  --threshold THRESHOLD\n                        threshold for training\n  --test_size TEST_SIZE\n                        size of test set\n  --save-model          For Saving the current Model\n  --log-interval LOG_INTERVAL\n                        logging training status interval\n```\n"
  },
  {
    "path": "mnist_forward_forward/main.py",
    "content": "# This code is based on the implementation of Mohammad Pezeshki available at\n# https://github.com/mohammadpz/pytorch_forward_forward and licensed under the MIT License.\n# Modifications/Improvements to the original code have been made by Vivek V Patel.\n\nimport argparse\nimport torch\nimport torch.nn as nn\nfrom torchvision.datasets import MNIST\nfrom torchvision.transforms import Compose, ToTensor, Normalize, Lambda\nfrom torch.utils.data import DataLoader\nfrom torch.optim import Adam\n\n\ndef get_y_neg(y):\n    y_neg = y.clone()\n    for idx, y_samp in enumerate(y):\n        allowed_indices = list(range(10))\n        allowed_indices.remove(y_samp.item())\n        y_neg[idx] = torch.tensor(allowed_indices)[\n            torch.randint(len(allowed_indices), size=(1,))\n        ].item()\n    return y_neg.to(device)\n\n\ndef overlay_y_on_x(x, y, classes=10):\n    x_ = x.clone()\n    x_[:, :classes] *= 0.0\n    x_[range(x.shape[0]), y] = x.max()\n    return x_\n\n\nclass Net(torch.nn.Module):\n    def __init__(self, dims):\n\n        super().__init__()\n        self.layers = []\n        for d in range(len(dims) - 1):\n            self.layers = self.layers + [Layer(dims[d], dims[d + 1]).to(device)]\n\n    def predict(self, x):\n        goodness_per_label = []\n        for label in range(10):\n            h = overlay_y_on_x(x, label)\n            goodness = []\n            for layer in self.layers:\n                h = layer(h)\n                goodness = goodness + [h.pow(2).mean(1)]\n            goodness_per_label += [sum(goodness).unsqueeze(1)]\n        goodness_per_label = torch.cat(goodness_per_label, 1)\n        return goodness_per_label.argmax(1)\n\n    def train(self, x_pos, x_neg):\n        h_pos, h_neg = x_pos, x_neg\n        for i, layer in enumerate(self.layers):\n            print(\"training layer: \", i)\n            h_pos, h_neg = layer.train(h_pos, h_neg)\n\n\nclass Layer(nn.Linear):\n    def __init__(self, in_features, out_features, bias=True, device=None, dtype=None):\n        super().__init__(in_features, out_features, bias, device, dtype)\n        self.relu = torch.nn.ReLU()\n        self.opt = Adam(self.parameters(), lr=args.lr)\n        self.threshold = args.threshold\n        self.num_epochs = args.epochs\n\n    def forward(self, x):\n        x_direction = x / (x.norm(2, 1, keepdim=True) + 1e-4)\n        return self.relu(torch.mm(x_direction, self.weight.T) + self.bias.unsqueeze(0))\n\n    def train(self, x_pos, x_neg):\n        for i in range(self.num_epochs):\n            g_pos = self.forward(x_pos).pow(2).mean(1)\n            g_neg = self.forward(x_neg).pow(2).mean(1)\n            loss = torch.log1p(\n                torch.exp(\n                    torch.cat([-g_pos + self.threshold, g_neg - self.threshold])\n                )\n            ).mean()\n            self.opt.zero_grad()\n            loss.backward()\n            self.opt.step()\n            if i % args.log_interval == 0:\n                print(\"Loss: \", loss.item())\n        return self.forward(x_pos).detach(), self.forward(x_neg).detach()\n\n\nif __name__ == \"__main__\":\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\n        \"--epochs\",\n        type=int,\n        default=1000,\n        metavar=\"N\",\n        help=\"number of epochs to train (default: 1000)\",\n    )\n    parser.add_argument(\n        \"--lr\",\n        type=float,\n        default=0.03,\n        metavar=\"LR\",\n        help=\"learning rate (default: 0.03)\",\n    )\n    parser.add_argument(\n        \"--no_accel\", action=\"store_true\", help=\"disables accelerator\"\n    )\n    parser.add_argument(\n        \"--seed\", type=int, default=1, metavar=\"S\", help=\"random seed (default: 1)\"\n    )\n    parser.add_argument(\n        \"--save_model\",\n        action=\"store_true\",\n        help=\"For saving the current Model\",\n    )\n    parser.add_argument(\n        \"--train_size\", type=int, default=50000, help=\"size of training set\"\n    )\n    parser.add_argument(\n        \"--threshold\", type=float, default=2, help=\"threshold for training\"\n    )\n    parser.add_argument(\"--test_size\", type=int, default=10000, help=\"size of test set\")\n    parser.add_argument(\n        \"--save-model\",\n        action=\"store_true\",\n        help=\"For Saving the current Model\",\n    )\n    parser.add_argument(\n        \"--log-interval\",\n        type=int,\n        default=10,\n        metavar=\"N\",\n        help=\"how many batches to wait before logging training status\",\n    )\n    args = parser.parse_args()\n    use_accel = not args.no_accel and torch.accelerator.is_available()\n    if use_accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n\n    train_kwargs = {\"batch_size\": args.train_size}\n    test_kwargs = {\"batch_size\": args.test_size}\n\n    if use_accel:\n        accel_kwargs = {\"num_workers\": 1, \"pin_memory\": True, \"shuffle\": True}\n        train_kwargs.update(accel_kwargs)\n        test_kwargs.update(accel_kwargs)\n\n    transform = Compose(\n        [\n            ToTensor(),\n            Normalize((0.1307,), (0.3081,)),\n            Lambda(lambda x: torch.flatten(x)),\n        ]\n    )\n    train_loader = DataLoader(\n        MNIST(\"./data/\", train=True, download=True, transform=transform), **train_kwargs\n    )\n    test_loader = DataLoader(\n        MNIST(\"./data/\", train=False, download=True, transform=transform), **test_kwargs\n    )\n    net = Net([784, 500, 500])\n    x, y = next(iter(train_loader))\n    x, y = x.to(device), y.to(device)\n    x_pos = overlay_y_on_x(x, y)\n    y_neg = get_y_neg(y)\n    x_neg = overlay_y_on_x(x, y_neg)\n    net.train(x_pos, x_neg)\n    print(\"train error:\", 1.0 - net.predict(x).eq(y).float().mean().item())\n    x_te, y_te = next(iter(test_loader))\n    x_te, y_te = x_te.to(device), y_te.to(device)\n    if args.save_model:\n        torch.save(net.state_dict(), \"mnist_ff.pt\")\n    print(\"test error:\", 1.0 - net.predict(x_te).eq(y_te).float().mean().item())\n"
  },
  {
    "path": "mnist_forward_forward/requirements.txt",
    "content": "torch\ntorchvision\n"
  },
  {
    "path": "mnist_hogwild/README.md",
    "content": "# MNIST Hogwild Example\n\n```bash\npip install -r requirements.txt\npython main.py\n```\n\nThe main.py script accepts the following arguments:\n\n```bash\noptional arguments:\n  -h, --help            show this help message and exit\n  --batch-size          input batch_size for training (default: 64)\n  --test-batch-size     input batch size for testing (default: 1000)\n  --epochs EPOCHS       number of epochs to train (default: 10)\n  --lr LR               learning rate (default: 0.01)\n  --momentum            SGD momentum (default: 0.5)\n  --seed SEED           random seed (default: 1)\n  --save_model          save the trained model to state_dict\n  --log-interval        how many batches to wait before logging training status (default: 10)\n  --num-processes       how many training processes to use (default: 2)\n  --cuda                enables CUDA training\n  --mps                 enables macos GPU training\n  --dry-run             quickly check a single pass\n```\n"
  },
  {
    "path": "mnist_hogwild/main.py",
    "content": "from __future__ import print_function\nimport argparse\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.multiprocessing as mp\nfrom torch.utils.data.sampler import Sampler\nfrom torchvision import datasets, transforms\n\nfrom train import train, test\n\n# Training settings\nparser = argparse.ArgumentParser(description='PyTorch MNIST Example')\nparser.add_argument('--batch-size', type=int, default=64, metavar='N',\n                    help='input batch size for training (default: 64)')\nparser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n                    help='input batch size for testing (default: 1000)')\nparser.add_argument('--epochs', type=int, default=10, metavar='N',\n                    help='number of epochs to train (default: 10)')\nparser.add_argument('--lr', type=float, default=0.01, metavar='LR',\n                    help='learning rate (default: 0.01)')\nparser.add_argument('--momentum', type=float, default=0.5, metavar='M',\n                    help='SGD momentum (default: 0.5)')\nparser.add_argument('--seed', type=int, default=1, metavar='S',\n                    help='random seed (default: 1)')\nparser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                    help='how many batches to wait before logging training status')\nparser.add_argument('--num-processes', type=int, default=2, metavar='N',\n                    help='how many training processes to use (default: 2)')\nparser.add_argument('--cuda', action='store_true', default=False,\n                    help='enables CUDA training')\nparser.add_argument('--mps', action='store_true', default=False,\n                    help='enables macOS GPU training')\nparser.add_argument('--save_model', action='store_true', default=False,\n                    help='save the trained model to state_dict')\nparser.add_argument('--dry-run', action='store_true', default=False,\n                    help='quickly check a single pass')\n\nclass Net(nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n        self.conv2_drop = nn.Dropout2d()\n        self.fc1 = nn.Linear(320, 50)\n        self.fc2 = nn.Linear(50, 10)\n\n    def forward(self, x):\n        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n        x = x.view(-1, 320)\n        x = F.relu(self.fc1(x))\n        x = F.dropout(x, training=self.training)\n        x = self.fc2(x)\n        return F.log_softmax(x, dim=1)\n\n\nif __name__ == '__main__':\n    args = parser.parse_args()\n\n    use_cuda = args.cuda and torch.cuda.is_available()\n    use_mps = args.mps and torch.backends.mps.is_available()\n    if use_cuda:\n        device = torch.device(\"cuda\")\n    elif use_mps:\n        device = torch.device(\"mps\")\n    else:\n        device = torch.device(\"cpu\")\n\n    transform=transforms.Compose([\n        transforms.ToTensor(),\n        transforms.Normalize((0.1307,), (0.3081,))\n        ])\n    dataset1 = datasets.MNIST('../data', train=True, download=True,\n                       transform=transform)\n    dataset2 = datasets.MNIST('../data', train=False,\n                       transform=transform)\n    kwargs = {'batch_size': args.batch_size,\n              'shuffle': True}\n    if use_cuda:\n        kwargs.update({'num_workers': 1,\n                       'pin_memory': True,\n                      })\n\n    torch.manual_seed(args.seed)\n    mp.set_start_method('spawn', force=True)\n\n    model = Net().to(device)\n    model.share_memory() # gradients are allocated lazily, so they are not shared here\n\n    processes = []\n    for rank in range(args.num_processes):\n        p = mp.Process(target=train, args=(rank, args, model, device,\n                                           dataset1, kwargs))\n        # We first train the model across `num_processes` processes\n        p.start()\n        processes.append(p)\n    for p in processes:\n        p.join()\n\n    if args.save_model:\n        torch.save(model.state_dict(), \"MNIST_hogwild.pt\")\n\n    # Once training is complete, we can test the model\n    test(args, model, device, dataset2, kwargs)\n"
  },
  {
    "path": "mnist_hogwild/requirements.txt",
    "content": "torch\ntorchvision==0.20.0\n"
  },
  {
    "path": "mnist_hogwild/train.py",
    "content": "import os\nimport torch\nimport torch.optim as optim\nimport torch.nn.functional as F\n\n\ndef train(rank, args, model, device, dataset, dataloader_kwargs):\n    torch.manual_seed(args.seed + rank)\n\n    train_loader = torch.utils.data.DataLoader(dataset, **dataloader_kwargs)\n\n    optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)\n    for epoch in range(1, args.epochs + 1):\n        train_epoch(epoch, args, model, device, train_loader, optimizer)\n\n\ndef test(args, model, device, dataset, dataloader_kwargs):\n    torch.manual_seed(args.seed)\n\n    test_loader = torch.utils.data.DataLoader(dataset, **dataloader_kwargs)\n\n    test_epoch(model, device, test_loader)\n\n\ndef train_epoch(epoch, args, model, device, data_loader, optimizer):\n    model.train()\n    pid = os.getpid()\n    for batch_idx, (data, target) in enumerate(data_loader):\n        optimizer.zero_grad()\n        output = model(data.to(device))\n        loss = F.nll_loss(output, target.to(device))\n        loss.backward()\n        optimizer.step()\n        if batch_idx % args.log_interval == 0:\n            print('{}\\tTrain Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                pid, epoch, batch_idx * len(data), len(data_loader.dataset),\n                100. * batch_idx / len(data_loader), loss.item()))\n            if args.dry_run:\n                break\n\n\ndef test_epoch(model, device, data_loader):\n    model.eval()\n    test_loss = 0\n    correct = 0\n    with torch.no_grad():\n        for data, target in data_loader:\n            output = model(data.to(device))\n            test_loss += F.nll_loss(output, target.to(device), reduction='sum').item() # sum up batch loss\n            pred = output.max(1)[1] # get the index of the max log-probability\n            correct += pred.eq(target.to(device)).sum().item()\n\n    test_loss /= len(data_loader.dataset)\n    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n        test_loss, correct, len(data_loader.dataset),\n        100. * correct / len(data_loader.dataset)))\n"
  },
  {
    "path": "mnist_rnn/README.md",
    "content": "# Example of MNIST using RNN\n\n## Motivation\nCreate pytorch example similar to Official Tensorflow Keras RNN example using MNIST [here](https://www.tensorflow.org/guide/keras/rnn) \n\n```bash\npip install -r requirements.txt\npython main.py\n# CUDA_VISIBLE_DEVICES=2 python main.py  # to specify GPU id to ex. 2\n```\n\n```bash\noptional arguments:\n  -h, --help            show this help message and exit\n  --batch_size          input batch_size for training (default:64)\n  --testing_batch_size  input batch size for testing (default: 1000)\n  --epochs EPOCHS       number of epochs to train (default: 14)\n  --lr LR               learning rate (default: 0.1)\n  --gamma               learning rate step gamma (default: 0.7)\n  --accel               enables accelerator\n  --seed SEED           random seed (default: 1)\n  --save_model          For saving the current Model\n  --log_interval        how many batches to wait before logging training status\n  --dry-run             quickly check a single pass\n```"
  },
  {
    "path": "mnist_rnn/main.py",
    "content": "from __future__ import print_function\n\nimport argparse\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import StepLR\nfrom torchvision import datasets, transforms\n\n\nclass Net(nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        self.rnn = nn.LSTM(input_size=28, hidden_size=64, batch_first=True)\n        self.batchnorm = nn.BatchNorm1d(64)\n        self.dropout1 = nn.Dropout2d(0.25)\n        self.dropout2 = nn.Dropout2d(0.5)\n        self.fc1 = nn.Linear(64, 32)\n        self.fc2 = nn.Linear(32, 10)\n\n    def forward(self, input):\n        # Shape of input is (batch_size,1, 28, 28)\n        # converting shape of input to (batch_size, 28, 28)\n        # as required by RNN when batch_first is set True\n        input = input.reshape(-1, 28, 28)\n        output, hidden = self.rnn(input)\n\n        # RNN output shape is (seq_len, batch, input_size)\n        # Get last output of RNN\n        output = output[:, -1, :]\n        output = self.batchnorm(output)\n        output = self.dropout1(output)\n        output = self.fc1(output)\n        output = F.relu(output)\n        output = self.dropout2(output)\n        output = self.fc2(output)\n        output = F.log_softmax(output, dim=1)\n        return output\n\n\ndef train(args, model, device, train_loader, optimizer, epoch):\n    model.train()\n    for batch_idx, (data, target) in enumerate(train_loader):\n        data, target = data.to(device), target.to(device)\n        optimizer.zero_grad()\n        output = model(data)\n        loss = F.nll_loss(output, target)\n        loss.backward()\n        optimizer.step()\n        if batch_idx % args.log_interval == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, batch_idx * len(data), len(train_loader.dataset),\n                       100. * batch_idx / len(train_loader), loss.item()))\n            if args.dry_run:\n                break\n\n\ndef test(args, model, device, test_loader):\n    model.eval()\n    test_loss = 0\n    correct = 0\n    with torch.no_grad():\n        for data, target in test_loader:\n            data, target = data.to(device), target.to(device)\n            output = model(data)\n            test_loss += F.nll_loss(output, target, reduction='sum').item()  # sum up batch loss\n            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability\n            correct += pred.eq(target.view_as(pred)).sum().item()\n            if args.dry_run:\n                break\n\n    test_loss /= len(test_loader.dataset)\n\n    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n        test_loss, correct, len(test_loader.dataset),\n        100. * correct / len(test_loader.dataset)))\n\n\ndef main():\n    # Training settings\n    parser = argparse.ArgumentParser(description='PyTorch MNIST Example using RNN')\n    parser.add_argument('--batch-size', type=int, default=64, metavar='N',\n                        help='input batch size for training (default: 64)')\n    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n                        help='input batch size for testing (default: 1000)')\n    parser.add_argument('--epochs', type=int, default=14, metavar='N',\n                        help='number of epochs to train (default: 14)')\n    parser.add_argument('--lr', type=float, default=0.1, metavar='LR',\n                        help='learning rate (default: 0.1)')\n    parser.add_argument('--gamma', type=float, default=0.7, metavar='M',\n                        help='learning rate step gamma (default: 0.7)')\n    parser.add_argument('--accel', action='store_true',\n                        help='enables accelerator')\n    parser.add_argument('--dry-run', action='store_true',\n                        help='quickly check a single pass')\n    parser.add_argument('--seed', type=int, default=1, metavar='S',\n                        help='random seed (default: 1)')\n    parser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                        help='how many batches to wait before logging training status')\n    parser.add_argument('--save-model', action='store_true',\n                        help='for Saving the current Model')\n    args = parser.parse_args()\n\n    if args.accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n\n    torch.manual_seed(args.seed)\n\n    kwargs = {'num_workers': 1, 'pin_memory': True} if args.accel else {}\n    train_loader = torch.utils.data.DataLoader(\n        datasets.MNIST('../data', train=True, download=True,\n                       transform=transforms.Compose([\n                           transforms.ToTensor(),\n                           transforms.Normalize((0.1307,), (0.3081,))\n                       ])),\n        batch_size=args.batch_size, shuffle=True, **kwargs)\n    test_loader = torch.utils.data.DataLoader(\n        datasets.MNIST('../data', train=False, transform=transforms.Compose([\n            transforms.ToTensor(),\n            transforms.Normalize((0.1307,), (0.3081,))\n        ])),\n        batch_size=args.test_batch_size, shuffle=True, **kwargs)\n\n    model = Net().to(device)\n    optimizer = optim.Adadelta(model.parameters(), lr=args.lr)\n\n    scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)\n    for epoch in range(1, args.epochs + 1):\n        train(args, model, device, train_loader, optimizer, epoch)\n        test(args, model, device, test_loader)\n        scheduler.step()\n\n    if args.save_model:\n        torch.save(model.state_dict(), \"mnist_rnn.pt\")\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "mnist_rnn/requirements.txt",
    "content": "torch\ntorchvision\n"
  },
  {
    "path": "regression/README.md",
    "content": "# Linear regression example\n\nTrains a single fully-connected layer to fit a 4th degree polynomial.\n"
  },
  {
    "path": "regression/main.py",
    "content": "#!/usr/bin/env python\nfrom __future__ import print_function\nfrom itertools import count\n\nimport torch\nimport torch.nn.functional as F\n\nPOLY_DEGREE = 4\nW_target = torch.randn(POLY_DEGREE, 1) * 5\nb_target = torch.randn(1) * 5\n\n\ndef make_features(x):\n    \"\"\"Builds features i.e. a matrix with columns [x, x^2, x^3, x^4].\"\"\"\n    x = x.unsqueeze(1)\n    return torch.cat([x ** i for i in range(1, POLY_DEGREE+1)], 1)\n\n\ndef f(x):\n    \"\"\"Approximated function.\"\"\"\n    return x.mm(W_target) + b_target.item()\n\n\ndef poly_desc(W, b):\n    \"\"\"Creates a string description of a polynomial.\"\"\"\n    result = 'y = '\n    for i, w in enumerate(W):\n        result += '{:+.2f} x^{} '.format(w, i + 1)\n    result += '{:+.2f}'.format(b[0])\n    return result\n\n\ndef get_batch(batch_size=32):\n    \"\"\"Builds a batch i.e. (x, f(x)) pair.\"\"\"\n    random = torch.randn(batch_size)\n    x = make_features(random)\n    y = f(x)\n    return x, y\n\n\n# Define model\nfc = torch.nn.Linear(W_target.size(0), 1)\n\nfor batch_idx in count(1):\n    # Get data\n    batch_x, batch_y = get_batch()\n\n    # Reset gradients\n    fc.zero_grad()\n\n    # Forward pass\n    output = F.smooth_l1_loss(fc(batch_x), batch_y)\n    loss = output.item()\n\n    # Backward pass\n    output.backward()\n\n    # Apply gradients\n    for param in fc.parameters():\n        param.data.add_(-0.1 * param.grad)\n\n    # Stop criterion\n    if loss < 1e-3:\n        break\n\nprint('Loss: {:.6f} after {} batches'.format(loss, batch_idx))\nprint('==> Learned function:\\t' + poly_desc(fc.weight.view(-1), fc.bias))\nprint('==> Actual function:\\t' + poly_desc(W_target.view(-1), b_target))\n"
  },
  {
    "path": "regression/requirements.txt",
    "content": "torch\n"
  },
  {
    "path": "reinforcement_learning/README.md",
    "content": "# Reinforcement learning training example\n\n```bash\npip install -r requirements.txt\n# For REINFORCE:\npython reinforce.py\n# For actor critic:\npython actor_critic.py\n```\n"
  },
  {
    "path": "reinforcement_learning/actor_critic.py",
    "content": "import argparse\nimport gymnasium as gym\nimport numpy as np\nfrom itertools import count\nfrom collections import namedtuple\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.distributions import Categorical\n\n# Cart Pole\n\nparser = argparse.ArgumentParser(description='PyTorch actor-critic example')\nparser.add_argument('--gamma', type=float, default=0.99, metavar='G',\n                    help='discount factor (default: 0.99)')\nparser.add_argument('--seed', type=int, default=543, metavar='N',\n                    help='random seed (default: 543)')\nparser.add_argument('--render', action='store_true',\n                    help='render the environment')\nparser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                    help='interval between training status logs (default: 10)')\nargs = parser.parse_args()\n\n\nrender_mode = \"human\" if args.render else None\nenv = gym.make('CartPole-v1', render_mode=render_mode)\nenv.reset(seed=args.seed)\ntorch.manual_seed(args.seed)\n\n\nSavedAction = namedtuple('SavedAction', ['log_prob', 'value'])\n\n\nclass Policy(nn.Module):\n    \"\"\"\n    implements both actor and critic in one model\n    \"\"\"\n    def __init__(self):\n        super(Policy, self).__init__()\n        self.affine1 = nn.Linear(4, 128)\n\n        # actor's layer\n        self.action_head = nn.Linear(128, 2)\n\n        # critic's layer\n        self.value_head = nn.Linear(128, 1)\n\n        # action & reward buffer\n        self.saved_actions = []\n        self.rewards = []\n\n    def forward(self, x):\n        \"\"\"\n        forward of both actor and critic\n        \"\"\"\n        x = F.relu(self.affine1(x))\n\n        # actor: choses action to take from state s_t\n        # by returning probability of each action\n        action_prob = F.softmax(self.action_head(x), dim=-1)\n\n        # critic: evaluates being in the state s_t\n        state_values = self.value_head(x)\n\n        # return values for both actor and critic as a tuple of 2 values:\n        # 1. a list with the probability of each action over the action space\n        # 2. the value from state s_t\n        return action_prob, state_values\n\n\nmodel = Policy()\noptimizer = optim.Adam(model.parameters(), lr=3e-2)\neps = np.finfo(np.float32).eps.item()\n\n\ndef select_action(state):\n    state = torch.from_numpy(state).float()\n    probs, state_value = model(state)\n\n    # create a categorical distribution over the list of probabilities of actions\n    m = Categorical(probs)\n\n    # and sample an action using the distribution\n    action = m.sample()\n\n    # save to action buffer\n    model.saved_actions.append(SavedAction(m.log_prob(action), state_value))\n\n    # the action to take (left or right)\n    return action.item()\n\n\ndef finish_episode():\n    \"\"\"\n    Training code. Calculates actor and critic loss and performs backprop.\n    \"\"\"\n    R = 0\n    saved_actions = model.saved_actions\n    policy_losses = [] # list to save actor (policy) loss\n    value_losses = [] # list to save critic (value) loss\n    returns = [] # list to save the true values\n\n    # calculate the true value using rewards returned from the environment\n    for r in model.rewards[::-1]:\n        # calculate the discounted value\n        R = r + args.gamma * R\n        returns.insert(0, R)\n\n    returns = torch.tensor(returns)\n    returns = (returns - returns.mean()) / (returns.std() + eps)\n\n    for (log_prob, value), R in zip(saved_actions, returns):\n        advantage = R - value.item()\n\n        # calculate actor (policy) loss\n        policy_losses.append(-log_prob * advantage)\n\n        # calculate critic (value) loss using L1 smooth loss\n        value_losses.append(F.smooth_l1_loss(value, torch.tensor([R])))\n\n    # reset gradients\n    optimizer.zero_grad()\n\n    # sum up all the values of policy_losses and value_losses\n    loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()\n\n    # perform backprop\n    loss.backward()\n    optimizer.step()\n\n    # reset rewards and action buffer\n    del model.rewards[:]\n    del model.saved_actions[:]\n\n\ndef main():\n    running_reward = 10\n\n    # run infinitely many episodes\n    for i_episode in count(1):\n\n        # reset environment and episode reward\n        state, _ = env.reset()\n        ep_reward = 0\n\n        # for each episode, only run 9999 steps so that we don't\n        # infinite loop while learning\n        for t in range(1, 10000):\n\n            # select action from policy\n            action = select_action(state)\n\n            # take the action\n            state, reward, terminated, truncated, _ = env.step(action)\n\n            model.rewards.append(reward)\n            ep_reward += reward\n            if terminated or truncated:\n                break\n\n        # update cumulative reward\n        running_reward = 0.05 * ep_reward + (1 - 0.05) * running_reward\n\n        # perform backprop\n        finish_episode()\n\n        # log results\n        if i_episode % args.log_interval == 0:\n            print(f'Episode {i_episode}\\tLast reward: {ep_reward:.2f}\\tAverage reward: {running_reward:.2f}')\n\n        # check if we have \"solved\" the cart pole problem\n        if running_reward > env.spec.reward_threshold:\n            print(f\"Solved! Running reward is now {running_reward} and \"\n                  f\"the last episode runs to {t} time steps!\")\n            break\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "reinforcement_learning/reinforce.py",
    "content": "import argparse\nimport gymnasium as gym\nimport numpy as np\nfrom itertools import count\nfrom collections import deque\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.distributions import Categorical\n\n\nparser = argparse.ArgumentParser(description='PyTorch REINFORCE example')\nparser.add_argument('--gamma', type=float, default=0.99, metavar='G',\n                    help='discount factor (default: 0.99)')\nparser.add_argument('--seed', type=int, default=543, metavar='N',\n                    help='random seed (default: 543)')\nparser.add_argument('--render', action='store_true',\n                    help='render the environment')\nparser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                    help='interval between training status logs (default: 10)')\nargs = parser.parse_args()\n\n\nrender_mode = \"human\" if args.render else None\nenv = gym.make('CartPole-v1', render_mode=render_mode)\nenv.reset(seed=args.seed)\ntorch.manual_seed(args.seed)\n\n\nclass Policy(nn.Module):\n    def __init__(self):\n        super(Policy, self).__init__()\n        self.affine1 = nn.Linear(4, 128)\n        self.dropout = nn.Dropout(p=0.6)\n        self.affine2 = nn.Linear(128, 2)\n\n        self.saved_log_probs = []\n        self.rewards = []\n\n    def forward(self, x):\n        x = self.affine1(x)\n        x = self.dropout(x)\n        x = F.relu(x)\n        action_scores = self.affine2(x)\n        return F.softmax(action_scores, dim=1)\n\n\npolicy = Policy()\noptimizer = optim.Adam(policy.parameters(), lr=1e-2)\neps = np.finfo(np.float32).eps.item()\n\n\ndef select_action(state):\n    state = torch.from_numpy(state).float().unsqueeze(0)\n    probs = policy(state)\n    m = Categorical(probs)\n    action = m.sample()\n    policy.saved_log_probs.append(m.log_prob(action))\n    return action.item()\n\n\ndef finish_episode():\n    R = 0\n    policy_loss = []\n    returns = deque()\n    for r in policy.rewards[::-1]:\n        R = r + args.gamma * R\n        returns.appendleft(R)\n    returns = torch.tensor(returns)\n    returns = (returns - returns.mean()) / (returns.std() + eps)\n    for log_prob, R in zip(policy.saved_log_probs, returns):\n        policy_loss.append(-log_prob * R)\n    optimizer.zero_grad()\n    policy_loss = torch.cat(policy_loss).sum()\n    policy_loss.backward()\n    optimizer.step()\n    del policy.rewards[:]\n    del policy.saved_log_probs[:]\n\n\ndef main():\n    running_reward = 10\n    for i_episode in count(1):\n        state, _ = env.reset()\n        ep_reward = 0\n        for t in range(1, 10000):  # Don't infinite loop while learning\n            action = select_action(state)\n            state, reward, terminated, truncated, _ = env.step(action)\n            if args.render:\n                env.render()\n            policy.rewards.append(reward)\n            ep_reward += reward\n            if terminated or truncated:\n                break\n\n        running_reward = 0.05 * ep_reward + (1 - 0.05) * running_reward\n        finish_episode()\n        if i_episode % args.log_interval == 0:\n            print(f'Episode {i_episode}\\tLast reward: {ep_reward:.2f}\\tAverage reward: {running_reward:.2f}')\n        if running_reward > env.spec.reward_threshold:\n            print(f\"Solved! Running reward is now {running_reward} and the last episode runs to {t} time steps!\")\n            break\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "reinforcement_learning/requirements.txt",
    "content": "torch\nnumpy\ngymnasium[classic-control]\n"
  },
  {
    "path": "run_cpp_examples.sh",
    "content": "#!/usr/bin/env bash\n# This script runs through the code in each of the cpp examples.\n# The purpose is just as an integration test, not to actually train models in any meaningful way.\n\n# Optionally specify a comma separated list of examples to run.\n# can be run as:\n# ./run_cpp_examples.sh \"get_libtorch,run_all,clean\"\n# To get libtorch, run all examples, and remove temporary/changed data files.\n\nBASE_DIR=`pwd`\"/\"`dirname $0`\necho \"BASE_DIR: $BASE_DIR\"\nEXAMPLES=`echo $1 | sed -e 's/ //g'`\nHOME_DIR=$HOME\nERRORS=\"\"\n\nfunction error() {\n  ERR=$1\n  ERRORS=\"$ERRORS\\n$ERR\"\n  echo $ERR\n}\n\nfunction get_libtorch() {\n  echo \"Getting libtorch\"\n  cd $HOME_DIR\n  if [ ! -d \"libtorch\" ]; then\n    wget https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip\n    unzip libtorch-cxx11-abi-shared-with-deps-latest.zip\n  fi\n\n  if [ $? -eq 0 ]; then\n    echo \"Successfully downloaded and extracted libtorch\"\n    LIBTORCH_PATH=\"$HOME_DIR/libtorch\"   # Store the LibTorch path in a variable.\n    echo \"LibTorch path: $LIBTORCH_PATH\" # Print the LibTorch path\n  else\n    error \"Failed to download or extract LibTorch\"\n  fi\n}\n\nfunction start() {\n  EXAMPLE=${FUNCNAME[1]}\n  cd $BASE_DIR/cpp/$EXAMPLE\n  echo \"Running example: $EXAMPLE\"\n}\n\nfunction check_run_success() {\n  if [ $? -eq 0 ]; then\n    echo \"Successfully ran $1\"\n  else\n    echo \"Failed to run $1\"\n    error \"Failed to run $1\"\n    exit 1\n  fi\n}\n\nfunction autograd() {\n  start\n  mkdir build\n  cd build\n  cmake -DCMAKE_PREFIX_PATH=$LIBTORCH_PATH ..\n  make\n  if [ $? -eq 0 ]; then\n    echo \"Successfully built $EXAMPLE\"\n    ./$EXAMPLE # Run the executable\n    check_run_success $EXAMPLE\n  else\n    error \"Failed to build $EXAMPLE\"\n    exit 1\n  fi\n}\n\nfunction custom-dataset() {\n  start\n  # Download the dataset and unzip it\n  if [ ! -d \"$BASE_DIR/cpp/$EXAMPLE/dataset\" ]; then\n    wget https://data.caltech.edu/records/mzrjq-6wc02/files/caltech-101.zip\n    unzip caltech-101.zip\n    cd caltech-101\n    tar -xzf 101_ObjectCategories.tar.gz\n    mv 101_ObjectCategories $BASE_DIR/cpp/$EXAMPLE/dataset\n  fi\n  # build the executable and run it\n  cd $BASE_DIR/cpp/$EXAMPLE\n  mkdir build\n  cd build\n  cmake -DCMAKE_PREFIX_PATH=$LIBTORCH_PATH ..\n  make\n  if [ $? -eq 0 ]; then\n    echo \"Successfully built $EXAMPLE\"\n    cd $BASE_DIR/cpp/$EXAMPLE\n    ./build/$EXAMPLE # Run the executable\n    check_run_success $EXAMPLE\n  else\n    error \"Failed to build $EXAMPLE\"\n    exit 1\n  fi\n}\nfunction dcgan() {\n  start\n  mkdir build\n  cd build\n  cmake -DCMAKE_PREFIX_PATH=$LIBTORCH_PATH ..\n  make\n  if [ $? -eq 0 ]; then\n    echo \"Successfully built $EXAMPLE\"\n    ./$EXAMPLE --epochs 5 # Run the executable with kNumberOfEpochs = 5\n    check_run_success $EXAMPLE\n  else\n    error \"Failed to build $EXAMPLE\"\n    exit 1\n  fi\n}\n\nfunction mnist() {\n  start\n  mkdir build\n  cd build\n  cmake -DCMAKE_PREFIX_PATH=$LIBTORCH_PATH ..\n  make\n  if [ $? -eq 0 ]; then\n    echo \"Successfully built $EXAMPLE\"\n    ./$EXAMPLE # Run the executable\n    check_run_success $EXAMPLE\n  else\n    error \"Failed to build $EXAMPLE\"\n    exit 1\n  fi\n}\n\nfunction regression() {\n  start\n  mkdir build\n  cd build\n  cmake -DCMAKE_PREFIX_PATH=$LIBTORCH_PATH ..\n  make\n  if [ $? -eq 0 ]; then\n    echo \"Successfully built $EXAMPLE\"\n    ./$EXAMPLE # Run the executable\n    check_run_success $EXAMPLE\n  else\n    error \"Failed to build $EXAMPLE\"\n    exit 1\n  fi\n}\n\nfunction clean() {\n  cd $BASE_DIR\n  echo \"Running clean to remove cruft\"\n  # Remove the build directories\n  find . -type d -name 'build' -exec rm -rf {} +\n  # Remove the libtorch directory\n  rm -rf $HOME_DIR/libtorch\n  rm -f $HOME_DIR/libtorch-shared-with-deps-latest.zip\n  echo \"Clean completed\"\n}\n\nfunction run_all() {\n  autograd\n  custom-dataset\n  regression\n  dcgan\n  mnist\n  \n}\n\n# by default, run all examples\nif [ \"\" == \"$EXAMPLES\" ]; then\n  run_all\nelse\n  for i in $(echo $EXAMPLES | sed \"s/,/ /g\")\n  do\n    echo \"Starting $i\"\n    $i\n    echo \"Finished $i, status $?\"\n  done\nfi\n\nif [ \"\" == \"$ERRORS\" ]; then\n  echo \"Completed successfully with status $?\"\nelse\n  echo \"Some examples failed:\"\n  printf \"$ERRORS\"\n  exit 1\nfi\n"
  },
  {
    "path": "run_distributed_examples.sh",
    "content": "#!/usr/bin/env bash\n#\n# This script runs through the code in each of the python examples.\n# The purpose is just as an integration test, not to actually train models in any meaningful way.\n# For that reason, most of these set epochs = 1 and --dry-run.\n#\n# Optionally specify a comma separated list of examples to run. Can be run as:\n# * To run all examples:\n#   ./run_distributed_examples.sh\n# * To run specific example:\n#   ./run_distributed_examples.sh \"distributed/tensor_parallelism,distributed/ddp\"\n#\n# To test examples on CUDA accelerator, run as:\n#   USE_CUDA=True ./run_distributed_examples.sh\n#\n# Script requires uv to be installed. When executed, script will install prerequisites from\n# `requirements.txt` for each example. If ran within activated virtual environment (uv venv,\n# python -m venv, conda) this might reinstall some of the packages. To change pip installation\n# index or to pass additional pip install options, run as:\n#   PIP_INSTALL_ARGS=\"--pre -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html\" \\\n#     ./run_python_examples.sh\n#\n# To force script to create virtual environment for each example, run as:\n#   VIRTUAL_ENV=\".venv\" ./run_distributed_examples.sh\n# Script will remove environments it creates in a teardown step after execution of each example.\n\nBASE_DIR=\"$(pwd)/$(dirname $0)\"\nsource $BASE_DIR/utils.sh\n\nUSE_CUDA=${USE_CUDA:-False}\ncase $USE_CUDA in\n  \"True\")\n    echo \"using cuda\"\n    CUDA=1\n    CUDA_FLAG=\"--cuda\"\n    ;;\n  \"False\")\n    echo \"not using cuda\"\n    CUDA=0\n    CUDA_FLAG=\"\"\n    ;;\n  \"\")\n    exit 1;\n    ;;\nesac\n\nfunction distributed_tensor_parallelism() {\n    uv run bash run_example.sh tensor_parallel_example.py || error \"tensor parallel example failed\"\n    uv run bash run_example.sh sequence_parallel_example.py || error \"sequence parallel example failed\"\n    uv run bash run_example.sh fsdp_tp_example.py || error \"2D parallel example failed\"\n}\n\nfunction distributed_FSDP2() {\n    uv run bash run_example.sh example.py || error \"FSDP2 example failed\"\n}\n\nfunction distributed_ddp() {\n    uv run bash run_example.sh example.py || error \"ddp example failed\"\n}\n\nfunction distributed_minGPT-ddp() {\n  uv run bash run_example.sh mingpt/main.py || error \"minGPT example failed\"\n}\n\nfunction distributed_rpc_ddp_rpc() {\n    uv run main.py || error \"ddp_rpc example failed\"\n}\n\nfunction distributed_rpc_rnn() {\n    uv run main.py || error \"rpc_rnn example failed\"\n}\n\nfunction run_all() {\n  run distributed/tensor_parallelism\n  run distributed/ddp\n  run distributed/minGPT-ddp\n  run distributed/rpc/ddp_rpc\n  run distributed/rpc/rnn\n}\n\n# by default, run all examples\nif [ \"\" == \"$EXAMPLES\" ]; then\n  run_all\nelse\n  for i in $(echo $EXAMPLES | sed \"s/,/ /g\")\n  do\n    echo \"Starting $i\"\n    run $i\n    echo \"Finished $i, status $?\"\n  done\nfi\n\nif [ \"\" == \"$ERRORS\" ]; then\n  echo \"Completed successfully with status $?\"\nelse\n  echo \"Some distributed examples failed:\"\n  printf \"$ERRORS\\n\"\n  #Exit with error (0-255) in case of failure in one of the tests.\n  exit 1\n\nfi\n"
  },
  {
    "path": "run_python_examples.sh",
    "content": "#!/bin/bash\n#\n# This script runs through the code in each of the python examples.\n# The purpose is just as an integration test, not to actually train models in any meaningful way.\n# For that reason, most of these set epochs = 1 and --dry-run.\n#\n# Optionally specify a comma separated list of examples to run. Can be run as:\n# * To run all examples:\n#   ./run_python_examples.sh\n# * To run few specific examples:\n#   ./run_python_examples.sh \"dcgan,fast_neural_style\"\n#\n# To test examples on CUDA accelerator, run as:\n#   USE_CUDA=True ./run_python_examples.sh\n#\n# To test examples on hardware accelerator (CUDA, MPS, XPU, etc.), run as:\n#   USE_ACCEL=True ./run_python_examples.sh\n# NOTE: USE_ACCEL relies on torch.accelerator API and not all examples are converted\n# to use it at the moment. Thus, expect failures using this flag on non-CUDA accelerators\n# and consider to run examples one by one.\n#\n# Script requires uv to be installed. When executed, script will install prerequisites from\n# `requirements.txt` for each example. If ran within activated virtual environment (uv venv,\n# python -m venv, conda) this might reinstall some of the packages. To change pip installation\n# index or to pass additional pip install options, run as:\n#   PIP_INSTALL_ARGS=\"--pre -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html\" \\\n#     ./run_python_examples.sh\n#\n# To force script to create virtual environment for each example, run as:\n#   VIRTUAL_ENV=\".venv\" ./run_python_examples.sh\n# Script will remove environments it creates in a teardown step after execution of each example.\n\nBASE_DIR=\"$(pwd)/$(dirname $0)\"\nsource $BASE_DIR/utils.sh\n\n# TODO: Leave only USE_ACCEL and drop USE_CUDA once all examples will be converted\n# to torch.accelerator API. For now, just add USE_ACCEL as an alias for USE_CUDA.\nif [ -n \"$USE_ACCEL\" ]; then\n  USE_CUDA=$USE_ACCEL\nfi\nUSE_CUDA=${USE_CUDA:-False}\ncase $USE_CUDA in\n  \"True\")\n    echo \"using cuda\"\n    CUDA=1\n    CUDA_FLAG=\"--cuda\"\n    ACCEL_FLAG=\"--accel\"\n    ;;\n  \"False\")\n    echo \"not using cuda\"\n    CUDA=0\n    CUDA_FLAG=\"\"\n    ACCEL_FLAG=\"\"\n    ;;\n  \"\")\n    exit 1;\n    ;;\nesac\n\nfunction dcgan() {\n  uv run main.py --dataset fake $ACCEL_FLAG --dry-run || error \"dcgan failed\"\n}\n\nfunction fast_neural_style() {\n  if [ ! -d \"saved_models\" ]; then\n    echo \"downloading saved models for fast neural style\"\n    uv run download_saved_models.py\n  fi\n  test -d \"saved_models\" || { error \"saved models not found\"; return; }\n\n  echo \"running fast neural style model\"\n  uv run neural_style/neural_style.py eval --content-image images/content-images/amber.jpg --model saved_models/candy.pth --output-image images/output-images/amber-candy.jpg $ACCEL_FLAG || error \"neural_style.py failed\"\n}\n\nfunction imagenet() {\n  if [[ ! -d \"sample/val\" || ! -d \"sample/train\" ]]; then\n    mkdir -p sample/val/n\n    mkdir -p sample/train/n\n    curl -O \"https://upload.wikimedia.org/wikipedia/commons/5/5a/Socks-clinton.jpg\" || { error \"couldn't download sample image for imagenet\"; return; }\n    mv Socks-clinton.jpg sample/train/n\n    cp sample/train/n/* sample/val/n/\n  fi\n  uv run main.py --epochs 1 sample/ || error \"imagenet example failed\"\n  uv run main.py --epochs 1 --gpu 0 sample/ || error \"imagenet example failed\"\n}\n\nfunction language_translation() {\n  uv run -m spacy download en || error \"couldn't download en package from spacy\"\n  uv run -m spacy download de || error \"couldn't download de package from spacy\"\n  uv run main.py -e 1 --enc_layers 1 --dec_layers 1 --backend cpu --logging_dir output/ --dry_run || error \"language translation example failed\"\n}\n\nfunction mnist() {\n  uv run main.py --epochs 1 --dry-run || error \"mnist example failed\"\n}\nfunction mnist_forward_forward() {\n  uv run main.py --epochs 1 --no_accel || error \"mnist forward forward failed\"\n\n}\nfunction mnist_hogwild() {\n  uv run main.py --epochs 1 --dry-run $CUDA_FLAG || error \"mnist hogwild failed\"\n}\n\nfunction mnist_rnn() {\n  uv run main.py --epochs 1 --dry-run || error \"mnist rnn example failed\"\n}\n\nfunction regression() {\n  uv run main.py --epochs 1 $CUDA_FLAG || error \"regression failed\"\n}\n\nfunction siamese_network() {\n  uv run main.py --epochs 1 --dry-run || error \"siamese network example failed\"\n}\n\nfunction reinforcement_learning() {\n  uv run reinforce.py || error \"reinforcement learning reinforce failed\"\n  uv run actor_critic.py || error \"reinforcement learning actor_critic failed\"\n}\n\nfunction snli() {\n  echo \"installing 'en' model if not installed\"\n  uv run -m spacy download en || { error \"couldn't download 'en' model needed for snli\";  return; }\n  echo \"training...\"\n  uv run train.py --epochs 1 --dev_every 1 --no-bidirectional --dry-run || error \"couldn't train snli\"\n}\n\nfunction fx() {\n  # uv run custom_tracer.py || error \"fx custom tracer has failed\" UnboundLocalError: local variable 'tabulate' referenced before assignment\n  uv run invert.py || error \"fx invert has failed\"\n  uv run module_tracer.py || error \"fx module tracer has failed\"\n  uv run primitive_library.py || error \"fx primitive library has failed\"\n  uv run profiling_tracer.py || error \"fx profiling tracer has failed\"\n  uv run replace_op.py || error \"fx replace op has failed\"\n  uv run subgraph_rewriter_basic_use.py || error \"fx subgraph has failed\"\n  uv run wrap_output_dynamically.py || error \"vmap output dynamically has failed\"\n}\n\nfunction super_resolution() {\n  uv run main.py --upscale_factor 3 --batchSize 4 --testBatchSize 100 --nEpochs 1 --lr 0.001 $ACCEL_FLAG || error \"super resolution failed\"\n  uv run super_resolve.py --input_image dataset/BSDS300/images/test/16077.jpg --model model_epoch_1.pth --output_filename out.png $ACCEL_FLAG || error \"super resolution upscaling failed\"\n}\n\nfunction time_sequence_prediction() {\n  uv run generate_sine_wave.py || { error \"generate sine wave failed\";  return; }\n  uv run train.py --steps 2 || error \"time sequence prediction training failed\"\n}\n\nfunction vae() {\n  uv run main.py --epochs 1 || error \"vae failed\"\n}\n\nfunction vision_transformer() {\n  uv run main.py --epochs 1 --dry-run || error \"vision transformer example failed\"\n}\n\nfunction word_language_model() {\n  uv run main.py --epochs 1 --dry-run $ACCEL_FLAG || error \"word_language_model failed\"\n  uv run generate.py $ACCEL_FLAG || error \"word_language_model generate failed\"\n  for model in \"RNN_TANH\" \"RNN_RELU\" \"LSTM\" \"GRU\" \"Transformer\"; do\n    uv run main.py --model $model --epochs 1 --dry-run $ACCEL_FLAG || error \"word_language_model failed\"\n    uv run generate.py $ACCEL_FLAG || error \"word_language_model generate failed\"\n  done\n}\n\nfunction gcn() {\n  uv run main.py --epochs 1 --dry-run || error \"graph convolutional network failed\"\n}\n\nfunction gat() {\n  uv run main.py --epochs 1 --dry-run || error \"graph attention network failed\"\n}\n\neval \"base_$(declare -f stop)\"\n\nfunction stop() {\n  cd $BASE_DIR\n  rm -rf dcgan/fake_samples_epoch_000.png \\\n    dcgan/netD_epoch_0.pth \\\n    dcgan/netG_epoch_0.pth \\\n    dcgan/real_samples.png \\\n    fast_neural_style/saved_models.zip \\\n    fast_neural_style/saved_models/ \\\n    imagenet/checkpoint.pth.tar \\\n    imagenet/lsun/ \\\n    imagenet/model_best.pth.tar \\\n    imagenet/sample/ \\\n    language_translation/output/ \\\n    snli/.data/ \\\n    snli/.vector_cache/ \\\n    snli/results/ \\\n    time_sequence_prediction/predict*.pdf \\\n    time_sequence_prediction/traindata.pt \\\n    word_language_model/model.pt \\\n    gcn/cora/ \\\n    gat/cora/ || error \"couldn't clean up some files\"\n\n  git checkout fast_neural_style/images/output-images/amber-candy.jpg || error \"couldn't clean up fast neural style image\"\n\n  base_stop \"$1\"\n}\n\nfunction run_all() {\n  # cpp moved to `run_cpp_examples.sh```\n  run dcgan\n  # distributed moved to `run_distributed_examples.sh`\n  run fast_neural_style\n  run imagenet\n  # language_translation\n  run mnist\n  run mnist_forward_forward\n  run mnist_hogwild\n  run mnist_rnn\n  run regression\n  run reinforcement_learning\n  run siamese_network\n  # run super_resolution - flaky\n  run time_sequence_prediction\n  run vae\n  # vision_transformer - example broken see https://github.com/pytorch/examples/issues/1184 and https://github.com/pytorch/examples/pull/1258 for more details\n  run word_language_model\n  run fx\n  run gcn\n  run gat\n}\n\n# by default, run all examples\nif [ \"\" == \"$EXAMPLES\" ]; then\n  run_all\nelse\n  for i in $(echo $EXAMPLES | sed \"s/,/ /g\")\n  do\n    echo \"Starting $i\"\n    run $i\n    echo \"Finished $i, status $?\"\n  done\nfi\n\nif [ \"\" == \"$ERRORS\" ]; then\n  echo \"Completed successfully with status $?\"\nelse\n  echo \"Some python examples failed:\"\n  printf \"$ERRORS\\n\"\n  #Exit with error (0-255) in case of failure in one of the tests.\n  exit 1\n\nfi\n"
  },
  {
    "path": "runtime.txt",
    "content": "3.9\n"
  },
  {
    "path": "siamese_network/README.md",
    "content": "# Siamese Network Example\nSiamese network for image similarity estimation.\nThe network is composed of two identical networks, one for each input.\nThe output of each network is concatenated and passed to a linear layer.\nThe output of the linear layer passed through a sigmoid function.\n[FaceNet](https://arxiv.org/pdf/1503.03832.pdf) is a variant of the Siamese network.\nThis implementation varies from FaceNet as we use the `ResNet-18` model from\n[Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385.pdf) as our feature extractor.\nIn addition, we aren't using `TripletLoss` as the MNIST dataset is simple, so `BCELoss` can do the trick.\n\n### Usage\n\nInstall the required dependencies:\n```bash\npip install -r requirements.txt\n```\n\nTo run the example, execute:\n```bahs\npython main.py\n# CUDA_VISIBLE_DEVICES=2 python main.py  # to specify GPU id to ex. 2\n```\n\nIf a hardware accelerator device is detected, the example will execute on the accelerator; otherwise, it will run on the CPU.\n\nTo force execution on the CPU, use `--no-accel` command line argument:\n\n```bash\npython main.py --no-accel\n```\n\nOptionally, you can add the following arguments to customize your execution.\n\n```bash\n--batch-size            input batch size for training (default: 64)\n--test-batch-size       input batch size for testing (default: 1000)\n--epochs                number of epochs to train (default: 14)\n--lr                    learning rate (default: 1.0)\n--gamma                 learning rate step gamma (default: 0.7)\n--no-accel              disables accelerator\n--dry-run               quickly check a single pass\n--seed                  random seed (default: 1)\n--log-interval          how many batches to wait before logging training status\n--save-model            Saving the current Model\n```\n"
  },
  {
    "path": "siamese_network/main.py",
    "content": "from __future__ import print_function\nimport argparse, random, copy\nimport numpy as np\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nimport torchvision\nfrom torch.utils.data import Dataset\nfrom torchvision import datasets\nfrom torchvision import transforms as T\nfrom torch.optim.lr_scheduler import StepLR\n\n\nclass SiameseNetwork(nn.Module):\n    \"\"\"\n        Siamese network for image similarity estimation.\n        The network is composed of two identical networks, one for each input.\n        The output of each network is concatenated and passed to a linear layer. \n        The output of the linear layer passed through a sigmoid function.\n        `\"FaceNet\" <https://arxiv.org/pdf/1503.03832.pdf>`_ is a variant of the Siamese network.\n        This implementation varies from FaceNet as we use the `ResNet-18` model from\n        `\"Deep Residual Learning for Image Recognition\" <https://arxiv.org/pdf/1512.03385.pdf>`_ as our feature extractor.\n        In addition, we aren't using `TripletLoss` as the MNIST dataset is simple, so `BCELoss` can do the trick.\n    \"\"\"\n    def __init__(self):\n        super(SiameseNetwork, self).__init__()\n        # get resnet model\n        self.resnet = torchvision.models.resnet18(weights=None)\n\n        # over-write the first conv layer to be able to read MNIST images\n        # as resnet18 reads (3,x,x) where 3 is RGB channels\n        # whereas MNIST has (1,x,x) where 1 is a gray-scale channel\n        self.resnet.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n        self.fc_in_features = self.resnet.fc.in_features\n        \n        # remove the last layer of resnet18 (linear layer which is before avgpool layer)\n        self.resnet = torch.nn.Sequential(*(list(self.resnet.children())[:-1]))\n\n        # add linear layers to compare between the features of the two images\n        self.fc = nn.Sequential(\n            nn.Linear(self.fc_in_features * 2, 256),\n            nn.ReLU(inplace=True),\n            nn.Linear(256, 1),\n        )\n\n        self.sigmoid = nn.Sigmoid()\n\n        # initialize the weights\n        self.resnet.apply(self.init_weights)\n        self.fc.apply(self.init_weights)\n        \n    def init_weights(self, m):\n        if isinstance(m, nn.Linear):\n            torch.nn.init.xavier_uniform_(m.weight)\n            m.bias.data.fill_(0.01)\n\n    def forward_once(self, x):\n        output = self.resnet(x)\n        output = output.view(output.size()[0], -1)\n        return output\n\n    def forward(self, input1, input2):\n        # get two images' features\n        output1 = self.forward_once(input1)\n        output2 = self.forward_once(input2)\n\n        # concatenate both images' features\n        output = torch.cat((output1, output2), 1)\n\n        # pass the concatenation to the linear layers\n        output = self.fc(output)\n\n        # pass the out of the linear layers to sigmoid layer\n        output = self.sigmoid(output)\n        \n        return output\n\nclass APP_MATCHER(Dataset):\n    def __init__(self, root, train, download=False):\n        super(APP_MATCHER, self).__init__()\n\n        # get MNIST dataset\n        self.dataset = datasets.MNIST(root, train=train, download=download)\n        \n        # as `self.dataset.data`'s shape is (Nx28x28), where N is the number of\n        # examples in MNIST dataset, a single example has the dimensions of\n        # (28x28) for (WxH), where W and H are the width and the height of the image. \n        # However, every example should have (CxWxH) dimensions where C is the number \n        # of channels to be passed to the network. As MNIST contains gray-scale images, \n        # we add an additional dimension to corresponds to the number of channels.\n        self.data = self.dataset.data.unsqueeze(1).clone()\n\n        self.group_examples()\n\n    def group_examples(self):\n        \"\"\"\n            To ease the accessibility of data based on the class, we will use `group_examples` to group \n            examples based on class. \n            \n            Every key in `grouped_examples` corresponds to a class in MNIST dataset. For every key in \n            `grouped_examples`, every value will conform to all of the indices for the MNIST \n            dataset examples that correspond to that key.\n        \"\"\"\n\n        # get the targets from MNIST dataset\n        np_arr = np.array(self.dataset.targets.clone(), dtype=None, copy=None)\n        \n        # group examples based on class\n        self.grouped_examples = {}\n        for i in range(0,10):\n            self.grouped_examples[i] = np.where((np_arr==i))[0]\n    \n    def __len__(self):\n        return self.data.shape[0]\n    \n    def __getitem__(self, index):\n        \"\"\"\n            For every example, we will select two images. There are two cases, \n            positive and negative examples. For positive examples, we will have two \n            images from the same class. For negative examples, we will have two images \n            from different classes.\n\n            Given an index, if the index is even, we will pick the second image from the same class, \n            but it won't be the same image we chose for the first class. This is used to ensure the positive\n            example isn't trivial as the network would easily distinguish the similarity between same images. However,\n            if the network were given two different images from the same class, the network will need to learn \n            the similarity between two different images representing the same class. If the index is odd, we will \n            pick the second image from a different class than the first image.\n        \"\"\"\n\n        # pick some random class for the first image\n        selected_class = random.randint(0, 9)\n\n        # pick a random index for the first image in the grouped indices based of the label\n        # of the class\n        random_index_1 = random.randint(0, self.grouped_examples[selected_class].shape[0]-1)\n        \n        # pick the index to get the first image\n        index_1 = self.grouped_examples[selected_class][random_index_1]\n\n        # get the first image\n        image_1 = self.data[index_1].clone().float()\n\n        # same class\n        if index % 2 == 0:\n            # pick a random index for the second image\n            random_index_2 = random.randint(0, self.grouped_examples[selected_class].shape[0]-1)\n            \n            # ensure that the index of the second image isn't the same as the first image\n            while random_index_2 == random_index_1:\n                random_index_2 = random.randint(0, self.grouped_examples[selected_class].shape[0]-1)\n            \n            # pick the index to get the second image\n            index_2 = self.grouped_examples[selected_class][random_index_2]\n\n            # get the second image\n            image_2 = self.data[index_2].clone().float()\n\n            # set the label for this example to be positive (1)\n            target = torch.tensor(1, dtype=torch.float)\n        \n        # different class\n        else:\n            # pick a random class\n            other_selected_class = random.randint(0, 9)\n\n            # ensure that the class of the second image isn't the same as the first image\n            while other_selected_class == selected_class:\n                other_selected_class = random.randint(0, 9)\n\n            \n            # pick a random index for the second image in the grouped indices based of the label\n            # of the class\n            random_index_2 = random.randint(0, self.grouped_examples[other_selected_class].shape[0]-1)\n\n            # pick the index to get the second image\n            index_2 = self.grouped_examples[other_selected_class][random_index_2]\n\n            # get the second image\n            image_2 = self.data[index_2].clone().float()\n\n            # set the label for this example to be negative (0)\n            target = torch.tensor(0, dtype=torch.float)\n\n        return image_1, image_2, target\n\n\ndef train(args, model, device, train_loader, optimizer, epoch):\n    model.train()\n\n    # we aren't using `TripletLoss` as the MNIST dataset is simple, so `BCELoss` can do the trick.\n    criterion = nn.BCELoss()\n\n    for batch_idx, (images_1, images_2, targets) in enumerate(train_loader):\n        images_1, images_2, targets = images_1.to(device), images_2.to(device), targets.to(device)\n        optimizer.zero_grad()\n        outputs = model(images_1, images_2).squeeze()\n        loss = criterion(outputs, targets)\n        loss.backward()\n        optimizer.step()\n        if batch_idx % args.log_interval == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, batch_idx * len(images_1), len(train_loader.dataset),\n                100. * batch_idx / len(train_loader), loss.item()))\n            if args.dry_run:\n                break\n\n\ndef test(model, device, test_loader):\n    model.eval()\n    test_loss = 0\n    correct = 0\n\n    # we aren't using `TripletLoss` as the MNIST dataset is simple, so `BCELoss` can do the trick.\n    criterion = nn.BCELoss()\n\n    with torch.no_grad():\n        for (images_1, images_2, targets) in test_loader:\n            images_1, images_2, targets = images_1.to(device), images_2.to(device), targets.to(device)\n            outputs = model(images_1, images_2).squeeze()\n            test_loss += criterion(outputs, targets).sum().item()  # sum up batch loss\n            pred = torch.where(outputs > 0.5, 1, 0)  # get the index of the max log-probability\n            correct += pred.eq(targets.view_as(pred)).sum().item()\n\n    test_loss /= len(test_loader.dataset)\n\n    # for the 1st epoch, the average loss is 0.0001 and the accuracy 97-98%\n    # using default settings. After completing the 10th epoch, the average\n    # loss is 0.0000 and the accuracy 99.5-100% using default settings.\n    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n        test_loss, correct, len(test_loader.dataset),\n        100. * correct / len(test_loader.dataset)))\n\n\ndef main():\n    # Training settings\n    parser = argparse.ArgumentParser(description='PyTorch Siamese network Example')\n    parser.add_argument('--batch-size', type=int, default=64, metavar='N',\n                        help='input batch size for training (default: 64)')\n    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n                        help='input batch size for testing (default: 1000)')\n    parser.add_argument('--epochs', type=int, default=14, metavar='N',\n                        help='number of epochs to train (default: 14)')\n    parser.add_argument('--lr', type=float, default=1.0, metavar='LR',\n                        help='learning rate (default: 1.0)')\n    parser.add_argument('--gamma', type=float, default=0.7, metavar='M',\n                        help='Learning rate step gamma (default: 0.7)')\n    parser.add_argument('--no-accel', action='store_true', \n                    help='disables accelerator')\n    parser.add_argument('--dry-run', action='store_true', default=False,\n                        help='quickly check a single pass')\n    parser.add_argument('--seed', type=int, default=1, metavar='S',\n                        help='random seed (default: 1)')\n    parser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                        help='how many batches to wait before logging training status')\n    parser.add_argument('--save-model', action='store_true', default=False,\n                        help='For Saving the current Model')\n    args = parser.parse_args()\n    \n    use_accel = not args.no_accel and torch.accelerator.is_available()\n\n    torch.manual_seed(args.seed)\n\n\n    if use_accel:\n        device = torch.accelerator.current_accelerator()\n    else:\n        device = torch.device(\"cpu\")\n    \n    print(f\"Using device: {device}\")\n\n    train_kwargs = {'batch_size': args.batch_size}\n    test_kwargs = {'batch_size': args.test_batch_size}\n    if use_accel:\n        accel_kwargs = {'num_workers': 1,\n                       'pin_memory': True,\n                       'shuffle': True}\n        train_kwargs.update(accel_kwargs)\n        test_kwargs.update(accel_kwargs)\n\n    train_dataset = APP_MATCHER('../data', train=True, download=True)\n    test_dataset = APP_MATCHER('../data', train=False)\n    train_loader = torch.utils.data.DataLoader(train_dataset,**train_kwargs)\n    test_loader = torch.utils.data.DataLoader(test_dataset, **test_kwargs)\n\n    model = SiameseNetwork().to(device)\n    optimizer = optim.Adadelta(model.parameters(), lr=args.lr)\n\n    scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)\n    for epoch in range(1, args.epochs + 1):\n        train(args, model, device, train_loader, optimizer, epoch)\n        test(model, device, test_loader)\n        scheduler.step()\n\n    if args.save_model:\n        torch.save(model.state_dict(), \"siamese_network.pt\")\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "siamese_network/requirements.txt",
    "content": "torch\ntorchvision\n"
  },
  {
    "path": "super_resolution/README.md",
    "content": "# Superresolution using an efficient sub-pixel convolutional neural network\n\nThis example illustrates how to use the efficient sub-pixel convolution layer described in [\"Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network\" - Shi et al.](https://arxiv.org/abs/1609.05158) for increasing spatial resolution within your network for tasks such as superresolution.\n\n```\nusage: main.py [-h] --upscale_factor UPSCALE_FACTOR [--batchSize BATCHSIZE]\n               [--testBatchSize TESTBATCHSIZE] [--nEpochs NEPOCHS] [--lr LR]\n               [--accel] [--threads THREADS] [--seed SEED]\n\nPyTorch Super Res Example\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --upscale_factor      super resolution upscale factor\n  --batchSize           training batch size\n  --testBatchSize       testing batch size\n  --nEpochs             number of epochs to train for\n  --lr                  Learning Rate. Default=0.01\n  --accel                use accelerator\n  --threads             number of threads for data loader to use Default=4\n  --seed                random seed to use. Default=123\n```\n\nThis example trains a super-resolution network on the [BSD300 dataset](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/), using crops from the 200 training images, and evaluating on crops of the 100 test images. A snapshot of the model after every epoch with filename `model_epoch_<epoch_number>.pth`.\n\n## Example Usage:\n\n### Train\n\n```bash\npython main.py --upscale_factor 3 --batchSize 4 --testBatchSize 100 --nEpochs 30 --lr 0.001 --accel\n```\n\n### Super Resolve\n\n```bash\npython super_resolve.py --input_image dataset/BSDS300/images/test/16077.jpg --model model_epoch_30.pth --output_filename out.png --accel\n```\n"
  },
  {
    "path": "super_resolution/data.py",
    "content": "from os.path import exists, join, basename\nfrom os import makedirs, remove\nfrom six.moves import urllib\nimport tarfile\nfrom torchvision.transforms import Compose, CenterCrop, ToTensor, Resize\n\nfrom dataset import DatasetFromFolder\n\n\ndef download_bsd300(dest=\"dataset\"):\n    output_image_dir = join(dest, \"BSDS300/images\")\n\n    if not exists(output_image_dir):\n        makedirs(dest)\n        url = \"http://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/BSDS300-images.tgz\"\n        print(\"downloading url \", url)\n\n        data = urllib.request.urlopen(url)\n\n        file_path = join(dest, basename(url))\n        with open(file_path, 'wb') as f:\n            f.write(data.read())\n\n        print(\"Extracting data\")\n        with tarfile.open(file_path) as tar:\n            for item in tar:\n                tar.extract(item, dest)\n\n        remove(file_path)\n\n    return output_image_dir\n\n\ndef calculate_valid_crop_size(crop_size, upscale_factor):\n    return crop_size - (crop_size % upscale_factor)\n\n\ndef input_transform(crop_size, upscale_factor):\n    return Compose([\n        CenterCrop(crop_size),\n        Resize(crop_size // upscale_factor),\n        ToTensor(),\n    ])\n\n\ndef target_transform(crop_size):\n    return Compose([\n        CenterCrop(crop_size),\n        ToTensor(),\n    ])\n\n\ndef get_training_set(upscale_factor):\n    root_dir = download_bsd300()\n    train_dir = join(root_dir, \"train\")\n    crop_size = calculate_valid_crop_size(256, upscale_factor)\n\n    return DatasetFromFolder(train_dir,\n                             input_transform=input_transform(crop_size, upscale_factor),\n                             target_transform=target_transform(crop_size))\n\n\ndef get_test_set(upscale_factor):\n    root_dir = download_bsd300()\n    test_dir = join(root_dir, \"test\")\n    crop_size = calculate_valid_crop_size(256, upscale_factor)\n\n    return DatasetFromFolder(test_dir,\n                             input_transform=input_transform(crop_size, upscale_factor),\n                             target_transform=target_transform(crop_size))\n"
  },
  {
    "path": "super_resolution/dataset.py",
    "content": "import torch.utils.data as data\n\nfrom os import listdir\nfrom os.path import join\nfrom PIL import Image\n\n\ndef is_image_file(filename):\n    return any(filename.endswith(extension) for extension in [\".png\", \".jpg\", \".jpeg\"])\n\n\ndef load_img(filepath):\n    img = Image.open(filepath).convert('YCbCr')\n    y, _, _ = img.split()\n    return y\n\n\nclass DatasetFromFolder(data.Dataset):\n    def __init__(self, image_dir, input_transform=None, target_transform=None):\n        super(DatasetFromFolder, self).__init__()\n        self.image_filenames = [join(image_dir, x) for x in listdir(image_dir) if is_image_file(x)]\n\n        self.input_transform = input_transform\n        self.target_transform = target_transform\n\n    def __getitem__(self, index):\n        input = load_img(self.image_filenames[index])\n        target = input.copy()\n        if self.input_transform:\n            input = self.input_transform(input)\n        if self.target_transform:\n            target = self.target_transform(target)\n\n        return input, target\n\n    def __len__(self):\n        return len(self.image_filenames)\n"
  },
  {
    "path": "super_resolution/main.py",
    "content": "from __future__ import print_function\nimport argparse\nfrom math import log10\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader\nfrom model import Net\nfrom data import get_training_set, get_test_set\n\n# Training settings\nparser = argparse.ArgumentParser(description='PyTorch Super Res Example')\nparser.add_argument('--upscale_factor', type=int, required=True, help=\"super resolution upscale factor\")\nparser.add_argument('--batchSize', type=int, default=64, help='training batch size')\nparser.add_argument('--testBatchSize', type=int, default=10, help='testing batch size')\nparser.add_argument('--nEpochs', type=int, default=2, help='number of epochs to train for')\nparser.add_argument('--lr', type=float, default=0.01, help='Learning Rate. Default=0.01')\nparser.add_argument('--accel', action='store_true', help='Enables acceleration for training, if available')\nparser.add_argument('--threads', type=int, default=4, help='number of threads for data loader to use')\nparser.add_argument('--seed', type=int, default=123, help='random seed to use. Default=123')\nopt = parser.parse_args()\n\nprint(opt)\n\ntorch.manual_seed(opt.seed)\n\n\nif opt.accel and torch.accelerator.is_available():\n    device = torch.accelerator.current_accelerator()\nelse:\n    device = torch.device(\"cpu\")\n\nprint('===> Loading datasets')\ntrain_set = get_training_set(opt.upscale_factor)\ntest_set = get_test_set(opt.upscale_factor)\ntraining_data_loader = DataLoader(dataset=train_set, num_workers=opt.threads, batch_size=opt.batchSize, shuffle=True)\ntesting_data_loader = DataLoader(dataset=test_set, num_workers=opt.threads, batch_size=opt.testBatchSize, shuffle=False)\n\nprint('===> Building model')\nmodel = Net(upscale_factor=opt.upscale_factor).to(device)\ncriterion = nn.MSELoss()\n\noptimizer = optim.Adam(model.parameters(), lr=opt.lr)\n\n\ndef train(epoch):\n    epoch_loss = 0\n    for iteration, batch in enumerate(training_data_loader, 1):\n        input, target = batch[0].to(device), batch[1].to(device)\n\n        optimizer.zero_grad()\n        loss = criterion(model(input), target)\n        epoch_loss += loss.item()\n        loss.backward()\n        optimizer.step()\n\n        print(\"===> Epoch[{}]({}/{}): Loss: {:.4f}\".format(epoch, iteration, len(training_data_loader), loss.item()))\n\n    print(\"===> Epoch {} Complete: Avg. Loss: {:.4f}\".format(epoch, epoch_loss / len(training_data_loader)))\n\n\ndef test():\n    avg_psnr = 0\n    with torch.no_grad():\n        for batch in testing_data_loader:\n            input, target = batch[0].to(device), batch[1].to(device)\n\n            prediction = model(input)\n            mse = criterion(prediction, target)\n            psnr = 10 * log10(1 / mse.item())\n            avg_psnr += psnr\n    print(\"===> Avg. PSNR: {:.4f} dB\".format(avg_psnr / len(testing_data_loader)))\n\n\ndef checkpoint(epoch):\n    model_out_path = \"model_epoch_{}.pth\".format(epoch)\n    torch.save(model, model_out_path)\n    print(\"Checkpoint saved to {}\".format(model_out_path))\n\nfor epoch in range(1, opt.nEpochs + 1):\n    train(epoch)\n    test()\n    checkpoint(epoch)\n"
  },
  {
    "path": "super_resolution/model.py",
    "content": "import torch\nimport torch.nn as nn\nimport torch.nn.init as init\n\n\nclass Net(nn.Module):\n    def __init__(self, upscale_factor):\n        super(Net, self).__init__()\n\n        self.relu = nn.ReLU()\n        self.conv1 = nn.Conv2d(1, 64, (5, 5), (1, 1), (2, 2))\n        self.conv2 = nn.Conv2d(64, 64, (3, 3), (1, 1), (1, 1))\n        self.conv3 = nn.Conv2d(64, 32, (3, 3), (1, 1), (1, 1))\n        self.conv4 = nn.Conv2d(32, upscale_factor ** 2, (3, 3), (1, 1), (1, 1))\n        self.pixel_shuffle = nn.PixelShuffle(upscale_factor)\n\n        self._initialize_weights()\n\n    def forward(self, x):\n        x = self.relu(self.conv1(x))\n        x = self.relu(self.conv2(x))\n        x = self.relu(self.conv3(x))\n        x = self.pixel_shuffle(self.conv4(x))\n        return x\n\n    def _initialize_weights(self):\n        init.orthogonal_(self.conv1.weight, init.calculate_gain('relu'))\n        init.orthogonal_(self.conv2.weight, init.calculate_gain('relu'))\n        init.orthogonal_(self.conv3.weight, init.calculate_gain('relu'))\n        init.orthogonal_(self.conv4.weight)\n"
  },
  {
    "path": "super_resolution/requirements.txt",
    "content": "six\ntorch\ntorchvision\n"
  },
  {
    "path": "super_resolution/super_resolve.py",
    "content": "from __future__ import print_function\nimport argparse\nimport torch\nfrom PIL import Image\nfrom torchvision.transforms import ToTensor\nfrom model import Net\n\nimport numpy as np\n\n# Training settings\nparser = argparse.ArgumentParser(description='PyTorch Super Res Example')\nparser.add_argument('--input_image', type=str, required=True, help='input image to use')\nparser.add_argument('--model', type=str, required=True, help='model file to use')\nparser.add_argument('--output_filename', type=str, help='where to save the output image')\nparser.add_argument('--accel', action='store_true', help='Enables acceleration device, if available')\nopt = parser.parse_args()\n\nprint(opt)\nimg = Image.open(opt.input_image).convert('YCbCr')\ny, cb, cr = img.split()\n\nwith open(opt.model, 'rb') as f:\n    safe_globals = [\n        Net,\n        torch.nn.modules.activation.ReLU,\n        torch.nn.modules.conv.Conv2d,\n        torch.nn.modules.pixelshuffle.PixelShuffle,\n    ]\n    with torch.serialization.safe_globals(safe_globals):\n        model = torch.load(f)\n\nimg_to_tensor = ToTensor()\ninput = img_to_tensor(y).view(1, -1, y.size[1], y.size[0])\n\nif opt.accel:\n    device = torch.accelerator.current_accelerator()\n    model = model.to(device)\n    input = input.to(device)\n\nout = model(input)\nout = out.cpu()\nout_img_y = out[0].detach().numpy()\nout_img_y *= 255.0\nout_img_y = out_img_y.clip(0, 255)\nout_img_y = Image.fromarray(np.uint8(out_img_y[0]), mode='L')\n\nout_img_cb = cb.resize(out_img_y.size, Image.BICUBIC)\nout_img_cr = cr.resize(out_img_y.size, Image.BICUBIC)\nout_img = Image.merge('YCbCr', [out_img_y, out_img_cb, out_img_cr]).convert('RGB')\n\nout_img.save(opt.output_filename)\nprint('output image saved to ', opt.output_filename)\n"
  },
  {
    "path": "time_sequence_prediction/README.md",
    "content": "# Time Sequence Prediction\n\nThis is a toy example for beginners to start with. It helps learn both PyTorch and time sequence prediction. Two LSTMCell units are used in this example to learn some sine wave signals starting at different phases. After learning the sine waves, the network tries to predict the signal values in the future. The results are shown in the picture below.\n\n## Usage\n\n```\npython generate_sine_wave.py\npython train.py\n```\n\n## Result\n\nThe initial signal and the predicted results are shown in the image. We first give some initial signals (full line). The network will subsequently give some predicted results (dash line). It can be concluded that the network can generate new sine waves.\n![image](https://cloud.githubusercontent.com/assets/1419566/24184438/e24f5280-0f08-11e7-8f8b-4d972b527a81.png)\n"
  },
  {
    "path": "time_sequence_prediction/generate_sine_wave.py",
    "content": "import numpy as np\nimport torch\n\nnp.random.seed(2)\n\nT = 20\nL = 1000\nN = 100\n\nx = np.empty((N, L), 'int64')\nx[:] = np.array(range(L)) + np.random.randint(-4 * T, 4 * T, N).reshape(N, 1)\ndata = np.sin(x / 1.0 / T).astype('float64')\ntorch.save(data, open('traindata.pt', 'wb'))\n"
  },
  {
    "path": "time_sequence_prediction/requirements.txt",
    "content": "torch<2.6\nmatplotlib\n"
  },
  {
    "path": "time_sequence_prediction/train.py",
    "content": "from __future__ import print_function\nimport argparse\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport numpy as np\nimport matplotlib\nmatplotlib.use('Agg')\nimport matplotlib.pyplot as plt\n\nclass Sequence(nn.Module):\n    def __init__(self):\n        super(Sequence, self).__init__()\n        self.lstm1 = nn.LSTMCell(1, 51)\n        self.lstm2 = nn.LSTMCell(51, 51)\n        self.linear = nn.Linear(51, 1)\n\n    def forward(self, input, future = 0):\n        outputs = []\n        h_t = torch.zeros(input.size(0), 51, dtype=torch.double)\n        c_t = torch.zeros(input.size(0), 51, dtype=torch.double)\n        h_t2 = torch.zeros(input.size(0), 51, dtype=torch.double)\n        c_t2 = torch.zeros(input.size(0), 51, dtype=torch.double)\n\n        for input_t in input.split(1, dim=1):\n            h_t, c_t = self.lstm1(input_t, (h_t, c_t))\n            h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))\n            output = self.linear(h_t2)\n            outputs += [output]\n        for i in range(future):# if we should predict the future\n            h_t, c_t = self.lstm1(output, (h_t, c_t))\n            h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))\n            output = self.linear(h_t2)\n            outputs += [output]\n        outputs = torch.cat(outputs, dim=1)\n        return outputs\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--steps', type=int, default=15, help='steps to run')\n    opt = parser.parse_args()\n    # set random seed to 0\n    np.random.seed(0)\n    torch.manual_seed(0)\n    # load data and make training set\n    data = torch.load('traindata.pt')\n    input = torch.from_numpy(data[3:, :-1])\n    target = torch.from_numpy(data[3:, 1:])\n    test_input = torch.from_numpy(data[:3, :-1])\n    test_target = torch.from_numpy(data[:3, 1:])\n    # build the model\n    seq = Sequence()\n    seq.double()\n    criterion = nn.MSELoss()\n    # use LBFGS as optimizer since we can load the whole data to train\n    optimizer = optim.LBFGS(seq.parameters(), lr=0.8)\n    #begin to train\n    for i in range(opt.steps):\n        print('STEP: ', i)\n        def closure():\n            optimizer.zero_grad()\n            out = seq(input)\n            loss = criterion(out, target)\n            print('loss:', loss.item())\n            loss.backward()\n            return loss\n        optimizer.step(closure)\n        # begin to predict, no need to track gradient here\n        with torch.no_grad():\n            future = 1000\n            pred = seq(test_input, future=future)\n            loss = criterion(pred[:, :-future], test_target)\n            print('test loss:', loss.item())\n            y = pred.detach().numpy()\n        # draw the result\n        plt.figure(figsize=(30,10))\n        plt.title('Predict future values for time sequences\\n(Dashlines are predicted values)', fontsize=30)\n        plt.xlabel('x', fontsize=20)\n        plt.ylabel('y', fontsize=20)\n        plt.xticks(fontsize=20)\n        plt.yticks(fontsize=20)\n        def draw(yi, color):\n            plt.plot(np.arange(input.size(1)), yi[:input.size(1)], color, linewidth = 2.0)\n            plt.plot(np.arange(input.size(1), input.size(1) + future), yi[input.size(1):], color + ':', linewidth = 2.0)\n        draw(y[0], 'r')\n        draw(y[1], 'g')\n        draw(y[2], 'b')\n        plt.savefig('predict%d.pdf'%i)\n        plt.close()\n"
  },
  {
    "path": "utils.sh",
    "content": "#!/usr/bin/env bash\n# This script contains utility functions and initialize exmaple scripts.\n# Eg: run_python_examples.sh, run_distributed_examples.sh\n\nBASE_DIR=\"$(pwd)/$(dirname $0)\"\nEXAMPLES=$(echo $1 | sed -e 's/ //g')\n\n# Redirect 'python' calls to 'python3'\npython() {\n    command python3 \"$@\"\n}\n\nERRORS=${ERRORS-\"\"}\n\nfunction error() {\n  ERR=$1\n  if [ \"\" == \"$ERRORS\" ]; then\n    ERRORS=\"$ERR\"\n  else\n    ERRORS=\"$ERRORS\\n$ERR\"\n  fi\n}\n\nfunction start() {\n  EXAMPLE=$1\n  cd $BASE_DIR/$EXAMPLE || { error \"$EXAMPLE: no such example\"; return 1; }\n  echo \"Install dependencies for $EXAMPLE\"\n  # Setting VIRTUAL_ENV=.venv externally will create uv virtual environment\n  # for each sample in start() and remove it in stop(). Note that this environment\n  # variable also forces other uv commands such as `uv pip...` and `uv run...` to\n  # use the specified environment.\n  if [ \"$VIRTUAL_ENV\" = \".venv\" ]; then\n    uv venv || { error \"$EXAMPLE: failed to create virtual environment\"; return 1; }\n  fi\n  uv pip install -r requirements.txt $PIP_INSTALL_ARGS || { error \"$EXAMPLE: failed to install requirements\"; return 1; }\n  echo \"Running example: $EXAMPLE\"\n}\n\nfunction stop() {\n  EXAMPLE=$1\n  if [ \"$VIRTUAL_ENV\" = \".venv\" ]; then\n    cd $BASE_DIR/$EXAMPLE && rm -rf .venv\n  fi\n}\n\nfunction run() {\n  EXAMPLE=$1\n  if start $EXAMPLE; then\n    # drop trailing slash (occurs due to auto completion in bash interactive mode)\n    # replace slashes with underscores: this allows to call nested examples\n    EXAMPLE_FN=$(echo $EXAMPLE | sed \"s@/\\$@@\" | sed 's@/@_@g')\n    $EXAMPLE_FN\n  fi\n  stop $EXAMPLE\n}\n"
  },
  {
    "path": "vae/README.md",
    "content": "# Basic VAE Example\n\nThis is an improved implementation of the paper [Auto-Encoding Variational Bayes](http://arxiv.org/abs/1312.6114) by Kingma and Welling.\nIt uses ReLUs and the adam optimizer, instead of sigmoids and adagrad. These changes make the network converge much faster.\n\n### Usage\nInstall the required dependencies:\n```bash\npip install -r requirements.txt\n```\n\nTo run the example, execute:\n```bash\npython main.py\n```\n\nIf a hardware accelerator device is detected, the example will execute on the accelerator; otherwise, it will run on the CPU.\n\nTo force execution on the CPU, use `--no-accel` command line argument:\n\n```bash\npython main.py --no-accel\n```\n\nThe main.py script accepts the following optional arguments:\n\n```bash\n--batch-size            input batch size for training (default: 128)\n--epochs                number of epochs to train (default: 10)\n--no-accel              disables accelerator\n--seed                  random seed (default: 1)\n--log-interval\t        how many batches to wait before logging training status\n```\n\n"
  },
  {
    "path": "vae/main.py",
    "content": "from __future__ import print_function\nimport argparse\nimport torch\nimport torch.utils.data\nfrom torch import nn, optim\nfrom torch.nn import functional as F\nfrom torchvision import datasets, transforms\nfrom torchvision.utils import save_image\n\n\nparser = argparse.ArgumentParser(description='VAE MNIST Example')\nparser.add_argument('--batch-size', type=int, default=128, metavar='N',\n                    help='input batch size for training (default: 128)')\nparser.add_argument('--epochs', type=int, default=10, metavar='N',\n                    help='number of epochs to train (default: 10)')\nparser.add_argument('--no-accel', action='store_true', \n                    help='disables accelerator')\nparser.add_argument('--seed', type=int, default=1, metavar='S',\n                    help='random seed (default: 1)')\nparser.add_argument('--log-interval', type=int, default=10, metavar='N',\n                    help='how many batches to wait before logging training status')\nargs = parser.parse_args()\n\nuse_accel = not args.no_accel and torch.accelerator.is_available()\n\ntorch.manual_seed(args.seed)\n\n\nif use_accel:\n    device = torch.accelerator.current_accelerator()\nelse:\n    device = torch.device(\"cpu\")\n\nprint(f\"Using device: {device}\")\n\nkwargs = {'num_workers': 1, 'pin_memory': True} if use_accel else {}\ntrain_loader = torch.utils.data.DataLoader(\n    datasets.MNIST('../data', train=True, download=True,\n                   transform=transforms.ToTensor()),\n    batch_size=args.batch_size, shuffle=True, **kwargs)\ntest_loader = torch.utils.data.DataLoader(\n    datasets.MNIST('../data', train=False, transform=transforms.ToTensor()),\n    batch_size=args.batch_size, shuffle=False, **kwargs)\n\n\nclass VAE(nn.Module):\n    def __init__(self):\n        super(VAE, self).__init__()\n\n        self.fc1 = nn.Linear(784, 400)\n        self.fc21 = nn.Linear(400, 20)\n        self.fc22 = nn.Linear(400, 20)\n        self.fc3 = nn.Linear(20, 400)\n        self.fc4 = nn.Linear(400, 784)\n\n    def encode(self, x):\n        h1 = F.relu(self.fc1(x))\n        return self.fc21(h1), self.fc22(h1)\n\n    def reparameterize(self, mu, logvar):\n        std = torch.exp(0.5*logvar)\n        eps = torch.randn_like(std)\n        return mu + eps*std\n\n    def decode(self, z):\n        h3 = F.relu(self.fc3(z))\n        return torch.sigmoid(self.fc4(h3))\n\n    def forward(self, x):\n        mu, logvar = self.encode(x.view(-1, 784))\n        z = self.reparameterize(mu, logvar)\n        return self.decode(z), mu, logvar\n\n\nmodel = VAE().to(device)\noptimizer = optim.Adam(model.parameters(), lr=1e-3)\n\n\n# Reconstruction + KL divergence losses summed over all elements and batch\ndef loss_function(recon_x, x, mu, logvar):\n    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')\n\n    # see Appendix B from VAE paper:\n    # Kingma and Welling. Auto-Encoding Variational Bayes. ICLR, 2014\n    # https://arxiv.org/abs/1312.6114\n    # 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)\n    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())\n\n    return BCE + KLD\n\n\ndef train(epoch):\n    model.train()\n    train_loss = 0\n    for batch_idx, (data, _) in enumerate(train_loader):\n        data = data.to(device)\n        optimizer.zero_grad()\n        recon_batch, mu, logvar = model(data)\n        loss = loss_function(recon_batch, data, mu, logvar)\n        loss.backward()\n        train_loss += loss.item()\n        optimizer.step()\n        if batch_idx % args.log_interval == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, batch_idx * len(data), len(train_loader.dataset),\n                100. * batch_idx / len(train_loader),\n                loss.item() / len(data)))\n\n    print('====> Epoch: {} Average loss: {:.4f}'.format(\n          epoch, train_loss / len(train_loader.dataset)))\n\n\ndef test(epoch):\n    model.eval()\n    test_loss = 0\n    with torch.no_grad():\n        for i, (data, _) in enumerate(test_loader):\n            data = data.to(device)\n            recon_batch, mu, logvar = model(data)\n            test_loss += loss_function(recon_batch, data, mu, logvar).item()\n            if i == 0:\n                n = min(data.size(0), 8)\n                comparison = torch.cat([data[:n],\n                                      recon_batch.view(args.batch_size, 1, 28, 28)[:n]])\n                save_image(comparison.cpu(),\n                         'results/reconstruction_' + str(epoch) + '.png', nrow=n)\n\n    test_loss /= len(test_loader.dataset)\n    print('====> Test set loss: {:.4f}'.format(test_loss))\n\nif __name__ == \"__main__\":\n    for epoch in range(1, args.epochs + 1):\n        train(epoch)\n        test(epoch)\n        with torch.no_grad():\n            sample = torch.randn(64, 20).to(device)\n            sample = model.decode(sample).cpu()\n            save_image(sample.view(64, 1, 28, 28),\n                       'results/sample_' + str(epoch) + '.png')\n"
  },
  {
    "path": "vae/requirements.txt",
    "content": "torch\ntorchvision\ntqdm\nsix\n"
  },
  {
    "path": "vae/results/.gitignore",
    "content": "*.png\n"
  },
  {
    "path": "word_language_model/README.md",
    "content": "# Word-level Language Modeling using RNN and Transformer\n\nThis example trains a multi-layer RNN (Elman, GRU, or LSTM) or Transformer on a language modeling task. By default, the training script uses the Wikitext-2 dataset, provided.\nThe trained model can then be used by the generate script to generate new text.\n\n```bash\npython main.py --accel --epochs 6           # Train a LSTM on Wikitext-2.\npython main.py --accel --epochs 6 --tied    # Train a tied LSTM on Wikitext-2.\npython main.py --accel --tied               # Train a tied LSTM on Wikitext-2for 40 epochs.\npython main.py --accel --epochs 6 --model Transformer --lr 5\n                                            # Train a Transformer model on Wikitext-2.\npython main.py --accel --epochs 6 --model Transformer --use-optimizer --lr 0.001\n                                            # Train a Transformer model with AdamW optimizer on Wikitext-2.\n\npython generate.py --accel                  # Generate samples from the default model checkpoint.\n```\n\n> [!NOTE] \n> Example supports running on acceleration devices (CUDA, MPS, XPU)\n\nThe model uses the `nn.RNN` module (and its sister modules `nn.GRU` and `nn.LSTM`) or Transformer module (`nn.TransformerEncoder` and `nn.TransformerEncoderLayer`) which will automatically use the cuDNN backend if run on CUDA with cuDNN installed.\n\nDuring training, if a keyboard interrupt (Ctrl-C) is received, training is stopped and the current model is evaluated against the test dataset.\n\nThe `main.py` script accepts the following arguments:\n\n```bash\noptional arguments:\n  -h, --help            show this help message and exit\n  --data DATA           location of the data corpus\n  --model MODEL         type of network (RNN_TANH, RNN_RELU, LSTM, GRU, Transformer)\n  --emsize EMSIZE       size of word embeddings\n  --nhid NHID           number of hidden units per layer\n  --nlayers NLAYERS     number of layers\n  --lr LR               initial learning rate\n  --clip CLIP           gradient clipping\n  --epochs EPOCHS       upper epoch limit\n  --batch_size N        batch size\n  --bptt BPTT           sequence length\n  --dropout DROPOUT     dropout applied to layers (0 = no dropout)\n  --tied                tie the word embedding and softmax weights\n  --seed SEED           random seed\n  --accel               use accelerator\n  --log-interval N      report interval\n  --save SAVE           path to save the final model\n  --onnx-export ONNX_EXPORT\n                        path to export the final model in onnx format\n  --nhead NHEAD         the number of heads in the encoder/decoder of the transformer model\n  --dry-run             verify the code and the model\n  --use-optimizer       specify whether to use an AdamW optimizer\n```\n\nWith these arguments, a variety of models can be tested.\nAs an example, the following arguments produce slower but better models:\n\n```bash\npython main.py --accel --emsize 650 --nhid 650 --dropout 0.5 --epochs 40\npython main.py --accel --emsize 650 --nhid 650 --dropout 0.5 --epochs 40 --tied\npython main.py --accel --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 40\npython main.py --accel --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 40 --tied\n```\n"
  },
  {
    "path": "word_language_model/data.py",
    "content": "import os\nfrom io import open\nimport torch\n\nclass Dictionary(object):\n    def __init__(self):\n        self.word2idx = {}\n        self.idx2word = []\n\n    def add_word(self, word):\n        if word not in self.word2idx:\n            self.idx2word.append(word)\n            self.word2idx[word] = len(self.idx2word) - 1\n        return self.word2idx[word]\n\n    def __len__(self):\n        return len(self.idx2word)\n\n\nclass Corpus(object):\n    def __init__(self, path):\n        self.dictionary = Dictionary()\n        self.train = self.tokenize(os.path.join(path, 'train.txt'))\n        self.valid = self.tokenize(os.path.join(path, 'valid.txt'))\n        self.test = self.tokenize(os.path.join(path, 'test.txt'))\n\n    def tokenize(self, path):\n        \"\"\"Tokenizes a text file.\"\"\"\n        assert os.path.exists(path)\n        # Add words to the dictionary\n        with open(path, 'r', encoding=\"utf8\") as f:\n            for line in f:\n                words = line.split() + ['<eos>']\n                for word in words:\n                    self.dictionary.add_word(word)\n\n        # Tokenize file content\n        with open(path, 'r', encoding=\"utf8\") as f:\n            idss = []\n            for line in f:\n                words = line.split() + ['<eos>']\n                ids = []\n                for word in words:\n                    ids.append(self.dictionary.word2idx[word])\n                idss.append(torch.tensor(ids).type(torch.int64))\n            ids = torch.cat(idss)\n\n        return ids\n"
  },
  {
    "path": "word_language_model/generate.py",
    "content": "###############################################################################\n# Language Modeling on Wikitext-2\n#\n# This file generates new sentences sampled from the language model.\n#\n###############################################################################\nimport argparse\nimport torch\n\nimport data\nfrom model import PositionalEncoding, RNNModel, TransformerModel\n\nparser = argparse.ArgumentParser(description='PyTorch Wikitext-2 Language Model')\n# Model parameters.\nparser.add_argument('--data', type=str, default='./data/wikitext-2',\n                    help='location of the data corpus')\nparser.add_argument('--checkpoint', type=str, default='./model.pt',\n                    help='model checkpoint to use')\nparser.add_argument('--outf', type=str, default='generated.txt',\n                    help='output file for generated text')\nparser.add_argument('--words', type=int, default='1000',\n                    help='number of words to generate')\nparser.add_argument('--seed', type=int, default=1111,\n                    help='random seed')\nparser.add_argument('--temperature', type=float, default=1.0,\n                    help='temperature - higher will increase diversity')\nparser.add_argument('--log-interval', type=int, default=100,\n                    help='reporting interval')\nparser.add_argument('--accel', action='store_true', default=False,\n                    help='use accelerator')\nargs = parser.parse_args()\n\n# Set the random seed manually for reproducibility.\ntorch.manual_seed(args.seed)\n\nif args.accel and torch.accelerator.is_available():\n    device = torch.accelerator.current_accelerator()\n\nelse:\n    device = torch.device(\"cpu\")\n\nif args.temperature < 1e-3:\n    parser.error(\"--temperature has to be greater or equal 1e-3.\")\n\nwith open(args.checkpoint, 'rb') as f:\n    safe_globals = [\n        PositionalEncoding,\n        RNNModel,\n        TransformerModel,\n        torch.nn.functional.relu,\n        torch.nn.modules.activation.MultiheadAttention,\n        torch.nn.modules.container.ModuleList,\n        torch.nn.modules.dropout.Dropout,\n        torch.nn.modules.linear.Linear,\n        torch.nn.modules.linear.NonDynamicallyQuantizableLinear,\n        torch.nn.modules.normalization.LayerNorm,\n        torch.nn.modules.sparse.Embedding,\n        torch.nn.modules.rnn.GRU,\n        torch.nn.modules.rnn.LSTM,\n        torch.nn.modules.rnn.RNN,\n        torch.nn.modules.transformer.TransformerEncoder,\n        torch.nn.modules.transformer.TransformerEncoderLayer,\n    ]\n    with torch.serialization.safe_globals(safe_globals):\n        model = torch.load(f, map_location=device)\nmodel.eval()\n\ncorpus = data.Corpus(args.data)\nntokens = len(corpus.dictionary)\n\nis_transformer_model = hasattr(model, 'model_type') and model.model_type == 'Transformer'\nif not is_transformer_model:\n    hidden = model.init_hidden(1)\ninput = torch.randint(ntokens, (1, 1), dtype=torch.long).to(device)\n\nwith open(args.outf, 'w') as outf:\n    with torch.no_grad():  # no tracking history\n        for i in range(args.words):\n            if is_transformer_model:\n                output = model(input, False)\n                word_weights = output[-1].squeeze().div(args.temperature).exp().cpu()\n                word_idx = torch.multinomial(word_weights, 1)[0]\n                word_tensor = torch.Tensor([[word_idx]]).long().to(device)\n                input = torch.cat([input, word_tensor], 0)\n            else:\n                output, hidden = model(input, hidden)\n                word_weights = output.squeeze().div(args.temperature).exp().cpu()\n                word_idx = torch.multinomial(word_weights, 1)[0]\n                input.fill_(word_idx)\n\n            word = corpus.dictionary.idx2word[word_idx]\n\n            outf.write(word + ('\\n' if i % 20 == 19 else ' '))\n\n            if i % args.log_interval == 0:\n                print('| Generated {}/{} words'.format(i, args.words))\n"
  },
  {
    "path": "word_language_model/main.py",
    "content": "# coding: utf-8\nimport argparse\nimport time\nimport math\nimport os\nimport torch\nimport torch.nn as nn\nimport torch.onnx\n\nimport data\nfrom model import PositionalEncoding, RNNModel, TransformerModel\n\nparser = argparse.ArgumentParser(description='PyTorch Wikitext-2 RNN/LSTM/GRU/Transformer Language Model')\nparser.add_argument('--data', type=str, default='./data/wikitext-2',\n                    help='location of the data corpus')\nparser.add_argument('--model', type=str, default='LSTM',\n                    help='type of network (RNN_TANH, RNN_RELU, LSTM, GRU, Transformer)')\nparser.add_argument('--emsize', type=int, default=200,\n                    help='size of word embeddings')\nparser.add_argument('--nhid', type=int, default=200,\n                    help='number of hidden units per layer')\nparser.add_argument('--nlayers', type=int, default=2,\n                    help='number of layers')\nparser.add_argument('--lr', type=float, default=20,\n                    help='initial learning rate')\nparser.add_argument('--clip', type=float, default=0.25,\n                    help='gradient clipping')\nparser.add_argument('--epochs', type=int, default=40,\n                    help='upper epoch limit')\nparser.add_argument('--batch_size', type=int, default=20, metavar='N',\n                    help='batch size')\nparser.add_argument('--bptt', type=int, default=35,\n                    help='sequence length')\nparser.add_argument('--dropout', type=float, default=0.2,\n                    help='dropout applied to layers (0 = no dropout)')\nparser.add_argument('--tied', action='store_true',\n                    help='tie the word embedding and softmax weights')\nparser.add_argument('--seed', type=int, default=1111,\n                    help='random seed')\nparser.add_argument('--log-interval', type=int, default=200, metavar='N',\n                    help='report interval')\nparser.add_argument('--save', type=str, default='model.pt',\n                    help='path to save the final model')\nparser.add_argument('--onnx-export', type=str, default='',\n                    help='path to export the final model in onnx format')\nparser.add_argument('--nhead', type=int, default=2,\n                    help='the number of heads in the encoder/decoder of the transformer model')\nparser.add_argument('--dry-run', action='store_true',\n                    help='verify the code and the model')\nparser.add_argument('--accel', action='store_true',\n                    help='Enables accelerated training')\nparser.add_argument('--use-optimizer', action='store_true',\n                    help='Uses AdamW optimizer for gradient updating')\nargs = parser.parse_args()\n\n# Set the random seed manually for reproducibility.\ntorch.manual_seed(args.seed)\n\nif args.accel and torch.accelerator.is_available():\n    device = torch.accelerator.current_accelerator()\n\nelse:\n    device = torch.device(\"cpu\")\n\nprint(\"Using device:\", device)\n\n###############################################################################\n# Load data\n###############################################################################\n\ncorpus = data.Corpus(args.data)\n\n# Starting from sequential data, batchify arranges the dataset into columns.\n# For instance, with the alphabet as the sequence and batch size 4, we'd get\n# ┌ a g m s ┐\n# │ b h n t │\n# │ c i o u │\n# │ d j p v │\n# │ e k q w │\n# └ f l r x ┘.\n# These columns are treated as independent by the model, which means that the\n# dependence of e. g. 'g' on 'f' can not be learned, but allows more efficient\n# batch processing.\n\ndef batchify(data, bsz):\n    # Work out how cleanly we can divide the dataset into bsz parts.\n    nbatch = data.size(0) // bsz\n    # Trim off any extra elements that wouldn't cleanly fit (remainders).\n    data = data.narrow(0, 0, nbatch * bsz)\n    # Evenly divide the data across the bsz batches.\n    data = data.view(bsz, -1).t().contiguous()\n    return data.to(device)\n\neval_batch_size = 10\ntrain_data = batchify(corpus.train, args.batch_size)\nval_data = batchify(corpus.valid, eval_batch_size)\ntest_data = batchify(corpus.test, eval_batch_size)\n\n###############################################################################\n# Build the model\n###############################################################################\n\nntokens = len(corpus.dictionary)\nif args.model == 'Transformer':\n    model = TransformerModel(ntokens, args.emsize, args.nhead, args.nhid, args.nlayers, args.dropout).to(device)\nelse:\n    model = RNNModel(args.model, ntokens, args.emsize, args.nhid, args.nlayers, args.dropout, args.tied).to(device)\n\ncriterion = nn.NLLLoss()\nif args.use_optimizer:\n    optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr)\n\n###############################################################################\n# Training code\n###############################################################################\n\ndef repackage_hidden(h):\n    \"\"\"Wraps hidden states in new Tensors, to detach them from their history.\"\"\"\n\n    if isinstance(h, torch.Tensor):\n        return h.detach()\n    else:\n        return tuple(repackage_hidden(v) for v in h)\n\n\n# get_batch subdivides the source data into chunks of length args.bptt.\n# If source is equal to the example output of the batchify function, with\n# a bptt-limit of 2, we'd get the following two Variables for i = 0:\n# ┌ a g m s ┐ ┌ b h n t ┐\n# └ b h n t ┘ └ c i o u ┘\n# Note that despite the name of the function, the subdivison of data is not\n# done along the batch dimension (i.e. dimension 1), since that was handled\n# by the batchify function. The chunks are along dimension 0, corresponding\n# to the seq_len dimension in the LSTM.\n\ndef get_batch(source, i):\n    seq_len = min(args.bptt, len(source) - 1 - i)\n    data = source[i:i+seq_len]\n    target = source[i+1:i+1+seq_len].view(-1)\n    return data, target\n\n\ndef evaluate(data_source):\n    # Turn on evaluation mode which disables dropout.\n    model.eval()\n    total_loss = 0.\n    ntokens = len(corpus.dictionary)\n    if args.model != 'Transformer':\n        hidden = model.init_hidden(eval_batch_size)\n    with torch.no_grad():\n        for i in range(0, data_source.size(0) - 1, args.bptt):\n            data, targets = get_batch(data_source, i)\n            if args.model == 'Transformer':\n                output = model(data)\n                output = output.view(-1, ntokens)\n            else:\n                output, hidden = model(data, hidden)\n                hidden = repackage_hidden(hidden)\n            total_loss += len(data) * criterion(output, targets).item()\n    return total_loss / (len(data_source) - 1)\n\n\ndef train():\n    # Turn on training mode which enables dropout.\n    model.train()\n    total_loss = 0.\n    start_time = time.time()\n    ntokens = len(corpus.dictionary)\n    if args.model != 'Transformer':\n        hidden = model.init_hidden(args.batch_size)\n    for batch, i in enumerate(range(0, train_data.size(0) - 1, args.bptt)):\n        data, targets = get_batch(train_data, i)\n        # Starting each batch, we detach the hidden state from how it was previously produced.\n        # If we didn't, the model would try backpropagating all the way to start of the dataset.\n        if args.use_optimizer:\n            optimizer.zero_grad()\n        else:\n            model.zero_grad()\n        if args.model == 'Transformer':\n            output = model(data)\n            output = output.view(-1, ntokens)\n        else:\n            hidden = repackage_hidden(hidden)\n            output, hidden = model(data, hidden)\n        loss = criterion(output, targets)\n        loss.backward()\n\n        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.\n        torch.nn.utils.clip_grad_norm_(model.parameters(), args.clip)\n        if args.use_optimizer:\n            optimizer.step()\n        else:\n            for p in model.parameters():\n                p.data.add_(p.grad, alpha=-lr)\n\n        total_loss += loss.item()\n\n        if batch % args.log_interval == 0 and batch > 0:\n            cur_loss = total_loss / args.log_interval\n            elapsed = time.time() - start_time\n            print('| epoch {:3d} | {:5d}/{:5d} batches | lr {:02.2f} | ms/batch {:5.2f} | '\n                    'loss {:5.2f} | ppl {:8.2f}'.format(\n                epoch, batch, len(train_data) // args.bptt, lr,\n                elapsed * 1000 / args.log_interval, cur_loss, math.exp(cur_loss)))\n            total_loss = 0\n            start_time = time.time()\n        if args.dry_run:\n            break\n\n\ndef export_onnx(path, batch_size, seq_len):\n    print('The model is also exported in ONNX format at {}.'.format(os.path.realpath(args.onnx_export)))\n    model.eval()\n    dummy_input = torch.LongTensor(seq_len * batch_size).zero_().view(-1, batch_size).to(device)\n    hidden = model.init_hidden(batch_size)\n    torch.onnx.export(model, (dummy_input, hidden), path)\n\n\n# Loop over epochs.\nlr = args.lr\nbest_val_loss = None\n\n# At any point you can hit Ctrl + C to break out of training early.\ntry:\n    for epoch in range(1, args.epochs+1):\n        epoch_start_time = time.time()\n        train()\n        val_loss = evaluate(val_data)\n        print('-' * 89)\n        print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '\n                'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),\n                                           val_loss, math.exp(val_loss)))\n        print('-' * 89)\n        # Save the model if the validation loss is the best we've seen so far.\n        if not best_val_loss or val_loss < best_val_loss:\n            with open(args.save, 'wb') as f:\n                torch.save(model, f)\n            best_val_loss = val_loss\n        else:\n            # Anneal the learning rate if no improvement has been seen in the validation dataset.\n            lr /= 4.0\nexcept KeyboardInterrupt:\n    print('-' * 89)\n    print('Exiting from training early')\n\n# Load the best saved model.\nwith open(args.save, 'rb') as f:\n    if args.model == 'Transformer':\n        safe_globals = [\n            PositionalEncoding,\n            TransformerModel,\n            torch.nn.functional.relu,\n            torch.nn.modules.activation.MultiheadAttention,\n            torch.nn.modules.container.ModuleList,\n            torch.nn.modules.dropout.Dropout,\n            torch.nn.modules.linear.Linear,\n            torch.nn.modules.linear.NonDynamicallyQuantizableLinear,\n            torch.nn.modules.normalization.LayerNorm,\n            torch.nn.modules.sparse.Embedding,\n            torch.nn.modules.transformer.TransformerEncoder,\n            torch.nn.modules.transformer.TransformerEncoderLayer,\n        ]\n    else:\n        safe_globals = [\n            RNNModel,\n            torch.nn.modules.dropout.Dropout,\n            torch.nn.modules.linear.Linear,\n            torch.nn.modules.rnn.GRU,\n            torch.nn.modules.rnn.LSTM,\n            torch.nn.modules.rnn.RNN,\n            torch.nn.modules.sparse.Embedding,\n        ]\n    with torch.serialization.safe_globals(safe_globals):\n        model = torch.load(f)\n    # after load the rnn params are not a continuous chunk of memory\n    # this makes them a continuous chunk, and will speed up forward pass\n    # Currently, only rnn model supports flatten_parameters function.\n    if args.model in ['RNN_TANH', 'RNN_RELU', 'LSTM', 'GRU']:\n        model.rnn.flatten_parameters()\n\n# Run on test data.\ntest_loss = evaluate(test_data)\nprint('=' * 89)\nprint('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(\n    test_loss, math.exp(test_loss)))\nprint('=' * 89)\n\nif len(args.onnx_export) > 0:\n    # Export the model in ONNX format.\n    export_onnx(args.onnx_export, batch_size=1, seq_len=args.bptt)\n"
  },
  {
    "path": "word_language_model/model.py",
    "content": "import math\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass RNNModel(nn.Module):\n    \"\"\"Container module with an encoder, a recurrent module, and a decoder.\"\"\"\n\n    def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0.5, tie_weights=False):\n        super(RNNModel, self).__init__()\n        self.ntoken = ntoken\n        self.drop = nn.Dropout(dropout)\n        self.encoder = nn.Embedding(ntoken, ninp)\n        if rnn_type in ['LSTM', 'GRU']:\n            self.rnn = getattr(nn, rnn_type)(ninp, nhid, nlayers, dropout=dropout)\n        else:\n            try:\n                nonlinearity = {'RNN_TANH': 'tanh', 'RNN_RELU': 'relu'}[rnn_type]\n            except KeyError as e:\n                raise ValueError( \"\"\"An invalid option for `--model` was supplied,\n                                 options are ['LSTM', 'GRU', 'RNN_TANH' or 'RNN_RELU']\"\"\") from e\n            self.rnn = nn.RNN(ninp, nhid, nlayers, nonlinearity=nonlinearity, dropout=dropout)\n        self.decoder = nn.Linear(nhid, ntoken)\n\n        # Optionally tie weights as in:\n        # \"Using the Output Embedding to Improve Language Models\" (Press & Wolf 2016)\n        # https://arxiv.org/abs/1608.05859\n        # and\n        # \"Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling\" (Inan et al. 2016)\n        # https://arxiv.org/abs/1611.01462\n        if tie_weights:\n            if nhid != ninp:\n                raise ValueError('When using the tied flag, nhid must be equal to emsize')\n            self.decoder.weight = self.encoder.weight\n\n        self.init_weights()\n\n        self.rnn_type = rnn_type\n        self.nhid = nhid\n        self.nlayers = nlayers\n\n    def init_weights(self):\n        initrange = 0.1\n        nn.init.uniform_(self.encoder.weight, -initrange, initrange)\n        nn.init.zeros_(self.decoder.bias)\n        nn.init.uniform_(self.decoder.weight, -initrange, initrange)\n\n    def forward(self, input, hidden):\n        emb = self.drop(self.encoder(input))\n        output, hidden = self.rnn(emb, hidden)\n        output = self.drop(output)\n        decoded = self.decoder(output)\n        decoded = decoded.view(-1, self.ntoken)\n        return F.log_softmax(decoded, dim=1), hidden\n\n    def init_hidden(self, bsz):\n        weight = next(self.parameters())\n        if self.rnn_type == 'LSTM':\n            return (weight.new_zeros(self.nlayers, bsz, self.nhid),\n                    weight.new_zeros(self.nlayers, bsz, self.nhid))\n        else:\n            return weight.new_zeros(self.nlayers, bsz, self.nhid)\n\n# Temporarily leave PositionalEncoding module here. Will be moved somewhere else.\nclass PositionalEncoding(nn.Module):\n    r\"\"\"Inject some information about the relative or absolute position of the tokens in the sequence.\n        The positional encodings have the same dimension as the embeddings, so that the two can be summed.\n        Here, we use sine and cosine functions of different frequencies.\n    .. math:\n        \\text{PosEncoder}(pos, 2i) = sin(pos/10000^(2i/d_model))\n        \\text{PosEncoder}(pos, 2i+1) = cos(pos/10000^(2i/d_model))\n        \\text{where pos is the word position and i is the embed idx)\n    Args:\n        d_model: the embed dim (required).\n        dropout: the dropout value (default=0.1).\n        max_len: the max. length of the incoming sequence (default=5000).\n    Examples:\n        >>> pos_encoder = PositionalEncoding(d_model)\n    \"\"\"\n\n    def __init__(self, d_model, dropout=0.1, max_len=5000):\n        super(PositionalEncoding, self).__init__()\n        self.dropout = nn.Dropout(p=dropout)\n\n        pe = torch.zeros(max_len, d_model)\n        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n        pe[:, 0::2] = torch.sin(position * div_term)\n        pe[:, 1::2] = torch.cos(position * div_term)\n        pe = pe.unsqueeze(0).transpose(0, 1)\n        self.register_buffer('pe', pe)\n\n    def forward(self, x):\n        r\"\"\"Inputs of forward function\n        Args:\n            x: the sequence fed to the positional encoder model (required).\n        Shape:\n            x: [sequence length, batch size, embed dim]\n            output: [sequence length, batch size, embed dim]\n        Examples:\n            >>> output = pos_encoder(x)\n        \"\"\"\n\n        x = x + self.pe[:x.size(0), :]\n        return self.dropout(x)\n\nclass TransformerModel(nn.Transformer):\n    \"\"\"Container module with an encoder, a recurrent or transformer module, and a decoder.\"\"\"\n\n    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5):\n        super(TransformerModel, self).__init__(d_model=ninp, nhead=nhead, dim_feedforward=nhid, num_encoder_layers=nlayers)\n        self.model_type = 'Transformer'\n        self.src_mask = None\n        self.pos_encoder = PositionalEncoding(ninp, dropout)\n\n        self.input_emb = nn.Embedding(ntoken, ninp)\n        self.ninp = ninp\n        self.decoder = nn.Linear(ninp, ntoken)\n\n        self.init_weights()\n\n    def _generate_square_subsequent_mask(self, sz):\n        return torch.log(torch.tril(torch.ones(sz,sz)))\n\n    def init_weights(self):\n        initrange = 0.1\n        nn.init.uniform_(self.input_emb.weight, -initrange, initrange)\n        nn.init.zeros_(self.decoder.bias)\n        nn.init.uniform_(self.decoder.weight, -initrange, initrange)\n\n    def forward(self, src, has_mask=True):\n        if has_mask:\n            device = src.device\n            if self.src_mask is None or self.src_mask.size(0) != len(src):\n                mask = self._generate_square_subsequent_mask(len(src)).to(device)\n                self.src_mask = mask\n        else:\n            self.src_mask = None\n\n        src = self.input_emb(src) * math.sqrt(self.ninp)\n        src = self.pos_encoder(src)\n        output = self.encoder(src, mask=self.src_mask)\n        output = self.decoder(output)\n        return F.log_softmax(output, dim=-1)\n"
  },
  {
    "path": "word_language_model/requirements.txt",
    "content": "torch>=2.6\n"
  }
]